Answers in seconds, not days

The Commercial team's answers were spread across five kinds of sources, and some of them couldn't be searched at all, so simple questions turned into multi-day requests.

What people asked

• Sales and performance numbers by region, product and period
• What a policy or process document says
• What a Power BI report shows, and why
• What was decided in a past meeting or email thread

Where the answers lived

• Documents and emails, including PDFs in many different layouts and PowerPoint decks full of charts and images, too varied to parse with a simple rule-based extractor
• Meeting recordings, whose content was effectively unsearchable
• SQL databases holding the structured commercial data
• Power BI reports built on top of that data

Why it took days

• Most data questions had to be queued with the analytics team.
• One answer often needed pieces from several sources at once.
• Some knowledge, such as what was said in a meeting or shown on a slide chart, couldn't be found by search at all.

One agent sits between the user and three tools, each built for a different kind of source.

• Document search over documents, emails and meeting recordings
• A SQL query tool for the structured commercial data
• The Power BI agent for questions about reports

The Power BI agent also has a direct path to the database, used when a visual isn't enough and to cross-check what the report shows.

I inherited a retrieval pipeline that found the right source only 68% of the time, and rebuilt ingestion and chunking to bring it above 95% on a hand-built evaluation set.

What was going wrong

• Broken text from PDFs. A plain Python PDF library pulled text out of files with very different layouts, leaving broken artifacts and mangled tables.
• Charts and images were invisible. Anything shown visually on a slide or in a PDF never reached the index.
• Fixed 500-word chunks cut across headings and paragraphs, splitting ideas in half.
• Chunks had no context. A chunk didn't say which document it came from or where it sat, so similar passages from different files were easy to confuse.

What I changed

1. Parse every page with a vision model. Each PDF page and slide is rendered and read by GPT-5.4 nano, so charts, images and odd layouts come through as text. Up to 20 pages run in parallel to keep ingestion fast.
2. Normalize everything to Markdown. Every file type stores information differently, so all of it is converted to one consistent Markdown format first.
3. Chunk by structure, not word count. Chunks follow headings and paragraphs instead of a fixed size.
4. Give every chunk its context. Before embedding, each chunk gets a <context> tag explaining where it sits in its document, so it can be found and understood on its own.

Measuring it

I built a Q&A evaluation set by hand, with each question tied to the source it should retrieve, and measured every version of the pipeline against it. The old pipeline scored 68%; the rebuilt one scores 95%+.

A single tool-using agent decides for itself which sources a question needs, so one answer can combine what was said in a meeting with what the numbers show.

• Tools, not a router. The agent has document search, a SQL query tool and the Power BI agent available, and calls whichever it needs. It can chain them, for example finding a decision in meeting notes, then pulling the figures that decision was about.
• Unstructured meets structured. Meeting notes and emails are linked to SQL data in the same answer, instead of the user stitching them together across tools.
• One place to ask. Reps and analysts use it through an internal chat web app.

The agent never sends SQL to the database. It writes a structured JSON request, my code validates it and compiles it into SQL, so model-written SQL can't reach the data.

1. The agent writes a JSON query request describing what it wants, rather than a raw SQL string.
2. The request is validated against a strict schema before anything runs. A malformed or out-of-bounds request is rejected.
3. Code compiles the JSON into parameterized SQL, so the shape of every query is controlled by the application, not the model.
4. It runs under a read-only role with row limits and timeouts, so even a valid query can't change data or run away.

This turns SQL safety from "hope the model behaves" into a guarantee enforced by code.

For questions about reports, a computer-use agent works the live Power BI report the way an analyst would, then checks what it sees against the database underneath.

• Navigates the report. It works from screenshots and clicks, moving between pages and setting filters and slicers.
• Reads the visuals. It interprets the charts and tables on screen to answer the question.
• Falls back to the database when a visual isn't enough to answer, querying the report's backend database directly.
• Cross-checks both. It compares what the report shows with the database result, so an answer about a dashboard is backed by the data behind it.

This resolves analytics questions end to end, without a request sitting in the analytics team's queue.

The Commercial team can now answer in seconds the questions that used to take days of back-and-forth with analysts.

• Days to seconds for lookups across documents, emails, meetings and data
• 500+ documents, emails and meeting recordings searchable in one place, including content that was previously unsearchable
• Retrieval accuracy from 68% to 95%+ on a hand-built evaluation set
• No model-written SQL ever reaches the database