AI engineering

Plain Questions, Valid GraphQL

Three versions of natural-language BI over a smart-district platform, and what it took to get from 70% to 96%.

A map before you read

One line per section, written by an AI model running entirely in your browser. Free, private, nothing leaves your device.

At Litehouse, part of my job was auditing a $6M PwC delivery before we accepted it. One piece of that delivery was natural-language BI for our smart-district platform, the one that started at the Honda Center. The idea is simple enough: an operator types a question in plain English and gets an answer back from the data. That data sits behind a federated GraphQL service that spans both the data warehouse and the application data, and the people running a district don’t write GraphQL. They shouldn’t have to.

It didn’t work. Most of the time it didn’t even give a wrong answer, it just couldn’t complete. What follows is what we found, the two versions we built after it, and what finally got it to a place we could ship.

Version One: Three Calls in a Row

PwC’s design was a pipeline of three LLM calls, run in sequence, each over different data, each one building on the output of the last. On a whiteboard this looks great. Every step is small and testable, and it’s easy to explain to a client.

The problem is that an LLM’s output is a prediction, and predictions are sometimes wrong. In a pipeline nothing catches a bad guess in the first call. It becomes the premise of the second call, and by the third you get either a confident wrong answer that nothing in the chain ever questioned or, more often for us, nothing at all. Worse, every call was a one-shot. We were trying to get it right the first time, and there was no path for the model to notice a mistake, its own or one in our one-shot user prompt, and correct it.

Underneath that were two more problems.

The first was that it had no sense of time or things. Ask “how many people came to the last concert?” and the model has no idea when the last concert was. It doesn’t know today’s date, let alone the event calendar. Ask how many hot dogs sold and, well, which hot dogs? There’s a stadium hot dog, a Polish sausage, a kosher dog and a chili dog, and each one has its own ID in the data. Nothing resolved either of these, so the model guessed.

The second was the schema. The pipeline took the entire GraphQL introspection and dumped it into the context window, and that schema was full of semantically similar queries. In fairness, this was early 2025, and the context windows we were working with were tight for something that size. But even when the context fits, it isn’t free. Attention gets diluted as context grows, and when you hand a model a pile of near-identical options and ask it to choose, more often than not ours picked the whole thing.

Some of this was the models of the day. Most of it was ours: how the steps were chained and what we were showing it.

Version Two: Resolve First, Then Ask

Our first rebuild was still a pipeline, but we stopped asking the model to guess anything a plain program could work out.

For time, we built a static temporal engine that resolves references like “the last concert” before the model ever sees the question. For things, we built an entity resolver: candidates come out of a vector database and reranking sorts out which ones the question actually means, so “hot dogs” turns into the four IDs that count. And instead of the whole introspection, we embedded the GraphQL queries and retrieved only the ones closest to the question.

To measure any of this we needed a test set, so our product manager put together the top questions we expected people to actually ask it. That became the curated set every version got scored against.

Version two came a lot closer. It also topped out at 70%.

Seventy percent sounds decent until you put it in front of someone. An operator can’t tell which three answers in ten are wrong, so tangibly, they can’t trust any of them. That isn’t usable in a product.

The ceiling was retrieval. Embeddings can find a query that looks like the question, but they can’t tell the model what that query means or when to use it. And because it was still a pipeline, whatever retrieval missed, the next step built on.

Version Three: Why Fight the Model?

By then something had shifted, and part of it was the field growing up. Best practices for building with AI were evolving fast, context windows kept growing, and so did the models’ intelligence. What was difficult in 2025, and took real engineering to get around, got easier in 2026. Most people put the inflection point right at the turn of the year: Latent Space described “a chorus of people calling out December 2025” as the turning point, and Martin Fowler dates it to November.

Skills in particular had become the de facto way of handing an agent a task, from Cursor to Claude Code to Codex, all built on a format Anthropic released as an open standard. And models had come to strongly prefer working a certain way: bash and Python style tools, skills, and an agentic loop. We could keep forcing our pipeline on them, or we could build in the shape they were already good at. We went with the latter.

We built a skills-based semantic layer, trimmed the GraphQL query schema down to what the model actually needed, and ran it all on Pi, a minimal, open-source agent harness that handles skills natively. We gave it two tools of our own: run_graphql to query the data, and search_entities to look things up in the vector database.

The other big change was who does the resolving. In version two we resolved time and entities up front and hoped we’d guessed what the question needed. In version three the model discovers them through tool calls as it goes. It looks up when the last concert was, it searches for which items count as a hot dog, and it only writes the query once it knows.

What a Skill Looks Like

Each skill is a Markdown file per area of the business, named in kebab-case: events.md, venues-and-districts.md, revenue.md. The harness only puts a short catalog in the prompt, each skill’s name and a one-line summary pulled from its first paragraph, and the model reads the whole guide when a question matches it. Pi calls this progressive disclosure, and it’s exactly what version one was missing. The skills carry the domain knowledge, not just the shape of the data.

Every guide has the same bones:

# Events

**Covers root fields:** `<rootField>`, `<rootField>`

One plain sentence on what this covers. It becomes the catalog summary.

## What an event is
## Key fields
Identity, timestamps, measures, dimensions.
## Joins
How events link to everything else, across queries.
## Filtering
The exact filter operators allowed.
## Sorting
The exact sort values allowed.
## Example
A working GraphQL query, pagination included.
## Gotchas
Fields you can’t filter on, complexity traps, PII rules.

The Covers root fields line is what keeps the skills honest. A drift check reads it against the GraphQL schema. Every root query field has to be covered by some guide, and a guide that names a field that doesn’t exist fails the check. That way the skills can’t quietly fall behind the schema, which matters a lot, because a model that trusts a stale guide is right back to guessing.

The Loop, the Cost, and the Model

The loop is what ties it together. With a pipeline or a one-shot, you get one chance to get it right. With a loop, the model gets feedback. If it writes a bad GraphQL call, the error comes back, it reasons about what went wrong, and it tries again.

The same goes for intent. Say someone asks about hot dogs, and search_entities comes back with the stadium hot dog, the Polish sausage, the kosher dog and the chili dog. Did they mean just the hot dog, or the hot dog and the Polish sausage? In a loop, the model can look at that list next to the rest of the question and make a call. A one-shot never sees the list, so it has nothing to reason off of.

On the same curated set, that took us from 70% to 96%, and that’s the version we shipped.

It isn’t free. A loop obviously spends more than a single call, and it takes a while to reach its conclusion, around 20 seconds of tool calls and reasoning. For BI, where the alternative is someone waiting on an analyst, that was a trade we were happy to make.

We ran it on GPT. For smaller, well-scoped tasks like these, we found it more mechanical and reliable than Claude, and mechanical is exactly what you want from something writing queries against your data. That isn’t just us; OpenAI says outright that its models are trained to follow instructions “more closely and more literally” than their predecessors. The caveat is size. On larger tasks that goes out the window. METR’s time-horizon measurements, which track how long a task an AI agent can finish half the time, put Claude Opus 4.6 at around 12 hours against roughly 6 for the best GPT model they measured.

Who’s Allowed to See What?

A BI tool that answers any question for anyone is a data leak with a chat box. Every AI tool sits behind an access-control proxy, so a question can only ever reach the data the person asking is allowed to see. The permissions live outside the model, so there’s nothing to talk your way past.

Takeaways

If I were starting this again, I’d audit the design before blaming the model. The models did get better while we worked on this, and that helped. But PwC’s pipeline didn’t fail because the model was weak. It failed because of how it was chained and what it was shown.

I’d still resolve whatever a plain program can resolve, but I’d expose it as tools and let the model ask, instead of front-loading answers and guessing the question. I’d build in the shape models are already fluent in, keep the skills honest with something like a drift check, and let it loop, budgeting for the time and tokens that costs. And I’d keep permissions out of the model entirely.

Some of it was the models catching up. The rest was what we handed them.

Sources

  1. OpenAI, GPT-4.1 Prompting Guide
  2. METR, Task-completion time horizons of frontier AI models
  3. Pi, the agent harness
  4. Latent Space, “WTF Happened in December 2025?”
  5. Martin Fowler, “November Inflection”
  6. Agent Skills, the open standard and its clients
  7. Pi, Skills documentation