Plain Questions, Valid GraphQL

Most AI features don’t fail because the model is dumb. They fail because of what we hand it.

I learned that one the expensive way. At Litehouse, part of my job was auditing a $6M PwC delivery before we accepted it. One piece of it was natural-language BI for our smart-district platform, the one proven at the Honda Center: an operator types a question in plain English and gets an answer back from the data. That data sits behind a federated GraphQL service spanning both the data warehouse and the application data. Operators don’t write GraphQL, and they shouldn’t have to.

It didn’t work. This is what we found, the two versions we built after it, and why only the third was good enough to put in front of a user.

Version One: Three Calls in a Row

PwC’s design was a pipeline. Three LLM calls ran in sequence, each over different data, each building on the output of the last.

Pipelines feel safe. Each step is small, easy to test, and easy to draw on a whiteboard. The problem is that an LLM’s output is a prediction, and predictions are sometimes wrong. In a pipeline, nothing catches a wrong guess in the first call. It becomes the premise of the second, and by the third it’s a confident wrong answer that nothing in the chain ever questioned.

That was the structural problem. Two more sat underneath it.

No sense of time or things. Ask “how many people came to the last concert?” and the model has no idea when the last concert was. Ask about hot dogs, and it has no idea what ID the data uses for hot dogs. Nothing resolved either one, so the model guessed.

The entire schema in the prompt. The pipeline dumped the whole GraphQL introspection into the context window, and the schema was full of semantically similar queries. Context windows weren’t big enough for that yet, and even when context fits, it isn’t free: attention gets diluted as context grows. Hand a model a pile of near-identical options, ask it to choose, and more often than not, ours picked the whole thing.

None of these are model problems. They’re design problems.

Version Two: Resolve First, Then Ask

Our first rebuild was still a pipeline, but it stopped asking the model to guess anything a plain program could work out.

• Time. A static temporal engine resolves references like “the last concert” before the model ever sees the question. The model gets dates, not riddles.
• Things. An entity resolver turns “hot dogs” into the ID the data actually uses. Candidates come out of a vector database, and reranking picks the right one.
• The schema. Instead of the whole introspection, we embedded the GraphQL queries and retrieved only the ones closest to the question.

It came a lot closer. It also topped out at 70%.

Seventy percent sounds decent until you put it in front of someone. An operator can’t tell which three answers in ten are wrong, so they can’t trust any of them. A BI tool you have to double-check isn’t a BI tool. 70% isn’t a product.

Retrieval was the ceiling. It can find a query that looks like the question; it can’t tell the model what that query means. And it was still a pipeline, so whatever retrieval missed, the next step built on.

Version Three: Meet the Model Where It’s Fluent

By then, models had come to strongly prefer working a certain way: bash and Python tools, skills, and an agentic loop. So we stopped fighting that and built for it:

• A skills-based semantic layer, instead of raw schema.
• A small set of tools for querying data over GraphQL or the vector database, in the shape the model already knows.
• A trimmed GraphQL query schema, cut down to what it actually needed.
• An agentic loop instead of a pipeline.

The loop matters most. A pipeline has no way back; a one-shot call has nothing to catch its mistakes with either. A loop can look at what came back, notice it’s wrong, and try again. The model checks its work against the real data instead of trusting its first guess.

That took us from 70% to 96%, measured over a curated set of questions. That’s the version we shipped.

Who’s Allowed to See What

A BI tool that answers any question for anyone is a data leak with a chat box. Every AI tool sits behind an access-control proxy, so a question can only reach the data the person asking is allowed to see. Permissions live outside the model, where no prompt can talk its way past them. The same push made the platform safe to sell: SOC 2, Auth0 in place of home-rolled login, and centralized secrets.

What I’d Tell the Next Team

• Audit the design before you blame the model. PwC’s pipeline didn’t fail because the model was weak. It failed because of how the steps were chained and what the model was shown.
• Resolve what you can deterministically. Time didn’t need a model, and names didn’t need a guess. Anything a plain program can answer, it should.
• Retrieval has a ceiling. It finds what looks right, not what’s meant.
• Build in the shape the model is fluent in. Tools and skills it already knows beat a schema it has to decode.
• Loop, don’t chain. Let the system check its own work instead of compounding its first mistake.
• Put permissions outside the model. Who can see what is a rule, not a prompt.

The model was never the hard part. What we handed it was.