On prompting, part 2 of 3

The Shape of a Prompt

Where the question goes, which tags wrap the documents, what lives in the system prompt: the labs disagree, and the research says the answer depends on the model. Treat the shape as a seam you test, not a rule you memorize.

In April 2025, OpenAI published a prompting guide for GPT-4.1 with a recommendation about where instructions belong in a long prompt. Ideally, it said, “place your instructions at both the beginning and end of the provided context,” and if you only want them once, “above the provided context works better than below” (OpenAI). Anthropic’s documentation says close to the opposite. Put long documents “near the top of your prompt, above your query, instructions, and examples,” because “queries at the end can improve response quality by up to 30 percent in tests” (Anthropic). Google’s guidance for Gemini 3 sides with Anthropic on the question, telling developers to “supply all the context first” and put the question “at the very end,” but sides with OpenAI on the rules, which it wants in the system instruction or “at the very beginning of the user prompt” (Google).

So which is it? Three frontier labs give three answers to where the words go, and none of them is wrong. Each one is describing its own models.

In the first part of this series I argued that prompt engineering is mostly finding the seams in a task, the places where a decision was left to the model that nobody on the team made, and that clever wording was never the lever. This part is about the layer underneath the wording: position, punctuation, whitespace, format and the system prompt. It’s the part of prompting that still looks like folklore, and the research of the last two years says something more useful than any single rule. The shape of a prompt is a seam between your system and a particular model, and it doesn’t carry over when the model changes.

Position

Notice where the three labs agree before getting to where they don’t. All three want long material, the documents, the transcripts, the retrieved passages, placed before the question about them. The disagreement is only about the instructions, and it makes sense once you think about what each lab is optimizing for. A question at the end is the last thing the model reads before it answers, so nothing dilutes it. A rule at the beginning frames how everything after it gets read. OpenAI’s “both” is a hedge that buys a little of each at the cost of saying everything twice.

Which one is right for a given system? It’s an empirical question, and a cheap one to answer. Anthropic’s 30 percent is a ceiling from its own tests on complex, multi-document inputs, not a number anyone should expect on a short prompt. The lesson from all three is narrower. On long inputs, position matters enough that each lab bothered to write it down, and they wrote down different things.

Punctuation and Whitespace

The strongest evidence that small things matter is still a 2023 study I cited in the first part, where changing only separators, casing and spacing in few-shot prompts moved LLaMA-2-13B’s accuracy by up to 76 points. Current models are far less fragile than that, but the reason it happened hasn’t gone away, and it’s worth understanding because it isn’t about meaning at all.

A model doesn’t read characters. It reads tokens, and the same string can be split into tokens more than one way. A 2025 paper presented at ACL makes the point with the word “penguin,” which a tokenizer normally splits as [p, enguin] but could just as validly split as [peng, uin]. The authors note that “current LLM pipelines account for only one possible tokenization for a given string,” yet models “still retain semantic understanding of other tokenizations” (Geh, Shao and Van den Broeck). Their concern was safety, since an unusual split can slip a request past alignment that only learned the usual one, but the lesson for everyday prompting is the same. A stray space or a different quote character doesn’t change what a sentence means to a person. It can change every token the model sees.

Punctuation also does more work inside the model than it looks like it should. Researchers behind SepLLM, presented at ICML 2025, found that “certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores” (Chen et al.), to the point that they could compress whole segments of text into the separator that ends them. The newline between two sections of a prompt isn’t decoration. It’s one of the places the model looks.

None of this means a prompt should be tuned character by character. It means the boundaries in a prompt are real to the model, so they should be deliberate, consistent and the same every time the prompt is built. A template that sometimes emits two blank lines and sometimes one, depending on whether a field was empty, is giving the model a slightly different prompt on every call.

Formats

The same is true one level up, where the question is whether to structure a prompt in Markdown, XML or JSON. A 2024 study from Microsoft formatted identical content as plain text, Markdown, JSON and YAML and found GPT-3.5’s performance on a code translation task varied “by up to 40%” depending on the template, while GPT-4 was “more robust to these variations” (He et al.). No format won everywhere.

The labs’ own guidance reflects that. OpenAI recommends starting with Markdown headings, reports that XML “performed well in our long context testing,” and says that for documents in long context, “JSON performed particularly poorly.” Anthropic recommends XML tags, because wrapping instructions, context, examples and inputs in their own tags “reduces misinterpretation.” Google says XML-style tags or Markdown headings are both effective and asks for only one thing: “Choose one format and use it consistently within a single prompt.” Google’s line is the most honest of the three. Consistency is the part all the evidence supports.

The bigger cost turns out to be on the other side, in what you ask the model to produce. In April 2026, researchers at UC San Diego and Wisconsin published a paper called The Format Tax, which found that requiring JSON, XML, LaTeX or Markdown output “substantially degrade[s] reasoning and writing performance across open-weight models.” The surprise was where the cost came from. The usual suspect is constrained decoding, but “format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied” (Lee, D’Antoni and Berg-Kirkpatrick). Asking a model to think and format at the same time costs it some of the thinking. Who would have guessed the cost was in the request rather than the decoder?

The same paper has the most useful caveat. Most recent closed-weight models “show little to no format tax.” So the effect is real and it isn’t universal, which is exactly the problem. A May 2026 study across 140,000 generations found that the accuracy range caused by nothing but the prompt wrapper “varies by over 30x across models and is largely explained by compliance failures” (Mehta). Some models barely notice the wrapper. Others lose points mostly because they stop producing answers that can be parsed. The fix the Format Tax authors recommend is to decouple the two jobs, letting the model reason freely and then reformatting in a second pass or with extended thinking, and it’s the same fix I described in Non-Determinism Is a Bug, Not a Feature: don’t make the reasoning carry the schema.

One more detail in Anthropic’s docs is easy to miss. “The formatting style used in your prompt might influence Claude’s response style,” and “removing markdown from your prompt can reduce the volume of markdown in the output.” The shape of the prompt leaks into the shape of the answer.

System and User

Most teams treat the system prompt as the place for rules and the user turn as the place for requests, on the assumption that the model ranks the first above the second. The research says that ranking is much weaker than it looks. In Control Illusion, accepted to AAAI 2026, researchers set system and user instructions against each other across six models and found that models “struggle with consistent instruction prioritization, even for simple formatting conflicts,” and that “the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy” (Geng et al.).

Their more interesting finding is what the models obeyed instead. Framings borrowed from society, like authority, expertise and consensus, “show stronger influence on model behavior than system/user roles.” A user message that sounds like it comes from an expert can outweigh a system prompt that only has its position going for it. That makes sense for a model trained on human text. In human text, who said something has always mattered more than which box it was typed into.

The labs are moving in the other direction, which complicates things. Anthropic notes that Claude Opus 4.5 and 4.6 are “more responsive to the system prompt than previous models,” to the point that prompts written to push older models now overshoot, and it recommends replacing “CRITICAL: You MUST use this tool when…” with plain “Use this tool when….” The all-caps habit from 2024 is now a source of bugs. So the system prompt is getting stronger as a suggestion, and the research says it still isn’t a boundary. Those are different things, and the difference is where prompt injection lives, which is the subject of the next part.

Three Labs, Three Dialects

Put the guidance side by side and the pattern is clear. Anthropic wants documents first, the question last and XML around everything. OpenAI wants Markdown headings and the instructions above the context, or on both sides of it. Google wants the rules in the system instruction, the question at the end and a single format used consistently. Each one is a reasonable description of how that lab’s models were trained, and none of them is a law.

The non-determinism essay made a similar point about structured output. A team supporting three providers ends up maintaining three dialects of the same contract, and each one fails differently. Prompt shape is the same problem one layer up. The meaning of the prompt, the definitions, the bar, the rubric and the authority from the first part, carries over between models. The shape of the prompt doesn’t.

Where the Shape Lives

That points to the same architecture argument as the first part. If the meaning of a prompt belongs to the people who own each decision, the shape belongs to the model, and it should be written down next to the model rather than scattered through every prompt. In practice that’s a thin rendering step, one template per model family, that takes the same instructions, documents and question and lays them out the way that model reads best. It means building prompts in code, so the whitespace and delimiters are identical on every call. And it means running the eval suite against each model whenever the template or the model changes, because a format that won on one model is only a hypothesis on the next.

If I were setting up a new system today, I’d put long material first and the question last unless an eval said otherwise, since that’s where two of the three labs and most of the evidence agree. I’d pick one delimiter style per model and never mix them. I’d keep the reasoning and the output format in separate steps whenever the format is strict. I’d write the system prompt in a normal voice and stop shouting at the model. And I’d stop treating the system prompt as a security boundary, because the research says it was never one.

The meaning of a prompt belongs to your team. The shape of it belongs to the model.