Production-Grade LLM Applications: Prompt Engineering Best Practices

A demo is one prompt, one happy path, and an operator who already knows the phrasing that works. Production is thousands of people who phrase it wrong, a proportion who phrase it adversarially, a model provider who will deprecate the version you tuned against, and a page that has to render inside the latency budget the rest of the product already committed to.

The variance that kills the feature is almost never in the instruction. It is in the input distribution. The same prompt that scores ninety per cent on the twelve examples you wrote by hand will score sixty on the first thousand real requests, because real requests contain abbreviations, three questions at once, pasted tables, the wrong language, and a fair number of people testing whether it will swear.

Teams routinely spend weeks on wording, days on retrieval, hours on evaluation and nothing at all on failure design. The right ratio is close to the reverse of that.

By the time a request leaves your server, the prompt is not a string somebody wrote. It is a compiled artefact, assembled per request out of half a dozen parts that were each decided by different code.

Seeing it that way changes the question. It stops being what should I write and becomes what should the compiler put in, in what order, and under what budget. Every one of those layers is testable on its own, and most of the quality lives in the two that nobody writes tests for.

The single highest-return change in most LLM features is to stop asking for an answer and start asking for a structure. Structured output, JSON schema, or a tool call — the mechanism differs by provider, the discipline does not. An unstructured answer makes every consumer downstream a parser of prose, and parsers of prose fail silently and at the worst possible moment.

Validate the result against a schema you own, in your own language, on your own side of the network. Do not trust that the provider enforced it; enforce it again. If validation fails, send one repair turn carrying the validation error, and if that fails, fail loudly. A retry loop with no ceiling is how a feature that cost two paise a call starts costing eleven.

Give the schema somewhere to put uncertainty. A field for confidence, an enumerated value for insufficient evidence, a list of the sources used. A model with no legal way to say it does not know will invent something that satisfies the schema, and you will have built a machine for producing well-formed nonsense.

In any retrieval-grounded system, retrieval quality dominates everything else, and it is not close. Most of what gets reported as hallucination is a retrieval miss the model papered over — the right passage was never in the window, and the model answered from its weights rather than admitting the gap. No amount of instruction wording fixes a document that is not there.

Position inside the window matters too. Models attend unevenly across long contexts, and material buried in the middle is used least reliably — the effect is well enough established to have its own name in the literature. Put the decisive passage at the start or the end. If you have twenty candidate passages and room for six, the job is to pick the right six, not to send all twenty and hope.

Chunk on structure rather than on a fixed token count. A chunk that begins mid-sentence and ends mid-table has lost the thing that made it answerable. Carry the heading path with each chunk, so the model knows that this paragraph is under Returns, under Policy, under the 2024 edition, and can say so when it cites.

Then give the model permission to come back empty-handed. An assistant that says the policy does not cover this is more useful than one that produces a confident paragraph about a policy that does not exist, and it is the only version you can put in front of a customer.

There is a long list of prompting techniques and a much shorter list that still earns its cost at scale. The difference is usually that the technique bought accuracy on a benchmark and buys latency in a product.

A prompt that lives in a database, edited through an admin screen by whoever is awake, is an unversioned production dependency. Prompts belong in the repository, in the same commit as the code that compiles them and the tests that score them, with a diff that a reviewer can read.

Pin the model version explicitly. A prompt is not portable across models and barely portable across versions of the same model; the pair is the artefact, not the string. When a version is deprecated — and it will be, on the provider's schedule rather than yours — you are migrating a program, not swapping a constant. Budget for a re-evaluation run and a fix window every time, and keep the previous version routable until the new one clears the same bar.

Everything above is guesswork until there is a harness. The golden set is the real specification of the feature — more so than any document, because it is the only description of the behaviour that can fail a build.

Cost and latency are product decisions disguised as infrastructure ones, and they should be written down before the prompt is. A feature that answers beautifully in nine seconds has been designed for a user who does not exist.

Every production LLM feature needs an answer to four questions before launch, and finding the answers after launch is considerably more expensive.

The first is abstention: what does it do when it does not know, and does that path look like a deliberate product state rather than an error. The second is escalation: at what confidence, on which intents, does a human take over, and does the handover carry the context or make the customer start again.

The third is injection. Retrieved documents and user messages are data, never instruction. A support article that contains the words ignore previous instructions must not be able to act as one, which means privilege separation in the prompt structure, not a sentence asking the model to be careful. Anything the model can do with a tool, an attacker who can write into your corpus can attempt.

The fourth is observability. Log the compiled prompt with secrets and personal data redacted, the model and version, the retrieved document identifiers, the raw response, the validation result and the latency. Without that, a complaint about one bad answer is unreproducible, and unreproducible failures never get fixed.

It is still worth doing well. A clear system contract, a schema that leaves room for uncertainty, exemplars chosen rather than pasted, and an ordering that lets the cache work — these are real gains and they are cheap. They are simply not the project.

The project is the compiler around the prompt, the harness that scores it, and the failure paths that catch it. Build those and the wording becomes a tuning exercise. Skip them and no amount of wording will get the thing past its first thousand real users.