Generative AI Architecture · Part 2 of 8

Prompt Engineering as Application Logic

Prompts as application code: structure, templates, context, caching, and testing.

It's tempting to treat a prompt as a one-off piece of text you type until the output looks right, then paste into the codebase. That habit works for a demo and fails in production: nobody wrote down what it's supposed to do, so nobody can tell when a change breaks it. A prompt that decides how your application behaves should be reviewed, versioned, and tested against representative cases before it ships.

The parts of a prompt

Most production prompts are assembled from a few distinct pieces:

• System instructions set the model's role and constraints for the whole conversation: what it is, what it should never do, what format its answers should take. These are usually set once by the application developer.
• The user prompt is the specific request or question for this turn.
• Few-shot examples are sample input/output pairs included in the prompt to demonstrate the format or style wanted, instead of describing it in the abstract. For example a model told “extract the date as YYYY-MM-DD”. Few shots improve accuracy and consistency but do not guarantee correct output.
• Retrieved or injected context is additional information supplied for a specific call (a document, a database row, a previous message) that the model needs but wasn't trained on. Retrieved content is the foundation for retrieval-augmented generation.

Structured output and prompt templates

When an application needs to parse a model's response programmatically, asking for a format in plain words is unreliable. Many model APIs support a structured output mode, where you supply a schema (a JSON schema, for instance) and the model's response is constrained to match it. This turns "please respond in JSON" from a request the model might ignore into output the serving infrastructure constrains to the schema, which removes the bugs where a parser breaks because the model added a stray sentence before the JSON. The schema guarantees valid structure, not correct values.

A prompt template is the reusable skeleton behind all of this: fixed instructions with placeholders filled in per call, kept in one place in the codebase instead of rebuilt inline at every call site. Small changes to wording can move output quality a lot, and one template is far easier to test and improve than the same instructions copy-pasted a dozen times with minor drift between the copies.

# One template, defined once and filled in per call
SUPPORT_TRIAGE_PROMPT = """
You are a support ticket triager. Given a ticket, output JSON with
a "category" ("account", "billing", "bug", or "other") and an "urgency" ("low", "medium", or "high").

Ticket: %s
"""

def build_prompt(ticket_text: str) -> str:
    return SUPPORT_TRIAGE_PROMPT % ticket_text

Context management

A long conversation, a large retrieved document, and a growing list of few-shot examples all compete for the same token budget in the context window, and a prompt that grows unbounded over a session eventually gets truncated, slows responses down, and costs more per call. Managing context means deciding what stays in the prompt as a conversation grows: summarizing older turns instead of keeping them verbatim, dropping retrieved content once it's no longer relevant to the current question, and trimming few-shot examples to the smallest set that reliably produces the right format.

Prompt caching

Production prompts often begin with the same long block on every call: system instructions, tool definitions, few-shot examples, sometimes a reference document. Prompt caching lets the provider reuse the processing it already did for that unchanged beginning of the prompt, the prefix, instead of repeating it on every request. Cached input tokens are billed at a reduced rate, and the model can start responding sooner.

Caching comes in two forms. With implicit caching the model/API or the service running the model tries to reuse eligible prefixes on its own. With explicit caching you mark where the reusable prefix ends, using what providers call a breakpoint or checkpoint, which gives you control over what is cached. Providers differ on which form they offer, and some offer both.

Whether a request hits the cache depends on how the prompt is built:

• Stable content first, variable content last. The prefix must match exactly. A value that changes on every request, such as a timestamp, near the top of the prompt prevents reuse of everything after it, and so does a change to the tool definitions.
• Long enough, and reused soon enough. Shorter prefixes aren't cached; providers document minimums from 512 to 4,096 tokens, depending on the model. An entry also expires after a period without use, a few minutes by default on some providers, and each reuse extends it.
• Enough reuse to cover the write cost. Some providers charge more than the normal input rate to write a cache entry, so a prefix that is never reused costs more with caching than without.

Testing prompts

A prompt change that looks like an improvement on the one example you tried it on can just as easily be a regression on ten others you didn't. The fix is a concrete eval set: a fixed list of representative inputs, each paired with the properties a correct output must have, checked automatically every time the prompt changes.

For the support-triage prompt above, the eval set checks properties instead of matching expected outputs word for word:

Running this set against every prompt or model change turns "does this still work" into a question with a yes-or-no answer, and it catches regressions on the edge cases (the ambiguous ticket, the one that fits no category) that a quick manual check skips, because nobody types them in while iterating.