Generative AI Architecture · Part 1 of 8

Foundation Models and How They're Served

A foundation model is a model trained on a broad enough slice of text, code, images, or audio that is useful for many downstream tasks. Understanding Transformers covers the mechanics of how one of these models learns and generates text. Here we discuss ways to use foundation models on an application.

Tokens and context windows

Text is first broken into tokens (roughly word pieces), and the model works in terms of token sequences. Its context window is the maximum number of tokens it can consider at once: the system prompt, the conversation so far, any retrieved documents, and the new response all have to fit inside that same budget. A model that handles a 200-word question comfortably can still fail when a large pile of documents is stuffed into the same window: the request errors out past the limit, or the model loses track of details buried in the middle.

Context windows have grown from a few thousand tokens to a million or more in several models, but a larger window costs money. Cost and latency both grow with how much text the model processes on every call, so an application that stuffs everything into the prompt pays twice: in the bill and in how long the user waits.

Inference, temperature, and sampling

Inference is running a trained model to produce output, as distinct from training, which is how it learned in the first place. Every API call to a model is an inference request. At each step the model computes a probability across the entire vocabulary, and decides which token gets chosen from that distribution.

Temperature is the knob that controls that decision. Near zero, the model almost always takes the single most likely next token, giving consistent, repeatable output. Turned up, it samples more freely from less likely tokens, producing output that varies more from run to run and reads as more exploratory and natural. Low temperature suits tasks with a single correct answer, like extracting a date from a document. Higher temperature suits open-ended generation where variety is a feature, like brainstorming names.

Multimodality and reasoning models

Modern foundation models are increasingly multimodal: the same model can accept and sometimes produce images, audio, or video alongside text, by representing all of them as sequences the same underlying architecture can process. A model that can look at a screenshot and describe a bug, or listen to an audio clip and transcribe it, is doing that through the same next-token mechanism as a text-only model, with a wider range of inputs mapped into the same numerical space.

A newer category, often called reasoning models, is trained and prompted to generate an extended internal chain of intermediate steps before producing its final answer. That extra step trades latency and cost for expected improved accuracy on tasks that need several steps of reasoning, like debugging a piece of logic or planning a multi-step task.

Closed and open-weight models

A model's weights are the learned parameters that encode the knowledge. With a closed model, the developer keeps the weights and offers the model only through an API, run by the developer or by a cloud platform. Anthropic's Claude and OpenAI's GPT models are examples. With an open-weight model, the developer publishes the weights, so anyone can download them. Moonshot AI's Kimi and Alibaba's Qwen models are examples. Some developers offer both kinds: OpenAI publishes gpt-oss, and Google offers its Gemini models through an API while publishing Gemma as open models.

Open weights are not the same as open source: open weights omit the training code, the training dataset, and details about the data. The license on each model sets what you may do with it.

To use open weight models you need to make them accessible.Some options include:

• Call a platform that already hosts the model. Cloud model platforms list open-weight models next to closed ones, billed per token like any hosted API. There is no server for you to run.
• Import your weights into a managed platform. Amazon Bedrock's Custom Model Import, for example, takes model files in Hugging Face format from S3 and serves them behind Bedrock's own API.
• Self host. A serving engine such as vLLM loads the weights onto your GPUs and exposes an HTTP API, including an OpenAI-compatible one, so your application calls it like any hosted model. In this case you operate the endpoint.

Common Model Deployment Options

The optiions below trade off cost, latency, control, privacy, customization, and operational burden:

Core platforms per cloud

Each major cloud vendor offers options in both the hosted-API and managed-platform categories.