Generative AI Architecture · Part 8 of 8

GPUs, TPUs, and AI Infrastructure

Hardware considerations for running generative AI models.

Large-model calls, a hosted API request or a self-hosted fine-tuned model alike, ultimately run on hardware. The right choice can help improve performance and reduce costs.

Why not a CPU

A CPU executes a small number of instruction streams flexibly, which makes it good at the kind of varied, branching logic traditional software is built from. A transformer's core operation is different, with the same simple operation (multiply and add) repeated an enormous number of times, in parallel, across huge grids of numbers. GPUs (graphics processing units, originally built for video rendering) and TPUs (tensor processing units, built by Google specifically for this kind of workload) are both designed around doing that one kind of math at massive scale, at the cost of being far less flexible than a CPU for general-purpose code. AWS builds its own accelerators: Trainium (for both training and inference), and Inferentia, an inference-focused line.

Training vs. inference

Training is the process that produces a model's weights: enormous amounts of data pushed through the model repeatedly, with every pass also computing how to adjust every parameter. It's the most hardware-intensive workload in a generative AI system; frontier models train on thousands of accelerators for weeks. Inference is running the already-trained model to produce a single response, several orders of magnitude cheaper per run but performed constantly, for every request, for as long as the system is live. Over a model's lifetime, inference usually accounts for more hardware spend than training, even though training gets the attention. Many startups never train a model from scratch and pay only for inference, or for a fine-tuning run.

Memory, batching, quantization, utilization

A few factors determine how efficiently that hardware is used:

• Memory. A model's weights, plus the working memory needed to process a request (the KV cache covered in Understanding Transformers is a major part of this at inference time), all have to fit on the accelerator. A model too large for a single chip's memory has to be split across several, adding communication overhead between them.
• Batching. Processing several requests together, instead of one at a time, uses the hardware's parallelism far more efficiently, since the same matrix operations run across a bigger batch at much less than the cost of running them one request at a time.
• Quantization. Storing and computing with lower-precision numbers (fewer bits per weight) shrinks a model's memory footprint and speeds up computation, usually at the cost of reduced accuracy.
• Utilization. The percentage of an accelerator's capacity doing useful work at any given moment. Idle accelerator time is waste and costly.

Why price per GPU-hour is the wrong metric

Comparing hardware options on the advertised price per GPU-hour ignores how differently the same task uses that hour. A cheaper accelerator that takes twice as long to finish a training run, or handles half as many requests per second, can be the more expensive choice once the job is accounted for. Better metrics measure the outcome of the work:

• Cost per completed training run, the total spend across every hour of hardware time a full training run took.
• Tokens per second, a direct throughput measurement of how much useful inference work a given setup produces.
• Cost per million successful requests, the "cost per successful task" idea applied to infrastructure.

Accelerator B looks two and a half times as expensive by the hour. It's the cheaper choice by the time the training run finishes, because it gets there in less than a third of the time. Tokens per second and cost per million successful requests can reverse a comparison the same way.

Common options