Generative AI Architecture · Part 7 of 8
Evaluating AI Systems
Why evals are needed and how they are built and run.
An eval is a repeatable test for an AI system: a dataset of cases, the system run on each case, and graders that score the outputs. The result is a set of scores that can be compared across runs.
Why generative AI needs evals
Outputs vary. Gen AI content is non-deterministic. The same input can produce different outputs. A conventional unit test asserts one exact output and passes or fails. An eval runs many cases, often with repeats of each, and reports the share of cases that met the criteria. Quality is also judged against properties (valid JSON, grounded in the sources, right urgency) rather than an exact string, which is why graders exist. Evals also serve as a regression test, because several parts of the system change over time:
| What changes | How |
|---|---|
| The model | You switch models, or the provider retires one. Different snapshots within one model family can also behave differently. |
| The prompt and configuration | Your own edits to prompts, tools, retrieval settings, or chunking. |
| The retrieved content | The documents behind a RAG (retrieval-augmented generation) system are added to, edited, and deleted. |
| The inputs | Users start asking things the original test set never covered. |
An eval run before a change shows whether scores moved. An eval run on a schedule, or on sampled production traffic, shows drift. Cloud evaluation tooling supports both: continuous evaluation of production traffic, and scheduled evaluation with test datasets.
Responsible use needs evidence. Responsible-AI requires fairness, safety, privacy and security, veracity, and robustness, all of which need to be measured to be enforced. The NIST AI Risk Management Framework says AI systems should be tested before deployment and regularly while in operation, and expects objective, repeatable testing processes to be in place, followed, and documented. For high-risk systems, the EU AI Act (Article 9) requires testing against prior defined metrics and probabilistic thresholds before the system is placed on the market. In an eval, that becomes a threshold agreed before the run that gates the release. A safety evaluator, for example, can score content on a severity scale, mark each response pass or fail against a threshold, and report the share of failing responses as a defect rate.
How an eval works
The workflow has thre main steps: describe the task as an eval, run it with test inputs, then analyze the results and iterate. In practice:
- Define success criteria. Criteria should be specific and measurable, with a threshold, such as fewer than 0.1% of 10,000 outputs flagged for toxicity by a content filter.
- Build the dataset. Each case has an input and either an expected output or the properties a good output must have. Cases should mirror the expected task distribution and include edge and ambiguous cases.
- Run the system on every case. The system under test is the whole application: prompt, retrieval, tools, and model.
- Grade each output (below), aggregate into rates and scores, and compare against the previous run.
Failures found in production can be added to the dataset, so the next change is tested against them.
Graders
| Grader | How it scores | Fits |
|---|---|---|
| Code | Exact or string match, overlap metrics such as ROUGE-L, embedding similarity, schema validation | Clear-cut, categorical answers; checking that output is valid JSON or that a tool was called with the right arguments |
| LLM judge | A second model scores the output against a rubric, on a scale or as yes or no | Qualities that are hard to compute, such as tone, completeness, or whether an answer is supported by its sources |
| Human | Humans rate outputs | Quality and preference ratings that code and judges can approximate only so far |
For example, exact match suits sentiment classification, embedding similarity suits consistency across paraphrased questions, ROUGE-L suits summaries, and LLM judges suit tone and use of context.
A judge is itself a model, and its reliability can be measured. A study of LLM judges (Zheng et al., 2023) found that strong judges reached over 80% agreement with human preferences, the level at which humans agree with each other, and also found position, verbosity, and self-enhancement biases and limited reasoning ability. Guidance for LLM grading:
- Use a different model to judge than the one that generated the output.
- Give the judge explicit scale anchors, such as “1: not at all” and “5: perfectly”.
- Constrain the reply to a parseable format, and validate it in code.
What gets measured
| Measure | What it checks | Graded by |
|---|---|---|
| Task success | Whether the system did what the user needed, by the task's own definition of done | Code or judge |
| Groundedness | For RAG, whether the answer is supported by the retrieved sources | Judge |
| Retrieval quality | Whether retrieval returned the relevant chunks, evaluated apart from generation | Judge against ground-truth passages |
| Tool correctness | For an agent, whether it chose the right tool with the right arguments | Code |
| Safety | Whether output stays within policy, including under adversarial input | Judge or classifier, plus red teaming |
| Fairness | Whether outputs treat social groups equitably, with scores compared across groups in the dataset | Judge or classifier |
| Cost and latency | What each interaction costs and how long it takes | Measured directly |
Offline and online evaluation
Offline evaluation runs the dataset before a release. Cloud evaluation tooling can add automated quality gates to CI/CD (continuous integration and continuous delivery) pipelines, so a change that lowers scores is caught before deployment. Online evaluation scores live traffic, usually a sample of it, and raises alerts when quality or safety scores cross a threshold.
Common evaluation tools
| Cloud | Service | What it provides |
|---|---|---|
| AWS | Amazon Bedrock Evaluations | Model evaluation jobs with built-in or custom prompt datasets, jobs scored by human workers, and jobs scored by a judge model with built-in or custom metrics. RAG evaluations, retrieve-only or retrieve-and-generate, run against a dataset with ground-truth passages and responses. Responses from models and RAG sources outside Bedrock can be evaluated by supplying them in the dataset. |
| GCP | Gen AI evaluation service, in Gemini Enterprise Agent Platform (formerly Vertex AI) | Pointwise and pairwise criteria, and rubric-based evaluation, where rubrics are generated for each entry and can be edited. Models from Google and other providers can be evaluated. Agent evaluation covers the trajectory of actions and tool calls as well as the final output. Results are tracked in Vertex AI Experiments. |
| Azure | Microsoft Foundry observability | Built-in evaluators for quality (coherence, fluency), RAG (groundedness, relevance), safety (hate and unfairness, violence, protected materials), and agents (tool call accuracy, task completion), plus custom evaluators. An AI red teaming agent built on PyRIT, CI/CD quality gates, continuous and scheduled evaluation of production, Azure Monitor alerts, and OpenTelemetry-based tracing. |
Cost per successful task
Cost per token can be misleading. A cheaper model lowers the cost of each call, but if it succeeds less often, the failures cost money too: a retry, a person correcting a wrong answer, a customer asking twice. Cost per successful task divides the total cost of all attempts by the number of tasks completed correctly. The example below is illustrative and assumes a failed attempt is retried until it succeeds.
| Model A | Model B | |
|---|---|---|
| Cost per call | $0.002 | $0.006 |
| Task success rate | 70% | 96% |
| Calls per successful task (failures retried) | 1.43 | 1.04 |
| Cost per successful task | $0.0029 | $0.0062 |
Model A is still cheaper per successful task here. If its success rate were 30% instead of 70%, its cost per successful task would be $0.002 ÷ 0.30 ≈ $0.0067, above Model B's $0.0062, a ranking cost per token alone would not show. The success rates come from the eval, which is why cost is tracked in it.