Create an Evaluation Set for Your AI App

An evaluation set is a collection of cases that show whether your AI application behaves as intended. It should include ordinary requests and failures you care about. Building it early gives you a reference when you change prompts, models, retrieval settings or the surrounding product.

Why it matters

A demo answers whether a system can work once. An evaluation asks where it works, where it fails and whether a revision improves the situations that matter to users.

A practical workflow

1. Collect realistic inputs with permission and remove sensitive information before storing reusable test cases.
2. Write expected properties for each result, including required evidence and behaviour when information is missing.
3. Include ambiguous, out-of-scope and adversarial inputs alongside straightforward examples from the main workflow.
4. Run the same set after meaningful changes and review serious failures individually rather than hiding them in an average.

An example

For a policy assistant, one case should contain an answer in the source, one should require an exception and one should have no supported answer. A system that replies to all three confidently is not necessarily performing well.

A mistake to avoid

Do not repeatedly tune against your entire evaluation set until it becomes a collection of memorised examples. Keep fresh cases aside and update the set when real users reveal a new failure pattern.

Try it and check the result

Compare the current version with one proposed change. Report which cases improved, which regressed and whether any critical failure remains, along with the amount of manual review the results required.

Reference and further reading

NIST AI Risk Management Framework.

The workflows and examples in this guide are illustrative. Product controls and availability can change; check the linked official guidance for your setup.

Continue with Build Your First Document Q&A Prototype Before Adding Complexity or explore all blog topics.