Hello Reader,

Six and a half years at AWS taught me a lot. But one pattern stood out above everything else.

Everyone is talking about which model scored higher on the latest benchmark.

Very few people are talking about the thing that determines whether any of those technologies actually create business value.

That thing is AI Evals. And it is the most important AI skill, even though fanicer names like MCP, Agents, Skills get all the attention.

Why AI Failures Are Different

Traditional software fails loudly - service crashes, timeout fires, an alert goes off.

AI systems fail quietly.

A customer support chatbot returns a fluent, confident answer that is factually wrong. A coding assistant generates code that compiles perfectly but introduces a subtle production bug. A financial assistant provides guidance that violates a compliance requirement, in polished, authoritative prose.

Without evaluation systems in place, teams mistake confidence for correctness.

Manual testing works for twenty users. Maybe fifty. Definitely not fifty thousand. The moment an AI application reaches production scale, manual inspection becomes impossible. You need a system built around measurement, not just a model built around capability.

What AI Evals Actually Are

AI Evals are structured methods for measuring whether an AI system is accomplishing its intended objective. Think of them as quality assurance for intelligence.

Traditional software teams run unit tests, integration tests, and security tests. AI teams need a parallel discipline:

Accuracy tests - Did the model return the correct answer, not just a plausible one?

Hallucination tests - Did the model invent information not grounded in its context?

Relevance tests - Did the system actually address the user's question, or just respond in the general topic area?

Retrieval tests - For RAG systems, did the right documents get retrieved in the first place?

Cost tests - Is the per-interaction cost sustainable at production scale?

Safety tests - Does the output meet compliance, legal, and policy requirements?

The goal is not to measure how smart the model is. The goal is to determine whether your product works.

The Core of AI Eval

To evaluate, first you need a reference.

You provide:

• A sample prompt (the input)
• A reference answer (what the correct or ideal response looks like)
• The LLM response being evaluated (what your production model actually returned)

A process then compares the production response against the reference answer and scores it on whatever dimensions you define: accuracy, relevance, completeness, safety, etc.

This process can be Bedrock evaluations, or LLM-as-a-Judge, or your own process using some standard eval library.

One important nuance: not every eval requires a reference answer. There are two main patterns:

Reference-based evaluation : You provide the ideal answer. The judge scores how close the model's response is to that reference. Best for factual, deterministic tasks like Q&A or compliance checks.

Reference-free evaluation : No ideal answer provided. The judge scores the response on its own merits: is it coherent, relevant, safe, complete? Best for open-ended tasks like summarization or creative generation where there is no single correct answer.

For most enterprise AWS use cases, you want both. Reference-based for your critical factual workflows, reference-free for everything else.

The AWS Eval Stack

Here is the good news: if you are already on AWS, you have native evaluation capabilities sitting in services you use every day. Most practitioners have never clicked on them.

Amazon Bedrock Evaluations

This is your starting point for model selection and RAG quality measurement. Bedrock Evaluations lets you run automatic evaluation jobs against foundation models. Bring your own prompt dataset tailored to your specific use case, or use built-in datasets (only for certain cases). It also covers end-to-end RAG workflow evaluation through Amazon Bedrock Knowledge Bases.

Bedrock Evaluations also supports LLM-as-a-Judge natively. This one is gaining popularity. You can designate a model to automatically score outputs for accuracy, relevance, and safety at scale, without manual review. Before committing to any model for a production use case, run them through Bedrock Evaluations against your actual business data. You can optionally provide reference response for this one.

Ahem, you also have the option to do this entirely by human, where a SME will rate the answers manually. This shows that your knowledge of business domain, and the system is still critical!

Amazon Bedrock AgentCore Evaluation

This is the newest addition and the most relevant for teams building agentic systems. Added at re:Invent 2025, AgentCore Evaluation gives you structured evaluation at the agent level: agent decisions, tool call behavior, and end-to-end task completion. That is a meaningfully different problem than evaluating a single model response, and now AWS has a native answer for it.

SageMaker Clarify

If you are training or fine-tuning your own models, Clarify is the evaluation layer to reach for. Clarify measures quality and responsibility metrics including accuracy, toxicity, semantic robustness, bias detection, and explainability. For organizations in healthcare, finance, or legal, these are not optional.

fmeval (AWS Open Source Library)

fmeval is AWS's open-source evaluation library for engineers who want evaluation logic embedded directly in code pipelines. Run it as part of your CI/CD pipeline so every code change triggers automatic quality checks before anything touches production. This is the same rigor as PyTest for traditional software, applied to your AI stack.

Amazon Nova as LLM-as-a-Judge on SageMaker

For teams who need more control over the judge workflow, AWS offers a code-driven implementation using Amazon Nova as the judge model on SageMaker. This runs inside a SageMaker training job on GPU instances unlike fully managed in Bedrock. You generate candidate responses from two or more models, pass them to Nova, and get back statistically rigorous comparison results helping you pick the better model.

How to Choose

Pick one entry point based on where you are:

Building a RAG system on Bedrock? Start with Bedrock Evaluations for end-to-end RAG workflow quality.

Training or fine-tuning your own models? Start with SageMaker Clarify.

Building agents on AWS? Start with Bedrock AgentCore Evaluation.

Engineering-led team with existing CI/CD culture? Add fmeval to your pipelines from day one.

Running comparison experiments across multiple models? Use LLM-as-a-Judge either on Bedrock or Sagemaker depending on which service you are using. Currently LLM-as-a-Judge in Bedrock can accept reference answer and Sagemaker can't.

What This Means for Your SA Career

Prompt engineering is getting commoditized. Model selection is getting easier. Frameworks are multiplying every quarter.

Evaluation remains difficult, because it requires judgment. You need to understand business requirements, user behavior, failure modes, system architecture, and cost tradeoffs simultaneously. That combination is hard to automate, which means it becomes increasingly valuable to the people who develop it.

In SA interviews, the difference between a good answer and a delightful answer on selecting models on this topic is significant.

An average answer: "I would use Bedrock and pick the best model based on published benchmarks"

A delightful answer: "I would run Bedrock Evaluations using LLM-as-a-Judge, scoring each candidate model against our actual business data on accuracy, relevance, and safety before selecting the model. As a SME, I will also manually validate few of the answers to ensure the eval job is rating the answers appropriately."

That is the answer that gets you hired.

What To Do Right Now

Open the Bedrock console and find the Model Evaluation tab. Most AWS practitioners have never clicked it. Spend 30 minutes running an evaluation job against two models on a dataset from your own domain. You will immediately understand why this matters more than reading another benchmark report.

If you want me to make a detailed video on it, please reply and let me know. We just ran a detailed Eval hands-on in the SA Bootcamp where students built and ran their own evaluation jobs - if you want that level of depth, check out sabootcamp.com.

Keep learning and keep rocking 🚀,

Raj

P.S. If you have found this newsletter helpful, and want to support me 🙏:

Checkout my bestselling courses on AWS, System Design, Kubernetes, DevOps, and more: Max discounted links​

Checkout my YouTube channel for Cloud Gen AI tutorial and interview prep videos: Here​

AWS SA Bootcamp with Live Classes, Mock Interviews, Hands-On, Resume Improvement and more: https://www.sabootcamp.com/​