Amazon Bedrock Evaluations

What it does

Bedrock evaluation jobs can compare models, prompts, knowledge bases, and external RAG sources using built-in metrics, custom datasets, judge models, or human workers. Jobs run through the console, CLI, or SDK and write reports to S3.

• Programmatic and human model-evaluation jobs with built-in or custom prompt datasets
• Judge-model evaluation with explanations and LLM-based RAG evaluation
• Support for Bedrock model types, prompt routers, console, CLI, SDK, and generated reports

Where it fits

AWS teams comparing Bedrock models or assessing knowledge bases and external RAG sources with automated metrics, LLM judges, or human reviewers.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Jobs require IAM roles and S3-backed inputs or outputs; human evaluation adds staffing and review overhead.
• RAG evaluation needs ground-truth retrieved text and responses, and model availability can differ between API and console surfaces.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.