Vertex AI Gen AI evaluation service

What it does

Vertex AI's evaluation service supports pointwise and pairwise model-based metrics, deterministic metrics, function-call checks, custom computation, and agent trajectory evaluation. It accepts data from pandas, Cloud Storage, or BigQuery.

• Pointwise, pairwise, custom-criteria, exact-match, ROUGE, BLEU, and function-call metrics
• Inference-and-evaluation workflows for supported platform models
• Final-response and trajectory metrics including match, precision, recall, latency, and failure

Where it fits

Vertex AI teams evaluating model outputs, prompts, RAG responses, function calls, and agent trajectories with managed or custom metrics.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Most managed model-based metrics use Gemini 2.5 Flash and consume its throughput quota.
• Agent execution and trajectory metrics have supported-agent and reference-trajectory prerequisites; the first call can incur propagation delay.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.