Arize Phoenix

What it does

Phoenix combines LLM and agent tracing with versioned datasets, experiments, LLM-based evaluation, retrieval evaluation, and prompt tooling. It can be self-hosted through Docker or Kubernetes and also connects to Arize's managed products.

• OpenTelemetry-based tracing for LLM and agent applications
• Response and retrieval evals tied to versioned datasets and experiments
• Prompt comparison, APIs, visual analysis, and provider-agnostic integrations

Where it fits

Teams that want evaluation connected to OpenTelemetry traces, datasets, experiments, prompt iterations, and production troubleshooting.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• The Elastic License 2.0 is source-available rather than OSI-approved open source and restricts managed-service redistribution.
• Phoenix is operationally heavier than an embedded test library, and configured external model providers can still receive evaluation data.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.