Evaluation platforms
MLflow GenAI evaluation
MLflow's GenAI stack stores datasets, expectations, outputs, traces, scorers, human feedback, and experiment results in the familiar MLflow tracking model. Databricks offers additional managed capabilities, while core MLflow remains open source.
- Maintainer
- MLflow Project
- Deployment
- Open-source self-hosting; managed offerings vary by provider
- Status
- Current as of September 22, 2026
What it does
MLflow's GenAI stack stores datasets, expectations, outputs, traces, scorers, human feedback, and experiment results in the familiar MLflow tracking model. Databricks offers additional managed capabilities, while core MLflow remains open source.
- Evaluation datasets with inputs, expectations, outputs, and traces
- Built-in judges, guideline judges, custom judges, and Python scorers
- Final-answer and intermediate-trace evaluation plus human feedback and monitoring
Where it fits
Teams extending an existing MLflow or MLOps stack to LLM and agent tracing, evaluation-driven development, human feedback, and monitoring.
Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.
Limitations and cautions
- A small SQLite server is not a production topology; serious deployments need deliberate storage, authentication, networking, and backups.
- Managed capabilities vary by provider, and users must supply judge models unless their chosen service includes them.
How to evaluate the evaluator
Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.