Langfuse

What it does

Langfuse is a self-hostable AI engineering platform with traces, prompt management, dataset experiments, LLM judges, code evaluators, annotation queues, and production scores. ClickHouse acquired Langfuse in January 2026 while retaining its MIT-licensed core.

• Offline dataset experiments and side-by-side prompt, model, or code comparisons
• LLM judges, Python and TypeScript evaluators, user feedback, and annotation queues
• Agent traces, CI experiment gates, score analytics, dashboards, and alerts

Where it fits

Teams prioritizing open-source data control and one workflow across tracing, prompts, datasets, experiments, annotation, and evaluation.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Langfuse observes and evaluates agents but does not host or execute them.
• Some governance features need an Enterprise license, and Docker Compose lacks production high availability, scaling, and backups.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.