LangSmith

What it does

LangSmith supports offline and online evaluation with code checks, human review, pairwise comparisons, and LLM judges. Production traces can be sampled, scored, inspected, and added back to datasets as regression cases.

• Offline experiments over curated, historical, or synthetic datasets
• Human, code, LLM-as-judge, and pairwise evaluators
• Production run and thread evaluation, sampling, alerts, and trace-to-dataset loops

Where it fits

Agent teams, especially LangChain and LangGraph users, that need tracing, datasets, experiments, human review, and production feedback in one system.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Self-hosting is an Enterprise add-on rather than a generally available open-source edition.
• The self-hosted stack includes several datastores and an arbitrary-code-execution backend; online judges require cost controls.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.