What this video shows
The speakers use a banking assistant to show the loop end to end. They start with tracing and mask personally identifiable information before it reaches the store, then build datasets from real traffic. Domain experts review outputs, and their comments become named failure modes rather than a single quality score.
They use both LLM judges and code-based checks, then run the suite automatically on every prompt change so a regression blocks the change. The tooling is Langfuse's, so the workflow doubles as a product walkthrough. The lasting value is the order: instrument first, collect cases, name failures, then gate releases. Treat the specific dashboards as one implementation of that order.
Read Langfuse's observability overview for the tracing model used in the workshop. Compare its evaluation overview for datasets, evaluators, and automated runs.
What you will learn
- Mask sensitive fields at the trace layer so evaluation does not create a new privacy problem.
- Build the dataset from real requests, then have experts label failures with names, not a single score.
- Use code checks for facts a program can verify and judges for qualities it cannot.
- Make the suite a release gate so a prompt change that lowers scores cannot merge.
How to apply this safely
- Turn on tracing and mask fields that identify a person before they are stored.
- Collect a few hundred real requests and have an expert label pass, fail, and failure type.
- Write one code evaluator and one judge, then check how well each matches the human labels.
- Run the suite on every prompt change and block merges that drop the score below the baseline.
Important limitations
- The workshop demonstrates Langfuse, so the interface and features describe that product rather than a neutral comparison.
- Judge-based scores drift with the judge model, so they need their own periodic review against human labels.
- The banking example is a teaching scenario, so its failure modes are illustrative rather than a real incident history.
Sources to check
- Langfuse observability Tracing, sessions, and PII handling for LLM applications.
- Langfuse evaluation Datasets, evaluators, and automated evaluation runs.
Continue learning on Learnetto
AI evals guide
Design datasets, graders, and regression tests for AI systems.
Best AI agent evaluation courses
Study tracing, scoring, and agent evaluation in depth.
Long context or retrieval
Test the retrieval choice on the same fixed question set.