Giskard

What it does

Giskard v3 is an Apache-2.0, async-first framework for scenario tests, natural-language judges, groundedness checks, and vulnerability scans. It targets agents and chat systems rather than the tabular-model focus of the older v2 product.

• Scenario-based pass/fail checks over single- and multi-turn interactions
• Python assertions, natural-language judges, groundedness, conformity, and semantic checks
• Prompt-injection, jailbreak, harmful-content, stereotype, and misinformation scans

Where it fits

Behavioral tests and adversarial scans of multi-turn agents, chatbots, and RAG systems from a pytest-compatible workflow.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Version 3 is a rewrite that requires Python 3.12 or newer; v2 is no longer actively maintained.
• Scans can consume many model calls, quality scans require a knowledge base, and optional telemetry is enabled by default.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.