Giskard is an open-source Python library for testing and evaluating agentic systems. The v3 architecture is a set of modular, focused packages, each carrying only the dependencies it needs, built from scratch to wrap any object: an LLM, a black-box agent, or a multi-step pipeline. These packages are built on three foundational libraries—giskard-core (shared utilities and telemetry), giskard-llm (provider-agnostic LLM routing), and giskard-agents (agent and workflow orchestration)—which are imported automatically and rarely used directly.
Main capabilities
Giskard Checks: Create and apply evaluations (evals) for testing agents. This is a lightweight library for creating evaluations for LLM-based systems, from simple assertions to LLM-as-judge evaluations. Unlike traditional unit tests, evaluations are designed for non-deterministic outputs, where the same input may produce different but valid responses. Uses include: catching regressions (verifying that a system still behaves correctly after changes), validating RAG quality (checking whether answers are grounded in the retrieved context), enforcing safety rules (ensuring outputs comply with content policies), and evaluating multi-turn agents (testing a full conversation, not just a single interaction). Built-in evaluations include string matching, comparison, regex, semantic similarity, and LLM-as-judge checks (Groundedness, Conformity, LLMJudge).
Giskard Scan: A vulnerability scanner for AI agents, the red-teaming and vulnerability scanning layer. It automatically generates adversarial test suites based on a natural language description of the agent, covering prompt injection, harmful content, stereotypes, misinformation, and more. Uses include: automatically generating adversarial inputs for red teaming under the OWASP LLM Top-10 threat categories; running prompt injection probes using the built-in injection payload dataset; and extending custom generators by passing a custom ScenarioGenerator instance to generate_suite, or by registering it on vulnerability_suite_generator_registry.
Key concepts and usage
Target: The system under test, any synchronous/asynchronous callable object, (inputs) -> outputs (optionally with trace). Scenario: A single evaluation, including interactions and checks. Check:Assertions or LLM judgments on traces. Suite: multiple scenarios run together.
giskard.agents.Generator is the LLM client used for workflows/judges, and is different from the input generator (LLMGenerator) in giskard.checks used to synthesize user messages. Groundedness is an LLM judge, requiring installation of a provider extension (for example pip install "giskard[openai]") and setting the corresponding API key. Default model: openai/gpt-4o-mini.
The Scan generator likewise requires an LLM provider extension and API key (the same as the Checks judges above).
Versions and compatibility
Giskard v2 includes Scan (automatic vulnerability detection) and RAGET (RAG evaluation test set generation). For LLM agents, both are replaced in v3 by giskard-scan: use vulnerability_scan instead of v2's LLM scan, and use quality_scan (with KnowledgeBase) instead of RAGET. v3 can also be used for ML models—wrap them as a target and evaluate with giskard-checks or giskard-scan. The automatic tabular scan unique to v2 (a suite of detectors that introspects a giskard.Model + giskard.Dataset to automatically detect performance, bias, and robustness issues), the giskard.testing ML test suite, and Giskard Hub are not planned for v3. Giskard v2 is still available, but is no longer actively maintained.
Environment requirements and telemetry
Requires Python 3.12+. Telemetry: aggregated analytics data is optionally collected through giskard-core, and prompts or outputs are not sent. You can opt out via export DO_NOT_TRACK=1 or export GISKARD_TELEMETRY_DISABLED=1 (or by setting the same keys in a .env file). Setting this before import skips creation of ~/.giskard/id; setting it later will still stop subsequent sending.
Other information
Wewelcome contributions from the AI community. The project provides documentation, a website, and a community (Discord). You can follow v3 announcements and the roadmap. The project accepts GitHub sponsorships; monthly sponsorship provides a sponsor badge, displays the company in the readme, and prioritizes bug reports; one-time sponsorships are also available for consulting projects, workshops, or corporate talks.
The example code blocks described in the original text as "Scan — automatically detect performance, bias & security issues" and "RAGET — generate evaluation datasets for RAG applications" are not provided in the supplied material, so they are not expanded here.
