giskard

Practical Jupyter notebooks from Andrew Ng and Giskard team's "Red Teaming LLM Applications" course on DeepLearning.AI.

✍️ Collaborate on writing technical content for the Giskard Community

Mutant is a red team and behavioral data generation library for LLMs, RAG pipelines, and AI Agents. It provides two powerful capabilities

Adversarial LLM red-teaming with Giskard: automated vulnerability scans of DeepSeek-R1, GPT-4o-mini & Llama 3.2 (prompt injection, hallucination, harmful output) plus tabular ML scanning, with per-model HTML reports

🎓 As part of a course project with a classmate, we explored prompt injection attacks by directly interacting with large language models (LLMs) to understand their vulnerabilities. We also presented how the open-source tool Giskard can help secure these models through automated testing and vulnerability detection.

LLM evaluation & AI safety suite for a live chatbot — custom LLM-as-a-Judge metrics, jailbreak/prompt-injection red teaming, RAG groundedness (Ragas, Giskard), Langfuse tracing, plus Playwright E2E and k6, gated nightly in CI.

Develop a system to generate diverse types of subjective questions from PDFs and automate the grading of answers.

A RAG system to answer questions about flights

Multi-agent LLM-driven SOC pipeline (n8n + Ollama), adversarially red-teamed against the CSA Agentic AI Red Teaming Guide.

Independent SR 11-7-style model validation of an ML trading strategy - deepchecks + giskard, formal report, reproducible. Verdict: not approved for deployment.

The complete pipeline for a fine-tuning of a classifier model.

PRML pre-registration for Giskard scenario results: commit an eval claim to a SHA-256 before the run, then verify the result against it.

Different approaches to evaluate RAG !!!

Security testing toolkit for AI agents