About
SafeRAG is an open-source security benchmarking framework developed collaboratively by researchers at Renmin University of China, the Institute for Advanced Algorithms Research (IAAR) in Shanghai, and Beihang University. It addresses the critical and growing need to evaluate the resilience of Retrieval-Augmented Generation (RAG) pipelines against adversarial threats before production deployment. The framework introduces a structured Threat Framework that executes four categories of attacks—Noise (misleading retrieved content), Conflict (contradictory document injection), Toxicity (harmful content insertion), and Denial-of-Service (DoS)—at multiple stages of a RAG pipeline. These attacks are purpose-built to bypass typical RAG security components and surface real-world vulnerabilities in LLM-based systems. SafeRAG ships with curated NCTD benchmark datasets, modular retrieval configurations, embedding utilities, knowledge base setups, task definitions, and a quantitative metrics layer—making security testing systematic, reproducible, and comparable across different RAG architectures. The tool is particularly valuable for AI researchers studying RAG robustness, ML engineers building production RAG applications, and security professionals conducting audits of enterprise LLM deployments. Teams can use it to proactively identify weaknesses in retrieval, augmentation, and generation phases, compare RAG configurations for relative security, and develop and validate new defenses before going live.
Key Features
- Comprehensive Threat Framework: Executes four attack categories—Noise, Conflict, Toxicity, and Denial-of-Service—at multiple stages of the RAG pipeline to systematically expose security weaknesses.
- NCTD Benchmark Datasets: Includes curated adversarial datasets (NCTD) specifically designed for reproducible RAG security evaluation and comparison.
- Modular & Configurable Pipeline: Provides plug-and-play retrieval modules, embedding utilities, knowledge base setups, and task definitions for flexible security testing across RAG configurations.
- Quantitative Security Metrics: Built-in metrics layer measures attack impact quantitatively, enabling data-driven comparison of RAG system resilience.
- Multi-Stage Attack Simulation: Targets vulnerabilities in retrieval, augmentation, and generation phases independently, providing granular insight into pipeline weak points.
Use Cases
- Evaluating the adversarial robustness of a production RAG pipeline before enterprise deployment
- Comparing the relative security of different RAG configurations and retrieval strategies
- Conducting academic research on LLM security, RAG attack vectors, and defense mechanisms
- Developing and validating new defenses against adversarial attacks in document retrieval systems
- Auditing enterprise LLM applications for vulnerabilities across retrieval, augmentation, and generation phases
Pros
- Broad Attack Coverage: Covers the full spectrum of known RAG adversarial attack types across all pipeline stages, offering thorough security evaluation.
- Research-Backed & Open-Source: Developed by leading academic institutions and freely available, ensuring credibility and accessibility for researchers and practitioners.
- Reproducible Benchmarking: Modular architecture with versioned configs, datasets, and evaluators ensures experiments are fully reproducible and results are comparable.
Cons
- High Technical Barrier: Designed exclusively for researchers and engineers; there is no GUI or beginner-friendly interface, requiring Python proficiency and ML knowledge.
- Small Community: As a specialized academic tool, it has a limited user base and support ecosystem, which may slow issue resolution and feature development.
- Python-Only: Currently restricted to Python environments, limiting adoption by teams working in other programming languages or tech stacks.
Frequently Asked Questions
What is SafeRAG?
SafeRAG is an open-source benchmarking framework that evaluates the security of Retrieval-Augmented Generation (RAG) systems by simulating adversarial attacks including Noise, Conflict, Toxicity, and Denial-of-Service (DoS) at various stages of the RAG pipeline.
What types of attacks does SafeRAG simulate?
SafeRAG simulates four attack categories: Noise (irrelevant or misleading retrieved content), Conflict (injection of contradictory documents), Toxicity (insertion of harmful content), and Denial-of-Service (DoS) attacks designed to degrade or disable the RAG system.
Who is SafeRAG built for?
SafeRAG targets AI researchers studying RAG robustness, ML engineers building production LLM applications, and security professionals auditing enterprise retrieval-augmented systems for vulnerabilities.
Is SafeRAG free to use?
Yes, SafeRAG is fully open-source and hosted on GitHub, free for both research and commercial use.
How do I get started with SafeRAG?
Clone the GitHub repository and run the provided quick_start_nctd.py script. The repo includes configuration files, NCTD datasets, retrieval modules, and evaluation scripts to guide you through your first security benchmark.
