What is red teaming for LLM systems?
LLM & AILevel: IntermediateRed Teaming & Safety Evaluation
What is red teaming for LLM systems?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Red teaming systematically tests an LLM system for safety vulnerabilities and harmful outputs by adversarially probing it.
Red teaming dimensions:
- Jailbreaks: prompts designed to bypass safety training ("DAN", "roleplay as evil AI", etc.)
- Prompt injection: injecting instructions through untrusted content
- Bias and stereotyping: demographic bias, racial/gender stereotyping
- Privacy violations: leakage of training data, PII extraction
- Hallucination induction: questions designed to elicit confident wrong answers
- Misuse: using the system for harmful purposes (harm instruction generation, CSAM, etc.)
Automated vs manual:
- Manual red teaming: security researchers with creative adversarial thinking
- Automated: use an LLM (attacker model) to generate adversarial prompts against the target model
- Hybrid: automated generation of candidates + human selection and refinement
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0