Skip to content
Quantoros

What is red teaming for LLM systems?

LLM & AILevel: IntermediateRed Teaming & Safety Evaluation

What is red teaming for LLM systems?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Red teaming systematically tests an LLM system for safety vulnerabilities and harmful outputs by adversarially probing it.

Red teaming dimensions:

  1. Jailbreaks: prompts designed to bypass safety training ("DAN", "roleplay as evil AI", etc.)
  2. Prompt injection: injecting instructions through untrusted content
  3. Bias and stereotyping: demographic bias, racial/gender stereotyping
  4. Privacy violations: leakage of training data, PII extraction
  5. Hallucination induction: questions designed to elicit confident wrong answers
  6. Misuse: using the system for harmful purposes (harm instruction generation, CSAM, etc.)

Automated vs manual:

  • Manual red teaming: security researchers with creative adversarial thinking
  • Automated: use an LLM (attacker model) to generate adversarial prompts against the target model
  • Hybrid: automated generation of candidates + human selection and refinement

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions