How do you prevent agents from taking irreversible harmful actions?
LLM & AILevel: IntermediateSafety & Reliability
How do you prevent agents from taking irreversible harmful actions?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
This is the most critical safety challenge for deployed agents.
1. Minimal capability: give agents only the tools they need for their specific task. A research agent doesn't need write access to production databases.
2. Action classification:
- Reversible: read operations, draft creation → execute freely
- Recoverable: writes with undo → log before execution
- Irreversible: delete, send, deploy → require explicit confirmation
3. Human-in-the-loop checkpoints: for irreversible actions, pause and request user confirmation. Use risk scoring to decide which actions require it.
4. Dry-run mode: before executing a consequential action, show the user what would happen. User confirms before real execution.
5. Sandboxing: execute code in isolated containers; use read replicas for database queries; never give production credentials to an agent running user-provided code.
6. Audit logging: every action logged with full context — what was executed, why, and by which agent. Essential for incident response.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0