What is Rotary Position Embedding (RoPE)?
LLM & AILevel: IntermediateAttention Mechanisms
What is Rotary Position Embedding (RoPE)?
Hints
- What does closeness mean in this representation and metric?
- Check normalization, exact versus approximate search and alignment with the training objective.
Answer
RoPE encodes positional information by rotating the Q and K vectors by an angle proportional to position:
- Position i: rotate Q by θᵢ; position j: rotate K by θⱼ
- The dot product QᵢKⱼ naturally encodes the relative position (i-j), not absolute positions
- Generalizes to longer sequences than trained on (with YaRN/RoPE scaling)
Used in LLaMA, Mistral, Falcon, Qwen — essentially all modern open-source LLMs. Replaced learned positional embeddings and sinusoidal encodings because of superior length generalization.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0