Skip to content
Quantoros

What is Rotary Position Embedding (RoPE)?

LLM & AILevel: IntermediateAttention Mechanisms

What is Rotary Position Embedding (RoPE)?

Hints
  • What does closeness mean in this representation and metric?
  • Check normalization, exact versus approximate search and alignment with the training objective.

Answer

RoPE encodes positional information by rotating the Q and K vectors by an angle proportional to position:

  • Position i: rotate Q by θᵢ; position j: rotate K by θⱼ
  • The dot product QᵢKⱼ naturally encodes the relative position (i-j), not absolute positions
  • Generalizes to longer sequences than trained on (with YaRN/RoPE scaling)

Used in LLaMA, Mistral, Falcon, Qwen — essentially all modern open-source LLMs. Replaced learned positional embeddings and sinusoidal encodings because of superior length generalization.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions