Skip to content
Quantoros

What is a Large Language Model (LLM) and how does it work?

LLM & AILevel: FoundationArchitecture & Core Concepts

What is a Large Language Model (LLM) and how does it work?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

An LLM is a neural network — primarily a Transformer decoder — trained on massive text corpora to predict the next token. Key mechanisms:

  1. Tokenization — text split into subword tokens via BPE/WordPiece
  2. Embeddings — tokens mapped to dense vectors via a lookup table
  3. Self-Attention — each token attends to all previous tokens to build contextual representations
  4. Feed-Forward layers — non-linear transformations applied token-wise
  5. Autoregressive generation — outputs one token at a time, conditioning on all previous tokens

Training uses next-token prediction (cross-entropy loss) over trillions of tokens, followed by SFT (instruction tuning) and RLHF/DPO for alignment.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions