Skip to content
Quantoros

How does the pretraining objective shape what a model is good at?

LLM & AILevel: IntermediateFine-tuning & Alignment

How does the pretraining objective shape what a model is good at?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

The objective is the exact prediction task you reward during pretraining, and it quietly decides the model's lifelong strengths. Causal LM predicts the next token from the past, which builds fluent generators.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions