How do you prepare a dataset for fine-tuning?
LLM & AILevel: IntermediateDataset Preparation
How do you prepare a dataset for fine-tuning?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Dataset quality matters more than quantity for fine-tuning. Key steps:
-
Define the task precisely — what is a perfect response? Document it as an evaluation rubric before collecting data.
-
Data format — instruction-response pairs (Alpaca format), chat format (ShareGPT), or completion format depending on the base model's expected format.
-
Quality filtering:
- Remove examples where response doesn't follow instructions
- Filter by response length (too short = low-quality; outliers = formatting issues)
- Deduplication (exact + near-duplicate removal)
- Language filter if needed
-
Size guidance: 1K examples for style/format tasks; 10K+ for complex behavior; 100K+ for domain-specific language models. Quality > quantity.
-
Train/validation split — hold out 5–10% for validation; ensure no leakage between splits.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0