Skip to content
Quantoros

How do you prepare a dataset for fine-tuning?

LLM & AILevel: IntermediateDataset Preparation

How do you prepare a dataset for fine-tuning?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

Dataset quality matters more than quantity for fine-tuning. Key steps:

  1. Define the task precisely — what is a perfect response? Document it as an evaluation rubric before collecting data.

  2. Data format — instruction-response pairs (Alpaca format), chat format (ShareGPT), or completion format depending on the base model's expected format.

  3. Quality filtering:

    • Remove examples where response doesn't follow instructions
    • Filter by response length (too short = low-quality; outliers = formatting issues)
    • Deduplication (exact + near-duplicate removal)
    • Language filter if needed
  4. Size guidance: 1K examples for style/format tasks; 10K+ for complex behavior; 100K+ for domain-specific language models. Quality > quantity.

  5. Train/validation split — hold out 5–10% for validation; ensure no leakage between splits.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions