Skip to content
Quantoros

Guides

LLM engineer roadmap: how to become an AI engineer

· 10 min read

LLM engineering is about making language models useful and reliable inside products. Prompting is the easy part; the job is retrieval, evaluation, fine-tuning, serving and knowing when the model is wrong. Here is what to learn and in what order.

What an LLM engineer does

An LLM or AI engineer designs systems around language models: chooses the model, connects it to data through retrieval, constrains outputs, adds tools, measures quality and cost, and ships it safely. The best engineers treat the model as an unreliable component and build evaluation and guardrails around it.

Prerequisites

  • Solid Python, APIs, async code, testing and basic web services.
  • ML basics: train/validation/test, metrics, overfitting.
  • Deep learning basics in PyTorch; ideally you have trained a small transformer yourself.

Understand the transformer

You do not need to pretrain large models, but you should know tokenisation, embeddings, attention, the decoder-only architecture, context windows, sampling (temperature, top-p) and why models hallucinate. Implementing a small GPT once makes every later decision clearer.

Retrieval-augmented generation (RAG)

  • Chunking strategies and metadata.
  • Embeddings, vector indexes and hybrid search (dense plus keyword).
  • Reranking with cross-encoders.
  • Citations: answers that point to verifiable sources.
  • Evaluating retrieval (recall@k) separately from answer quality.

Structured outputs, tools and agents

  • JSON schemas and validated structured output.
  • Function calling and tool design with clear contracts.
  • Agent loops with step limits, timeouts and safe, permissioned tools.
  • Prompt injection and data exfiltration risks, and how to contain them.

Fine-tuning

Fine-tune when prompting and retrieval are not enough: for a consistent format, a domain style or a smaller, cheaper model. Learn supervised fine-tuning (SFT), parameter-efficient methods such as LoRA and QLoRA, dataset curation, and how to prove the adapter improved quality without breaking other behaviour.

Evaluation and regression testing

  • Golden datasets built from real user questions.
  • Automatic metrics where possible, LLM-as-judge with calibration against human labels.
  • Regression tests that run before every prompt or model change.
  • Tracking cost and latency alongside quality.

Serving and operations

  • Quantisation (for example GGUF or 4-bit) and its effect on quality.
  • Inference servers such as vLLM, batching and the KV cache.
  • Observability: traces, failures, user feedback and reproducible releases.

Portfolio projects

  • A RAG assistant over a real document set, with retrieval and answer evaluations.
  • A LoRA fine-tune with a before/after evaluation report.
  • A tool-using agent with explicit limits and a test suite of failure cases.
  • A locally served model with a latency and cost benchmark.

Working with a mentor

Quantoros has an LLM & AI Engineering track of 8 sprints — from the transformer to controllable generation, SFT and LoRA, structured output and function calling, RAG with verifiable sources, evaluation before release, local serving with GGUF and vLLM, safe agents and observability. It builds on the Deep Learning stage, where you write a small GPT yourself. Mentoring is 1:1 with code review and mock interviews.

Resources

Questions

Do I need to be an ML engineer first?

Not necessarily, but ML basics and some deep learning make you much stronger, especially for evaluation and fine-tuning.

Is prompt engineering a career?

On its own it is narrow. Employers hire engineers who can build retrieval, evaluation, tools and serving around models.

Should I use closed APIs or open models?

Learn both. Open models teach you serving, quantisation and fine-tuning; APIs are common in products.

What is the most important skill?

Evaluation: being able to measure whether a change made the system better.

Mentoring directions