Skip to content
Quantoros

Article

Training is sexy, inference pays the bills

Why much of the interesting AI engineering in the coming years will happen around inference: memory bandwidth, KV-cache, batching, quantization and the real cost of serving.

For the past few years, the AI industry has been focused on training ever larger models. More parameters, more GPUs, more compute in general. But judging by where things are heading, a significant share of the interesting engineering over the next few years will happen around inference optimization.

Inference is eating more and more compute

McKinsey recently published a study projecting that by 2030 around 60% of AI's total compute demand will go to inference, with the remaining 40% going to training.

That's to be expected. Interacting with an LLM used to mean a handful of queries to ChatGPT; now we have agents, reasoning models, multimodal systems, and so on and so forth. A single user task can trigger dozens of model calls, process huge contexts and generate thousands of intermediate tokens. The number of users may barely change while compute consumption grows several-fold.

Training and inference hit different limits

Now for the technical part. Training and inference often run into completely different bottlenecks. In training, large matrix operations keep the GPU's compute units well utilized. In autoregressive generation, especially at small batch sizes, the bottleneck is often memory bandwidth. At every step you have to read the model weights and work with the KV-cache just to produce one next token. You can end up with a GPU rated at hundreds of TFLOPS that spends most of its time waiting on data from HBM.

You can buy a GPU that's twice as powerful and get nowhere near a matching speedup, because the problem isn't compute at all. Or you can increase the batch size, cut down the amount of data moved, or organize memory access properly, and get a far more noticeable gain on the same hardware.

Hence the whole story with continuous batching, PagedAttention, quantization, speculative decoding, custom CUDA kernels and distributed inference. And speeding things up doesn't always mean diving into low-level code. The vLLM authors reported 2-4x throughput gains largely just by no longer wasting so much memory on the KV-cache. The model didn't get any smarter, no extra GPUs appeared, but you can serve significantly more requests.

A faster kernel is not a faster service

And there's another problem. Speeding up an individual CUDA kernel and speeding up a real service are two entirely different tasks. You can spend a week optimizing a matrix multiplication, celebrate a 2x speedup in benchmarks, and then find out the gain in production was 3%. Because half the time goes into waiting for requests, forming batches, moving data, or inter-GPU communication. Meanwhile, changing the request scheduler sometimes gives you more than all that kernel tinkering.

Quantization is much the same story. In theory, INT4 lets you store weights several times more compactly than FP16, but that doesn't automatically mean a 4x speedup. There's dequantization overhead, not every operation runs efficiently at low precision, and the system as a whole may be bottlenecked somewhere else entirely. So looking only at model size or tokens per second on a single request is meaningless.

Cheaper tokens, bigger bill

By the way, there's one more point. The cheaper inference gets, the more compute we end up consuming. Cut the cost of generation in half? Now you can let the agent make twenty model calls instead of five. Or run several models in parallel, extend the context, add extra checks. In the end each token is cheaper, yet the total GPU bill somehow only keeps growing.

More broadly, the whole race for parameter count is starting to look a bit one-sided. Say you have two models. One is slightly better on benchmarks, but costs three times as much to serve, and its latency under load becomes unacceptable. The other is a bit weaker, but handles requests reliably and meets the product's requirements. Which one is better? For a research paper, maybe the first. For a business, that's far from certain.

So amid the race for new architectures and benchmark records, it's easy to miss a fairly simple point: making a model smarter is one problem, and making it affordable to actually use is a completely different one. And it's clearly too early to call the second one solved.

Sources

New articles by email