AI economics is usually discussed as a training story: giant clusters, months of compute, eye-watering electricity bills. Training is expensive, but it is a one-time capital cost. Inference — answering a request — is a recurring operating cost that scales with every user, every token and every retry.
Training: a large, one-off bill
Pretraining runs the model over trillions of tokens, updating billions of parameters. The cost is dominated by GPU time, high-bandwidth memory, networking and the data pipeline. Fine-tuning and preference training add more, but usually at a fraction of the pretraining bill.
The important property is that training does not scale with usage. Once the weights exist, you can serve one user or a million without retraining. That is why labs can spend heavily upfront and still believe the unit economics will work later.
Inference: rent, not purchase
Every request requires a forward pass through the model. For a chat product, the model generates one token at a time, and each token requires another pass. Longer answers, longer contexts and more concurrent users all increase the load. The bill arrives continuously, in GPU-hours and electricity.
Three things dominate inference cost:
- Model size. Bigger weights mean more memory bandwidth per token.
- Context length. Attention grows with the square of the sequence, and the KV cache grows linearly with every stored token.
- Utilisation. GPUs are expensive whether or not they are busy. Serving at low utilisation is the fastest way to lose money.
Batching and the KV cache
Inference engines batch many requests together so one GPU pass serves several users. That improves throughput and lowers cost per token. But batching increases latency and memory pressure. The KV cache — the stored keys and values for earlier tokens — is often the limiting factor: a long conversation can occupy more memory than the model’s weights.
This is why providers charge differently for input and output tokens, why cached prefixes are cheaper, and why very long contexts carry a premium. It is also why “unlimited” chat plans tend to acquire fair-use limits.
Making a token cheaper
There are four common levers. Quantization stores weights in 8 or 4 bits instead of 16, cutting memory and bandwidth. Distillation trains a smaller model to imitate a larger one. Speculative decoding lets a small draft model propose tokens that the large model verifies. Caching avoids recomputing shared prefixes, system prompts and retrieved documents.
Each lever trades quality, latency or engineering complexity for cost. The right mix depends on the product: a coding assistant can tolerate more latency than a voice agent; a batch summariser can tolerate much more than either.
Why the headline numbers keep moving
Training costs are visible and dramatic, so they get the coverage. Inference costs are diffuse — spread across cloud bills, GPU rentals and electricity — but they compound. As models become good enough for everyday use, the balance shifts: the lifetime cost of serving a popular model can exceed the cost of building it.
For anyone reading AI business claims, the useful question is not only “how big was the training run?” but “how many tokens does this product serve, at what utilisation, and who pays for the idle capacity?” That is where the business actually lives.