The first question people ask about local models is “how big a model can I run?” The honest answer is a memory calculation, not a brand name. Once you can do that calculation, the marketing around “runs on a laptop” becomes much easier to read.
Parameters are not bytes
A 7-billion-parameter model stores 7 billion numbers. In 16-bit precision that is about 14 GB before any overhead. In 8-bit it is roughly 7 GB. In 4-bit it is roughly 3.5–4 GB. Quantization compresses the weights by storing each number with fewer bits; the model does not shrink in parameter count, only in storage.
Add the KV cache for the context, the runtime’s own overhead and a little headroom, and a 7B model at 4-bit fits comfortably on a 8 GB machine for modest contexts. A 13B model at 4-bit wants 8–10 GB; a 70B model wants 40 GB or more even at 4-bit.
Memory bandwidth is the real bottleneck
Token generation is mostly a memory-bandwidth problem. For every token, the model reads its weights once. A GPU with more bandwidth can feed the arithmetic units faster; a CPU with fast RAM can still be surprisingly usable, but it will not match a modern accelerator. This is why a small model on a good GPU often feels better than a larger model on a slow one.
What small models are good at
- Drafting and rewriting — emails, summaries, documentation, commit messages.
- Classification and extraction — routing tickets, pulling fields out of text, tagging.
- Code completion in a narrow stack, especially with retrieval of local files.
- Private workloads where data should not leave the machine.
They are weaker at long multi-step reasoning, broad world knowledge, rare languages and tasks that need a very large context. A small model with good retrieval can outperform a large model with none for document questions; a small model alone will struggle with open-ended research.
The software layer
Runtimes such as llama.cpp, MLX, Ollama and vLLM handle quantization, batching and GPU offload. The practical differences are platform support, how much of the model fits in VRAM and how well they manage the KV cache. For a single user on a laptop, a simple 4-bit GGUF file and a local server are usually enough. For many concurrent users, batching and cache management matter more than the raw model choice.
Context is a budget, not a feature
A 128k-token context sounds generous until you price the cache. Long contexts consume memory and slow every subsequent token. For local use, keep prompts lean, retrieve only the relevant passages and reset conversations when they stop being useful. A 8k context used well often beats a 128k context used as a dumping ground.
A sensible starting point
Pick a 7–8B instruction-tuned model at 4-bit, run it locally, and test it on your actual tasks for a week. Measure quality, speed and memory. If it is close, try a 14B model; if it is not, add retrieval or a narrower fine-tune before jumping to a larger model. The goal is not the biggest model that fits — it is the smallest one that does the job reliably.