Shawn NgLocal Models vs Frontier APIs: An Honest Accounting
Last quarter I watched a team spend three weeks getting Llama 3.3 70B running on a pair of A6000s, complete with quantization experiments, a vLLM deployment, and a Grafana dashboard, to replace an API workload that cost them $214 a month. The engineering time alone cost more than a decade of API calls. Two floors down, another team was paying five figures a month to a frontier API to classify support tickets into six categories, a task a fine-tuned 8B model handles at 98% parity for approximately the cost of electricity. Both teams were smart. Both had the wrong mental model of where the local-vs-API line actually sits. After a few years of running models both ways in production, here is my honest accounting.
Whether a Model Fits Is Arithmetic, Not Vibes
The first question is brutally simple: do the weights fit in memory? A parameter is a number, and the memory cost is parameters x bytes per parameter.
At FP16, every parameter costs 2 bytes:
Llama 3.1 8B @ FP16: 8B x 2 bytes = 16 GB
Qwen2.5 32B @ FP16: 32B x 2 bytes = 64 GB
Llama 3.3 70B @ FP16: 70B x 2 bytes = 140 GB
DeepSeek-V3 @ FP8: 671B x 1 byte = ~671 GB (37B active per token, but ALL experts must be resident)
That 140 GB figure for a 70B model is why nobody runs FP16 locally. Quantization is the whole game. Squeeze each weight down and the math changes:
70B @ 8-bit (Q8_0): ~70 GB
70B @ 4-bit (Q4_K_M): ~40-42 GB
70B @ 4-bit (IQ4_XS): ~36 GB
8B @ 4-bit: ~4.7 GB
A 4-bit 70B fits on a 48 GB card or a 64 GB Mac. A 4-bit 8B runs on basically anything made in the last five years. This is the arithmetic that determines your entire local strategy, and people routinely skip it and go straight to arguing about benchmarks.
But weights are not the end of the story. The KV cache grows linearly with context length, and it is the silent killer of long-context local inference. For each token, you store keys and values for every layer and every KV head:
KV bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes
Llama 3.1 70B uses grouped-query attention (80 layers, 8 KV heads, head_dim 128), so at FP16 that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes, roughly 320 KB per token. A 128K-token context costs about 40 GB of KV cache alone, as much as the quantized weights themselves. GQA already bought a 8x reduction versus full multi-head attention; DeepSeek's multi-head latent attention (MLA) compresses further, and this is not an academic detail. It is the difference between "fits" and "OOM" the first time someone pastes a long document. Quantizing the KV cache to 8-bit or 4-bit (llama.cpp supports both) halves or quarters this, with modest quality cost.
Memory Bandwidth Is the Speed Limit, Not FLOPS
Here is the fact that reorganized how I think about inference hardware: for single-stream generation, your tokens per second is bounded by memory bandwidth, not compute.
Autoregressive decoding generates one token at a time, and each token requires reading every active weight from memory once. The GPU does a trivial amount of math per byte loaded, an arithmetic intensity of roughly 1 FLOP per byte, while modern accelerators are built for hundreds. Decoding is memory-bound almost by definition, which gives you a napkin formula that is startlingly accurate:
tokens/sec ≈ memory bandwidth / bytes per token
≈ bandwidth / (model size in bytes)
Run the numbers for an M-series Mac. An M4 Max has 546 GB/s of unified memory bandwidth:
8B @ 4-bit (~4.7 GB): 546 / 4.7 ≈ 116 t/s theoretical, ~60-80 real
70B @ 4-bit (~40 GB): 546 / 40 ≈ 13.7 t/s theoretical, ~8-10 real
Real numbers land at 60-75% of theoretical because of KV cache reads, attention overhead, and imperfect kernels, but the ceiling is set by bandwidth, full stop. An RTX 4090 (1,008 GB/s) roughly doubles the Mac. An H100 (3.35 TB/s) roughly triples the 4090. Meanwhile the compute units on all three sit mostly idle during decode.
This has three practical consequences. First, quantization makes models faster, not just smaller. Half the bytes per token means double the tokens per second, which is a large part of why 4-bit is so popular. Second, Apple Silicon punches absurdly above its weight for local inference: unified memory means a 128 GB M4 Max can hold a 70B model that no single consumer GPU can, and 546 GB/s makes it usable, if not fast. Third, prompt processing (prefill) is a different regime entirely. Prefill processes all input tokens in parallel and is compute-bound, which is where NVIDIA hardware demolishes Macs. On a long-document workload, an M-series machine will crawl through ingesting the prompt and then generate at a perfectly respectable pace.
Batching changes the physics too. If you serve 32 concurrent requests, you still read the weights once per forward pass but produce 32 tokens, so aggregate throughput scales with batch size until you hit compute or KV-cache memory limits. This is why vLLM, with its PagedAttention KV-cache management and continuous batching, exists as a separate species from llama.cpp. llama.cpp (and Ollama, which wraps it in a pleasant docker-style UX with a model registry) is optimized for one user on one machine, with GGUF quantizations and CPU offload. vLLM is optimized for saturating a GPU across many concurrent streams, routinely delivering 10-20x the aggregate throughput on the same silicon. Using llama.cpp to serve a team, or vLLM for a single laptop user, is the most common ecosystem mismatch I see.
What 4-Bit Actually Costs You
Quantization skeptics point at perplexity numbers; quantization evangelists point at benchmarks. Both are half right.
Measured carefully, modern 4-bit schemes (Q4_K_M in llama.cpp, AWQ, GPTQ with activation ordering) cost a 70B model on the order of 1-3% on standard benchmarks like MMLU, with perplexity deltas of a few percent. The k-quants and importance-matrix quants got clever about it: they keep the most sensitive weights (as identified by a calibration set) at higher precision and spend fewer bits where activations are flat. Below 4 bits the curve stops being gentle. At 3-bit you notice; at 2-bit the model is visibly lobotomized, occasionally emitting confident nonsense mid-sentence. Four bits is a genuine sweet spot, not a marketing convention: it is roughly where the quality-per-byte curve has its knee for dense transformer weights.
Two honest caveats. First, degradation is not uniform across capabilities. Perplexity is an average, and averages hide tails. Quantized models degrade disproportionately on the hardest tasks: multi-step math, long chains of tool calls, code that must compile. A 1% MMLU drop can coexist with a 5-8% drop on a hard coding eval. Second, a bigger model quantized beats a smaller model at full precision almost every time: Llama 70B at 4-bit comfortably outperforms Llama 8B at FP16 while using only 2.5x the memory. If you have the RAM, spend it on parameters, not precision.
The Gap Narrowed Everywhere Except Where You Need It Most
Here is the intellectually honest read on capability, as of early 2026.
On single-turn tasks, the open-weight frontier is genuinely close. Llama 3.3 70B, Qwen 2.5 72B, Mistral Large, and especially DeepSeek-V3 and DeepSeek-R1 sit within a few points of frontier APIs on MMLU, GSM8K-class math, summarization, extraction, translation, and everyday coding assistance. DeepSeek-R1's release was the moment open-weight reasoning stopped being a punchline; distilled R1 variants brought real chain-of-thought competence down to sizes that fit on a laptop. For a huge slice of production workloads, classification, structured extraction, RAG answering over your own documents, template-ish code generation, the gap is functionally zero and has been for a while.
Where the gap has not closed, and stubbornly refuses to, is long-horizon agentic work. Ask a model to execute a 40-step task: explore an unfamiliar codebase, form a plan, edit eight files, run tests, interpret failures, and recover from its own mistakes without human intervention. Frontier models fail at this sometimes. Open-weight models fail at it compoundingly. If a model's per-step reliability is p, the chance of surviving n dependent steps is roughly p^n; a gap of 0.98 vs 0.90 per step is invisible in a chat window and catastrophic at step 30 (0.98^30 ≈ 0.55 vs 0.90^30 ≈ 0.04). Benchmarks like SWE-bench Verified make this legible: the spread between the best open models and the best APIs on multi-step software tasks remains far wider than any single-turn benchmark suggests. The same pattern shows up in complex multi-constraint reasoning, subtle instruction adherence across long contexts, and knowing when to say "I'm not sure" instead of confabulating.
My working heuristic: the gap is small on tasks where errors are independent, and large on tasks where errors compound.
The Decision Is a Matrix, Not a Preference
Strip away the tribalism and the choice decomposes into five axes:
Privacy and data governance. If the data legally cannot leave your infrastructure, HIPAA, ITAR, certain financial contexts, air-gapped environments, local is not a preference, it is the only option, and the entire rest of this post is moot. This is the one non-negotiable column of the matrix.
Cost at volume. API pricing is a per-token rental; hardware is a capital expense amortized over utilization. Run the arithmetic. A used RTX 3090 (~$700) serving an 8B model via vLLM can push millions of tokens per day. Small-model API tiers now price at roughly $0.05-0.15 per million input tokens, so the API on a comparable-quality small model can cost less than your electricity at low volume. The flip happens when you have sustained, high-utilization, small-model workloads: at hundreds of millions of tokens per day, 24/7, owned hardware wins decisively. At bursty or low volume, it never does, because your GPU depreciates whether or not it is generating tokens. Most teams who "save money going local" are not counting the engineering salary spent on the serving stack.
Latency. Local wins on time-to-first-token, no network round trip, no queue, which matters enormously for interactive completions and voice. APIs generally win on tokens/sec for large models, because they run on H100/B200-class hardware you cannot buy for a workstation.
Fine-tuning and control. This is local's quiet superpower. A LoRA fine-tune of Qwen 2.5 7B on 10,000 examples of your task, your log formats, your ticket taxonomy, your codebase conventions, routinely beats a frontier model prompted with the same information, at 1/50th the inference cost. You own the weights, the sampler, the logit biases, the whole stack. No deprecation emails, no silent model updates changing behavior under you.
Offline and edge. Ships, aircraft, factory floors, field hardware, developer laptops on planes. If the network is unreliable, the API's capability is irrelevant.
The pattern that actually works in production, and the one I recommend, is boring: route by task. A cheap local (or small-API) model handles the 80% of traffic that is classification, extraction, and short-form generation; the frontier API handles the 20% that is genuinely hard, agentic, or novel. Cascading with a confidence check, escalate when the small model's self-consistency or logprobs look shaky, cuts frontier spend by 60-90% in every deployment I have measured, with negligible quality loss.
Distillation Is Why the Small Models Keep Getting Better
The reason this routing strategy keeps improving is structural. Every frontier model is, involuntarily, a teacher. Distillation, training a small model on outputs (or logits) of a large one, transfers a shocking fraction of narrow-task capability into a fraction of the parameters. DeepSeek demonstrated this at scale by distilling R1's reasoning traces into Qwen and Llama backbones as small as 7B, producing models that outperform much larger non-distilled ones on math and code. The mechanism is intuitive: a small model has enough capacity for your task; what it lacks is the frontier model's breadth. Distillation lets you trade breadth you do not need for depth you do.
This creates a permanent conveyor belt: whatever the frontier can do reliably today becomes distillable into a local model within months. The frontier keeps its lead on the newest capabilities, the long-horizon agentic work, the hardest reasoning, but each capability it pioneers eventually slides down the ramp into open weights and onto your GPU. Local models are not catching up to the frontier; they are trailing it at a roughly constant distance while both move forward, and everything behind that moving line becomes commodity.
So the real question was never "local or API." It is: which of your workloads have already crossed the commodity line, and do you have the routing layer to exploit it? The teams that build that layer now will spend the next five years paying frontier prices only for frontier problems, and watching everything else get cheaper by the quarter.