Small language models are eating the edge

Frontier models are extraordinary and expensive. Most product surfaces do not need one.

The jobs SLMs win - Classification, routing and intent detection - Structured extraction from a known document type - Autocomplete, rewrite and summarize on short spans - Anything latency-critical or privacy-sensitive

The economics A quantized 3B model on modern mobile silicon answers in under 300ms with zero marginal cost and no data leaving the device. The same call to a hosted frontier model costs money on every keystroke and adds a network round trip.

The pattern we ship Run the small model first. Score its confidence. Escalate to a hosted model only when confidence is low or the task is open-ended. In most deployments this routes 70-85% of traffic on-device.

Watch the traps Quantization changes behaviour, not just speed. Re-run your evaluation suite against the exact quantized artifact you plan to ship, on the exact hardware tiers you support.