While the headlines focus on trillion-parameter cloud models, the quieter and arguably more consequential AI shift of 2026 is happening at the edge. Apple Intelligence runs locally on iPhone 15 Pro+ and every M-series Mac. Gemini Nano ships on Pixel 9 and 10. Microsoft Copilot+ PCs ship with 45 TOPS NPUs. Snapdragon X laptops are everywhere. On-device LLMs are not the future, they are 2026.
Why on-device suddenly makes sense
- Latency. A 7B model on a modern NPU runs in 30–60ms per token. No round-trip. Conversational features feel instant.
- Privacy. The user's email, photos, and chats never leave the device. This is the only viable architecture for many regulated industries.
- Cost. Cloud inference is your largest AI bill. Moving 80% of trivial requests on-device drops a SaaS company's AI COGS by 50–70% in our deployments.
- Offline. Flaky network? On-device AI doesn't care.
What runs locally now
Smartphones (Pixel 9+, iPhone 15 Pro+, Galaxy S24+) comfortably run quantised 7B-class models. Laptops with NPUs run 13B–30B comfortably, 70B with quantisation. The capability gap between "local" and "frontier" has narrowed from "absurd" to "real but manageable."
The hybrid architecture we now ship
Every new mobile or desktop app we build follows the same pattern:
- Triage on-device. Classify intent. Most intents are simple, handle locally.
- Escalate when needed. Hard reasoning, long context, or domain knowledge → cloud frontier model.
- Cache aggressively. Common cloud responses cached locally; future similar queries served from the device.
End result: median latency under 100ms, p95 under 500ms, cloud spend cut 4–6×.
Frameworks we use
- Core ML + Apple Foundation Models (iOS / macOS), first-class for Apple Intelligence integration.
- Android AI Core + Gemini Nano, exposed via Android Studio APIs since 2025.
- llama.cpp / MLC, for cross-platform, custom models.
- ONNX Runtime + DirectML, for Windows Copilot+ PCs.
What still needs the cloud
- Complex multi-step reasoning ("explain this contract").
- Anything that needs hundreds of pages of context.
- Real-time access to fresh external data.
- Best-in-class quality where the user notices.
The mental model for product teams
Pick on-device when latency, privacy, or cost dominates. Pick cloud when quality dominates. Most real products need both, and architecting for both from day one is significantly easier than retrofitting later.
Building a mobile or desktop AI product? See our mobile app services or talk to our AI team.
