A 35B Model That Fits in Your Pocket
POCKET is a family of quantized builds derived from Darwin-36B-Opus — a 34.66-billion-parameter sparse Mixture-of-Experts model that activates only ~3 billion parameters per token. The pitch sounds impossible until you see the numbers: a 13 GB build generates around 20 tokens per second on a mini-PC with no GPU. A language-pruned phone build fits into roughly 5 GB on an iPhone.
No CUDA. No cloud API. No custom runtime fork. It runs on stock llama.cpp, LM Studio, Ollama, PocketPal, and MLX — the tools people already use every day.
Released by FINAL-Bench (VIDRAFT) under an Apache-2.0 license, POCKET keeps 88% of the full model’s quality at its most practical Q2_K setting. That tradeoff — give up 12% quality to run a 35B-class model without a datacenter — is one of the most consequential engineering results of the year for anyone building AI products.
Why MoE Changes the On-Device Math
The reason POCKET works where earlier attempts didn’t comes down to one architectural decision: sparse Mixture-of-Experts.
In a dense model, every token passes through every parameter. A 35B dense model means 35B parameters of compute per token — far too much for a phone or a CPU-only mini-PC.
In a sparse MoE model, each token activates only a small subset of experts. Darwin-36B-Opus activates roughly 3 billion parameters per token, even though the full model contains 34.66 billion. The file on disk is large (the model needs all experts available for routing), but the per-token compute cost is small.
This is the unlock. You still need enough storage to hold the full model, and enough memory bandwidth to load the right experts — but the actual computation per generated token is modest. On a 12-core mini-PC, POCKET generates text faster than you can read it.
The Three-Layer Engineering Stack
POCKET layers three techniques to hit these targets:
- Darwin-lineage checkpoint — the base model is a sparse MoE designed for efficient routing from the start
- Domain expert pruning — keeping only the experts a specific language actually fires, removing dead weight
- MoE-aware mixed precision — keeping the shared path at high precision while pushing routing experts to sub-2-bit
The pruning step produced a finding worth pausing on: Korean survives pruning but dies under quantization, while English is the opposite. Different languages stress the model in different directions, and a one-size-fits-all compression strategy will break at least one of them. This is a genuinely novel insight for anyone building multilingual on-device systems.
How POCKET Stacks Up Against Bonsai
The most useful comparison is against Bonsai, a dense 27B model built for similar on-device use cases. At matched quality:
- 2.7× faster on CPU generation
- 2.2× faster on GPU generation
- Runs on upstream
llama.cpp— Bonsai requires a custom fork
That last point matters more than the speed numbers. A custom fork means every upstream update needs manual porting, every bug fix has to be re-applied, and your build is permanently coupled to someone else’s fork maintenance. POCKET runs on stock llama.cpp, which means it inherits every performance improvement, bug fix, and feature the community ships — for free.
What This Means for AI Product Builders
If you’re building an AI-powered app today, your architecture probably looks like this: a thin client sends requests to a cloud-hosted model, you pay per token, and your users need an internet connection for every interaction.
POCKET makes a different architecture credible: bundle the model with your app, run inference locally, never send data to a server.
Privacy Becomes the Default, Not a Feature
Health data, financial records, confidential business documents, private conversations — all of these can now be processed by a 35B-class model without leaving the device. You don’t need a privacy policy that says “we may share your data with our AI provider.” You don’t need a data processing agreement. The data never moves.
For regulated industries — healthcare, finance, legal — this is the difference between “we can’t use AI for this” and “we can ship this tomorrow.”
Cost Structure Flips From Variable to Fixed
Cloud inference is pay-per-token. Every user interaction costs money. As your user base grows, your inference bill grows proportionally — and frontier model pricing has not been trending toward affordability at the rate people hoped.
On-device inference is a fixed cost: the model ships with the app, the user’s hardware does the work. Your marginal cost per interaction is zero. The tradeoff is a larger app download and a device that needs enough RAM — but those are one-time costs, not recurring ones.
Offline Use Cases Become Real
Field workers, travelers, rural users, anyone in a low-connectivity environment — these use cases have been a bullet point in pitch decks for years without a credible implementation. A 35B-class model that runs on a phone changes that from aspiration to shipping product.
The Strategic Takeaway: Track the Bundling, Not Just the Models
The broader signal here isn’t POCKET specifically — it’s that the on-device AI stack is maturing fast enough to be a real product strategy, not a demo. Three things need to converge for on-device AI to go mainstream:
- Model efficiency (sparse MoE, aggressive quantization, expert pruning) ✅ happening now
- Runtime compatibility (stock
llama.cpp, not custom forks) ✅ POCKET delivers this - Developer tooling (LM Studio, Ollama, MLX, PocketPal) ✅ already mature
All three are in place as of July 2026. The question for builders isn’t whether on-device AI is viable — it’s which of your features you move local first.
Start with the features that handle sensitive data. Then move the ones that need to work offline. Then move everything else, because the cloud inference cost you’re paying per token is a cost your competitors may not be paying at all.
POCKET is available on Hugging Face under an Apache-2.0 license. The original research and benchmarks are documented in FINAL-Bench’s release on Hugging Face.
