Optimizations
S-MoE is measurement-driven. Every design decision on this page is held in place by an instrumented number, and anything the numbers reject is removed from the engine — not kept out of sentiment. This page describes the optimizations as they exist in the engine today; the story of how they were found belongs to written articles, not to the documentation.
All numbers: canonical benchmark — 45-token chat prompt, Qwen3-235B Q4 vault, greedy, seed 1337, 48 GB Apple Silicon reference machine.
| Metric | Measured |
|---|---|
| Time-to-first-token, cold prompt | ~29.4 s |
| Time-to-first-token, follow-up turn (server mode) | ~14 s — constant, independent of conversation length |
| Decode throughput | 1.43 t/s |
The Instrument Panel
--instrument buckets every decode millisecond on the main thread — dense path, dispatch, I/O spin, GPU wait, LM head — plus NVMe bytes per token, adjacent-token expert overlap, and the gate-ranking coverage curve. Measurement before belief: this flag is the engine's only law, and every section below carries the number that justifies it.
The decode budget it reports today (~700 ms/token):
| Bucket | ms/token | What it is |
|---|---|---|
| io-spin | ~490 | Demand misses — a true NVMe bandwidth floor: ~3.07 GB of misses per token at ~6.3 GB/s, saturated during miss windows, unchanged at 4/8/16 workers |
| dense | ~155 | QKV/O projection weight bandwidth (~13.4 GB of bf16 per token through the coalesced matvec kernel) plus the CPU-side routing gate and norms |
| gpu-wait | ~46 | Expert FFN kernels executing on late-arriving misses |
| lm-head | ~8 | The GPU LM head matvec |
The budget begins, unambiguously, at the SSD — which is why the horizon below does too.
Prefill Reads Each Expert Once, Not Once Per Token
Prompt chunks of up to 64 tokens traverse the model layer by layer; each layer's routed experts are deduplicated into a union that is read once per chunk. Prefill I/O does not scale with prompt length. The dense math is batched to match: all tokens' Q/K/V projections ride one command buffer per layer and all o_proj rows another — two GPU syncs per layer instead of two per token — and each expert fires at the streamer the moment the first token routes to it, so the NVMe starts on a layer's union while later tokens are still routing. Full treatment: Layer-Major Batched Prefill.
Prompt positions before the last skip the LM head and the sampler entirely — their next token is already known, and the 151,936 × 4,096 unembedding matvec would be ~622M discarded MACs per position.
Routing Is Exact — Prompt and Generation Alike
Every MoE layer evaluates the real router gate on the true hidden state — top-8 selection and norm_topk_prob mixing weights exactly as Qwen3 defines them. There is no routing predictor in the engine, because the free oracle is unbeatable: the retained ring covers 46.4% of each token's experts from the recent past, and the current token's top-16 gate ranking covers 63.7% of the next token's top-8 — both at zero compute. (For comparison: the model's own dense trunk, evaluated as a predictor, measures 51.5% coverage at ~265 ms/token — worse than free.) The experts doing the thinking are the ones Qwen3 was trained to use.
The Ring Is an LRU Cache
Released slots are retained with their data and expert identity; a re-request for the same (layer, expert) is a cache hit with zero bytes read. Eviction is lazy least-recently-used, respects live GPU references, and stamps fresh ticks on new loads. There is no wholesale pruning anywhere — retention survives across tokens and across serve-mode turns, because expert weights are immutable and a retained slot can never go stale. This is what converts the measured 46.4% adjacent-token overlap into skipped reads.
Persistent Server Mode
--serve keeps the engine — KV-cache, ring, Scout — alive across turns. Each turn sends the full templated conversation; the engine prefix-matches against its stored stream (including its own generated tokens) and prefills only the new suffix. Turn-N latency is not a function of conversation length. Protocol: The Chat Console.
The Subtractive Ring Budget
The auto-tuner asks the OS its own question (kern.memorystatus_level: how much memory exists before pressure), subtracts the Scout file, a fixed engine overhead, and an OS floor, and gives everything that remains to the ring — ~15 GB on a 48 GB machine — floored at a conservative 25%-of-free fallback and capped so the ring can never touch swap. The ring holds roughly two generated tokens' expert working sets at Q4.
GPU-Resident Decode Attention
Each layer's entire attention block — QKV projections, per-head QK-RMSNorm, RoPE, KV append, causal GQA attention, o_proj — executes inside one Metal command buffer with a single CPU sync: 94 sync points per token instead of the 188 that separate QKV/o_proj submissions would cost. The KV ring is registered once at boot as a single zero-copy buffer, written by the GPU during decode and by the CPU during prefill, coherently. Attention is O(context) work that does not grow on the CPU. Details: Metal Compute Kernels.
The NEON Dense Path
The only CPU matvec left in decode — the exact-routing gate, ~94 MB of bf16 per token — runs on NEON (vshll_n_u16 converts and widens bf16 in one instruction, four FMA accumulator lanes), and the dot/axpy helpers vectorise the CPU-fallback attention paths. What remains of the dense bucket is weight bandwidth, not scalar arithmetic.
Coalesced GPU Kernels — Expert FFN and Dense Matvec
The hot GPU kernels assign one simdgroup per output row: lanes read consecutive 16-byte chunks (perfectly coalesced), the input vector is staged once per 8-row threadgroup, and the inner loop is pure fma. The quantised expert FFN kernels add a per-chunk affine dequant epilogue and compute gate+up fused in one pass — measured 158 GB/s effective on an M4 Pro, ~48 ms/token of GPU-side FFN. The bf16 dense matvec twin (scout_matvec_bf16_sg) carries decode's QKV/O projections at 207–285 GB/s and the LM head at 234 GB/s — the K/V projections, whose 512-row shapes strand a thread-per-row grid at 2 threadgroups, gain 23×. Details: Metal Compute Kernels.
Zero Allocations, Zero Locks in the Token Loop
The golden rules are enforced in the hot path, not just declared: sampling is a single vocab pass into a pre-allocated top-K min-heap (one RNG draw per call); expert dispatch parameters travel by setBytes, never a fresh MTLBuffer; a layer's ready experts are grouped into one command buffer per claim sweep; all activation planes are posix_memalign(16384) at startup and Metal-registered once. Synchronisation is std::atomic with release/acquire fences — no mutexes, no condition variables, anywhere.
The Small Blades
Precomputed RoPE tables (the angle depends only on position and dimension — never head, never layer); sequential background prefault of the 16 GB Scout mmap, so cold starts stream at full NVMe bandwidth instead of random-faulting mid-prefill.
Armed and Waiting: the Q2 Levers
Two mechanisms ship complete, verified, and deliberately dormant at Q4 — both priced by the instrument panel and both waiting for vault bytes to halve:
- Idle-time popularity prewarm. The engine records a per-(layer, expert) claim histogram (
vault/expert_freq.bin) and streams the historically hottest experts into the ring whenever it is idle, pausing instantly when a request lands. At Q4 the ring holds well under 10% of a turn's expert working set, so the measured latency effect is neutral — the histogram is recording, the machinery is live, its moment arrives with Q2. - Speculative prefetch (
--spec N, default 0). Each layer's exact gate ranking extends past rank 8 for free, and ranks 9–16 cover 63.7% of the next token's top-8. A low-priority streamer queue can fire them as bets that never starve demand — but at Q4 it measures net-negative (+242 ms/token of memory-bandwidth contention, +144 ms/token of worker occupancy), because speculation bids against an already-saturated bus. It stays off until Q2 halves the price of every speculative byte.
The Horizon
Ordered by the instrument panel's budget: the SSD first, then the dense path.
Q2 Vaults — the Next 2×, Twice Over
Both regimes are NVMe-bound at their floor: prefill reads each routed expert once per chunk, decode reads only its retained-ring misses, and the io-spin bucket is bandwidth, not scheduling. SMOE-Q2 halves every blob — halving both bills and doubling ring slots, which arms the popularity prewarm, the enlarged ring, and speculative prefetch in one stroke. The dequant path is in place; what remains is the Sculptor run itself — re-shattering from the original checkpoint.
The Dense Ceiling — GPU-Resident Epilogue
With the matvecs coalesced, what remains of the dense bucket is round-trips and the CPU work between them: every per-layer command buffer still ends in a CPU sync so the residual-add, RMSNorm and router-gate can run on the CPU before the next layer begins. Moving that epilogue into the same per-layer command buffer — GPU-resident residual, norm, and gate with only the top-k selection returning to the CPU — removes the last per-layer round-trip. The simdgroup-per-row kernel layout also awaits the token-batch prefill FFN kernels, which keep the older thread-per-row contract.
PagedAttention KV-Cache Offloading
A natively sovereign, tiered KV-cache: hot tokens in UMA, cold history paged to a .kvcache file via 16 KB-aligned F_NOCACHE reads. Infinite context on any RAM configuration — a 16 GB Mac maintaining 128K-token conversations.
Adaptive Quantisation (AWQ Integration)
Activation-aware bit allocation in the Sculptor: high-sensitivity, popular experts keep SMOE-Q4; niche experts compress to SMOE-Q2. Frontier coherence without growing the average blob.
Multi-Model Vault Switching
Two dense backbones resident at once — a light one for fast replies, the frontier one for deep reasoning — with the engine shifting between vaults inside a single session, on one consumer machine.