Qwen3.8-Flash-Next on a 128 GB Box: How to Leave 51B of It on the SSD

01 The One Idea That Makes It Fit

Qwen3.8-Flash-Next has ~180B parameters, but they aren't all the same kind. 128.8B are compute — the MoE experts, attention, Gated DeltaNet — and they're read densely, every token, so they must be resident. The other 51.2B are a single tensor:

the N-gram / PLE embedding table — a second memory layer, not a weight matrix

A weight matrix in a matmul is read in its entirety every use — if it isn't resident, every token pays the full transfer and speed collapses to storage bandwidth. An embedding table is different in two ways that make offloading free:

• Sparsely accessed. Per token the model gathers ~16 rows out of 320,001,536. It never touches the other 99.99995% for that token.
• Deterministically addressed. The rows are known from the token ids (via an n-gram hash) before the gather runs — a lookup, not a computation over the tensor. So the per-token traffic is a handful of tiny reads, not a 26.8 GiB stream.

Reads like that are exactly what an OS page cache over NVMe is good at. Qwen's own tech report makes the point explicitly: the N-gram vocabulary can be scaled up precisely because the resulting table can be served from off-accelerator storage. That's the whole trick, and it's why the rest of this article is mostly bookkeeping around two flags.

02 The Memory Math

At the UD-Q4_K_XL quant the model file is 103.7 GiB (4 shards). Here's where each byte ends up on a 128 GB box:

source: 0xbakeer/qwen38-flash-next-spark, docs/how-it-works.md, measured on a 121 GiB-usable GB10

The 26.8 GiB is simply 103.7 − 76.9, and it matches the size of the sequential read that warms the whole table. Steady state measured: ~95 GiB used overall, ~26 GiB in the page cache (the table, once touched), and that's it. The margin is what buys the long context, because the KV cache is shockingly small.

unsloth quant docs · 0xbakeer recipe preflight (warns below ~100 GiB)

03 The Exact Command

This is the whole trick, verbatim from the reference recipe that all the published numbers were measured on. Build a current llama.cpp first — qwen4exp support merged upstream on Aug 27, 2026 (PR #27742), so anything older dies with unknown model architecture: 'qwen4exp'.

two flags do the offload; everything else is standard serving config

• -lm mmap — memory-map the model so the table's pages are only faulted in as rows are actually gathered, rather than being read into RAM up front.
• -ot "per_layer_token_embd=CPU" — pin that one tensor to the CPU backend so it's never copied to accelerator-resident memory. With mmap, it stays a file the OS serves on demand.
• --n-gpu-layers 999 — push the compute layers to the GPU / unified memory. The one tensor you just pinned stays out.
• --ctx-size 262144 — enable the full window. It's only ~6 GiB, so there's no trade-off.
• --parallel 2 — slots. llama.cpp divides ctx-size across them, so this is a standing claim on context (2 → 131,072 tokens each), not a free concurrency knob. 2 costs nothing at one caller and is worth ~1.25× under load.
• --spec-type ngram-mod — speculative decoding that drafts spans from context repetition. It's exact (the target verifies every token, so output is unchanged), free (no draft model, no memory), and a big win when you're rewriting text that already exists. A no-op on free-form prose. Disable with --spec-type none.

04 The Three Routes

Same idea, three ways in. Pick by hardware and how much setup you want to do.

0xbakeer/qwen38-flash-next-spark (MIT) · sglang-ssd-stream (Apache-2.0) · llama.cpp PR #27742

Route A — the turnkey DGX Spark one

The most-documented path. One repo, two recipes that are genuinely good at different things:

both answer an OpenAI-compatible API on :8000/v1 — so Open WebUI, coding agents, your scripts all just work

Not sure which? Start with longctx — it's the more even performer and its fast document reading is the first difference you'll feel. The two share the GPU, so only one runs at a time.

05 The Numbers

The honest story: the trick removes the memory obstacle; it doesn't make 180B fast. Here's what "running it" actually feels like.

DGX Spark (128 GB, GB10), by task

0xbakeer/qwen38-flash-next-spark, docs/measurements.md · speeds are meaningless without the task attached

Read that column for editing and the spread is the whole architecture: ngram-mod drafts from your prompt, so when you paste a file and change one thing, almost every answer token already exists and it leaps ahead 60 tokens at a time. Ask it to write something new and there's nothing to copy, so it falls back to ~28. A speed figure for this model is meaningless without the workload next to it.

Why the SSD doesn't show up on a GPU

sglang-ssd-stream public benchmarks, matched 1,024-token completions · independent community measurement

Move a 47.6 GiB table to disk and it got faster, using ~64 MiB of RAM. On the GPU, the 160-byte random reads hide entirely behind compute. The same physics holds on a CPU box — decode time is dominated by reading the activated compute weights out of ~273 GB/s unified memory, and the few-KB table reads largely hide behind that.

06 Skip the Warming

The most-repeated tip in the community is to "warm" the table — read it sequentially once so the hot rows land in the page cache. On the current build, it does nothing, and you shouldn't bother.

An early measurement claimed +42% on copy-heavy work after a 27-second warm. It was retracted. Re-measured on the current build with CUDA graph capture engaging, residency changed nothing: prose ran 27.8 tok/s at both 17.7% and 58.1% cached, and file reproduction was 88.5 cold vs 86.2 warm — a difference inside the noise. A third-party box (six vLLM runs) confirmed the floor: identical runs differ by ~7%, which is bigger than any warm-vs-cold gap. The recipe ships WARM=0 for this exact reason.

07 What It Does Not Solve

• It doesn't make 180B fast. ~22 tok/s single-stream on the Spark at Q4. Speculative decoding is where the headroom is, not the table location.
• Prefill is a clock problem. A Strix Halo box handled a 201,647-token prompt in 17.3 minutes to first token. Long context stopped being a memory wall — now it's just slow to chew through.
• Thinking burns tokens you don't see. The model thinks before it answers, and in one measurement 86% of generated tokens were reasoning. Turning thinking off got the same answer in 15 s instead of 55 — not faster tokens, far fewer of them. Add {"chat_template_kwargs": {"enable_thinking": false}} for everything but hard reasoning and maths.
• 24 GB cards don't qualify. Even at 1-bit the floor is 75 GB. This is a 96–128 GB unified-memory model, full stop.
• QSA is approximate by design. Sparse attention can diverge slightly on ~3% of positions past the 8K sparse-attention budget. Full context works; don't assume bit-exactness at the edge.

08 The Bottom Line

• 128 GB unified memory (DGX Spark, Strix Halo, big DDR5 tower + NVMe): this is your model. Two flags, swap off, full 262K context for 6 GiB. The first frontier-class open model that was actually designed for this tier.
• 96 GB: fits at Q4, tight on the KV headroom. 75 GB at 1-bit if you need the margin.
• A single 24 GB GPU: no. Wrong tier — keep a dense 27B there. This needs unified memory, not VRAM.
• NVIDIA datacenter (RTX PRO / N150): use sglang-ssd-stream, not llama.cpp — the io_uring + FP8 path is what gets you to 64 MiB of RAM and the ~165 tok/s figure.
• Everyone: skip the table-warming, turn thinking off for most requests, and attach the workload to every speed number you quote.