# Qwen3.8-Flash-Next on a 128 GB Box: How to Leave 51B of It on the SSD

180B parameters, 128 GB of memory, and it fits. The exact llama.cpp command that streams Qwen3.8-Flash-Next's 51B N-gram table from NVMe, the memory math that proves it, and the three routes to get there.

## 01 The One Idea That Makes It Fit

Qwen3.8-Flash-Next has ~180B parameters, but they aren't all the same kind. 128.8B are compute — the MoE experts, attention, Gated DeltaNet — and they're read densely, every token, so they must be resident. The other 51.2B are a single tensor:

per_layer_token_embd.weight shape [160, 320001536]
# 160 × 320,001,536 = 51.2G elements — 28% of the model, one tensor

the N-gram / PLE embedding table — a second memory layer, not a weight matrix

A weight matrix in a matmul is read *in its entirety* every use — if it isn't resident, every token pays the full transfer and speed collapses to storage bandwidth. An embedding table is different in two ways that make offloading free:

- **Sparsely accessed.** Per token the model gathers ~16 rows out of 320,001,536. It never touches the other 99.99995% for that token.

- **Deterministically addressed.** The rows are known from the token ids (via an n-gram hash) *before* the gather runs — a lookup, not a computation over the tensor. So the per-token traffic is a handful of tiny reads, not a 26.8 GiB stream.

Reads like that are exactly what an OS page cache over NVMe is good at. Qwen's own tech report makes the point explicitly: the N-gram vocabulary can be scaled up *precisely because* the resulting table can be served from off-accelerator storage. That's the whole trick, and it's why the rest of this article is mostly bookkeeping around two flags.

## 02 The Memory Math

At the `UD-Q4_K_XL` quant the model file is **103.7 GiB** (4 shards). Here's where each byte ends up on a 128 GB box:

| **Portion** | **Size** | **Where it lives**|
--- | --- | ---
| Compute weights (128.8B) | ~76.9 GiB | Resident, in unified memory|
| N-gram / PLE table (51.2B) | ~26.8 GiB | On NVMe, served via page cache|
| Process RSS | ~1.4 GiB | Weights are mmapped, so almost nothing is process-private|

source: 0xbakeer/qwen38-flash-next-spark, docs/how-it-works.md, measured on a 121 GiB-usable GB10

The 26.8 GiB is simply 103.7 − 76.9, and it matches the size of the sequential read that warms the whole table. Steady state measured: ~95 GiB used overall, ~26 GiB in the page cache (the table, once touched), and that's it. The margin is what buys the long context, because the KV cache is shockingly small.

Why 262K context costs only 6 GiB
There are 48 layers, but only **12** use full attention with a growing KV cache — the other 36 are GDN-style layers whose state is constant-size regardless of context. The full-attention layers use GQA with just 2 KV heads. Net: **~24 KB per token**, so the entire 262,144-token window is `262,144 × 24 KB = 6 GiB`. That's why you can enable max context by default instead of trading it away. One catch: KV must stay `f16` — quantised KV aborts on this architecture.

| **Minimum to run** | **RAM** | **Why**|
--- | --- | ---
| 1-bit quant | 75 GB | Smallest. Table held at 4-bit floor — random-access tables degrade under heavy quant|
| 2-bit | 79 GB | |
| Q4 (measured recipe) | ~100 GB usable | 76.9 GiB resident + room for the page cache. This is the 128 GB path|

unsloth quant docs · 0xbakeer recipe preflight (warns below ~100 GiB)

## 03 The Exact Command

This is the whole trick, verbatim from the reference recipe that all the published numbers were measured on. Build a current `llama.cpp` first — `qwen4exp` support merged upstream on Aug 27, 2026 (PR #27742), so anything older dies with `unknown model architecture: 'qwen4exp'`.

llama-server \
-m model/gguf-model-q4_K_m.gguf \
-lm mmap \
-ot "per_layer_token_embd=CPU" \
--n-gpu-layers 999 \
--ctx-size 262144 \
--parallel 2 \
--spec-type ngram-mod \
--temp 1.0 --top-p 0.95 --top-k 20 \
--host 127.0.0.1 --port 30000

two flags do the offload; everything else is standard serving config

- `-lm mmap` — memory-map the model so the table's pages are only faulted in as rows are actually gathered, rather than being read into RAM up front.

- `-ot "per_layer_token_embd=CPU"` — pin that one tensor to the CPU backend so it's never copied to accelerator-resident memory. With mmap, it stays a file the OS serves on demand.

- `--n-gpu-layers 999` — push the *compute* layers to the GPU / unified memory. The one tensor you just pinned stays out.

- `--ctx-size 262144` — enable the full window. It's only ~6 GiB, so there's no trade-off.

- `--parallel 2` — slots. llama.cpp divides `ctx-size` across them, so this is a standing claim on context (2 → 131,072 tokens each), not a free concurrency knob. 2 costs nothing at one caller and is worth ~1.25× under load.

- `--spec-type ngram-mod` — speculative decoding that drafts spans from context repetition. It's exact (the target verifies every token, so output is unchanged), free (no draft model, no memory), and a big win when you're rewriting text that already exists. A no-op on free-form prose. Disable with `--spec-type none`.

Before you run it
**Swap must be off** (`sudo swapoff -a`). You want cold table pages evicted from the page cache, never resident compute weights pushed into swap — that would destroy the whole arrangement. And keep the model file on NVMe (not a SATA SSD); the read latency only hides behind compute when the drive can stream ~1 GiB/s.

## 04 The Three Routes

Same idea, three ways in. Pick by hardware and how much setup you want to do.

| **Route** | **Best for** | **How the table is served**|
--- | --- | ---
| **DIY llama.cpp** | Any Linux box, 75+ GB RAM, one GPU or none | The two flags above. `-ot per_layer_token_embd=CPU` + `-lm mmap`. The page cache does the rest.|
| **0xBakeer Spark recipe** | DGX Spark / GB10, 128 GB, want it turnkey | `./run.sh edit serve` or `./run.sh longctx serve`. Wraps the same flags, pins the exact commit, serves an OpenAI API on `:8000`.|
| **sglang-ssd-stream** | NVIDIA datacenter (RTX PRO 6000 / N150), max headroom | FP8 table to a read-only sidecar file, `io_uring` concurrent reads, conversion on a side CUDA stream. The 64 MiB-RAM measurement below.|

0xbakeer/qwen38-flash-next-spark (MIT) · sglang-ssd-stream (Apache-2.0) · llama.cpp PR #27742

### Route A — the turnkey DGX Spark one

The most-documented path. One repo, two recipes that are genuinely good at different things:

git clone https://github.com/0xBakeer/qwen38-flash-next-spark.git
cd qwen38-flash-next-spark

# coding agents that rewrite files → edit (~105 GB)
./run.sh edit setup
./run.sh edit serve

# chat, reasoning, long documents → longctx (~126 GB, vLLM + NVFP4 + MTP)
./run.sh longctx setup
./run.sh longctx serve

both answer an OpenAI-compatible API on :8000/v1 — so Open WebUI, coding agents, your scripts all just work

Not sure which? Start with `longctx` — it's the more even performer and its fast document reading is the first difference you'll feel. The two share the GPU, so only one runs at a time.

## 05 The Numbers

The honest story: the trick removes the *memory* obstacle; it doesn't make 180B fast. Here's what "running it" actually feels like.

### DGX Spark (128 GB, GB10), by task

| **Workload** | **Editing recipe** | **Long-ctx recipe**|
--- | --- | ---
| Rewrite a file, one change | 88 tok/s | 39|
| Fix a bug in a file | 46 | 35|
| Write new prose | 28 | 32|
| Read a 128K-token document (prefill) | ~7 min | ~56 s|
| Concurrent requests | 2 (half context each) | 16|

0xbakeer/qwen38-flash-next-spark, docs/measurements.md · speeds are meaningless without the task attached

Read that column for editing and the spread is the whole architecture: `ngram-mod` drafts from your prompt, so when you paste a file and change one thing, almost every answer token already exists and it leaps ahead 60 tokens at a time. Ask it to write something new and there's nothing to copy, so it falls back to ~28. A speed figure for this model is meaningless without the workload next to it.

### Why the SSD doesn't show up on a GPU

| **RTX PRO 6000 (96 GB), NVFP4** | **RAM used** | **Decode**|
--- | --- | ---
| Table in RAM | 47.68 GiB | 148.5–156.2 tok/s|
| Table on SSD (sglang-ssd-stream) | ~64 MiB | 164.7 tok/s · 47.6 GiB freed|

sglang-ssd-stream public benchmarks, matched 1,024-token completions · independent community measurement

Move a 47.6 GiB table to disk and it got *faster*, using ~64 MiB of RAM. On the GPU, the 160-byte random reads hide entirely behind compute. The same physics holds on a CPU box — decode time is dominated by reading the activated compute weights out of ~273 GB/s unified memory, and the few-KB table reads largely hide behind that.

## 06 Skip the Warming

The most-repeated tip in the community is to "warm" the table — read it sequentially once so the hot rows land in the page cache. **On the current build, it does nothing, and you shouldn't bother.**

An early measurement claimed +42% on copy-heavy work after a 27-second warm. It was retracted. Re-measured on the current build with CUDA graph capture engaging, residency changed nothing: prose ran 27.8 tok/s at both 17.7% and 58.1% cached, and file reproduction was 88.5 cold vs 86.2 warm — a difference inside the noise. A third-party box (six vLLM runs) confirmed the floor: identical runs differ by ~7%, which is bigger than any warm-vs-cold gap. The recipe ships `WARM=0` for this exact reason.

Why it stopped mattering
Warming only helps when page faults dominate the per-token cost. Once CUDA graph capture engages, the cost the faults used to sit on top of is much smaller, so residency stops being the bottleneck. The table being on NVMe at all is, for all practical purposes, free — which is the point of the whole article.

## 07 What It Does Not Solve

- **It doesn't make 180B fast.** ~22 tok/s single-stream on the Spark at Q4. Speculative decoding is where the headroom is, not the table location.

- **Prefill is a clock problem.** A Strix Halo box handled a 201,647-token prompt in 17.3 minutes to first token. Long context stopped being a memory wall — now it's just slow to chew through.

- **Thinking burns tokens you don't see.** The model thinks before it answers, and in one measurement **86% of generated tokens were reasoning**. Turning thinking off got the same answer in 15 s instead of 55 — not faster tokens, far fewer of them. Add `{"chat_template_kwargs": {"enable_thinking": false}}` for everything but hard reasoning and maths.

- **24 GB cards don't qualify.** Even at 1-bit the floor is 75 GB. This is a 96–128 GB unified-memory model, full stop.

- **QSA is approximate by design.** Sparse attention can diverge slightly on ~3% of positions past the 8K sparse-attention budget. Full context works; don't assume bit-exactness at the edge.

## 08 The Bottom Line

- **128 GB unified memory (DGX Spark, Strix Halo, big DDR5 tower + NVMe):** this is your model. Two flags, swap off, full 262K context for 6 GiB. The first frontier-class open model that was actually designed for this tier.

- **96 GB:** fits at Q4, tight on the KV headroom. 75 GB at 1-bit if you need the margin.

- **A single 24 GB GPU:** no. Wrong tier — keep a dense 27B there. This needs unified memory, not VRAM.

- **NVIDIA datacenter (RTX PRO / N150):** use sglang-ssd-stream, not llama.cpp — the io_uring + FP8 path is what gets you to 64 MiB of RAM and the ~165 tok/s figure.

- **Everyone:** skip the table-warming, turn thinking off for most requests, and attach the workload to every speed number you quote.

Caveats
All speeds and the memory arithmetic are from community measurements (0xBakeer Spark repo, sglang-ssd-stream, abliter8-ai Strix Halo), not vendor benchmarks — treat them as representative, not absolute. The +42% warming claim and its retraction are reproduced here exactly as the repo documents them. The model carries the Qwen Community License, which has its own conditions.

## ↺ Related Articles

[The Review: Qwen3.8-Flash-Next — 180B, 6B Active, and the N-gram Table That Streams From SSD](/qwen3-8-flash-next-ngram-ssd-streaming.html)
[Qwen4: The Blueprint Behind This Model (and Where the Rumor Stops)](/qwen4-preview-architecture-blueprint.html)
[Qwen3.8: Alibaba's 2.4T Open-Weight Bet — What We Actually Know](/qwen3-8-alibaba-2-4t-open-weight-frontier-model.html)
[DeepSeek-V4-Flash: The Official Release That Ships DSpark Built In](/deepseek-v4-flash-official-release-dspark-built-in.html)

**Sources:** Qwen3.8-Flash-Next technical report (Aug 2026) · 0xBakeer/qwen38-flash-next-spark (README, how-it-works.md, measurements.md, inference-atlas) · llama.cpp PR #27742 (qwen4exp, merged Aug 27, 2026) · sglang-ssd-stream public benchmarks · Unsloth documentation & quant tables · abliter8-ai Strix Halo deployment.

This article was drafted with AI assistance and reviewed by a human editor.

ZVHH — AI Tools & Software Reviews © 2026
[Home](/)[Agent feed](/index.md)