Jason Belnick

Writing · · 3 min read

Fitting Qwen3.8 Flash-Next on a 64 GB Mac Studio

How a 68 GB-resident model runs in 42 GB at better-than-reference speed with a row-granular NVMe loader for its 51B n-gram table

Qwen3.8 Flash-Next is a 125B MoE with 6B active per token and a 51B-parameter n-gram embedding table. Its REAP-288 4-bit MLX build, sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit, arrives at about 68 GB resident. Stock mlx-vlm maps everything, so this machine swaps. The transformer weights are ~39 GB. The table is the problem.

The measured result, before the story: 41.7 GB peak MX memory against ~68 GB stock, 32.2 tok/s decode against the 28 tok/s in-memory reference, ~29.5 GB max process RSS, and bit-exact output versus the in-memory QuantizedEmbedding path. All numbers are local runs on this box.

The table nobody needed hot

The n-gram table is a hashed embedding at layer 2 of the decoder. Sixteen n-gram heads hash the last two or three tokens, and each head fetches one row. Sixteen rows of 160 dimensions feed the 2560-wide residual stream. The table is split across 128 shard files, each 2,500,012 rows by 160 dims, affine 4-bit with group size 32, about 100 bytes per row.

The model card is direct about this design: the table does not need to be resident, and the card's own "39 GB mode" serves it from NVMe with a row-granular disk-read patch. That patch is the model author's and has not been upstreamed, so I reimplemented it. Each token touches sixteen rows, roughly 1.6 KB of reads; keeping 51B parameters hot for that is waste.

The memory move

How 68 GB became 41.7 GB

Final state
How a 68 GB Qwen3.8 Flash-Next load fits in 41.7 GB Before: stock mlx-vlm maps about 39 GB of transformer weights and about 29 GB of n-gram table, about 68 GB total, over the 64 GB Mac Studio limit, so it swaps. Move: the 51B n-gram table leaves unified memory for 128 memory-mapped NVMe shards while about 39 GB of weights stay resident. Per token: a decode step sends 16 row IDs to NVMe; about 1.6 KB of quantized rows returns, dequantizes, and joins the resident path. Result: measured peak MX memory is 41.7 GB, throughput is 32.2 tok/s versus the 28 tok/s reference, and correctness is bit-exact. 01 / BEFORE Stock mlx-vlm everything mapped resident 64 GB Mac Studio ~39 GB transformer weights ~29 GB n-gram table mapped in memory ~68 GB total exceeds 64 GB · swaps 02 / MOVE Split the cold tensor one owner stays in memory unified memory ~39 GB weights stay resident NVMe 51B n-gram table 128 mmap shards 128 memory-mapped shards on NVMe ~39 GB weights remain resident 03 / PER TOKEN One sparse gather request → rows → join decode step sends 16 row IDs ID ID ID NVMe row store only the requested rows return ~1.6 KB quantized Q Q Q dequantize + join into resident 39 GB path 16 rows · affine 4-bit → residual stream 04 / RESULT Working set settles measured on the box 41.7 GB peak MX memory ~42 GB working set 32.2 tok/s versus 28 tok/s reference bit-exact vs QuantizedEmbedding correctness gate
The sequence shows the stock 68 GB residency, the 51B n-gram table leaving unified memory for 128 NVMe shards, and each token’s 16-row request/return. Only ~1.6 KB comes back to dequantize beside the resident ~39 GB weights; the measured result is 41.7 GB peak MX, 32.2 tok/s, and bit-exact output.

Leave it on NVMe

A patched mlx_vlm/models/qwen4_exp/ngram_disk.py reimplements the card's 39 GB mode.

  • Gather path. A shared per-file numpy memmap reads the sixteen rows; mx.dequantize expands them. MLX_VLM_NGRAM_DISK_CACHE_ROWS defaults to 131,072 rows for repeats. One memmap per shard file: per-tensor memmaps exhaust the default 256-descriptor limit.
  • Opt-in. MLX_VLM_NGRAM_ON_DISK=1; fitting models keep the old path.
  • Correctness gate. Synthetic and real shard gathers are bit-exact with QuantizedEmbedding, covered by pytest before serving.

Disk-backed mmap, dequantize, and cache changed the result: 32.2 tok/s versus the 28 tok/s in-memory reference, 41.7 GB peak MX memory, and ~29.5 GB process RSS. The working set falls from ~68 GB to ~42 GB. NVMe reads for sixteen short rows cost less than compressed-memory pressure.

One memory owner per box

Making the loader work was only half. A launchd sidecar on :8080 first served the patched module as a dedicated Studio provider. Then a headless job loaded a second 36 GB model in another server. No process owned total memory; the guard aborted, the other server crashed, and the dashboard still said "No resident model" while 41.6 GB was gone. Two servers that each believe they own the box will meet in the swap file.

Current state: REAP-288 runs in oMLX on :8000 with thinkingFormat: qwen-chat-template. The sidecar stays retired. One memory owner per box.

Honest edges

  • 262k context is config-verified, not stress-tested; no prefill beyond 32k has run. The memory guard is the backstop.
  • REAP is not pinned. Another model evicts it; the next call pays a ~14 s reload.
  • oMLX throughput is ~20-29 tok/s without the MTP drafter. The model card reports speculative decoding as roughly break-even on M4-class machines and 1.5-2.6x on M5-class; this box is an M4 Max, so I will measure before counting the drafter as a win.
  • Reasoning effort is partial. The template accepts xhigh (default), medium, and low; high returns a 500.

The transferable part

Find the tensor that breaks residency. Here it was the 51B n-gram table, with sixteen rows touched per token. For sparse access, memmap from NVMe, gather by row, dequantize on read, keep one mmap per file, and bound the cache. Prove bit-exactness against the resident path before serving.

A model that does not fit can run on the box when the access pattern, not the whole tensor, determines residency.