Qwen3.8-Flash-Next on 2x3090: expert cache moved off GPU during prefill, 2.2-2.5x faster

A Reddit user running Qwen3.8-Flash-Next on a dual-RTX 3090 box reports 2.2 to 2.5x faster prefill after changing when the expert cache occupies GPU memory, according to the fourth post in their ongoing series on the setup [s1]. Prefill had been the weak point of the build: more than 80 seconds before the first token on an 8k prompt, and 24 minutes on a 119k prompt [s1].

The hardware is unchanged from earlier parts except for one swap. The box is two RTX 3090s on dual Broadwell Xeon CPUs, running llama.cpp with UD-Q4_K_XL and a Q8 MTP head on the second card, all expert layers pinned in host RAM, a 150-slot expert cache, 261k context and f16 KV [s1]. Since part 2 the author replaced the LRDIMMs with 6x32 GB DDR4-2133 ECC, and says which benchmark numbers come from the 4-DIMM configuration and which from the 6-DIMM one rather than mixing them [s1].

The prefill cost traces back to a tradeoff made in part 2. The author ran a 512-token micro-batch because a 2048-token micro-batch needs roughly 7.3 GiB of compute buffer per GPU while 512 needs 1.9 GiB, and the difference is about 50 expert-cache slots they wanted to keep for decode [s1]. That choice quietly cost about 3x on prefill at the time [s1]. The mechanism is that experts get streamed from host memory to GPU0 once per micro-batch, so the upload repeats with every micro-batch [s1].

The new change kicks the expert cache off the GPU while the prompt is being processed, so the compute buffer can be larger during prefill [s1]. The author frames the result as 2.2 to 2.5x faster prefill [s1].

This is the fourth installment in a series that has been building the same machine up incrementally. Part 1 took the box from 17 to 25-29 tokens per second with an expert-cache pull request [s1]. Part 2 reached 37-41 t/s after switching to UD-Q4_K_XL and stacking MTP on the cache [s1]. Part 3 addressed a top-k fallback that was sorting more than it needed to [s1]. A separate post from the same author reported that replacing the CUDA top-k fallback lifted median decode from about 30.2 to 33.3 t/s at roughly 119k context, a 9 to 12 percent gain, on the same GGUF, MTP settings and binary with the change toggled off and on [s2]. That run used 128 GB DDR4-2133 across four DIMMs, 150 expert-cache slots, MTP-3 and an allocated context of 261,888 tokens [s2].

The prefill work sits alongside a broader push on Qwen3.8 27B quantization quality. One author published a task-aware quantization pipeline claiming 82.81 percent on reasoning versus 77.34 percent for the byte-matched Unsloth UD IQ2_S and 83.59 percent for BF16, describing it as 99 percent of BF16 reasoning performance at 15 percent of the size, with coding as the next target [s3]. A separate benchmark of Unsloth GGUF quantizations found Q4_K_M at 17 GB matching full BF16 on Terminal-Bench 2.1 and fitting a 24 GB RTX 4090 with room for about 64k tokens of context, while 1-bit UD-IQ1_S at 6.2 GB performed around random chance on GPQA Diamond and got worse with longer reasoning [s4]. That benchmark used F16 KV cache regardless of model quantization, about 2.3 GB per 32k tokens, and ran on a llama.cpp build from 16 August 2026 because earlier builds did not work for the model [s4].

What remains unverified is the prefill claim itself. The 2.2 to 2.5x figure comes from a single user's box with no independent replication, and the source text available for this post is a partial excerpt, so the full measurement methodology, seed counts and comparison runs are not visible here [s1]. The author's own framing also leaves open how much of the gain is attributable to the 6-DIMM memory change rather than the cache change, since both landed between parts 2 and 4 [s1].