MLX-serve adds Qwen3.8-Flash-Next support sustaining 1M context on Apple silicon
A developer released Qwen3.8-Flash-Next engine support for MLX-serve, tuned to run the model at up to 1M context on an M5 Max with 128GB of unified memory. The co-creator of the engine support said generation sustains through the full 1M window at roughly 40 tokens per second on prose and 75 tokens per second on coding, using temperature 1.0 sampling rather than short-context greedy benchmarks. The build quantizes dense layers to 8 bits and expert layers to 4 bits to preserve quality, and peaks near 117GB of memory, so users must set iogpu.wired_limit_mb to 120000 before attempting the full context. Weights are published on Hugging Face and the engine on GitHub, alongside an OpenCode2 plugin that displays an MLX Serve Monitor. The developer acknowledged bugs remain and asked for reports.
The release lands as the same model family draws engine-comparison work elsewhere. One benchmark on a 96GB VRAM workstation measured time to first token at the full 262K window at 35.4 seconds in SGLang, 80.4 seconds in FreeToken, and 210.2 seconds in llama.cpp with an MTP fork, testing speculative decoding and newer PR patches.
The MLX-serve build shows long-context Qwen3.8-Flash-Next inference is becoming practical on Apple silicon, while separate engine benchmarks show wide variance in time-to-first-token across llama.cpp, SGLang, and FreeToken on the same hardware.