Windows LLM inference slows 2-3x when server window loses focus; headless run fixes it

A Windows 11 user has traced a 2-3x slowdown in local LLM inference to a surprising cause: whether the server's console window has focus. On an RTX 5090 running a roughly 27B NVFP4 model through ninfer, u/koloved measured 130-200 tokens per second when the terminal was focused, but only 50-60 tok/s the moment the window lost focus. Clicking back on the terminal restored full speed immediately, and the pattern reproduced with a native Windows build, ruling out a WSL2-specific quirk.

The user initially suspected GPU throttling, but monitoring showed the opposite. During the slow runs, the SM clock was actually higher at 2550-2600 MHz versus 2100-2200 MHz when focused, while the power limit stayed at 400 W. There was no PCIe power-state drop, and decode-host time held around 390-494 microseconds. The decisive difference was in the serving loop's wait time: 38-41 ms per token when unfocused versus 16-17 ms when focused. The GPU was not getting slower; the CPU-side serving loop was being delayed.

The fix was to run the server detached or headless, for example with docker run -d, rather than keeping it attached to a console window. That kept wait time around 17 ms and throughput at 130-200 tok/s regardless of which window had focus. The user attributes the behavior to Windows foreground and background CPU scheduling, and asked whether others have seen the same effect with local inference on Windows 11.

The report is a single anecdote from one Reddit thread, and no other coverage of the same subject was available to corroborate it. The underlying mechanism is not fully explained: the user did not identify which Windows scheduling policy or setting causes the delay, and it is unclear whether the effect is specific to ninfer, to the RTX 5090, or to certain model sizes. The post does not specify the exact Windows 11 build or driver version in use.

If the finding generalizes, it has practical implications for anyone benchmarking or serving local models on Windows, since throughput numbers can swing by a factor of two or three depending on nothing more than which window is active. The suggested workaround, running the inference server headless or detached, is simple to test on other hardware and software stacks.

What remains unknown is whether the slowdown stems from Windows' default background throttling of console processes, from a timer or message-pump behavior tied to window focus, or from something in the serving framework itself. The poster has not yet reported follow-up tests on other GPUs, models, or inference engines, and no maintainer of ninfer or Windows has commented on the thread.