Qwen3 27B Hits 50 Tokens/Sec at 256K Context on a Single 24GB GPU
A new blog post details running Qwen3's 27B-parameter model with a full 256K token context window on an RTX PRO 4000 SFF, a small-form-factor workstation GPU with just 24GB of VRAM and 432 GB/s of memory bandwidth. Despite the modest hardware, the setup achieves roughly 50 tokens per second by leveraging multi-token prediction (MTP), a technique that lets the model generate several tokens per forward pass instead of one at a time.
The result is notable because large context windows are usually memory-hungry, and 256K tokens is enough to hold entire codebases or lengthy documents in a single prompt. The author walks through the quantization and memory-management choices needed to fit both the model weights and the KV cache for that context length into 24GB, a constraint that normally forces users toward smaller models or datacenter-grade cards.