Same GPU. The RAM budget of a TQ–32.
These runs used our in-house RTX 5090 (32 GB) test machine with a 64 GB-class RAM budget, of which 62 GB is usable. That is the same GPU and RAM budget as a TQ–32. It is not an assembled TQ–32, so this note says nothing about a TQ–32’s thermals, noise or long-term reliability. The machine was also running other workloads during the tests, sharing its RAM and CPU; the GPU was not in use by anything else. A dedicated machine may do somewhat better, but we have not measured that.
- Model: Qwen3.8-27B in NVFP4 (RadixArk checkpoint), FP8 KV cache, multi-token prediction off.
- Runtime: vLLM 0.28.0 from the official
vllm/vllm-openai:v0.28.0container image. - Key settings:
--gpu-memory-utilization 0.95,--max-model-len 131072,--max-num-seqs 16, and--kv-offloading-size 24for a 24 GiB CPU KV-cache tier in host RAM. vLLM reported a GPU KV-cache pool of about 202K tokens. - Memory used: the serving container peaked at about 28 GiB of host RAM, including the 24 GiB tier.
These are the same model, runtime and cache settings as the CPU-tier results in our first Qwen note. The main difference is the host RAM.
How we measured.
For each test cell, every session first builds up its history one at a time. Then all sessions send a short follow-up at the same moment, and each generates 128 tokens. We repeat that follow-up twice. The history is synthetic filler text, not real coding tasks.
Tok/s per user is 128 generated tokens divided by that user’s own wall-clock time. It includes waiting in the queue, restoring cached history from RAM, prefill and decode. It is not pure decode speed, and it is not the combined output divided by the number of users.
Results: two runs, two days.
Each bar pair: repeat 1 (solid) and repeat 2 (light) of the same follow-up
4 × 32K 128K live tokens
8 × 16K 128K live tokens
8 × 32K 256K live tokens
16 × 16K 256K live tokens
Reference: 4 × 32K, GPU cache only earlier test on a rented cloud server; not re-measured here
Bar scale 0–50 tok/s. The reference row used the same model, runtime and follow-up test without the CPU tier, on a different machine with a 116 GiB RAM cap.
| Sessions × history | Run 1 · 26 Sep | Run 2 · 28 Sep | Slowest–fastest user |
|---|---|---|---|
| 4 × 32K | 42.2 / 45.8 | 42.1 / 45.8 | 41.8–46.0 |
| 8 × 16K | 39.9 / 45.6 | 44.1 / 46.4 | 39.7–46.6 |
| 8 × 32K | 33.1 / 35.5 | 35.1 / 35.5 | 22.0–43.7 |
| 16 × 16K | 28.6 / 33.1 | 31.5 / 34.7 | 19.9–45.0 |
All requests in both runs completed. The second run, two days later with the same settings and cell order, stayed within about 4 tok/s of the first in every cell. That is two runs, not a confidence interval.
The four-session row is the one to compare with our earlier note. There, four 32K sessions with no CPU tier managed about 7 tok/s each, because history kept falling out of the GPU cache and had to be read again. With the 24 GiB tier, the same kind of follow-up ran at 42–46 tok/s, here on a machine with the RAM budget of a TQ–32.
Past the GPU cache, service gets uneven.
Eight sessions × 32K is 256K tokens of live history. That is more than the roughly 202K tokens the GPU cache can hold, so some sessions have to be restored from the RAM tier before they can continue.
The result was not an even slowdown. At 8 × 32K, the same pattern showed up in all four repeats: three users got about 22–23 tok/s, and the other five got about 39–44. At 16 × 16K, seven users got about 20–23 tok/s and the rest 30–45. The mean per user, 29–36 tok/s, describes neither group well. That is why we show the slowest and fastest users as well as the average.
For planning, the useful question is how slow the slowest session gets, not just the average. Here, the slowest user at eight long sessions still got about 22 tok/s on a 128-token reply.
A separate test: part of the cache on an SSD.
We also tried a second cache tier on the machine’s NVMe SSD, behind the 24 GiB RAM tier. This used vLLM 0.30.0, a newer release than the runs above, so its numbers are not directly comparable with them. It is one run.
To force the SSD tier to be used, we ran 12 sessions × 64K tokens (768K tokens of history), more than the GPU cache and RAM tier hold together. Each session had unique random-token history, so sessions could not share cached blocks.
| Follow-up | Mean tok/s per user | Median | Slowest–fastest user |
|---|---|---|---|
| Repeat 1 | 7.8 | 7.2 | 5.4–12.1 |
| Repeat 2 | 17.5 | 12.4 | 7.9–34.9 |
It works: all 12 sessions completed, and vLLM’s own metrics show 5.4 GB of cache read back from the SSD tier. It is also slow: about 8 tok/s per user on the first follow-up, with a wide spread. About a third of the requested history was not found in any cache tier and had to be processed again.
We have not yet run the same cell with the RAM tier only, so this test does not show how much the SSD tier helped compared with recomputing the history. Treat it as “possible, and slow”, not as a recommendation.
What this does not show.
- Not a TQ–32 qualification. Same GPU and RAM budget, but not an assembled TQ–32. Thermals, noise, power and long-term reliability are untested here.
- Not real work. The history is synthetic filler text and the replies are 128 tokens. Real coding sessions add tool calls, longer replies and pauses. These results are not agent task success rates.
- Small sample. Two runs of the main test on different days, and one run of the SSD test. Per-user speed varied widely in the larger cells.
- One model and runtime. Qwen3.8-27B NVFP4 on vLLM with these settings. Another model, quantisation or runtime can behave differently.
Download the results (CSV), including every user’s tok/s in every repeat, so you can check the means, medians and ranges yourself.