A fast first answer can hide a slow next turn.
We built up prompt history, then sent a continuation with 12 new input tokens and an output cap of 128 tokens. On this hybrid model, keeping that history available mattered: an evicted cache meant reading the context again.
| Scripted sessions × history | GPU cache only, repeats 1 / 2 | CPU tier, repeat 1 | CPU tier, repeat 2 |
|---|---|---|---|
| 3 × 32K | 47.5 / 45.2 | 46.4 | 48.4 |
| 4 × 32K | 7.0 / 7.6 | 35.8 | 44.1 |
| 8 × 16K | 8.3 / 8.8 | 31.6 | 37.4 |
| 16 × 8K | 9.0 / 9.8 | 34.4 | 37.4 |
These selected rows are transcribed from experiments 8a and 8c in the retained benchmark notes. Effective speed includes the request’s waiting and generation time; it is not pure decode speed. Both configurations have two successive repeats, not a confidence interval or a large sample.
The tier used --kv-offloading-size 24: 24 GiB of host RAM, not SSD storage. Repeat one could pay the CPU-to-GPU restore cost; repeat two could reuse a GPU-resident cache when the working set fit. The recorded container had a 116 GiB RAM cap for this experiment. Do not treat that as the specification of the workstation below.
Our takeaway is narrower than “CPU offload makes everything faster”: the 3 × 32K row barely changed. It helped when the tested history was being evicted. Include cold and warm follow-up turns in a capacity test, and record the cache configuration.
Download these cache results (CSV). The separate agent runs below used different workloads and serving settings.
What we observed.
The internal tests ran on 2 September 2026. At 16 agent sessions, all 16 tasks completed in the test log, with a median of 7.5 minutes. At 30 sessions, 27 completed, two failed in the harness and one was still running when observation ended. Completed tasks had a median of 20.7 minutes.
| Sessions | Recorded outcome | How to interpret it |
|---|---|---|
| 8 | Recorded parallel task excerpt. | Source summaries differ; no completion-rate claim is made here. |
| 16 | 16 completed; median 7.5 minutes. | A completed run under the recorded configuration. |
| 30 | 27 completed; 2 harness failures; 1 unfinished. Median 20.7 minutes for completed tasks. | Stress test beyond practical capacity for this workload. |
The larger run included continuing and fresh tasks. It is not an experiment that isolates session count as the only variable, and the median excludes the unfinished task. These observations do not establish a supported number of customers.
Medians are recalculated from the listed per-session times: 447.5 seconds for the 16 completed tasks and 1,240 seconds for the 27 completed tasks. “Completed” is the harness outcome, not a claim that every result passed an independent correctness test. Download the recorded completion times (CSV).
The recorded setup.
- One RTX 5090 on a remote test server, with 32 GB VRAM.
- Qwen3.8-27B in NVFP4, served with vLLM 0.28 and FP8 KV cache.
- An engineering agent harness performing small coding tasks, including file edits and checks.
- CPU cache offloading; serving settings varied between runs.
- Test host RAM: rented cloud servers with container RAM caps of about 116 GiB (125 GB). The TQ–32 has 64 GB of RAM. A later in-house test with the same GPU and a 64 GB-class RAM budget is in lab note 04; results measured on an assembled TQ–32 are pending.
These are historical test settings, not a current installation recipe. The visible harness is not a promise of the interface supplied with a TensorQuay workstation. A session may be waiting for the model, using a tool or checking its output.
Watch the evidence.
Each public clip is a continuous, silent 60-second excerpt at original speed. The full frame is retained, including failures or retries visible during the selected interval. A short excerpt cannot show that every task in the full run completed.
- 8-session recording — source interval 00:15–01:15.
- 16-session recording — source interval 00:45–01:45.
- 30-session stress recording — source interval 15:00–16:00.
For the player and session selector, watch the recorded tests on the homepage. Completion summaries above come from the internal test log, experiments 9h and 9j; they are not inferred from the one-minute clips.
Measure the experience as well as throughput.
Combined tokens per second describe server output. They do not tell you whether a user received a correct result or how long an agent spent waiting, retrying and running tools.
vLLM’s benchmark documentation distinguishes the wait for first output, gaps between streamed outputs and per-request output timing. Prefix-cache reuse can also change a repeated benchmark. Record the measurement method and cache conditions alongside the result. See the vLLM benchmark guide.
For your own workload, keep a log of accepted tasks, total completion time, errors, unfinished tasks and human corrections. Repeat the same task set before comparing configurations. Report the slow cases as well as the typical one.
Our separate short-request throughput measurements used up to 512 output tokens at 8, 16 and 32 concurrent requests. At 8 requests, combined output was about 445 tok/s and per-session output about 57 tok/s. Per-session speed is measured over each request’s own start-to-finish time, so it is not the combined figure divided by eight. The 32-request result is not the 30-session agent test. Combining those labels would misrepresent the evidence.
What this does not prove.
The tests do not establish workstation thermals, acoustics, power consumption, long-term reliability or performance on your documents. They do not prove the quality of another model or a newly trained model. They also do not establish that 16 simultaneous customers will have an acceptable experience.
Our practical conclusion is to size and test for the work people need to finish. Raising the number of open sessions is useful only if the completion time and results remain acceptable.
Continue with the small task-based evaluation worksheet or the guide to hardware and capacity.