13
Alibaba9B paramsBF16Dense

Qwen-3.5-9B

The best quality per parameter in this arena, and you pay for it in throughput.

Measured on 2026-08-07 with vLLM v0.26.0. In the chat test with ten concurrent requests this profile reaches 12.51 tokens/s per user, with an average of 1.82 seconds to the first token. At 25k context that average wait rises to 37.12 seconds. These runs measure speed and waiting, not the quality of the answers.

62.6
Suitability score (indicative)
84
Throughput tok/s
18 GB
VRAM
11/11
Reliable benches
Hugging Face →·vLLM v0.26.0·DGX Spark, NVIDIA GB10, 128 GB unified memory·Measured with context 128K·Last measured 7 August 2026
02Vendor quality · reference only

Knowledge: MMLU-Pro; reasoning: GPQA-Diamond; coding: LiveCodeBench v6. Each value names its exact test. Other tests and missing measurements do not qualify for the total score. External figures are not our own evaluation of this precision. Source: vendor model card ↗

3/3
Coverage
82.5
MMLU-Pro
81.7
GPQA-Diamond
65.6
LiveCodeBench v6
03Performance · BF16

Decode throughput · total t/s · c=10

■ BF16
1k ctx BF16123 t/s
8k ctx BF1684 t/s
4k+turn BF16113 t/s
25k ctx BF1630 t/s
04Test suite · 11 benchmarks

6 closed-loop tests with llama-benchy and 5 open-loop tests with vllm bench serve. Only complete runs without failed sanity checks are eligible; open-loop tests require at least 99% successful requests. Raw results remain visible. Expand “view command” for the reconstructed command. Methodology →

01 · llama-benchyclosed-loop

Chat

Short prompt, long answer. The shape that has to feel like ordinary chat; TTFT decides whether it feels snappy.

pp (prompt)
1024
tg (gen)
1024
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
12.5t/s
TTFT · mean
1.82s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model qwen3.5-9b-bf16 \
  --pp 1024 \
  --tg 1024 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
02 · llama-benchyclosed-loop

RAG · 8k context

Medium context, a few document chunks with an answer of normal length. Shows prefill cost without hitting the wall.

pp (prompt)
8192
tg (gen)
512
depth
0
concurrency
5 · 10 · 20
repeats
3
Tokens/sec · per user
10.3t/s
TTFT · mean
11.33s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model qwen3.5-9b-bf16 \
  --pp 8192 \
  --tg 512 \
  --depth 0 \
  --concurrency 5 10 20 \
  --runs 3 \
  --latency-mode generation \
  --format md
03 · llama-benchyclosed-loop

Long output and agents

Short instruction, lots of output. Code generation, reports or structured agent output. A stress test for decode throughput.

pp (prompt)
256
tg (gen)
4096
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
12.6t/s
TTFT · mean
637ms
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model qwen3.5-9b-bf16 \
  --pp 256 \
  --tg 4096 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
04 · llama-benchyclosed-loop

Multi-turn office work

Five turns per conversation, ten conversations in parallel. Close to how a team actually uses this, with context growing every turn.

pp (prompt)
2048
tg (gen)
512
depth
4
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
12.1t/s
TTFT · mean
3.13s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model qwen3.5-9b-bf16 \
  --pp 2048 \
  --tg 512 \
  --depth 4 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
05 · llama-benchyclosed-loop

Large context · 25k

A stress test with large prompts. Not necessarily chat material, but exactly where the prefill wall shows up and TTFT collapses.

pp (prompt)
4096 · 8192 · 16384 · 25000
tg (gen)
256
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
5.9t/s
TTFT · mean
37.12s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model qwen3.5-9b-bf16 \
  --pp 4096 8192 16384 25000 \
  --tg 256 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
06 · llama-benchyclosed-loop

Concurrency under pressure

25k context with twenty requests at once. Five and ten are already covered by test 05; twenty shows where the scheduler gives up.

pp (prompt)
25000
tg (gen)
256
depth
0
concurrency
20
repeats
3
Tokens/sec · per user
3.8t/s
TTFT · mean
71.44s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model qwen3.5-9b-bf16 \
  --pp 25000 \
  --tg 256 \
  --depth 0 \
  --concurrency 20 \
  --runs 3 \
  --latency-mode generation \
  --format md
07 · vllm bench serveopen-loop

Realistic office baseline

Random dataset, 4000 tokens in and 500 out, arrival rate 0.3 with burstiness 0.7. A quiet office.

dataset
random
input / output
4000 / 500
rate (req/s)
0.3
prompts
200
burstiness
0.7
Tokens/sec · per user
37.6t/s
TTFT · p50
1.87s
Request success 100% · Included
200 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 200 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 07-office-baseline.json
08 · vllm bench serveopen-loop

Real conversations · ShareGPT

ShareGPT V3, averaging 228 tokens per turn, varying naturally per conversation. What real users do, not a synthetic distribution.

dataset
sharegpt
rate (req/s)
0.3
prompts
250
burstiness
0.7
Tokens/sec · per user
9.2t/s
TTFT · p50
251ms
Request success 100% · Included
250 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name sharegpt \
  --dataset-path /tmp/ShareGPT_V3.json \
  --num-prompts 250 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 08-sharegpt.json
09 · vllm bench serveopen-loop

Reasoning workload

Long chains of thought, 1k in and 4k out, a slow arrival rate of 0.2 because every request costs a lot of decode budget. Tests whether TTFT stays stable.

dataset
random
input / output
1024 / 4096
rate (req/s)
0.2
prompts
50
burstiness
1
Tokens/sec · per user
6.1t/s
TTFT · p50
531ms
Request success 100% · Included
50 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 4096 \
  --random-range-ratio 0.9 \
  --num-prompts 50 \
  --request-rate 0.2 \
  --burstiness 1.0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 09-reasoning.json
10 · vllm bench serveopen-loop

Monday morning peak

Random, 4000 in and 500 out, arrival rate 1.5 per second with burstiness 1.0 and at most 25 in parallel. A heavy, independent request stream.

dataset
random
input / output
4000 / 500
rate (req/s)
1.5
prompts
300
burstiness
1
max parallel
25
Tokens/sec · per user
49.3t/s
TTFT · p50
1.65s
Request success 100% · Included
300 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 300 \
  --request-rate 1.5 \
  --burstiness 1.0 \
  --max-concurrency 25 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 10-monday-peak.json
11 · vllm bench serveopen-loop

Capacity · rate sweep

Six increasing arrival rates, from 0.1 to 1.0 requests per second. At each rung we check whether p95 TTFT still stays under the threshold.

dataset
random
input / output
4000 / 500
rates (req/s)
0.1 · 0.2 · 0.3 · 0.5 · 0.7 · 1.0
prompts
100 · 100 · 100 · 125 · 175 · 250
burstiness
0.7
Req/s · TTFT < 5s
0.24req/s
Req/s · TTFT < 10s
0.47req/s
Request success 100% · Included
6 rates · seed 42view command →hide command ↑ 
# 0.1 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.1 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.1.json

# 0.2 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.2 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.2.json

# 0.3 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.3.json

# 0.5 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 125 \
  --request-rate 0.5 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.5.json

# 0.7 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 175 \
  --request-rate 0.7 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.7.json

# 1.0 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen3.5-9B \
  --tokenizer Qwen/Qwen3.5-9B \
  --served-model-name qwen3.5-9b-bf16 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 250 \
  --request-rate 1.0 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-1.0.json
05Notes on this run

Chat

Chat with ten concurrent requests

With 1024 input and 1024 output tokens at ten concurrent requests I measure an average of 12.51 tokens/s per user. The average wait for the first token is 1.82 seconds.

Long context

Wait at 25k context

With 25000 input tokens and 256 output tokens at ten concurrent requests the average wait for the first token is 37.12 seconds. After that the model generates an average of 5.91 tokens/s per user. The measurement alone does not show whether memory, prompt processing or scheduling causes the delay.

Peak load

Requests and wait under peak load

The run handles 0.32 requests/s at a configured arrival rate of 1.5 requests/s. 300 of 300 requests succeed. The p95 wait for the first token is 5.59 seconds.

Measurement coverage

Which measurements count

11 of 11 tests have a measurement that counts under the current Arena rules. Output for the sanity check was stored but has not been reviewed yet. The current count therefore does not mean that check passed.

More numbers?
Read the full article.

RunSource data and metadata→WeightsHugging Face→BackTo the arena→
MarkdownSource JSON
Explanation