The suite defines the same 11 benchmarks for every model on the same DGX Spark: 6 closed-loop tests for controlled throughput and 5 open-loop tests for behaviour under load. Each run records which tests completed, stopped early or have no data. Part of the guide Running LLMs on the DGX Spark.
A single speed measurement does not show how a model responds to a different load. A model with high tokens/sec on one stream can slow down sharply under twenty concurrent requests. Another may retain throughput while making new requests wait for their first token. Fine for batch, frustrating for chat.
I therefore measure throughput at fixed concurrency (closed-loop) and with independently arriving requests (open-loop). I also vary prompt and response lengths. This shows how speed and latency change across scenarios.
How does throughput scale at a fixed number of concurrent streams?
Closed-loop keeps one request active per stream: the next starts as soon as one finishes. The suite tests fixed concurrency values from 1 to 20. Every cell runs three times and is reported as mean ± stddev.
Ideal for: a single user, batch processing, code completion. Not representative of: a chat app with many concurrent sessions.
How does it hold up under X concurrent users who don't wait for each other?
Open-loop schedules arrivals independently of what the server is already doing. Request rate and burstiness define the pattern; at 0.3 req/s the average gap is about 3.3 seconds. The generator uses seed 42 so the configured load is repeatable.
Result: p50/p95 TTFT and per-user tokens/sec under load. I do report TTFT numbers (above 2s your app starts to feel slow), but they don't drop models from the ranking. An open-loop test only counts when at least 99% of requests complete successfully.
Every card comes directly from the versioned benchmark contract. Expand "view command" for the reproducible command template and replace ORG/MODEL and SERVED_MODEL_NAME. Model pages fill those values from the run metadata.
Short prompt, long answer. The shape that has to feel like ordinary chat; TTFT decides whether it feels snappy.
tokens/sec · ttft p50
uvx llama-benchy==0.4.0 \
--base-url http://localhost:8000/v1 \
--model SERVED_MODEL_NAME \
--pp 1024 \
--tg 1024 \
--depth 0 \
--concurrency 1 5 10 \
--runs 3 \
--latency-mode generation \
--format mdMedium context, a few document chunks with an answer of normal length. Shows prefill cost without hitting the wall.
tokens/sec · ttft p50
uvx llama-benchy==0.4.0 \
--base-url http://localhost:8000/v1 \
--model SERVED_MODEL_NAME \
--pp 8192 \
--tg 512 \
--depth 0 \
--concurrency 5 10 20 \
--runs 3 \
--latency-mode generation \
--format mdShort instruction, lots of output. Code generation, reports or structured agent output. A stress test for decode throughput.
tokens/sec · ttft p50
uvx llama-benchy==0.4.0 \
--base-url http://localhost:8000/v1 \
--model SERVED_MODEL_NAME \
--pp 256 \
--tg 4096 \
--depth 0 \
--concurrency 1 5 10 \
--runs 3 \
--latency-mode generation \
--format mdFive turns per conversation, ten conversations in parallel. Close to how a team actually uses this, with context growing every turn.
tokens/sec · ttft p50
uvx llama-benchy==0.4.0 \
--base-url http://localhost:8000/v1 \
--model SERVED_MODEL_NAME \
--pp 2048 \
--tg 512 \
--depth 4 \
--concurrency 1 5 10 \
--runs 3 \
--latency-mode generation \
--format mdA stress test with large prompts. Not necessarily chat material, but exactly where the prefill wall shows up and TTFT collapses.
tokens/sec · ttft p50
uvx llama-benchy==0.4.0 \
--base-url http://localhost:8000/v1 \
--model SERVED_MODEL_NAME \
--pp 4096 8192 16384 25000 \
--tg 256 \
--depth 0 \
--concurrency 1 5 10 \
--runs 3 \
--latency-mode generation \
--format md25k context with twenty requests at once. Five and ten are already covered by test 05; twenty shows where the scheduler gives up.
tokens/sec · ttft p50
uvx llama-benchy==0.4.0 \
--base-url http://localhost:8000/v1 \
--model SERVED_MODEL_NAME \
--pp 25000 \
--tg 256 \
--depth 0 \
--concurrency 20 \
--runs 3 \
--latency-mode generation \
--format mdRandom dataset, 4000 tokens in and 500 out, arrival rate 0.3 with burstiness 0.7. A quiet office.
ttft p50/p95 · tokens/sec per user
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 200 \
--request-rate 0.3 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 07-office-baseline.jsonShareGPT V3, averaging 228 tokens per turn, varying naturally per conversation. What real users do, not a synthetic distribution.
ttft p50/p95 · tokens/sec per user
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name sharegpt \
--dataset-path /tmp/ShareGPT_V3.json \
--num-prompts 250 \
--request-rate 0.3 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 08-sharegpt.jsonLong chains of thought, 1k in and 4k out, a slow arrival rate of 0.2 because every request costs a lot of decode budget. Tests whether TTFT stays stable.
ttft p50 · sustained tokens/sec
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 4096 \
--random-range-ratio 0.9 \
--num-prompts 50 \
--request-rate 0.2 \
--burstiness 1.0 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 09-reasoning.jsonRandom, 4000 in and 500 out, arrival rate 1.5 per second with burstiness 1.0 and at most 25 in parallel. A heavy, independent request stream.
ttft p95/p99 · queue depth
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 300 \
--request-rate 1.5 \
--burstiness 1.0 \
--max-concurrency 25 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 10-monday-peak.jsonSix increasing arrival rates, from 0.1 to 1.0 requests per second. At each rung we check whether p95 TTFT still stays under the threshold.
req/s under p95 TTFT threshold
# 0.1 req/s
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 100 \
--request-rate 0.1 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 11-rate-sweep-0.1.json
# 0.2 req/s
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 100 \
--request-rate 0.2 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 11-rate-sweep-0.2.json
# 0.3 req/s
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 100 \
--request-rate 0.3 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 11-rate-sweep-0.3.json
# 0.5 req/s
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 125 \
--request-rate 0.5 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 11-rate-sweep-0.5.json
# 0.7 req/s
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 175 \
--request-rate 0.7 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 11-rate-sweep-0.7.json
# 1.0 req/s
docker exec vllm-bench vllm bench serve \
--backend openai-chat \
--base-url http://localhost:8000 \
--endpoint /v1/chat/completions \
--model ORG/MODEL \
--tokenizer ORG/MODEL \
--served-model-name SERVED_MODEL_NAME \
--dataset-name random \
--random-input-len 4000 \
--random-output-len 500 \
--random-range-ratio 0.9 \
--num-prompts 250 \
--request-rate 1.0 \
--burstiness 0.7 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,90,95,99 \
--seed 42 \
--save-result \
--result-dir /tmp \
--result-filename 11-rate-sweep-1.0.jsonCommands are built deterministically from the versioned contract and each run's metadata.
First start an OpenAI-compatible vLLM server with the model and server configuration from the run metadata. Then use the command on the model page. Test 08 expects ShareGPT V3 at /tmp/ShareGPT_V3.json; the other open-loop tests generate synthetic random prompts.
Historical runs did not store the original argv verbatim. Published commands are therefore labelled reconstructions from the contract and run metadata. The source link pins both Git revisions, so you can inspect exactly what the reconstruction uses.
Send a Hugging Face link or vLLM configuration. If the model fits on the Spark, I can compare it using the same tests.