—
NVIDIA4B paramsFP8Dense

Nemotron-3-Nano-4B

Measured on 2026-08-08 with vLLM v0.26.0. In the chat test with ten concurrent requests this profile reaches 34.89 tokens/s per user, with an average of 0.93 seconds to the first token. At 25k context that average wait rises to 12.4 seconds. These runs measure speed and waiting, not the quality of the answers.

—
Suitability score (indicative)
201
Throughput tok/s
4 GB
VRAM
11/11
Reliable benches
Hugging Face →·vLLM v0.26.0·DGX Spark, NVIDIA GB10, 128 GB unified memory·Measured with context 48K·Last measured 8 August 2026
02Vendor quality · reference only

Knowledge: MMLU-Pro; reasoning: GPQA-Diamond; coding: LiveCodeBench v6. Each value names its exact test. Other tests and missing measurements do not qualify for the total score. External figures are not our own evaluation of this precision. Source: vendor model card ↗

3/3
Coverage
18.1
MMLU-Pro
51.3
GPQA-Diamond
51.8
LiveCodeBench
03Performance · FP8

Decode throughput · total t/s · c=10

■ FP8
1k ctx FP8308 t/s
8k ctx FP8225 t/s
4k+turn FP8300 t/s
25k ctx FP887 t/s
04Test suite · 11 benchmarks

6 closed-loop tests with llama-benchy and 5 open-loop tests with vllm bench serve. Only complete runs without failed sanity checks are eligible; open-loop tests require at least 99% successful requests. Raw results remain visible. Expand “view command” for the reconstructed command. Methodology →

01 · llama-benchyclosed-loop

Chat

Short prompt, long answer. The shape that has to feel like ordinary chat; TTFT decides whether it feels snappy.

pp (prompt)
1024
tg (gen)
1024
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
34.9t/s
TTFT · mean
931ms
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-4b-fp8 \
  --pp 1024 \
  --tg 1024 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
02 · llama-benchyclosed-loop

RAG · 8k context

Medium context, a few document chunks with an answer of normal length. Shows prefill cost without hitting the wall.

pp (prompt)
8192
tg (gen)
512
depth
0
concurrency
5 · 10 · 20
repeats
3
Tokens/sec · per user
27.6t/s
TTFT · mean
3.85s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-4b-fp8 \
  --pp 8192 \
  --tg 512 \
  --depth 0 \
  --concurrency 5 10 20 \
  --runs 3 \
  --latency-mode generation \
  --format md
03 · llama-benchyclosed-loop

Long output and agents

Short instruction, lots of output. Code generation, reports or structured agent output. A stress test for decode throughput.

pp (prompt)
256
tg (gen)
4096
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
37.1t/s
TTFT · mean
331ms
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-4b-fp8 \
  --pp 256 \
  --tg 4096 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
04 · llama-benchyclosed-loop

Multi-turn office work

Five turns per conversation, ten conversations in parallel. Close to how a team actually uses this, with context growing every turn.

pp (prompt)
2048
tg (gen)
512
depth
4
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
32.6t/s
TTFT · mean
1.25s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-4b-fp8 \
  --pp 2048 \
  --tg 512 \
  --depth 4 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
05 · llama-benchyclosed-loop

Large context · 25k

A stress test with large prompts. Not necessarily chat material, but exactly where the prefill wall shows up and TTFT collapses.

pp (prompt)
4096 · 8192 · 16384 · 25000
tg (gen)
256
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
16.4t/s
TTFT · mean
12.4s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-4b-fp8 \
  --pp 4096 8192 16384 25000 \
  --tg 256 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
06 · llama-benchyclosed-loop

Concurrency under pressure

25k context with twenty requests at once. Five and ten are already covered by test 05; twenty shows where the scheduler gives up.

pp (prompt)
25000
tg (gen)
256
depth
0
concurrency
20
repeats
3
Tokens/sec · per user
10.1t/s
TTFT · mean
24.24s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-4b-fp8 \
  --pp 25000 \
  --tg 256 \
  --depth 0 \
  --concurrency 20 \
  --runs 3 \
  --latency-mode generation \
  --format md
07 · vllm bench serveopen-loop

Realistic office baseline

Random dataset, 4000 tokens in and 500 out, arrival rate 0.3 with burstiness 0.7. A quiet office.

dataset
random
input / output
4000 / 500
rate (req/s)
0.3
prompts
200
burstiness
0.7
Tokens/sec · per user
102t/s
TTFT · p50
733ms
Request success 99.5% · Included
200 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 200 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 07-office-baseline.json
08 · vllm bench serveopen-loop

Real conversations · ShareGPT

ShareGPT V3, averaging 228 tokens per turn, varying naturally per conversation. What real users do, not a synthetic distribution.

dataset
sharegpt
rate (req/s)
0.3
prompts
250
burstiness
0.7
Tokens/sec · per user
16.5t/s
TTFT · p50
107ms
Request success 100% · Included
250 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name sharegpt \
  --dataset-path /tmp/ShareGPT_V3.json \
  --num-prompts 250 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 08-sharegpt.json
09 · vllm bench serveopen-loop

Reasoning workload

Long chains of thought, 1k in and 4k out, a slow arrival rate of 0.2 because every request costs a lot of decode budget. Tests whether TTFT stays stable.

dataset
random
input / output
1024 / 4096
rate (req/s)
0.2
prompts
50
burstiness
1
Tokens/sec · per user
15.3t/s
TTFT · p50
467ms
Request success 100% · Included
50 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 4096 \
  --random-range-ratio 0.9 \
  --num-prompts 50 \
  --request-rate 0.2 \
  --burstiness 1.0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 09-reasoning.json
10 · vllm bench serveopen-loop

Monday morning peak

Random, 4000 in and 500 out, arrival rate 1.5 per second with burstiness 1.0 and at most 25 in parallel. A heavy, independent request stream.

dataset
random
input / output
4000 / 500
rate (req/s)
1.5
prompts
300
burstiness
1
max parallel
25
Tokens/sec · per user
113t/s
TTFT · p50
774ms
Request success 100% · Included
300 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 300 \
  --request-rate 1.5 \
  --burstiness 1.0 \
  --max-concurrency 25 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 10-monday-peak.json
11 · vllm bench serveopen-loop

Capacity · rate sweep

Six increasing arrival rates, from 0.1 to 1.0 requests per second. At each rung we check whether p95 TTFT still stays under the threshold.

dataset
random
input / output
4000 / 500
rates (req/s)
0.1 · 0.2 · 0.3 · 0.5 · 0.7 · 1.0
prompts
100 · 100 · 100 · 125 · 175 · 250
burstiness
0.7
Req/s · TTFT < 5s
0.87req/s
Req/s · TTFT < 10s
0.87req/s
Request success 100% · Included
6 rates · seed 42view command →hide command ↑ 
# 0.1 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.1 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.1.json

# 0.2 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.2 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.2.json

# 0.3 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.3.json

# 0.5 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 125 \
  --request-rate 0.5 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.5.json

# 0.7 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 175 \
  --request-rate 0.7 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.7.json

# 1.0 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --tokenizer nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --served-model-name nemotron-3-nano-4b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 250 \
  --request-rate 1.0 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-1.0.json
05Notes on this run

Chat

Chat with ten concurrent requests

With 1024 input and 1024 output tokens at ten concurrent requests I measure an average of 34.89 tokens/s per user. The average wait for the first token is 0.93 seconds.

Long context

Wait at 25k context

With 25000 input tokens and 256 output tokens at ten concurrent requests the average wait for the first token is 12.4 seconds. After that the model generates an average of 16.4 tokens/s per user. The measurement alone does not show whether memory, prompt processing or scheduling causes the delay.

Peak load

Requests and wait under peak load

The run handles 0.7 requests/s at a configured arrival rate of 1.5 requests/s. 300 of 300 requests succeed. The p95 wait for the first token is 1.73 seconds.

Measurement coverage

Which measurements count

11 of 11 tests have a measurement that counts under the current Arena rules. No sanity check was recorded for this historical run. That proof is missing, even where the speed measurement counts under the current rules.

More numbers?
Read the full article.

RunSource data and metadata→WeightsHugging Face→BackTo the arena→
MarkdownSource JSON
Explanation