—
nvidia30B-A3B paramsFP8MoE

Nemotron-3 Nano Omni 30B-A3B

Measured on 2026-08-06 with vLLM v0.26.0. In the chat test with ten concurrent requests this profile reaches 16.85 tokens/s per user, with an average of 1.42 seconds to the first token. At 25k context that average wait rises to 14.32 seconds. These runs measure speed and waiting, not the quality of the answers.

—
Suitability score (indicative)
109
Throughput tok/s
33 GB
VRAM
9/11
Reliable benches
Hugging Face →·vLLM v0.26.0·DGX Spark, NVIDIA GB10, 128 GB unified memory·Measured with context 128K·Last measured 6 August 2026
02Vendor quality · reference only

Knowledge: MMLU-Pro; reasoning: GPQA-Diamond; coding: LiveCodeBench v6. Each value names its exact test. Other tests and missing measurements do not qualify for the total score. External figures are not our own evaluation of this precision. Source: vendor model card ↗

0/3
Coverage
—
knowledge
—
science
—
coding
03Performance · FP8

Decode throughput · total t/s · c=10

■ FP8
1k ctx FP8145 t/s
8k ctx FP8117 t/s
4k+turn FP8137 t/s
25k ctx FP863 t/s
04Test suite · 11 benchmarks

6 closed-loop tests with llama-benchy and 5 open-loop tests with vllm bench serve. Only complete runs without failed sanity checks are eligible; open-loop tests require at least 99% successful requests. Raw results remain visible. Expand “view command” for the reconstructed command. Methodology →

01 · llama-benchyclosed-loop

Chat

Short prompt, long answer. The shape that has to feel like ordinary chat; TTFT decides whether it feels snappy.

pp (prompt)
1024
tg (gen)
1024
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
16.9t/s
TTFT · mean
1.42s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-omni-30b-fp8 \
  --pp 1024 \
  --tg 1024 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
02 · llama-benchyclosed-loop

RAG · 8k context

Medium context, a few document chunks with an answer of normal length. Shows prefill cost without hitting the wall.

pp (prompt)
8192
tg (gen)
512
depth
0
concurrency
5 · 10 · 20
repeats
3
Tokens/sec · per user
13.4t/s
TTFT · mean
5.16s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-omni-30b-fp8 \
  --pp 8192 \
  --tg 512 \
  --depth 0 \
  --concurrency 5 10 20 \
  --runs 3 \
  --latency-mode generation \
  --format md
03 · llama-benchyclosed-loop

Long output and agents

Short instruction, lots of output. Code generation, reports or structured agent output. A stress test for decode throughput.

pp (prompt)
256
tg (gen)
4096
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
18.8t/s
TTFT · mean
863ms
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-omni-30b-fp8 \
  --pp 256 \
  --tg 4096 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
04 · llama-benchyclosed-loop

Multi-turn office work

Five turns per conversation, ten conversations in parallel. Close to how a team actually uses this, with context growing every turn.

pp (prompt)
2048
tg (gen)
512
depth
4
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
14.8t/s
TTFT · mean
2s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-omni-30b-fp8 \
  --pp 2048 \
  --tg 512 \
  --depth 4 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
05 · llama-benchyclosed-loop

Large context · 25k

A stress test with large prompts. Not necessarily chat material, but exactly where the prefill wall shows up and TTFT collapses.

pp (prompt)
4096 · 8192 · 16384 · 25000
tg (gen)
256
depth
0
concurrency
1 · 5 · 10
repeats
3
Tokens/sec · per user
9.7t/s
TTFT · mean
14.32s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-omni-30b-fp8 \
  --pp 4096 8192 16384 25000 \
  --tg 256 \
  --depth 0 \
  --concurrency 1 5 10 \
  --runs 3 \
  --latency-mode generation \
  --format md
06 · llama-benchyclosed-loop

Concurrency under pressure

25k context with twenty requests at once. Five and ten are already covered by test 05; twenty shows where the scheduler gives up.

pp (prompt)
25000
tg (gen)
256
depth
0
concurrency
20
repeats
3
Tokens/sec · per user
6.3t/s
TTFT · mean
26.83s
3 repeats · mean ± stddevview command →hide command ↑ 
uvx llama-benchy==0.4.0 \
  --base-url http://localhost:8000/v1 \
  --model nemotron-3-nano-omni-30b-fp8 \
  --pp 25000 \
  --tg 256 \
  --depth 0 \
  --concurrency 20 \
  --runs 3 \
  --latency-mode generation \
  --format md
07 · vllm bench serveopen-loop

Realistic office baseline

Random dataset, 4000 tokens in and 500 out, arrival rate 0.3 with burstiness 0.7. A quiet office.

dataset
random
input / output
4000 / 500
rate (req/s)
0.3
prompts
200
burstiness
0.7
Tokens/sec · per user
54t/s
TTFT · p50
1.28s
Request success 100% · Included
200 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 200 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 07-office-baseline.json
08 · vllm bench serveopen-loop

Real conversations · ShareGPT

ShareGPT V3, averaging 228 tokens per turn, varying naturally per conversation. What real users do, not a synthetic distribution.

dataset
sharegpt
rate (req/s)
0.3
prompts
250
burstiness
0.7
Tokens/sec · per user
13.1t/s
TTFT · p50
196ms
Request success 100% · Included
250 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name sharegpt \
  --dataset-path /tmp/ShareGPT_V3.json \
  --num-prompts 250 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 08-sharegpt.json
09 · vllm bench serveopen-loop

Reasoning workload

Long chains of thought, 1k in and 4k out, a slow arrival rate of 0.2 because every request costs a lot of decode budget. Tests whether TTFT stays stable.

dataset
random
input / output
1024 / 4096
rate (req/s)
0.2
prompts
50
burstiness
1
Tokens/sec · per user
8.2t/s
TTFT · p50
816ms
Request success 100% · Included
50 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 4096 \
  --random-range-ratio 0.9 \
  --num-prompts 50 \
  --request-rate 0.2 \
  --burstiness 1.0 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 09-reasoning.json
10 · vllm bench serveopen-loop

Monday morning peak

Random, 4000 in and 500 out, arrival rate 1.5 per second with burstiness 1.0 and at most 25 in parallel. A heavy, independent request stream.

dataset
random
input / output
4000 / 500
rate (req/s)
1.5
prompts
300
burstiness
1
max parallel
25
Tokens/sec · per user
65.3t/s
TTFT · p50
1.23s
Request success 66% · Too few successful requests
300 prompts · seed 42view command →hide command ↑ 
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 300 \
  --request-rate 1.5 \
  --burstiness 1.0 \
  --max-concurrency 25 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 10-monday-peak.json
11 · vllm bench serveopen-loop

Capacity · rate sweep

Six increasing arrival rates, from 0.1 to 1.0 requests per second. At each rung we check whether p95 TTFT still stays under the threshold.

dataset
random
input / output
4000 / 500
rates (req/s)
0.1 · 0.2 · 0.3 · 0.5 · 0.7 · 1.0
prompts
100 · 100 · 100 · 125 · 175 · 250
burstiness
0.7
Req/s · TTFT < 5s
—req/s
Req/s · TTFT < 10s
—req/s
No measurement
6 rates · seed 42view command →hide command ↑ 
# 0.1 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.1 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.1.json

# 0.2 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.2 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.2.json

# 0.3 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 100 \
  --request-rate 0.3 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.3.json

# 0.5 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 125 \
  --request-rate 0.5 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.5.json

# 0.7 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 175 \
  --request-rate 0.7 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-0.7.json

# 1.0 req/s
docker exec vllm-bench vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --endpoint /v1/chat/completions \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --tokenizer nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-3-nano-omni-30b-fp8 \
  --dataset-name random \
  --random-input-len 4000 \
  --random-output-len 500 \
  --random-range-ratio 0.9 \
  --num-prompts 250 \
  --request-rate 1.0 \
  --burstiness 0.7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --seed 42 \
  --save-result \
  --result-dir /tmp \
  --result-filename 11-rate-sweep-1.0.json
05Notes on this run

Chat

Chat with ten concurrent requests

With 1024 input and 1024 output tokens at ten concurrent requests I measure an average of 16.85 tokens/s per user. The average wait for the first token is 1.42 seconds.

Long context

Wait at 25k context

With 25000 input tokens and 256 output tokens at ten concurrent requests the average wait for the first token is 14.32 seconds. After that the model generates an average of 9.69 tokens/s per user. The measurement alone does not show whether memory, prompt processing or scheduling causes the delay.

Peak load

Requests and wait under peak load

The run handles 0.41 requests/s at a configured arrival rate of 1.5 requests/s. 198 of 300 requests succeed. The p95 wait for the first token is 2.96 seconds.

Measurement coverage

Which measurements count

9 of 11 tests have a measurement that counts under the current Arena rules. No sanity check was recorded for this historical run. That proof is missing, even where the speed measurement counts under the current rules. No source data for: 11-rate-sweep.

More numbers?
Read the full article.

RunSource data and metadata→WeightsHugging Face→BackTo the arena→
MarkdownSource JSON
Explanation