Nemotron-3 on the DGX Spark: BF16 vs FP8 vs NVFP4

This run is the foundation: what happens when the prompt gets longer, while the number of users climbs from one to ten? That touches office work directly. A short chat is easy. A RAG question with 25k context and several people at once is where the Spark shows how much room is really left.

Here I look at two things. First decode per user: how fast does text come back once generation is running? Then TTFT: how long do you wait for the first token? With long context TTFT is often the pain users feel first. They see no tokens, so it feels like the system is stuck.

Single-user is mostly a pure speed measurement. There NVFP4 nearly doubles BF16. At ten users it gets more interesting: the smaller weights give vLLM more room to batch, and then BF16 just gets heavy.

Decode/user (tg256), c=1

BF16 stays neatly flat around 28-29 tokens per second. That is stable, but not fast. FP8 puts about 50 t/s against it. NVFP4 sits around 55-60 t/s. For a single user that is the difference between “fine” and “this feels local but not local-slow”.

Decode/user (tg256), c=10

At ten users NVFP4 is not “a bit faster”. It is a different class. At 25k context BF16 does 5.56 tok/s/user. NVFP4 does 12.99. That is still no cloud-GPU cluster, but the difference in feel is large: BF16 becomes waiting, NVFP4 keeps working.

TTFT (first token), c=10

This is the table I take most seriously for real users. At 25k context and ten users you wait almost 20 seconds for the first token with BF16. With NVFP4 that is 12.7 seconds. Still long, but not the same kind of long.

Run A shows how context length scales. Run B keeps the context heavy and only raises the concurrency. This is the “everyone asks a big question at the same time” test.

In practice this does not happen every hour. Ten to twenty people rarely click send at exactly the same moment with 25k context. But if you put a local AI machine in front of a team, you want to know how it fails. Calmly getting slower is acceptable. A queue that feels dead is not.

NVFP4 keeps the most air here. Not because the model gets smarter, but because the server with smaller weights has more room for batching and KV-cache.

Twenty users with 25k context is deliberately unkind. Still, it is useful. BF16 sits at 37.88 seconds TTFT. That feels broken. NVFP4 sits at 24.08 seconds. Also not cozy, but still a good thirteen seconds faster.

Aggregate decode shows the same picture:

The ceiling shifts from 44 t/s to 84 t/s. For a single user that is abstract. For a team it means the queue drains faster.

This is the workload for agents, code generation and longer answers: little input, a lot of output. The prompt is only 1024 tokens, so prefill is not the problem here. The question is mostly how fast the model keeps ticking once the output gets long.

So here I look at decode per user. TTFT has to stay low, but the real difference you feel only after a few hundred tokens. A model that starts fast but then hangs at 8 tok/s still feels slow.

NVFP4 clearly wins here. At ten parallel users the model stays at 22.90 tok/s/user. BF16 drops to 7.84. That is still readable, but for an agent flow it feels like someone is typing along by hand.

For this workload NVFP4 is the logical default. FP8 is fine, but here you mostly give up speed without tail-latency playing the lead role.

Multi-turn is closer to real use than one isolated prompt. Five turns per conversation, several conversations in parallel. That resembles an employee who does not ask one question, but keeps asking, corrects, and carries context along.

Here I do not just want to see high throughput. I mostly want the server not to feel like it comes out of a cold start every turn. With ten conversations at once that becomes relevant: the context grows per conversation, the scheduler has to keep sharing, and the user expects the chat to keep running.

This is the most important office run for me. Not because it is perfectly real, but because it comes closest to “25 people use this spread across the day”.

At ten parallel conversations NVFP4 sits at 21.58 tok/s/user. FP8 sits at 14.88. BF16 at 7.68. That last one works technically, but it no longer feels like a snappy chat. NVFP4 stays well above the line where you experience an answer as fluent.

RAG is usually not 25k context, but not a short chat either. This run uses an 8k prompt and 512 output tokens. Think four chunks of about 2k tokens, plus question and instruction.

With RAG prefill counts more than in Run C. You push a sizeable slab of context into the model each time before anything comes back. After that you want enough decode left to make the answer usefully fast.

So the question is: does quantization keep helping when the prompt gets heavier? Yes. NVFP4 stays clearly ahead, even at twenty users.

At twenty users NVFP4 delivers 14.09 tok/s/user. BF16 sits at 5.51. For batch processing that can still work. For real-time RAG in an office BF16 feels tight, certainly when documents are messy and prompts get longer than you had hoped. They always do.

Run G resembles Run C, but pulls the output much further: 4096 tokens. This is the shape of agents that write out plans, generate code, make long analyses or summarize multiple files.

For this kind of workload the first token is almost a side issue. If the answer is long, decode speed determines the experience. Ten seconds of difference at the start is annoying. Waiting on output for minutes is worse.

NVFP4 stays strongest here. More important: it also stays above 25 tok/s/user at ten users. For local hardware on a desk machine that is simply usable.

For agent flows this is fairly hard: BF16 is not broken, but you pay for every long output twice. First in memory, then in waiting time.

From here the interpretation changes. The previous runs push controlled batches through the model. Run H uses open-loop traffic: requests come in according to a Poisson distribution. So the server has to deal with arrivals that do not neatly wait for the previous one to finish.

This resembles an office more. Not perfect, but better than everyone at once or fully sequential. The metrics are different too. TPOT tells how fast tokens come once it is your turn. TTFT P50 tells the normal experience. TTFT P99 tells what the unlucky one notices.

Here FP8 gets interesting. NVFP4 wins the median and TPOT, but FP8 wins the tail. That is exactly why I do not want to end with “NVFP4 is always better”.

That peak concurrent of BF16 looks good on paper, but it is not. The queue grows because BF16 drains it less quickly. NVFP4 processes faster, so fewer requests are open at the same time. That is not lower capacity, that is less of a line.

The real choice is between NVFP4 and FP8. Want the best median and fastest output, then NVFP4. Want the cleanest P99 on this workload, then FP8.

ShareGPT replay is messier and therefore useful. Real conversations have varying lengths, follow-up questions, short answers, long answers and prompts that have not been neatly smoothed out by a benchmark author.

This is the run I trust most for chat feel. Not for company documents, but for the question: how does this feel when several people hold conversations throughout the day?

The pattern from Run H holds. NVFP4 is fastest for the average user. FP8 has the better P99.

NVFP4 feels instant for most users: 157 ms TTFT P50 and 26 ms TPOT P50. But the P99 is 1361 ms, where FP8 stays at 422 ms. That is a hefty difference.

For an internal chat where a single slower request is no disaster, I pick NVFP4. For a product UI with a hard latency promise I would take FP8 more seriously.

Run J is oversubscribe. The target is 1.5 requests per second with a concurrency cap of 25. This is not the normal workday. This is the test for what happens when demand is bigger than the server can neatly keep up with.

With oversubscribe I look at achieved RPS first. Not at configured RPS, because that is the same for everyone. The question is how many requests the server actually processes while it is under pressure.

There NVFP4 wins clearly. FP8 keeps the tail cleaner, but NVFP4 gets much more work through the machine.

Concretely: NVFP4 processes about 35 requests per minute. BF16 about 15. That is the difference between a queue that slowly drains and a queue that makes users wonder whether they should click again. Do not click. That second click never helps.