Running LLMs on the DGX Spark

The DGX Spark is a compact NVIDIA AI computer with a GB10 superchip. CPU and GPU share memory. You can run language models locally and place the machine next to your desk.

The interesting detail is in the FP4 compute. NVIDIA advertises the Spark with a petaFLOP of FP4, but on sm_121 it lacks the instruction path that NVFP4 needs. vLLM warns about this itself and falls back to Marlin, which dequantizes the 4-bit weights to BF16. So you store 4-bit, and that memory win is real, but you compute at a higher level. For pure compute that's a downside, for memory and bandwidth it isn't, and on this hardware that's where the win sits.

The rule of thumb: model size in billions of parameters, times the bytes per parameter, is what you spend in memory. BF16 costs 2 bytes per parameter, FP8 one, NVFP4 half. So a 30B model in BF16 is about 60 GB of weights, in NVFP4 just 15. On top of that comes the KV cache, and that grows with your context length.

What fits in 128 GB depends on model size and precision. After loading the weights, there must still be room for the KV cache. Precision therefore also affects the context you can use.

Prefill processes your prompt; decode generates the answer tokens. With short prompts, output speed is most noticeable. Long prompts can take substantial time to process before output starts. In my Spark tests, this waiting time rose sharply with large contexts.

In my overload test, requests stayed in the queue and the server did not crash. Waiting time did increase. Alongside throughput, check how long users wait for answers under heavy load.

The exact numbers per model and context length are in the benchmark suite, run on a single Spark with a fixed measurement protocol. The reasoning behind these three numbers is in a separate essay.

I use vLLM to serve models as an API. It supports the NVFP4 configuration I test and can process long prompts in chunks using chunked prefill.

My configuration enables chunked prefill. I tune max-num-batched-tokens and memory utilisation to the desired context length. These settings affect stability and throughput. The configuration is in the Gemma-4 build log.

A cloud API can be more practical if you are allowed to process data there and do not want to manage a model server yourself. A Spark is an option when you want to keep processing under your own control.

Hardware is only one part of the decision. The Local AI decision guide also compares quality, data boundaries, operations, cost and change.

Think of SMBs and organizations that work with personal data, internal documents or customer data that has to stay close under GDPR. Then the question isn't "what's the fastest model", but "which part is even allowed to leave". A Spark under your own control answers that a lot more easily than a contract with a cloud provider.

With your own hardware, you choose the model version and when to update it. Your machine limits capacity. You also handle maintenance and updates yourself.

All figures on this site come from one Spark, using a fixed protocol: the same prompts, the same seeds and three runs per measurement. Configuration, prompts and raw output are public on GitHub.

Getting different results on another Spark or vLLM version? Send your settings and measurements so we can investigate the differences.