{"name":"Gemma-4-26B-A4B-it","aaUrl":"https://artificialanalysis.ai/models/gemma-4-26b-a4b","arena":{"id":"gemma-4-nvfp4","name":"Gemma-4 26B-A4B","vram":24,"params":26,"vendor":"google","quality":{"coding":{"bench":"LiveCodeBench v6","value":79.8},"science":{"bench":"GPQA-Diamond","value":79.9},"knowledge":{"bench":"MMLU-Pro","value":84.8}},"contextK":256,"baseModelId":"gemma-4-26b-a4b-it"},"notes":{"en":"NVIDIA NVFP4 re-quant on vLLM v0.23.0. KV-cache fp8, prefix caching off, gpu-memory-utilization 0.85. Full suite run on 2026-06-23.","fr":"Re-quant NVIDIA NVFP4 sur vLLM v0.23.0. KV-cache fp8, prefix caching désactivé, gpu-memory-utilization 0.85. Suite complète exécutée le 2026-06-23.","nl":"Model-ID: nvidia/Gemma-4-26B-A4B-NVFP4. Container: vllm/vllm-openai:v0.26.0-aarch64-cu129-ubuntu2404. llama-benchy 0.4.0. Bronrun: 2026-08-06T11:38:17+02:00. KV-cache fp8; max_model_len 131072; gpu_memory_utilization 0,85. Prefix caching uit; async scheduling aan. Extra serverflags: --quantization modelopt --moe-backend marlin --max-num-batched-tokens 8192. 11 van de 11 tests hebben een meting die volgens de huidige Arena-regels meetelt. Bij deze historische run is geen sanity-check vastgelegd. Dat bewijs ontbreekt, ook wanneer de snelheidsmeting volgens de huidige regels meetelt."},"order":17,"engine":"vLLM v0.26.0","vramGb":24,"verdict":{"en":"Best practical v23 profile for this Spark. Chat reaches 21.59 t/s/user, multi-turn 20.01 t/s/user and ShareGPT replay stays at p95 TTFT 225.09 ms. 25k context is still prefill pain, but normal chat and agents feel fast locally.","fr":"Meilleur profil v23 pratique pour ce Spark. Le chat atteint 21.59 t/s/user, le multi-turn 20.01 t/s/user et le replay ShareGPT reste à p95 TTFT 225.09 ms. Le contexte 25k reste douloureux en prefill, mais le chat normal et les agents semblent rapides en local.","nl":"Gemeten op 2026-08-06 met vLLM v0.26.0. In de chattest met tien gelijktijdige verzoeken haalt dit profiel 21,1 tokens/s per gebruiker, met gemiddeld 1,28 seconden tot het eerste token. Bij 25k context stijgt die gemiddelde wachttijd naar 38,8 seconden. Bij deze historische run is geen sanity-check vastgelegd. Dat bewijs ontbreekt, ook wanneer de snelheidsmeting volgens de huidige regels meetelt. De URL bevat nog v23, maar de huidige bronrun is uitgevoerd met vLLM v0.26.0."},"findings":{"broke":{"body":{"en":"Even NVFP4 sits at 38.58s average TTFT for 25k and c=10. Serving profile helps decode, not the wait before large prompts.","fr":"Même NVFP4 est à 38.58s de TTFT moyen en 25k et c=10. Le profil de serving aide le decode, pas l’attente avant les grands prompts.","nl":"Bij 25000 inputtokens en 256 outputtokens met tien gelijktijdige verzoeken is de gemiddelde wachttijd voor het eerste token 38,8 seconden. Daarna genereert het model gemiddeld 7,24 tokens/s per gebruiker. De meting alleen toont niet of geheugen, promptverwerking of scheduling de oorzaak van vertraging is."},"title":{"en":"25k context is still prefill pain","fr":"Le contexte 25k reste douloureux en prefill","nl":"Wachttijd bij 25k context"}},"costly":{"body":{"en":"MTP beats BF16 on decode, but under Monday peak load its p95 TTFT and p95 TPOT are worse than BF16. Percentiles still matter.","fr":"MTP bat BF16 en decode, mais sous le pic du lundi son p95 TTFT et son p95 TPOT sont pires que BF16. Les percentiles restent nécessaires.","nl":"De run verwerkt 0,43 verzoeken/s bij een ingestelde aankomstsnelheid van 1,5 verzoeken/s. 300 van de 300 verzoeken slagen. De p95-wachttijd voor het eerste token is 2,5 seconden."},"title":{"en":"MTP buys decode, not perfect tail","fr":"MTP achète du decode, pas une tail parfaite","nl":"Verzoeken en wachttijd bij piekbelasting"}},"worked":{"body":{"en":"Chat at 21.59 t/s/user and multi-turn at 20.01 t/s/user at c=10. For local office chat this does not feel like a compromise.","fr":"Chat à 21.59 t/s/user et multi-turn à 20.01 t/s/user à c=10. Pour du chat local de bureau, cela ne ressemble pas à un compromis.","nl":"Bij 1024 input- en 1024 outputtokens met tien gelijktijdige verzoeken meet ik gemiddeld 21,1 tokens/s per gebruiker. De gemiddelde wachttijd voor het eerste token is 1,28 seconden."},"title":{"en":"NVFP4 is the practical default","fr":"NVFP4 est le choix pratique","nl":"Chat bij tien gelijktijdige verzoeken"}},"surprised":{"body":{"en":"NVFP4 completes 250/250 requests with p95 TTFT 225.09 ms and p95 TPOT 45.30 ms. Real short conversations are much lighter than random 4k.","fr":"NVFP4 termine 250/250 requêtes avec p95 TTFT 225.09 ms et p95 TPOT 45.30 ms. Les vraies conversations courtes sont bien plus légères que random 4k.","nl":"11 van de 11 tests hebben een meting die volgens de huidige Arena-regels meetelt. Bij deze historische run is geen sanity-check vastgelegd. Dat bewijs ontbreekt, ook wanneer de snelheidsmeting volgens de huidige regels meetelt."},"title":{"en":"ShareGPT replay is extremely friendly","fr":"Le replay ShareGPT est très favorable","nl":"Welke metingen meetellen"}}},"hardware":"DGX Spark, NVIDIA GB10, 128 GB unified memory","modelUrl":"https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4","provider":"NVIDIA (re-quant van Google)","precision":"NVFP4","parameters":"26B-A4B","provenance":{"modelId":"nvidia/Gemma-4-26B-A4B-NVFP4","runPath":"results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23","validity":{"tests":{"01-chat":"complete","02-rag-8k":"complete","08-sharegpt":"complete","09-reasoning":"complete","04-multi-turn":"complete","11-rate-sweep":"complete","03-long-output":"complete","05-big-context":"complete","10-monday-peak":"complete","07-office-baseline":"complete","06-concurrency-stress":"complete"},"sanity":"not_recorded"},"sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","servedName":"gemma-4-26b-a4b-nvfp4","generatedAt":"2026-08-06T11:38:17+02:00","runRevision":"a948cec23b054c62fef595bad3cdb284066922ac","vllmVersion":"0.26.0","suiteVersion":"2026-08","repositoryUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks","contractVersion":1,"contractRevision":"a948cec23b054c62fef595bad3cdb284066922ac","commandProvenance":"reconstructed_from_raw_artifacts","llamaBenchyVersion":"0.4.0"},"architecture":"MoE","measuredContextK":128,"benchmarkResults":[{"id":"gemma-4-26b-a4b-it-nvfp4-v23-chat-1k","date":"2026-08-06","runs":3,"metrics":{"ttfrSec":{"mean":1.278,"stddev":0.396},"decodeTokensPerSecondTotal":{"mean":140.63,"stddev":21.1},"prefillTokensPerSecondTotal":{"mean":6273.65,"stddev":39.43},"decodeTokensPerSecondPerUser":{"mean":21.1,"stddev":1.12}},"sourceRun":"Test 01","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","workloadId":"chat-1k","sourceLabel":"nvidia/Gemma-4-26B-A4B-NVFP4 nvfp4-v23 op de DGX Spark","sourceStatus":"complete","sourceTestId":"01-chat","prefixCaching":false},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-rag-8k","date":"2026-08-06","runs":3,"metrics":{"ttfrSec":{"mean":8.173,"stddev":4.231},"decodeTokensPerSecondTotal":{"mean":123.09,"stddev":3.85},"prefillTokensPerSecondTotal":{"mean":5280.96,"stddev":8.23},"decodeTokensPerSecondPerUser":{"mean":16.06,"stddev":2.1}},"sourceRun":"Test 02","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","workloadId":"rag-8k","sourceLabel":"nvidia/Gemma-4-26B-A4B-NVFP4 nvfp4-v23 op de DGX Spark","sourceStatus":"complete","sourceTestId":"02-rag-8k","prefixCaching":false},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-long-output-agents","date":"2026-08-06","runs":3,"metrics":{"ttfrSec":{"mean":0.382,"stddev":0.005},"decodeTokensPerSecondTotal":{"mean":161.48,"stddev":0.99},"prefillTokensPerSecondTotal":{"mean":6088.59,"stddev":36.66},"decodeTokensPerSecondPerUser":{"mean":22.94,"stddev":0.76}},"sourceRun":"Test 03","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","workloadId":"long-output-agents","sourceLabel":"nvidia/Gemma-4-26B-A4B-NVFP4 nvfp4-v23 op de DGX Spark","sourceStatus":"complete","sourceTestId":"03-long-output","prefixCaching":false},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-multi-turn-office","date":"2026-08-06","runs":3,"metrics":{"ttfrSec":{"mean":2.017,"stddev":0.833},"decodeTokensPerSecondTotal":{"mean":172.32,"stddev":7.46},"prefillTokensPerSecondTotal":{"mean":6402.91,"stddev":84.95},"decodeTokensPerSecondPerUser":{"mean":19.5,"stddev":0.84}},"sourceRun":"Test 04","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","workloadId":"multi-turn-office","sourceLabel":"nvidia/Gemma-4-26B-A4B-NVFP4 nvfp4-v23 op de DGX Spark","sourceStatus":"complete","sourceTestId":"04-multi-turn","prefixCaching":false},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-large-context-25k","date":"2026-08-06","runs":3,"metrics":{"ttfrSec":{"mean":38.796,"stddev":19.794},"decodeTokensPerSecondTotal":{"mean":32.67,"stddev":0.71},"prefillTokensPerSecondTotal":{"mean":3259.42,"stddev":4.09},"decodeTokensPerSecondPerUser":{"mean":7.24,"stddev":4.17}},"sourceRun":"Test 05","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","workloadId":"large-context-25k","sourceLabel":"nvidia/Gemma-4-26B-A4B-NVFP4 nvfp4-v23 op de DGX Spark","sourceStatus":"complete","sourceTestId":"05-big-context","prefixCaching":false},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-concurrency-stress-25k","date":"2026-08-06","runs":3,"metrics":{"ttfrSec":{"mean":73.575,"stddev":39.897},"decodeTokensPerSecondTotal":{"mean":33.16,"stddev":0.03},"prefillTokensPerSecondTotal":{"mean":3251.76,"stddev":5.2},"decodeTokensPerSecondPerUser":{"mean":3.99,"stddev":2.6}},"sourceRun":"Test 06","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","workloadId":"concurrency-stress-25k","sourceLabel":"nvidia/Gemma-4-26B-A4B-NVFP4 nvfp4-v23 op de DGX Spark","sourceStatus":"complete","sourceTestId":"06-concurrency-stress","prefixCaching":false}],"openLoopResults":[{"id":"gemma-4-26b-a4b-it-nvfp4-v23-office-random-4k","date":"2026-08-06","metrics":{"e2eMs":{"p50":30837.53,"p90":55307.6,"p95":62486.42,"p99":65774.24,"mean":32126.06},"tpotMs":{"p50":64.69,"p90":76.16,"p95":80.42,"p99":89.65,"mean":64.13},"ttftMs":{"p50":1056.91,"p90":2044.43,"p95":2437.1,"p99":3072.59,"mean":1161.74}},"sourceRun":"Test 07","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","scenarioId":"office-random-4k","achievedRps":0.286,"sourceStatus":"complete","sourceTestId":"07-office-baseline","configuredRps":0.3,"prefixCaching":false,"totalRequests":200,"successfulRequests":200,"totalTokenThroughput":1304.52,"peakConcurrentRequests":16},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-sharegpt-replay","date":"2026-08-06","metrics":{"e2eMs":{"p50":5112.3,"p90":19542.36,"p95":23492.28,"p99":33090.76,"mean":8017.45},"tpotMs":{"p50":38.11,"p90":44.77,"p95":46.36,"p99":48.75,"mean":38.39},"ttftMs":{"p50":153.51,"p90":214.91,"p95":231.19,"p99":257.95,"mean":157.26}},"sourceRun":"Test 08","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","scenarioId":"sharegpt-replay","achievedRps":0.299,"sourceStatus":"complete","sourceTestId":"08-sharegpt","configuredRps":0.3,"prefixCaching":false,"totalRequests":250,"successfulRequests":250,"totalTokenThroughput":133.79,"peakConcurrentRequests":10},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-reasoning","date":"2026-08-06","metrics":{"e2eMs":{"p50":310371.19,"p90":461439.5,"p95":506584.13,"p99":525691.5,"mean":307127.18},"tpotMs":{"p50":75.02,"p90":83.12,"p95":83.47,"p99":87.85,"mean":74.24},"ttftMs":{"p50":338.5,"p90":467.16,"p95":479,"p99":581.94,"mean":346.1}},"sourceRun":"Test 09","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","scenarioId":"reasoning","achievedRps":0.074,"sourceStatus":"complete","sourceTestId":"09-reasoning","configuredRps":0.2,"prefixCaching":false,"totalRequests":50,"successfulRequests":50,"totalTokenThroughput":392.1,"peakConcurrentRequests":44},{"id":"gemma-4-26b-a4b-it-nvfp4-v23-monday-peak-random-4k","date":"2026-08-06","metrics":{"e2eMs":{"p50":55233.59,"p90":100785.92,"p95":104109.74,"p99":109981.41,"mean":56220.85},"tpotMs":{"p50":111.8,"p90":121.2,"p95":127.23,"p99":152.8,"mean":111.58},"ttftMs":{"p50":957.85,"p90":1870.43,"p95":2499.81,"p99":6285.99,"mean":1199.34}},"sourceRun":"Test 10","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","scenarioId":"monday-peak-random-4k","achievedRps":0.428,"sourceStatus":"complete","sourceTestId":"10-monday-peak","configuredRps":1.5,"prefixCaching":false,"totalRequests":300,"successfulRequests":300,"totalTokenThroughput":1929.26,"peakConcurrentRequests":28}],"rateSweep":{"date":"2026-08-06","capacity":{"slo2000":{"outputTps":47.7,"ttftP95Ms":1973.2,"achievedRps":0.097,"configuredRps":0.1,"peakConcurrent":8},"slo5000":{"outputTps":267.5,"ttftP95Ms":3817.6,"achievedRps":0.547,"configuredRps":0.7,"peakConcurrent":83},"slo10000":{"outputTps":329.9,"ttftP95Ms":6096.1,"achievedRps":0.671,"configuredRps":1,"peakConcurrent":190}},"sourceRun":"Test 11","sourceUrl":"https://github.com/djangodevreng/dgx-spark-benchmarks/tree/a948cec23b054c62fef595bad3cdb284066922ac/results/gemma-4/gemma-4-26b-a4b-it/nvfp4-v23/","abortedAtRps":null,"queueKneeRps":0.7,"sourceStatus":"complete","sourceTestId":"11-rate-sweep","totalRequests":850,"successfulRequests":850}}