Running Qwen3-8B on DGX Spark

My DGX Spark recently arrived! That’s some 40k HKD I hope well spent :)

I wanted to play with deploying a small LLM locally. I picked Qwen3-8B and tried BF16, FP8, NVFP4, SGLang, vLLM, and DSpark. This post is just a record of the numbers.

Summary

All servers use a 32,768-token context limit and CUDA Graph. SGLang uses FlashInfer; vLLM uses its V2 runner, compiled kernels, and full plus piecewise CUDA Graph. Memory figures below are model-weight footprints reported by the server or checkpoint sizes, not total system memory usage.

DSpark uses exact target verification, so its quality is that of the FP8 or NVFP4 target rather than a separate model.

The 32K context is a deliberately identical server setting, not a datatype limit. Quantization mainly increases total KV capacity and therefore the number of concurrent long requests. It does not automatically change Qwen3-8B’s configured per-request context window.

The earlier draft also mixed checkpoint size with runtime allocation. With the same checkpoint, vLLM and SGLang load slightly different amounts because their quantized-weight packing, scale metadata, allocator, and kernel workspaces are different. The table now reports both quantities separately.

Prefill Throughput

These are aggregate input token/s with one generated token. Each value is BF16 / FP8 / NVFP4. Every batch has an SGLang row and a vLLM row.

FP8 prefill is not especially good in this stack: SGLang reports no GB10-specific W8A8 tuning configuration and uses a default kernel. NVFP4 completes FlashInfer SM121 autotuning and is much faster.

Decode Throughput

These are steady aggregate output token/s for 128 generated tokens per request, calculated from TPOT. Each value is BF16 / FP8 / NVFP4.

At batch one, FP8 is almost exactly 2x BF16 and NVFP4 is about 2.8x. The advantage shrinks with long contexts and large batches as KV-cache traffic and attention account for more work.

DSpark

This test uses vLLM, 20 natural-language prompts, greedy decoding, 256 generated tokens, and concurrency one.

DSpark is useful for low-concurrency decode, but not for a saturated server. At batch 64 and context 128, plain FP8 reaches 1,098 tok/s while DSpark reaches 688; plain NVFP4 reaches 1,749 tok/s while DSpark reaches 623.

Compute and Memory Roofs

I measured both an 8192-cubed GEMM and a Qwen-like M=8192, K=4096, N=12288 projection. The best median roof for each datatype is 100.5 TFLOP/s BF16, 198.9 TFLOP/s FP8, and 234.5 TFLOP/s NVFP4. The local 2-GiB CUDA copy reaches 232.4 GB/s; NVIDIA advertises 273 GB/s LPDDR5X.

Effective model FLOP/s counts dense Qwen operations completed per second, not physical low-precision instructions. For prefill, starred bandwidth is only the checkpoint-once lower bound and deliberately excludes activation traffic; compute is the relevant comparison. For decode, effective weight bandwidth is checkpoint size times token rate. The FP8 estimate above 273 GB/s shows the approximation error: checkpoint bytes are not identical to DRAM bytes and caching is ignored. The relevant, bound-setting columns are bold.

Takeaways

• FP8 is the default I would use. It roughly doubles batch-one decode and did not show a statistically significant quality loss here.
• NVFP4 is faster, but the loss is visible. C-Eval drops 4.49 points and GSM8K drops 6.00 points relative to BF16.
• DSpark is for interactive generation. FP8 + DSpark reaches 52 tok/s without the NVFP4 quality penalty, but plain decoding wins at high batch.
• Batch-one decode is already at the memory roof. The simple estimates are around 89–101% of the advertised LPDDR bandwidth.
• SGLang FP8 prefill needs better GB10 tuning. It reaches 41.5% of the measured FP8 GEMM roof in the large prefill case, versus 62.0% for vLLM.