Primarily, I got a dual GPU setup to fit bigger models, and after that was working well enough, I got an itch to see if I could improve utilization. I had that working for a while, and it made me wonder if both GPUs could be utilized more instead of handing the baton to each one after each layer finishes.

As of now, each GPU is not even utilized at 50% most of the time:

And this is the performance (the baseline for this experiment):

llama-bench -hf unsloth/gemma-4-31B-it-qat-GGUF:UD-Q4_K_XL --split-mode layer -n 512 -d 8192,32768,65536,131072

But after switching to --split-mode tensor, I got almost 100% GPU utilization, and almost double inference speeds as well:

llama-bench -hf unsloth/gemma-4-31B-it-qat-GGUF:UD-Q4_K_XL --split-mode tensor -n 512 -d 8192,32768,65536,131072

Which is around a 1.8x improvement over the layer baseline across all context windows I was intending to use.

When compiling llamacpp, I got a warning that: NCCL not found, performance for multiple CUDA GPUs will be suboptimal. However, from my perspective in my specific setup, this performs much better than using the dedicated multi GPU library.

From now on, I will be using --split-mode tensor as the default.

References

• Hugging Face gemma-4-31B-it-GGUF
• Ollama gemma4
• llamacpp GitHub
• llamacpp server readme
• llamacpp split modes
• Gemma 4 QAT release
• QAT quantization aware training
• Quantization aware training for LLMs
• CUDA Compute Capabilities table
• CUDA SDK toolkit download page
• UD - Unsloth Dynamic
• NVIDIA Collective Communication Library (NCCL)
• Perplexity

Related articles

• Building Llama.cpp from source
• Ollama vs Llama.cpp
• Switching from Gemma 4 to Gemma 4 QAT
• Compressing KV cache