gemma4-research/docs/reference/gpu-bakeoff-2026-04-20.md

GPU Bakeoff — Gemma 4 Throughput: 3090 Ti vs Strix Halo

Date: 2026-04-20 Host matrix: steel141 (RTX 3090 Ti) · strix-halo (AMD Strix Halo iGPU) Models: gemma4:26b (MoE Q4_K_M) · gemma4:31b-it-q4_K_M (dense Q4_K_M) Harness: scripts/gpu-bakeoff/harness.py Raw data: scripts/gpu-bakeoff/runs/

---

TL;DR

GPU	26B (MoE) decode	31B (dense) decode	Long-prompt prefill (26B)
RTX 3090 Ti (steel141)	128 tok/s	27 tok/s	23,849 tok/s
AMD Strix Halo iGPU (strix-halo)	54 tok/s (42%)	11 tok/s (39%)	14,326 tok/s (60%)

Headline findings

1. MoE changes everything. gemma4:26b decodes ~4.7× faster than gemma4:31b on every GPU tested, because only ~4 B of its 25.8 B parameters activate per token. Total parameter counts (26 B vs 31 B) don't predict latency; active parameters do.
2. 3090 Ti wins decisively on decode. For inference workloads the memory-bandwidth-flop ratio of consumer Ampere GDDR6X is hard to beat at this price point.
3. Strix Halo punches above its bandwidth. Gets 42 % of 3090 Ti decode speed on only ~25 % of the memory bandwidth (~256 GB/s vs ~1008 GB/s) — good SIMD utilization, especially on the MoE model.

---

Hardware inventory

Host	GPU	VRAM	Bandwidth	Compute cap	Notes
steel141	RTX 3090 Ti	24 GB GDDR6X	~1008 GB/s	8.6 (Ampere)	Workstation. Also has a GTX 1660 SUPER as aux display card — not used for inference. Ollama on localhost.
strix-halo	AMD Strix Halo (Radeon 890M iGPU + XDNA 2 NPU)	Shared LPDDR5X	~256 GB/s	—	Unified memory lets it fit models a 24 GB card can't. Ollama accessed via Tailscale.

---

Methodology

• Each (host × model × prompt-length) cell:
  • 1 warm-up call (discarded, absorbs model load time and JIT warm-up)
  • 3 measurement calls
  • temperature: 0.0, top_k: 1 (greedy), num_predict: 256, num_ctx: 4096
  • keep_alive: 10m so the model stays resident between runs
• Two prompt lengths:
  • short (~15 tokens) — isolates decode performance, prefill time is negligible
  • long (~500 tokens) — stresses prefill (prompt evaluation)
• All timings come from Ollama's own /api/generate response fields (prompt_eval_duration, eval_duration, etc.), so HTTP and wall-clock jitter are excluded from the rates.
• Median of the 3 measurement runs is reported in tables; min/max are in the raw JSON.

---

Full results

Decode rate (tok/s, median of 3 runs)

Decode is the metric that matters most for interactive LLM use — it's the speed of token generation after the prompt has been processed.

Model	3090 Ti	Strix Halo
gemma4:26b (MoE, ~4 B active)	128.20	53.86
gemma4:31b (dense, 31.3 B active)	27.15	10.64

Prefill rate (tok/s, long ~500-token prompt, median)

Prefill is the cost of ingesting the prompt and populating the KV cache before decode begins. Batched per-token, so short-prompt prefill numbers are noisy (dominated by fixed overhead — see raw JSON for those); the long-prompt numbers below are the ones to reason from.

Model	3090 Ti	Strix Halo
gemma4:26b (long)	23,849	14,326
gemma4:31b (long)	7,716	3,278

Short-prompt prefill (for reference)

On a 15-token prompt, prefill tokens/sec is meaningless — prompt is too small to amortize overhead. Included only to confirm no regression.

Model	3090 Ti	Strix Halo
gemma4:26b (short)	2,063	1,276
gemma4:31b (short)	661	292

---

Why 26B decodes 4.7× faster than 31B

gemma4:26b is the MoE variant ("A4B" in Google's naming = activated 4B). Per-token inference routes through only ~4 B of its 25.8 B total parameters. gemma4:31b is dense: every one of its 31.3 B parameters participates in every token's forward pass. Memory bandwidth is the binding constraint for decode, so the ratio of active params is what you actually pay for.

Rough math (3090 Ti, 1008 GB/s, Q4_K_M ≈ 0.5 bytes/param):

• 26B MoE: 4 B × 0.5 B = 2 GB per token. Theoretical max ≈ 504 tok/s. Observed 128 tok/s = 25 % utilization.
• 31B dense: 31.3 B × 0.5 B = 15.65 GB per token. Theoretical max ≈ 64 tok/s. Observed 27 tok/s = 42 % utilization.

So dense workloads actually extract higher bandwidth utilization — they're less overhead-dominated per token. But in absolute terms, MoE wins by a large factor because the active-parameter bill is much smaller. For interactive chat this is decisive: Seth's mort-bot running gemma4:26b gets ~4.7× the responsiveness it would on gemma4:31b, even though the models are near-equal in total params.

Why the ratio holds on both GPUs: memory bandwidth is the bottleneck on both cards. Strix gets 42 % of 3090 Ti on 26B and 39 % of 3090 Ti on 31B — nearly identical ratios — because it has ~25 % of the bandwidth and matches or slightly exceeds proportionally.

---

When to use which GPU

Interactive chat / agent workloads (decode-heavy).

• Primary: 3090 Ti — by a wide margin. 128 tok/s on 26B is comfortable for real-time responses.
• Fallback: Strix Halo — 54 tok/s is usable. Benefit is unified memory can host larger models the 24 GB 3090 Ti can't.

Long-context / prompt-heavy workloads (prefill-heavy).

• Primary: 3090 Ti again — 23,849 tok/s prefill means a 500-token prompt ingests in ~21 ms.
• Strix at 14,326 tok/s is ~35 ms — still interactive.

Running models that don't fit on discrete cards.

• Strix Halo. Unified LPDDR5X can hold 80 GB+ models that a 24 GB 3090 Ti can't — at the cost of lower bandwidth.
• The largest model tested here (gemma4:31b Q4 at 19.9 GB) fits both. Q8 variants (28 GB+) only fit Strix in this matrix.

Fine-tuning / training.

• Not measured here. 3090 Ti's 24 GB limits batch size on 20 B+ models.

---

Open questions / follow-ups

1. Strix max-model fit. Strix can host models that wouldn't fit the 3090 Ti. A follow-up would pull a larger model (70 B+ quantized) on strix-halo and measure the Strix-only performance ceiling.
2. Q8 vs Q4 on Strix. Same model, two quantizations — quality/speed tradeoff characterization.

---

Raw data

All per-run JSON traces are under scripts/gpu-bakeoff/runs/:

runs/
├── steel141/
│   ├── gemma4-26b/{short,long}.json
│   └── gemma4-31b/{short,long}.json
└── strix-halo/
    ├── gemma4-26b/{short,long}.json
    └── gemma4-31b/{short,long}.json

Each JSON contains the warmup call and all 3 measurement calls with every field Ollama's /api/generate returns (token counts, durations, loaded-at, context length), plus a summary with min/median/max for prefill and decode rates.