Mortdecai
5011059f5d
Architecture specs, benchmarks, gotchas, Ollama settings, tool calling format, and implementation patterns from Simon and AI_Visualizer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
56 lines
2.1 KiB
Markdown
56 lines
2.1 KiB
Markdown
# Gemma 4 Capabilities Reference
|
|
|
|
## Modalities
|
|
|
|
### Text (all variants)
|
|
- Standard instruction-following, chat, completion
|
|
- System prompt support (critical — see synthesis)
|
|
- 128K context window (training length)
|
|
- 262K vocabulary
|
|
|
|
### Vision (all variants)
|
|
- **Tested and verified working** (Seth, 2026-04-10)
|
|
- Accurately described colors, shapes, composition in 256x256 test image
|
|
- ~25 tok/s, ~24s end-to-end on pve197 V100
|
|
- Input: base64-encoded image in `images` field of Ollama API
|
|
- Vision encoder: 16x16 patches, 2D RoPE, variable aspect ratio
|
|
- Token budgets scale with resolution (70-1120 soft tokens)
|
|
- Used in AI_Visualizer for SDXL frame quality criticism
|
|
|
|
### Audio (E2B/E4B only)
|
|
- **Not tested by Seth** — status unknown in practice
|
|
- Conformer architecture (~300M params), mel-spectrogram input
|
|
- **Trained on SPEECH ONLY — not music or environmental sounds**
|
|
- Maximum 30 seconds per clip
|
|
- NOT available on 26B or 31B variants
|
|
- AI_Visualizer explicitly rejected audio for music analysis (DECISIONS S2) — correct call, model wasn't trained for it
|
|
|
|
### Video (all variants)
|
|
- E2B/E4B: video WITH audio (`load_audio_from_video=True`)
|
|
- 31B/26B: video WITHOUT audio
|
|
- Not explicitly post-trained on video but works
|
|
- Maximum 60 seconds at 1 frame/second
|
|
- Not tested by Seth
|
|
|
|
### Tool Calling / Function Calling
|
|
- **Verified reliable** in both Simon and AI_Visualizer
|
|
- Ollama native tool format (OpenAI-compatible function calling)
|
|
- Simon: 6 genealogy tools, up to 12 sequential iterations
|
|
- Supports parallel tool calls in single response
|
|
- Weak at deeply nested JSON schemas -> prefer sequential calls
|
|
|
|
## Benchmark Context (vs Gemma 3)
|
|
|
|
- 31B replaces Gemma 3 27B (60 layers vs 62, but wider)
|
|
- MoE variant (26B) is new — no Gemma 3 equivalent
|
|
- E-series with PLE is new — on-device focus
|
|
- Proportional RoPE replaces linear frequency scaling -> better long-context
|
|
- Shared KV cache is new -> more efficient inference
|
|
|
|
## What Gemma 4 Does NOT Do
|
|
|
|
- No native code execution / sandboxing
|
|
- No web browsing or retrieval
|
|
- Audio only on E-series (not the models most people run)
|
|
- No built-in RAG — tool calling can implement it
|