feat: CLI coding agent bakeoff — 26b reproducibly silent-stops at write_file
Ran minimal agent loop (Ollama /api/chat + read_file/write_file/run_bash) on steel141 3090 Ti against 3 models on a broken-median-function task: - gemma4:31b-it-q4_K_M: PASS (8 iters, 1 write, 44s) — textbook trace - qwen3-coder:30b: PASS (15 iters, 1 write, 22s) — correct but chatty - gemma4:26b: FAIL (6 iters, 0 writes) — silently stops with eval=4 after reading source. Reproduced on second run. One-shot probe confirms 26b CAN produce the correct fix — failure is specifically at the write_file tool-call argument boundary. Updates GOTCHAS with a new HIGH-severity entry, SYNTHESIS model-selection table, CORPUS_cli_coding_agent.md empirical-follow-up pointer, and adds docs/reference/bakeoff-2026-04-18.md with the full writeup.
This commit is contained in:
Mortdecai
@@ -7,6 +7,9 @@ __pycache__/
|
||||
*.pyc
|
||||
.pytest_cache/
|
||||
|
||||
# Bakeoff work directories — recreated from task_seed/ each run; logs preserved separately
|
||||
scripts/bakeoff/runs/*/work/
|
||||
|
||||
# Editor / OS
|
||||
.DS_Store
|
||||
*.swp
|
||||
|
||||
@@ -6,6 +6,14 @@
|
||||
> `IMPLEMENTATIONS.md` chat-agent patterns (Simon) and pipeline patterns
|
||||
> (AI_Visualizer).
|
||||
|
||||
> **Empirical follow-up:** `docs/reference/bakeoff-2026-04-18.md` — real runs of
|
||||
> `gemma4:26b`, `gemma4:31b-it-q4_K_M`, and `qwen3-coder:30b` against a custom
|
||||
> minimal CLI-agent harness on a fix-the-median-bug task. Key findings:
|
||||
> **31B clean (8 iters, 1 write), Qwen3-Coder correct but chatty (15 iters),
|
||||
> 26B reproducibly silent-stops at the `write_file` tool call boundary** even
|
||||
> though it can produce the fix in a direct one-shot call. Read when: scoping
|
||||
> which model to point an agent at, or hitting an unexpected tool-call halt.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- Gemma 4 is Google's **first Gemma with trained (not proof-of-concept) tool use**. LiveCodeBench v6 = 80.0% (31B) / 77.1% (26B). Codeforces ELO = 2150 / 1718. That's frontier-open territory on the reported benchmarks.
|
||||
|
||||
+31
@@ -64,6 +64,37 @@ Ollama defaults `num_predict` to 128 tokens. Almost any useful Gemma 4 output ex
|
||||
|
||||
**Fix:** Always set `num_predict` explicitly. Minimum recommended: 512. For JSON output: 2048+.
|
||||
|
||||
## HIGH: 26B Silent-Stops at `write_file` Tool Boundary (reproducible)
|
||||
|
||||
**Severity: HIGH — agent-loop failure, silent**
|
||||
|
||||
Reproduced on 2026-04-18 against `gemma4:26b` via Ollama 0.20.4 on a 3090 Ti
|
||||
(steel141). Agent harness exposed `read_file` / `write_file` / `run_bash` tools
|
||||
and asked the model to fix a failing Python test.
|
||||
|
||||
Observed pattern (both runs identical):
|
||||
1. Model reads README, runs pytest (sees failures), reads the buggy source file
|
||||
2. Next turn: **empty content, no tool calls, `eval_count=4`** — model silently exits
|
||||
3. Zero writes ever emitted
|
||||
|
||||
Isolation: a direct one-shot call asking 26B to rewrite the same function
|
||||
returned the correct fix (`eval_count=81`). So diagnosis and code generation are
|
||||
intact — failure is at the `write_file(path, content)` tool-call argument
|
||||
boundary, where `content` is a ~500-char string. Consistent with the "Weak at
|
||||
Long/Nested JSON" gotcha below: a long string inside a tool-call argument is
|
||||
structurally the same problem.
|
||||
|
||||
`gemma4:31b-it-q4_K_M` on the same harness completed the task cleanly
|
||||
(`eval_count=330` on the write turn). `qwen3-coder:30b` also completed.
|
||||
|
||||
**Fix:**
|
||||
- For 26B in an agent loop, prefer a **patch/diff tool surface**
|
||||
(`apply_patch(path, old, new)`) over a **full-content write** (`write_file(path, full_content)`).
|
||||
Delta-sized arguments are inside the model's comfort zone.
|
||||
- Or use 31B for the agent and keep 26B for single-shot tasks where the full
|
||||
response is the output, not a tool-call argument.
|
||||
- See `docs/reference/bakeoff-2026-04-18.md` for the full trace.
|
||||
|
||||
## MEDIUM: Weak at Long/Nested JSON
|
||||
|
||||
**Severity: MEDIUM — causes parse failures**
|
||||
|
||||
@@ -15,6 +15,7 @@ Research corpus and implementation guidance for Google Gemma 4, based on product
|
||||
| `CORPUS_benchmarks.md` | Full benchmark table vs Gemma 3, arena scores, agentic scores | When comparing Gemma 4 to alternatives |
|
||||
| `CORPUS_tool_calling_format.md` | Native token format + JSON API format for function calling | When implementing tool calling |
|
||||
| `CORPUS_cli_coding_agent.md` | Positioning Gemma 4 for CLI coding agent use (openclaw / open code / pi / hermes / aider style). Honest take on what Google did and didn't measure, head-to-head with `qwen3-coder:30b`, homelab setup pointer | When scoping a CLI coding agent or deciding Gemma 4 vs Qwen3-Coder |
|
||||
| `docs/reference/bakeoff-2026-04-18.md` | Raw results: CLI-coding-agent bakeoff of gemma4:26b / gemma4:31b / qwen3-coder:30b on steel141 3090 Ti. **31B clean, Qwen3-Coder correct but chatty, 26B reproducibly silent-stops at write_file.** Harness at `scripts/bakeoff/` | When deciding which model to back a CLI agent with, or debugging a similar tool-call halt |
|
||||
| `tooling/` | **Canonical upstream tooling** — real scripts, notebooks, model cards, and configs pulled from Google / HF / framework maintainers (147 files). Subdirs: `google-official/`, `huggingface/`, `inference-frameworks/`, `gemma-family/`, `fine-tuning/`. See `tooling/README.md` for index and findings that update the older `CORPUS_*` docs | When you need authoritative source material — model cards, chat templates, fine-tuning recipes, serving commands for vLLM / llama.cpp / MLX, or to scope a specialized sibling (ShieldGemma, EmbeddingGemma, etc.) |
|
||||
|
||||
## Source Projects
|
||||
|
||||
+1
-1
@@ -176,7 +176,7 @@ Vision is on ALL Gemma 4 variants (E2B, E4B, 26B, 31B). Audio is E-series only.
|
||||
| Maximum quality (single-model GPU) | `gemma4:31b-it-q4_K_M` | Dense 31B, sharpest but 5x slower, more VRAM pressure |
|
||||
| Rapid prototyping / testing | `gemma4:26b` | Fast enough for interactive dev |
|
||||
| Retrieval / embeddings | `embeddinggemma` (308M, separate model) | Gemma 4 has no embedding mode; use the sibling |
|
||||
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:26b` (or compare to `qwen3-coder:30b`) | Trained tool use + strong LiveCodeBench, but Google didn't publish SWE-bench — see `CORPUS_cli_coding_agent.md` for the honest positioning and the homelab bakeoff plan |
|
||||
| CLI coding agent (openclaw / open code / pi / hermes / aider) | `gemma4:31b-it-q4_K_M` (verified), fallback `qwen3-coder:30b` | 2026-04-18 bakeoff on 3090 Ti: 31B passes cleanly (8 iters, 1 write), Qwen3-Coder passes verbosely (15 iters), **26B reproducibly silent-stops at the write_file tool call** — see `CORPUS_cli_coding_agent.md` and `docs/reference/bakeoff-2026-04-18.md` |
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
# CLI Coding Agent Bakeoff — 2026-04-18
|
||||
|
||||
> Empirical follow-up to `CORPUS_cli_coding_agent.md`. Runs a minimal CLI coding
|
||||
> agent loop against three candidate models on identical hardware and an
|
||||
> identical broken-code task. **n=1 per model** (plus one re-run to check
|
||||
> reproducibility of a failure). Treat as a smoke test, not a benchmark.
|
||||
|
||||
## Setup
|
||||
|
||||
- **Host:** steel141 (Seth's local box)
|
||||
- **GPU:** NVIDIA RTX 3090 Ti, 24 GiB, ~22.7 GiB free
|
||||
- **Ollama:** 0.20.4
|
||||
- **Harness:** `scripts/bakeoff/harness.py` — custom minimal agent loop, **not** openclaw / open code / aider / pi / hermes. Protocol: Ollama `/api/chat` with `tools=[read_file, write_file, run_bash]`, non-streaming, `think: false`, `num_ctx: 32768`, `num_predict: 4096`, `temperature: 0.3`. Iteration cap = 15.
|
||||
- **Task:** `scripts/bakeoff/task_seed/` — Python package with buggy `median()` function. 3 of 7 pytest tests fail on even-length inputs. Fix is ~5 lines.
|
||||
- **System prompt:** generic CLI-agent template (identity + allowed tools + rules: "never modify tests", "prefer minimal edits"). Not tuned per model.
|
||||
|
||||
All three models pulled from steel141's local Ollama, swapped in/out of GPU as each run started. First iteration per run pays the load cost; later iterations are hot.
|
||||
|
||||
## Results
|
||||
|
||||
| Model | Pass | Iterations | write_file | read_file | run_bash | Wall clock | Halt reason |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `gemma4:26b` | **Fail** | 6 | **0** | 2 | 3 | 10.9s | `no_tool_calls` (silent empty response) |
|
||||
| `gemma4:26b` (retry) | **Fail** | 6 | **0** | 2 | 3 | 11.4s | `no_tool_calls` (reproduces exactly) |
|
||||
| `gemma4:31b-it-q4_K_M` | **Pass** | 8 | 1 | 2 | 4 | 44.1s | `no_tool_calls` (clean summary turn) |
|
||||
| `qwen3-coder:30b` | **Pass** | 15 (cap) | 1 | 4 | 8 | 22.6s | `no_tool_calls` (at iteration cap) |
|
||||
|
||||
### Gemma 4 31B — clean run
|
||||
|
||||
Textbook agent trace:
|
||||
|
||||
1. `read_file README.md`
|
||||
2. `pytest` (exit=2, module not found — pytest needs PYTHONPATH)
|
||||
3. `ls -R`
|
||||
4. `PYTHONPATH=. pytest` → sees 3 failures
|
||||
5. `read_file calc/stats.py`
|
||||
6. `write_file calc/stats.py` (eval_count=330, 13.4s) — correct fix
|
||||
7. `PYTHONPATH=. pytest` → all green
|
||||
8. summary: *"I updated the `median` function in `calc/stats.py` to correctly calculate the average of the two middle elements..."*
|
||||
|
||||
Zero wasted turns. One write. Minimal edit.
|
||||
|
||||
### Qwen3-Coder 30B — correct but chatty
|
||||
|
||||
Passed, but used all 15 iterations:
|
||||
|
||||
- Narrated every step ("I'll help you...", "Now let's look at...")
|
||||
- Tried to read a non-existent file (`test_calc.py`) — wasted iter 2
|
||||
- Tried to `read_file` on a directory (`calc`) — wasted iter 6
|
||||
- Ran several redundant bash calls (`pwd && pytest`, etc.)
|
||||
- Emitted a ceremonial `echo "All tests pass..."` bash call at iter 14
|
||||
- Final turn was a polite summary
|
||||
|
||||
The fix itself (iter 12) was correct on the first write. Quality is fine; efficiency isn't. Per-iteration it was fast (many 20-40 token turns) — total wall clock 22.6s beat Gemma 31B despite using nearly 2× the iterations.
|
||||
|
||||
### Gemma 4 26B — reproducible silent stop
|
||||
|
||||
Both runs followed an identical trajectory:
|
||||
|
||||
1. `ls -R`
|
||||
2. `read_file README.md`
|
||||
3. `pytest` (exit=2)
|
||||
4. `PYTHONPATH=. pytest` → sees 3 failures
|
||||
5. `read_file calc/stats.py`
|
||||
6. **Empty response. `eval_count=4`. No tool calls. Loop terminates.**
|
||||
|
||||
Zero writes. The model saw all the context it needed (failing tests + buggy source) and then silently declined to act.
|
||||
|
||||
### Isolating the failure — one-shot probe
|
||||
|
||||
To check whether 26B can produce the fix at all, I ran a single-turn call with no tool loop:
|
||||
|
||||
```
|
||||
prompt: "The following function is buggy — median([1,2,3,4]) returns 3
|
||||
but should return 2.5. Rewrite it correctly. [buggy code]"
|
||||
```
|
||||
|
||||
Response (eval_count=81):
|
||||
|
||||
```python
|
||||
def median(numbers):
|
||||
s = sorted(numbers)
|
||||
n = len(s)
|
||||
if n % 2 == 1:
|
||||
return s[n // 2]
|
||||
else:
|
||||
return (s[n // 2 - 1] + s[n // 2]) / 2
|
||||
```
|
||||
|
||||
**Correct.** So 26B's diagnosis and code generation are intact. The failure is specifically at the **tool-call-boundary** — when the model needs to emit a `write_file(path, content)` call where the `content` argument is a several-hundred-character string, it aborts with eval=4 instead.
|
||||
|
||||
This aligns with `GOTCHAS.md` § "Weak at Long/Nested JSON". A `write_file` tool call argument with a ~500-char string is structurally similar to a long JSON value. Gemma 4 31B handles the same surface reliably (eval=330 on that turn); the 26B MoE does not.
|
||||
|
||||
## Interpretation
|
||||
|
||||
### What this is evidence for
|
||||
|
||||
- **Gemma 4 31B is a viable CLI-coding-agent backing model on this class of task.** Clean trace, minimal wasted turns, correct fix on first write.
|
||||
- **Qwen3-Coder 30B also works**, at the cost of more iterations and looser discipline. Diff quality was fine; agentic efficiency wasn't.
|
||||
- **Gemma 4 26B has a reproducible failure mode** at tool-call-argument emission. It can reason. It can code. It struggles to deliver code through a `write_file` tool call when the content is non-trivial.
|
||||
|
||||
### What this is NOT evidence for
|
||||
|
||||
- **This is not a representative benchmark.** n=1 per model. One task. One fix. One harness. Do not conclude "Gemma 4 26B is broken for coding agents" — conclude "Gemma 4 26B failed this specific setup reproducibly; investigate further before relying on it."
|
||||
- **This harness is not openclaw / open code / aider / pi / hermes.** Production agents wrap prompts, retries, and tool surfaces differently. The 26B failure may be avoided in a harness that:
|
||||
- Uses a **patch/diff tool** (`apply_patch(old, new)`) instead of `write_file(full_content)` — smaller argument surface, matches the "sequential tool calls" pattern from `SYNTHESIS.md`
|
||||
- Adds a **retry on empty response** (same as Simon's streaming-fallback pattern in `IMPLEMENTATIONS.md`)
|
||||
- Provides fewer but richer tools (a dedicated `fix_file` that re-prompts internally)
|
||||
- **This compares agent behavior, not raw performance.** Wall clock is noisy (model load, context size, token rate all differ). Per-iteration latency is more meaningful — but that only matters for throughput, not correctness.
|
||||
|
||||
### Recommendations
|
||||
|
||||
1. **For a CLI coding agent on Seth's hardware:** start with `gemma4:31b-it-q4_K_M`. Clean behavior, modest wall clock (44s for a simple fix), no retry needed.
|
||||
2. **For comparison or backup:** `qwen3-coder:30b` is equally correct, roughly half the per-iteration cost, ~2× the iteration count. In a longer session those extra turns add up.
|
||||
3. **Do not default to `gemma4:26b` for this pattern.** Two tests in a row silent-stopped at the write boundary. If you want to use the 26B MoE (it's strong on `LiveCodeBench v6` at 77.1%), validate it against your specific agent framework first — especially whether the framework uses `write_file` (full content) or `apply_patch` (delta) as its edit primitive.
|
||||
4. **Test with the real harness you plan to use in production** (openclaw2, open code, etc.) before committing. A handful of this style of run takes minutes on the 3090 Ti and will tell you more than any benchmark card.
|
||||
|
||||
## Honest caveats
|
||||
|
||||
- **Stochasticity.** Only 26B was re-run. 31B and Qwen3-Coder might hit failure modes on a different seed or a different task. Temperature 0.3 is low but not zero.
|
||||
- **System prompt bias.** "Start by reading README.md" steered all three models similarly; a different prompt skeleton would produce different traces. I did not tune per model — deliberately — because a production agent won't either.
|
||||
- **The 26B silent-stop hypothesis (tool-arg emission failure) is inferred, not proven.** A clean confirmation would require running the same task with a smaller-surface edit tool (`apply_patch(path, old, new)` instead of `write_file(path, full_content)`) and showing 26B succeeds. That's the obvious follow-up.
|
||||
- **Ollama 0.20.4** is between the 0.20.0/0.20.1 known-broken-streaming range and whatever is current. Non-streaming tool calls worked cleanly for 31B and Qwen; 26B's failure looks model-specific, not Ollama-specific, but I didn't test on a different Ollama version.
|
||||
- **No openclaw / open code / aider runs.** Those are the frameworks named in the HF launch blog. This was a synthetic harness; transfer is plausible but unverified.
|
||||
|
||||
## Artifacts
|
||||
|
||||
- `scripts/bakeoff/harness.py` — the agent loop
|
||||
- `scripts/bakeoff/task_seed/` — the broken-code seed (reset between runs)
|
||||
- `scripts/bakeoff/runs/gemma4-26b/log.json` — full turn-by-turn trace
|
||||
- `scripts/bakeoff/runs/gemma4-26b-retry/log.json`
|
||||
- `scripts/bakeoff/runs/gemma4-31b/log.json`
|
||||
- `scripts/bakeoff/runs/qwen3-coder-30b/log.json`
|
||||
|
||||
Each log records per-turn: content, tool calls, results (truncated to 800 chars), prompt/eval token counts, wall time. Final block records halt reason, pass/fail, iteration count, tool-call totals, total wall clock.
|
||||
|
||||
## Reproducing
|
||||
|
||||
```bash
|
||||
cd scripts/bakeoff
|
||||
python3 harness.py gemma4:31b-it-q4_K_M runs/gemma4-31b/work runs/gemma4-31b/log.json
|
||||
python3 harness.py qwen3-coder:30b runs/qwen3-coder-30b/work runs/qwen3-coder-30b/log.json
|
||||
python3 harness.py gemma4:26b runs/gemma4-26b/work runs/gemma4-26b/log.json
|
||||
```
|
||||
|
||||
Each invocation resets the work directory from `task_seed/`, runs the loop, writes the log, and prints a one-line summary.
|
||||
@@ -0,0 +1,301 @@
|
||||
"""CLI coding agent bakeoff harness.
|
||||
|
||||
Minimal agent loop: Ollama /api/chat with tools [read_file, write_file, run_bash].
|
||||
Non-streaming, think=false. Workdir sandboxed; path traversal blocked; bash
|
||||
timeout 30s; iteration cap 15.
|
||||
|
||||
Invocation:
|
||||
python3 harness.py <model-tag> <workdir> <log-path>
|
||||
|
||||
Example:
|
||||
python3 harness.py gemma4:26b runs/gemma4-26b/work runs/gemma4-26b/log.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from urllib import request as urlreq
|
||||
|
||||
OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://127.0.0.1:11434")
|
||||
MAX_ITERATIONS = 15
|
||||
BASH_TIMEOUT_S = 30
|
||||
REQUEST_TIMEOUT_S = 540 # 9 min — keep inside the 10-min Bash ceiling
|
||||
|
||||
SYSTEM_PROMPT = """You are a terminal coding agent.
|
||||
|
||||
## What you do
|
||||
- Read source and test files to understand the code
|
||||
- Make targeted edits to fix bugs so the tests pass
|
||||
- Run pytest to verify your fix
|
||||
- Stop once all tests pass and reply with a one-sentence summary
|
||||
|
||||
## What you do NOT do
|
||||
- Never modify files under tests/
|
||||
- Never disable, skip, or delete tests
|
||||
- Never write outside the working directory
|
||||
- Never call tools after all tests pass — just reply with the summary and stop
|
||||
|
||||
## Available tools
|
||||
- read_file(path): read a file relative to the working directory
|
||||
- write_file(path, content): overwrite a file relative to the working directory
|
||||
- run_bash(command): run a shell command in the working directory
|
||||
|
||||
## Rules
|
||||
- Start by reading README.md
|
||||
- Prefer minimal edits. Do not refactor unrelated code.
|
||||
- Run the full test suite after each edit to verify.
|
||||
"""
|
||||
|
||||
USER_PROMPT = "Make the failing tests pass. Begin."
|
||||
|
||||
TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "read_file",
|
||||
"description": "Read a file. Path is relative to the working directory.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"path": {"type": "string"}},
|
||||
"required": ["path"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "write_file",
|
||||
"description": "Overwrite a file with new content. Path is relative to the working directory.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {"type": "string"},
|
||||
"content": {"type": "string"},
|
||||
},
|
||||
"required": ["path", "content"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run_bash",
|
||||
"description": "Run a shell command in the working directory. Returns stdout, stderr, and exit code.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"command": {"type": "string"}},
|
||||
"required": ["command"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def safe_path(workdir: Path, rel: str) -> Path:
|
||||
p = (workdir / rel).resolve()
|
||||
if not str(p).startswith(str(workdir.resolve())):
|
||||
raise ValueError(f"path escapes workdir: {rel}")
|
||||
return p
|
||||
|
||||
|
||||
def tool_read_file(workdir: Path, args: dict) -> str:
|
||||
p = safe_path(workdir, args["path"])
|
||||
if not p.exists():
|
||||
return f"ERROR: {args['path']} does not exist"
|
||||
return p.read_text()
|
||||
|
||||
|
||||
def tool_write_file(workdir: Path, args: dict) -> str:
|
||||
p = safe_path(workdir, args["path"])
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(args["content"])
|
||||
return f"wrote {args['path']} ({len(args['content'])} bytes)"
|
||||
|
||||
|
||||
def tool_run_bash(workdir: Path, args: dict) -> str:
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["bash", "-c", args["command"]],
|
||||
cwd=workdir,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=BASH_TIMEOUT_S,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return f"ERROR: command timed out after {BASH_TIMEOUT_S}s"
|
||||
head = (
|
||||
f"exit={r.returncode}\n"
|
||||
f"--- stdout ---\n{r.stdout[-4000:]}\n"
|
||||
f"--- stderr ---\n{r.stderr[-2000:]}"
|
||||
)
|
||||
return head
|
||||
|
||||
|
||||
TOOL_DISPATCH = {
|
||||
"read_file": tool_read_file,
|
||||
"write_file": tool_write_file,
|
||||
"run_bash": tool_run_bash,
|
||||
}
|
||||
|
||||
|
||||
def ollama_chat(model: str, messages: list) -> dict:
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"tools": TOOLS,
|
||||
"stream": False,
|
||||
"think": False,
|
||||
"keep_alive": "10m",
|
||||
"options": {
|
||||
"num_ctx": 32768,
|
||||
"num_predict": 4096,
|
||||
"temperature": 0.3,
|
||||
},
|
||||
}
|
||||
data = json.dumps(payload).encode()
|
||||
req = urlreq.Request(
|
||||
f"{OLLAMA_HOST}/api/chat",
|
||||
data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with urlreq.urlopen(req, timeout=REQUEST_TIMEOUT_S) as resp:
|
||||
return json.loads(resp.read())
|
||||
|
||||
|
||||
def pytest_passes(workdir: Path) -> bool:
|
||||
r = subprocess.run(
|
||||
["python3", "-m", "pytest", "tests/", "-q"],
|
||||
cwd=workdir,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
)
|
||||
return r.returncode == 0
|
||||
|
||||
|
||||
def run_bakeoff(model: str, workdir: Path, log_path: Path) -> dict:
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": SYSTEM_PROMPT},
|
||||
{"role": "user", "content": USER_PROMPT},
|
||||
]
|
||||
|
||||
trace = {
|
||||
"model": model,
|
||||
"workdir": str(workdir),
|
||||
"started_at": time.time(),
|
||||
"turns": [],
|
||||
"final": None,
|
||||
}
|
||||
|
||||
tool_call_counts = {"read_file": 0, "write_file": 0, "run_bash": 0}
|
||||
halt_reason = None
|
||||
|
||||
for iteration in range(1, MAX_ITERATIONS + 1):
|
||||
turn_start = time.time()
|
||||
try:
|
||||
response = ollama_chat(model, messages)
|
||||
except Exception as e:
|
||||
halt_reason = f"chat_error: {e}"
|
||||
trace["turns"].append(
|
||||
{"iteration": iteration, "error": str(e), "elapsed_s": time.time() - turn_start}
|
||||
)
|
||||
break
|
||||
|
||||
assistant_msg = response.get("message", {})
|
||||
content = assistant_msg.get("content", "") or ""
|
||||
tool_calls = assistant_msg.get("tool_calls", []) or []
|
||||
|
||||
turn = {
|
||||
"iteration": iteration,
|
||||
"elapsed_s": round(time.time() - turn_start, 2),
|
||||
"content": content,
|
||||
"tool_calls": [],
|
||||
"prompt_eval_count": response.get("prompt_eval_count"),
|
||||
"eval_count": response.get("eval_count"),
|
||||
}
|
||||
|
||||
messages.append({"role": "assistant", "content": content, "tool_calls": tool_calls})
|
||||
|
||||
if not tool_calls:
|
||||
trace["turns"].append(turn)
|
||||
halt_reason = "no_tool_calls"
|
||||
break
|
||||
|
||||
for tc in tool_calls:
|
||||
fn = tc.get("function", {})
|
||||
name = fn.get("name")
|
||||
args = fn.get("arguments") or {}
|
||||
if isinstance(args, str):
|
||||
try:
|
||||
args = json.loads(args)
|
||||
except Exception:
|
||||
args = {"_raw": args}
|
||||
|
||||
if name not in TOOL_DISPATCH:
|
||||
result = f"ERROR: unknown tool {name}"
|
||||
else:
|
||||
try:
|
||||
result = TOOL_DISPATCH[name](workdir, args)
|
||||
tool_call_counts[name] = tool_call_counts.get(name, 0) + 1
|
||||
except Exception as e:
|
||||
result = f"ERROR: {e}"
|
||||
|
||||
turn["tool_calls"].append({"name": name, "arguments": args, "result": result[:800]})
|
||||
messages.append({"role": "tool", "content": result})
|
||||
|
||||
trace["turns"].append(turn)
|
||||
|
||||
if iteration == MAX_ITERATIONS:
|
||||
halt_reason = "iteration_cap"
|
||||
break
|
||||
|
||||
final_pass = pytest_passes(workdir)
|
||||
trace["final"] = {
|
||||
"halt_reason": halt_reason,
|
||||
"tests_pass": final_pass,
|
||||
"iterations_used": len(trace["turns"]),
|
||||
"tool_call_counts": tool_call_counts,
|
||||
"wall_clock_s": round(time.time() - trace["started_at"], 2),
|
||||
}
|
||||
|
||||
log_path.write_text(json.dumps(trace, indent=2, default=str))
|
||||
return trace
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 4:
|
||||
print(__doc__, file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
model, workdir_s, log_s = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
workdir = Path(workdir_s).resolve()
|
||||
log_path = Path(log_s).resolve()
|
||||
|
||||
seed = Path(__file__).parent / "task_seed"
|
||||
if workdir.exists():
|
||||
shutil.rmtree(workdir)
|
||||
shutil.copytree(seed, workdir)
|
||||
|
||||
result = run_bakeoff(model, workdir, log_path)
|
||||
final = result["final"]
|
||||
print(
|
||||
f"model={model} pass={final['tests_pass']} "
|
||||
f"iters={final['iterations_used']} "
|
||||
f"read={final['tool_call_counts'].get('read_file', 0)} "
|
||||
f"write={final['tool_call_counts'].get('write_file', 0)} "
|
||||
f"bash={final['tool_call_counts'].get('run_bash', 0)} "
|
||||
f"halt={final['halt_reason']} "
|
||||
f"wall={final['wall_clock_s']}s"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,106 @@
|
||||
{
|
||||
"model": "gemma4:26b",
|
||||
"workdir": "/home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b-retry/work",
|
||||
"started_at": 1776533063.5394974,
|
||||
"turns": [
|
||||
{
|
||||
"iteration": 1,
|
||||
"elapsed_s": 6.72,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "ls -R"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\n.:\ncalc\nREADME.md\ntests\n\n./calc:\n__init__.py\n__pycache__\nstats.py\n\n./calc/__pycache__:\n__init__.cpython-313.pyc\nstats.cpython-313.pyc\n\n./tests:\n__pycache__\ntest_stats.py\n\n./tests/__pycache__:\ntest_stats.cpython-313-pytest-9.0.2.pyc\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 389,
|
||||
"eval_count": 21
|
||||
},
|
||||
{
|
||||
"iteration": 2,
|
||||
"elapsed_s": 0.64,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
},
|
||||
"result": "# Bakeoff Task\n\nA tiny Python package (`calc/`) with a statistics module. Run `pytest` from this\ndirectory \u2014 two tests currently fail because `median` returns the upper-middle\nelement instead of averaging the two middle elements on even-length inputs.\n\nYour job: make all tests pass. Do not disable or modify the tests.\n\nAllowed tools:\n- `read_file(path)` \u2014 read a file (relative to this directory)\n- `write_file(path, content)` \u2014 overwrite a file (relative to this directory)\n- `run_bash(command)` \u2014 run a shell command (cwd is this directory)\n\nWhen all tests pass, reply with a short summary of the fix and stop calling tools.\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 530,
|
||||
"eval_count": 21
|
||||
},
|
||||
{
|
||||
"iteration": 3,
|
||||
"elapsed_s": 0.66,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
},
|
||||
"result": "exit=2\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b-retry/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollected 0 items / 1 error\n\n==================================== ERRORS ====================================\n_____________________ ERROR collecting tests/test_stats.py _____________________\nImportError while importing test module '/home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b-retry/work/tests/test_stats.py'.\nHint: make sure your test modules/packages have valid Python na"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 711,
|
||||
"eval_count": 19
|
||||
},
|
||||
{
|
||||
"iteration": 4,
|
||||
"elapsed_s": 0.88,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "export PYTHONPATH=$PYTHONPATH:. && pytest"
|
||||
},
|
||||
"result": "exit=1\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b-retry/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollected 7 items\n\ntests/test_stats.py ..FFF.. [100%]\n\n=================================== FAILURES ===================================\n_______________________________ test_median_even _______________________________\n\n def test_median_even():\n> assert median([1, 2, 3, 4]) == 2.5\nE assert 3 == 2.5\nE + where 3 = median([1, 2,"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1147,
|
||||
"eval_count": 26
|
||||
},
|
||||
{
|
||||
"iteration": 5,
|
||||
"elapsed_s": 1.12,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "calc/stats.py"
|
||||
},
|
||||
"result": "\"\"\"Basic statistics helpers.\"\"\"\n\n\ndef mean(numbers):\n \"\"\"Arithmetic mean of a non-empty list.\"\"\"\n return sum(numbers) / len(numbers)\n\n\ndef median(numbers):\n \"\"\"Return the median of a list of numbers.\"\"\"\n s = sorted(numbers)\n n = len(s)\n return s[n // 2]\n\n\ndef mode(numbers):\n \"\"\"Return the most common value. Ties broken by first occurrence.\"\"\"\n counts = {}\n for x in numbers:\n counts[x] = counts.get(x, 0) + 1\n best = None\n best_count = -1\n for x in numbers:\n if counts[x] > best_count:\n best = x\n best_count = counts[x]\n return best\n\n\ndef variance(numbers):\n \"\"\"Sample variance (divides by n-1).\"\"\"\n if len(numbers) < 2:\n raise ValueError(\"variance requires at least 2 values\")\n m = mean(numbers)\n return"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1786,
|
||||
"eval_count": 23
|
||||
},
|
||||
{
|
||||
"iteration": 6,
|
||||
"elapsed_s": 0.6,
|
||||
"content": "",
|
||||
"tool_calls": [],
|
||||
"prompt_eval_count": 2075,
|
||||
"eval_count": 4
|
||||
}
|
||||
],
|
||||
"final": {
|
||||
"halt_reason": "no_tool_calls",
|
||||
"tests_pass": false,
|
||||
"iterations_used": 6,
|
||||
"tool_call_counts": {
|
||||
"read_file": 2,
|
||||
"write_file": 0,
|
||||
"run_bash": 3
|
||||
},
|
||||
"wall_clock_s": 11.36
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
{
|
||||
"model": "gemma4:26b",
|
||||
"workdir": "/home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b/work",
|
||||
"started_at": 1776532893.203386,
|
||||
"turns": [
|
||||
{
|
||||
"iteration": 1,
|
||||
"elapsed_s": 5.99,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "ls -R"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\n.:\ncalc\nREADME.md\ntests\n\n./calc:\n__init__.py\n__pycache__\nstats.py\n\n./calc/__pycache__:\n__init__.cpython-313.pyc\nstats.cpython-313.pyc\n\n./tests:\n__pycache__\ntest_stats.py\n\n./tests/__pycache__:\ntest_stats.cpython-313-pytest-9.0.2.pyc\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 389,
|
||||
"eval_count": 21
|
||||
},
|
||||
{
|
||||
"iteration": 2,
|
||||
"elapsed_s": 0.65,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
},
|
||||
"result": "# Bakeoff Task\n\nA tiny Python package (`calc/`) with a statistics module. Run `pytest` from this\ndirectory \u2014 two tests currently fail because `median` returns the upper-middle\nelement instead of averaging the two middle elements on even-length inputs.\n\nYour job: make all tests pass. Do not disable or modify the tests.\n\nAllowed tools:\n- `read_file(path)` \u2014 read a file (relative to this directory)\n- `write_file(path, content)` \u2014 overwrite a file (relative to this directory)\n- `run_bash(command)` \u2014 run a shell command (cwd is this directory)\n\nWhen all tests pass, reply with a short summary of the fix and stop calling tools.\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 530,
|
||||
"eval_count": 21
|
||||
},
|
||||
{
|
||||
"iteration": 3,
|
||||
"elapsed_s": 0.68,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
},
|
||||
"result": "exit=2\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollected 0 items / 1 error\n\n==================================== ERRORS ====================================\n_____________________ ERROR collecting tests/test_stats.py _____________________\nImportError while importing test module '/home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b/work/tests/test_stats.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceba"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 711,
|
||||
"eval_count": 19
|
||||
},
|
||||
{
|
||||
"iteration": 4,
|
||||
"elapsed_s": 0.95,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "export PYTHONPATH=$PYTHONPATH:. && pytest"
|
||||
},
|
||||
"result": "exit=1\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-26b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollected 7 items\n\ntests/test_stats.py ..FFF.. [100%]\n\n=================================== FAILURES ===================================\n_______________________________ test_median_even _______________________________\n\n def test_median_even():\n> assert median([1, 2, 3, 4]) == 2.5\nE assert 3 == 2.5\nE + where 3 = median([1, 2, 3, 4]"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1139,
|
||||
"eval_count": 26
|
||||
},
|
||||
{
|
||||
"iteration": 5,
|
||||
"elapsed_s": 1.28,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "calc/stats.py"
|
||||
},
|
||||
"result": "\"\"\"Basic statistics helpers.\"\"\"\n\n\ndef mean(numbers):\n \"\"\"Arithmetic mean of a non-empty list.\"\"\"\n return sum(numbers) / len(numbers)\n\n\ndef median(numbers):\n \"\"\"Return the median of a list of numbers.\"\"\"\n s = sorted(numbers)\n n = len(s)\n return s[n // 2]\n\n\ndef mode(numbers):\n \"\"\"Return the most common value. Ties broken by first occurrence.\"\"\"\n counts = {}\n for x in numbers:\n counts[x] = counts.get(x, 0) + 1\n best = None\n best_count = -1\n for x in numbers:\n if counts[x] > best_count:\n best = x\n best_count = counts[x]\n return best\n\n\ndef variance(numbers):\n \"\"\"Sample variance (divides by n-1).\"\"\"\n if len(numbers) < 2:\n raise ValueError(\"variance requires at least 2 values\")\n m = mean(numbers)\n return"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1775,
|
||||
"eval_count": 23
|
||||
},
|
||||
{
|
||||
"iteration": 6,
|
||||
"elapsed_s": 0.64,
|
||||
"content": "",
|
||||
"tool_calls": [],
|
||||
"prompt_eval_count": 2064,
|
||||
"eval_count": 4
|
||||
}
|
||||
],
|
||||
"final": {
|
||||
"halt_reason": "no_tool_calls",
|
||||
"tests_pass": false,
|
||||
"iterations_used": 6,
|
||||
"tool_call_counts": {
|
||||
"read_file": 2,
|
||||
"write_file": 0,
|
||||
"run_bash": 3
|
||||
},
|
||||
"wall_clock_s": 10.87
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
{
|
||||
"model": "gemma4:31b-it-q4_K_M",
|
||||
"workdir": "/home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-31b/work",
|
||||
"started_at": 1776532955.296148,
|
||||
"turns": [
|
||||
{
|
||||
"iteration": 1,
|
||||
"elapsed_s": 15.8,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
},
|
||||
"result": "# Bakeoff Task\n\nA tiny Python package (`calc/`) with a statistics module. Run `pytest` from this\ndirectory \u2014 two tests currently fail because `median` returns the upper-middle\nelement instead of averaging the two middle elements on even-length inputs.\n\nYour job: make all tests pass. Do not disable or modify the tests.\n\nAllowed tools:\n- `read_file(path)` \u2014 read a file (relative to this directory)\n- `write_file(path, content)` \u2014 overwrite a file (relative to this directory)\n- `run_bash(command)` \u2014 run a shell command (cwd is this directory)\n\nWhen all tests pass, reply with a short summary of the fix and stop calling tools.\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 389,
|
||||
"eval_count": 21
|
||||
},
|
||||
{
|
||||
"iteration": 2,
|
||||
"elapsed_s": 1.5,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
},
|
||||
"result": "exit=2\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-31b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollected 0 items / 1 error\n\n==================================== ERRORS ====================================\n_____________________ ERROR collecting tests/test_stats.py _____________________\nImportError while importing test module '/home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-31b/work/tests/test_stats.py'.\nHint: make sure your test modules/packages have valid Python names.\nTraceba"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 570,
|
||||
"eval_count": 19
|
||||
},
|
||||
{
|
||||
"iteration": 3,
|
||||
"elapsed_s": 2.42,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "ls -R"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\n.:\ncalc\nREADME.md\ntests\n\n./calc:\n__init__.py\n__pycache__\nstats.py\n\n./calc/__pycache__:\n__init__.cpython-313.pyc\nstats.cpython-313.pyc\n\n./tests:\n__pycache__\ntest_stats.py\n\n./tests/__pycache__:\ntest_stats.cpython-313-pytest-9.0.2.pyc\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 998,
|
||||
"eval_count": 21
|
||||
},
|
||||
{
|
||||
"iteration": 4,
|
||||
"elapsed_s": 1.69,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "export PYTHONPATH=$PYTHONPATH:. && pytest"
|
||||
},
|
||||
"result": "exit=1\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-31b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollected 7 items\n\ntests/test_stats.py ..FFF.. [100%]\n\n=================================== FAILURES ===================================\n_______________________________ test_median_even _______________________________\n\n def test_median_even():\n> assert median([1, 2, 3, 4]) == 2.5\nE assert 3 == 2.5\nE + where 3 = median([1, 2, 3, 4]"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1139,
|
||||
"eval_count": 26
|
||||
},
|
||||
{
|
||||
"iteration": 5,
|
||||
"elapsed_s": 3.27,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "calc/stats.py"
|
||||
},
|
||||
"result": "\"\"\"Basic statistics helpers.\"\"\"\n\n\ndef mean(numbers):\n \"\"\"Arithmetic mean of a non-empty list.\"\"\"\n return sum(numbers) / len(numbers)\n\n\ndef median(numbers):\n \"\"\"Return the median of a list of numbers.\"\"\"\n s = sorted(numbers)\n n = len(s)\n return s[n // 2]\n\n\ndef mode(numbers):\n \"\"\"Return the most common value. Ties broken by first occurrence.\"\"\"\n counts = {}\n for x in numbers:\n counts[x] = counts.get(x, 0) + 1\n best = None\n best_count = -1\n for x in numbers:\n if counts[x] > best_count:\n best = x\n best_count = counts[x]\n return best\n\n\ndef variance(numbers):\n \"\"\"Sample variance (divides by n-1).\"\"\"\n if len(numbers) < 2:\n raise ValueError(\"variance requires at least 2 values\")\n m = mean(numbers)\n return"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1775,
|
||||
"eval_count": 23
|
||||
},
|
||||
{
|
||||
"iteration": 6,
|
||||
"elapsed_s": 13.42,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"content": "\"\"\"Basic statistics helpers.\"\"\"\n\n\ndef mean(numbers):\n \"\"\"Arithmetic mean of a non-empty list.\"\"\"\n return sum(numbers) / len(numbers)\n\n\ndef median(numbers):\n \"\"\"Return the median of a list of numbers.\"\"\"\n s = sorted(numbers)\n n = len(s)\n if n % 2 == 1:\n return s[n // 2]\n else:\n return (s[n // 2 - 1] + s[n // 2]) / 2\n\n\ndef mode(numbers):\n \"\"\"Return the most common value. Ties broken by first occurrence.\"\"\"\n counts = {}\n for x in numbers:\n counts[x] = counts.get(x, 0) + 1\n best = None\n best_count = -1\n for x in numbers:\n if counts[x] > best_count:\n best = x\n best_count = counts[x]\n return best\n\n\ndef variance(numbers):\n \"\"\"Sample variance (divides by n-1).\"\"\"\n if len(numbers) < 2:\n raise ValueError(\"variance requires at least 2 values\")\n m = mean(numbers)\n return sum((x - m) ** 2 for x in numbers) / (len(numbers) - 1)\n",
|
||||
"path": "calc/stats.py"
|
||||
},
|
||||
"result": "wrote calc/stats.py (937 bytes)"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 2064,
|
||||
"eval_count": 330
|
||||
},
|
||||
{
|
||||
"iteration": 7,
|
||||
"elapsed_s": 2.52,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "export PYTHONPATH=$PYTHONPATH:. && pytest"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/gemma4-31b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollected 7 items\n\ntests/test_stats.py ....... [100%]\n\n============================== 7 passed in 0.01s ===============================\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 2411,
|
||||
"eval_count": 27
|
||||
},
|
||||
{
|
||||
"iteration": 8,
|
||||
"elapsed_s": 2.49,
|
||||
"content": "I updated the `median` function in `calc/stats.py` to correctly calculate the average of the two middle elements when the input list has an even number of elements.",
|
||||
"tool_calls": [],
|
||||
"prompt_eval_count": 2622,
|
||||
"eval_count": 41
|
||||
}
|
||||
],
|
||||
"final": {
|
||||
"halt_reason": "no_tool_calls",
|
||||
"tests_pass": true,
|
||||
"iterations_used": 8,
|
||||
"tool_call_counts": {
|
||||
"read_file": 2,
|
||||
"write_file": 1,
|
||||
"run_bash": 4
|
||||
},
|
||||
"wall_clock_s": 44.06
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
{
|
||||
"model": "qwen3-coder:30b",
|
||||
"workdir": "/home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work",
|
||||
"started_at": 1776533012.2470767,
|
||||
"turns": [
|
||||
{
|
||||
"iteration": 1,
|
||||
"elapsed_s": 10.69,
|
||||
"content": "I'll help you make the failing tests pass. First, let's understand what we're working with by reading the README.md file to get context about the project.",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
},
|
||||
"result": "# Bakeoff Task\n\nA tiny Python package (`calc/`) with a statistics module. Run `pytest` from this\ndirectory \u2014 two tests currently fail because `median` returns the upper-middle\nelement instead of averaging the two middle elements on even-length inputs.\n\nYour job: make all tests pass. Do not disable or modify the tests.\n\nAllowed tools:\n- `read_file(path)` \u2014 read a file (relative to this directory)\n- `write_file(path, content)` \u2014 overwrite a file (relative to this directory)\n- `run_bash(command)` \u2014 run a shell command (cwd is this directory)\n\nWhen all tests pass, reply with a short summary of the fix and stop calling tools.\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 625,
|
||||
"eval_count": 55
|
||||
},
|
||||
{
|
||||
"iteration": 2,
|
||||
"elapsed_s": 0.42,
|
||||
"content": "Now let's look at the test file to understand what's failing:",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "test_calc.py"
|
||||
},
|
||||
"result": "ERROR: test_calc.py does not exist"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 834,
|
||||
"eval_count": 37
|
||||
},
|
||||
{
|
||||
"iteration": 3,
|
||||
"elapsed_s": 0.36,
|
||||
"content": "Let me check what files are available in the directory:",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "ls -la"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\ntotal 24\ndrwxrwxr-x 5 claude claude 4096 Apr 18 13:19 .\ndrwxrwxr-x 3 claude claude 4096 Apr 18 13:23 ..\ndrwxrwxr-x 3 claude claude 4096 Apr 18 13:19 calc\ndrwxrwxr-x 3 claude claude 4096 Apr 18 13:19 .pytest_cache\n-rw-rw-r-- 1 claude claude 637 Apr 18 13:19 README.md\ndrwxrwxr-x 3 claude claude 4096 Apr 18 13:20 tests\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 892,
|
||||
"eval_count": 37
|
||||
},
|
||||
{
|
||||
"iteration": 4,
|
||||
"elapsed_s": 0.35,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "ls -la tests/"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\ntotal 16\ndrwxrwxr-x 3 claude claude 4096 Apr 18 13:20 .\ndrwxrwxr-x 5 claude claude 4096 Apr 18 13:19 ..\ndrwxrwxr-x 2 claude claude 4096 Apr 18 13:20 __pycache__\n-rw-rw-r-- 1 claude claude 591 Apr 18 13:20 test_stats.py\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1132,
|
||||
"eval_count": 27
|
||||
},
|
||||
{
|
||||
"iteration": 5,
|
||||
"elapsed_s": 0.41,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "tests/test_stats.py"
|
||||
},
|
||||
"result": "from calc.stats import mean, median, mode, variance\n\n\ndef test_mean_basic():\n assert mean([1, 2, 3, 4, 5]) == 3.0\n\n\ndef test_median_odd():\n assert median([1, 2, 3]) == 2\n\n\ndef test_median_even():\n assert median([1, 2, 3, 4]) == 2.5\n\n\ndef test_median_unsorted():\n assert median([3, 1, 4, 1, 5, 9, 2, 6]) == 3.5\n\n\ndef test_median_floats():\n assert median([1.0, 2.0, 3.0, 4.0]) == 2.5\n\n\ndef test_mode_basic():\n assert mode([1, 2, 2, 3]) == 2\n\n\ndef test_variance_basic():\n # sample variance (n-1) of [1, 2, 3, 4, 5] is 10/4 = 2.5\n assert variance([1, 2, 3, 4, 5]) == 2.5\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1305,
|
||||
"eval_count": 24
|
||||
},
|
||||
{
|
||||
"iteration": 6,
|
||||
"elapsed_s": 0.43,
|
||||
"content": "Now let's look at the calc module to see the current implementation:",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "calc"
|
||||
},
|
||||
"result": "ERROR: [Errno 21] Is a directory: '/home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work/calc'"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1593,
|
||||
"eval_count": 35
|
||||
},
|
||||
{
|
||||
"iteration": 7,
|
||||
"elapsed_s": 0.33,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "ls -la calc/"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\ntotal 16\ndrwxrwxr-x 3 claude claude 4096 Apr 18 13:19 .\ndrwxrwxr-x 5 claude claude 4096 Apr 18 13:19 ..\n-rw-rw-r-- 1 claude claude 0 Apr 18 13:19 __init__.py\ndrwxrwxr-x 2 claude claude 4096 Apr 18 13:19 __pycache__\n-rw-rw-r-- 1 claude claude 857 Apr 18 13:19 stats.py\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1683,
|
||||
"eval_count": 27
|
||||
},
|
||||
{
|
||||
"iteration": 8,
|
||||
"elapsed_s": 0.33,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "calc/stats.py"
|
||||
},
|
||||
"result": "\"\"\"Basic statistics helpers.\"\"\"\n\n\ndef mean(numbers):\n \"\"\"Arithmetic mean of a non-empty list.\"\"\"\n return sum(numbers) / len(numbers)\n\n\ndef median(numbers):\n \"\"\"Return the median of a list of numbers.\"\"\"\n s = sorted(numbers)\n n = len(s)\n return s[n // 2]\n\n\ndef mode(numbers):\n \"\"\"Return the most common value. Ties broken by first occurrence.\"\"\"\n counts = {}\n for x in numbers:\n counts[x] = counts.get(x, 0) + 1\n best = None\n best_count = -1\n for x in numbers:\n if counts[x] > best_count:\n best = x\n best_count = counts[x]\n return best\n\n\ndef variance(numbers):\n \"\"\"Sample variance (divides by n-1).\"\"\"\n if len(numbers) < 2:\n raise ValueError(\"variance requires at least 2 values\")\n m = mean(numbers)\n return"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 1885,
|
||||
"eval_count": 23
|
||||
},
|
||||
{
|
||||
"iteration": 9,
|
||||
"elapsed_s": 1.32,
|
||||
"content": "Now I understand the issue. Looking at the failing tests, specifically `test_median_even()` and `test_median_unsorted()`, I can see that the median function is returning the upper-middle element instead of averaging the two middle elements for even-length lists.\n\nIn the current implementation:\n- For `[1, 2, 3, 4]`, it returns `s[4 // 2]` = `s[2]` = `3` (upper-middle)\n- But it should return `(2 + 3) / 2` = `2.5` (average of two middle elements)\n\nLet me run the tests first to confirm the current failures:",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest -v"
|
||||
},
|
||||
"result": "exit=2\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0 -- /usr/bin/python3\ncachedir: .pytest_cache\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollecting ... collected 0 items / 1 error\n\n==================================== ERRORS ====================================\n_____________________ ERROR collecting tests/test_stats.py _____________________\nImportError while importing test module '/home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work/tests/test_stats.py'.\nHint: "
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 2143,
|
||||
"eval_count": 163
|
||||
},
|
||||
{
|
||||
"iteration": 10,
|
||||
"elapsed_s": 0.61,
|
||||
"content": "I need to run the tests from the correct directory. Let me check the current working directory and run pytest properly:",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pwd && python -m pytest tests/ -v"
|
||||
},
|
||||
"result": "exit=127\n--- stdout ---\n/home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work\n\n--- stderr ---\nbash: line 1: python: command not found\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 2693,
|
||||
"eval_count": 56
|
||||
},
|
||||
{
|
||||
"iteration": 11,
|
||||
"elapsed_s": 0.39,
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pwd && python3 -m pytest tests/ -v"
|
||||
},
|
||||
"result": "exit=1\n--- stdout ---\n/home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0 -- /usr/bin/python3\ncachedir: .pytest_cache\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollecting ... collected 7 items\n\ntests/test_stats.py::test_mean_basic PASSED [ 14%]\ntests/test_stats.py::test_median_odd PASSED [ 28%]\ntests/test_stats.py::test_median_even FAILED [ 42%]\nt"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 2812,
|
||||
"eval_count": 34
|
||||
},
|
||||
{
|
||||
"iteration": 12,
|
||||
"elapsed_s": 3.11,
|
||||
"content": "Perfect! Now I can see the exact failures. The issue is in the `median` function in `calc/stats.py`. For even-length lists, it should return the average of the two middle elements, but currently it's just returning the upper-middle element.\n\nLet me fix the median function:",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "calc/stats.py",
|
||||
"content": "\"\"\"Basic statistics helpers.\"\"\"\n\n\ndef mean(numbers):\n \"\"\"Arithmetic mean of a non-empty list.\"\"\"\n return sum(numbers) / len(numbers)\n\n\ndef median(numbers):\n \"\"\"Return the median of a list of numbers.\"\"\"\n s = sorted(numbers)\n n = len(s)\n if n % 2 == 1:\n # Odd length: return the middle element\n return s[n // 2]\n else:\n # Even length: return the average of the two middle elements\n mid1 = s[n // 2 - 1]\n mid2 = s[n // 2]\n return (mid1 + mid2) / 2\n\n\ndef mode(numbers):\n \"\"\"Return the most common value. Ties broken by first occurrence.\"\"\"\n counts = {}\n for x in numbers:\n counts[x] = counts.get(x, 0) + 1\n best = None\n best_count = -1\n for x in numbers:\n if counts[x] > best_count:\n best = x\n best_count = counts[x]\n return best\n\n\ndef variance(numbers):\n \"\"\"Sample variance (divides by n-1).\"\"\"\n if len(numbers) < 2:\n raise ValueError(\"variance requires at least 2 values\")\n m = mean(numbers)\n return sum((x - m) ** 2 for x in numbers) / (len(numbers) - 1)"
|
||||
},
|
||||
"result": "wrote calc/stats.py (1093 bytes)"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 3563,
|
||||
"eval_count": 387
|
||||
},
|
||||
{
|
||||
"iteration": 13,
|
||||
"elapsed_s": 0.44,
|
||||
"content": "Now let's run the tests to see if they pass:",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "python3 -m pytest tests/ -v"
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\n============================= test session starts ==============================\nplatform linux -- Python 3.13.5, pytest-9.0.2, pluggy-1.6.0 -- /usr/bin/python3\ncachedir: .pytest_cache\nrootdir: /home/claude/bin/gemma4-research/scripts/bakeoff/runs/qwen3-coder-30b/work\nplugins: asyncio-1.3.0, anyio-4.12.1\nasyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function\ncollecting ... collected 7 items\n\ntests/test_stats.py::test_mean_basic PASSED [ 14%]\ntests/test_stats.py::test_median_odd PASSED [ 28%]\ntests/test_stats.py::test_median_even PASSED [ 42%]\ntests/test_stats.py::test_median_unsorted PASSED [ 5"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 3974,
|
||||
"eval_count": 44
|
||||
},
|
||||
{
|
||||
"iteration": 14,
|
||||
"elapsed_s": 1.75,
|
||||
"content": "Perfect! All tests are now passing. Let me summarize what I did to fix the issue:\n\nI fixed the `median` function in `calc/stats.py` to properly handle even-length lists. The original implementation was returning just the upper-middle element for even-length lists, but according to statistical convention and the test expectations, it should return the average of the two middle elements. My fix checks if the list length is odd or even, and for even lengths, it calculates the average of the two middle elements correctly.\n\nThe fix involved modifying the median function to:\n1. Sort the input list\n2. Check if the length is odd or even\n3. For odd lengths, return the middle element (as before)\n4. For even lengths, return the average of the two middle elements\n\nThis ensures that all the median tests now pass correctly.",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "echo \"All tests pass - median function now correctly handles even-length lists by averaging the two middle elements\""
|
||||
},
|
||||
"result": "exit=0\n--- stdout ---\nAll tests pass - median function now correctly handles even-length lists by averaging the two middle elements\n\n--- stderr ---\n"
|
||||
}
|
||||
],
|
||||
"prompt_eval_count": 4319,
|
||||
"eval_count": 213
|
||||
},
|
||||
{
|
||||
"iteration": 15,
|
||||
"elapsed_s": 0.65,
|
||||
"content": "I've successfully fixed the failing tests by correcting the median function in `calc/stats.py`. The issue was that for even-length lists, the median function was returning the upper-middle element instead of averaging the two middle elements. I modified the function to properly handle both odd and even-length lists, ensuring all tests now pass.",
|
||||
"tool_calls": [],
|
||||
"prompt_eval_count": 4574,
|
||||
"eval_count": 66
|
||||
}
|
||||
],
|
||||
"final": {
|
||||
"halt_reason": "no_tool_calls",
|
||||
"tests_pass": true,
|
||||
"iterations_used": 15,
|
||||
"tool_call_counts": {
|
||||
"read_file": 4,
|
||||
"write_file": 1,
|
||||
"run_bash": 8
|
||||
},
|
||||
"wall_clock_s": 22.58
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
# Bakeoff Task
|
||||
|
||||
A tiny Python package (`calc/`) with a statistics module. Run `pytest` from this
|
||||
directory — two tests currently fail because `median` returns the upper-middle
|
||||
element instead of averaging the two middle elements on even-length inputs.
|
||||
|
||||
Your job: make all tests pass. Do not disable or modify the tests.
|
||||
|
||||
Allowed tools:
|
||||
- `read_file(path)` — read a file (relative to this directory)
|
||||
- `write_file(path, content)` — overwrite a file (relative to this directory)
|
||||
- `run_bash(command)` — run a shell command (cwd is this directory)
|
||||
|
||||
When all tests pass, reply with a short summary of the fix and stop calling tools.
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Basic statistics helpers."""
|
||||
|
||||
|
||||
def mean(numbers):
|
||||
"""Arithmetic mean of a non-empty list."""
|
||||
return sum(numbers) / len(numbers)
|
||||
|
||||
|
||||
def median(numbers):
|
||||
"""Return the median of a list of numbers."""
|
||||
s = sorted(numbers)
|
||||
n = len(s)
|
||||
return s[n // 2]
|
||||
|
||||
|
||||
def mode(numbers):
|
||||
"""Return the most common value. Ties broken by first occurrence."""
|
||||
counts = {}
|
||||
for x in numbers:
|
||||
counts[x] = counts.get(x, 0) + 1
|
||||
best = None
|
||||
best_count = -1
|
||||
for x in numbers:
|
||||
if counts[x] > best_count:
|
||||
best = x
|
||||
best_count = counts[x]
|
||||
return best
|
||||
|
||||
|
||||
def variance(numbers):
|
||||
"""Sample variance (divides by n-1)."""
|
||||
if len(numbers) < 2:
|
||||
raise ValueError("variance requires at least 2 values")
|
||||
m = mean(numbers)
|
||||
return sum((x - m) ** 2 for x in numbers) / (len(numbers) - 1)
|
||||
@@ -0,0 +1,30 @@
|
||||
from calc.stats import mean, median, mode, variance
|
||||
|
||||
|
||||
def test_mean_basic():
|
||||
assert mean([1, 2, 3, 4, 5]) == 3.0
|
||||
|
||||
|
||||
def test_median_odd():
|
||||
assert median([1, 2, 3]) == 2
|
||||
|
||||
|
||||
def test_median_even():
|
||||
assert median([1, 2, 3, 4]) == 2.5
|
||||
|
||||
|
||||
def test_median_unsorted():
|
||||
assert median([3, 1, 4, 1, 5, 9, 2, 6]) == 3.5
|
||||
|
||||
|
||||
def test_median_floats():
|
||||
assert median([1.0, 2.0, 3.0, 4.0]) == 2.5
|
||||
|
||||
|
||||
def test_mode_basic():
|
||||
assert mode([1, 2, 2, 3]) == 2
|
||||
|
||||
|
||||
def test_variance_basic():
|
||||
# sample variance (n-1) of [1, 2, 3, 4, 5] is 10/4 = 2.5
|
||||
assert variance([1, 2, 3, 4, 5]) == 2.5
|
||||
Reference in New Issue
Block a user