mortdecai-model-analysis/training-data-analysis.md

Training Data Analysis: merged_training_v06.jsonl

Date: 2026-03-26 Source: /home/claude/bin/Mincecraft-AI-model/data/processed/merged_training_v06.jsonl Examples: 7,256 total Training script: training/scripts/train_lora.py (Unsloth + TRL SFTTrainer)

---

Dataset Structure

Metric	Value
Total examples	7,256
Parse errors	0
Message format	"conversations" (7,254) + "text" (2)
Multi-turn (>3 msgs)	4,006 (55.2%)
Has system prompt	100%
Has /no_think prefix	3,459 (47.7%)
Avg system prompt length	21,358 chars (~5,300 tokens)
System prompts >20K chars	6,526 (89.9%)
JSON responses	3,248 (44.8%)
tool_call responses	4,006 (55.2%)
Contains <think> tags	52
Tool role messages	12,155
Contaminated JSON (tool_call inside)	84
Empty commands arrays	387

System Prompt Variants

Variant	Count
script_writer (no /no_think)	3,559
/no_think + paper_server	2,942
/no_think + other	338
paper_server (no /no_think)	236
/no_think + server_admin	179
god_persona	6

JSON Response Schema Variants

Key Combination	Count
{commands, risk_level}	1,112
{commands, reasoning, risk_level}	911
{commands, message, risk_level}	495
{commands, message, reasoning}	338
{commands, message, reasoning, risk_level}	243
{commands, reasoning}	149

Tool Usage in Training Data

Tool	Calls
rcon.execute	7,220
script.validate	1,496
script.write	1,493
script.execute	1,485
journal.read	110
world.player_info	70
journal.write	33
world.scan_area	29
minecraft.wiki_lookup	28
world.nearby_entities	25
Others (14 tools)	<25 each

---

Six Compounding Failure Modes

1. System Prompt Truncation (The Killer)

max_seq_len = 2048 tokens. Average system prompt = ~5,300 tokens. 90% of examples have system prompts that exceed the entire sequence length by 2.5x.

With packing enabled, the trainer stuffs multiple examples per 2048-token window. The system prompt alone doesn't fit — so the model trained on truncated system prompts with no user input and no assistant response in most examples. It learned system prompt fragments, not task behavior.

2. Mixed Response Paradigm

44.8% of examples teach: return clean JSON {"commands": [...]}. 55.2% of examples teach: emit <tool_call>{"name": "rcon.execute",...}</tool_call>.

No clear signal distinguishes when to use each format. The system prompts differ but get truncated (Issue 1), so the model never sees the disambiguation.

3. Inconsistent JSON Schema

6 different key combinations across the JSON responses. No single schema dominates enough to become the learned default.

4. Contaminated Examples

• 84 examples have <tool_call> strings inside JSON responses (pipeline leakage)
• 387 examples have empty commands: [] (teaches returning nothing is acceptable)
• 2 raw text format entries with literal <|im_start|> tokens

5. Tool Role Incompatibility

4,006 examples use custom tool names (rcon.execute, script.validate) that aren't in Qwen's pretrained vocabulary. The model needs to learn these from scratch, but with truncated sequences it never sees enough context.

6. /no_think Misuse

/no_think is a Qwen inference-time directive, not a trainable behavior. Including it in 48% of training data wastes tokens and doesn't transfer to the fine-tuned model's behavior (confirmed by probes).

---

Training Pipeline Details

Data Flow

Raw sources (seed, tool, audit, self_play, distilled, etc.)
  → merge_datasets.py (normalize, dedup, 95/5 split)
  → merged_training_v06.jsonl
  → train_lora.py
    ├─ load_seed_dataset() → conversations format
    ├─ load_tool_dataset() → messages/text format
    └─ formatting_func() → tokenizer.apply_chat_template()
  → SFTTrainer (Unsloth, QLoRA 4-bit)
  → LoRA merge → GGUF → quantize → Ollama

Training Config

• Base models: Qwen3-8B, Qwen3.5-9B, Qwen3.5-14B
• Method: QLoRA (4-bit base + FP16 LoRA)
• LoRA: rank 16-64, alpha 32-128, targets q/k/v/o/gate/up/down_proj
• Batch: 2 x 4 grad accum = effective 8
• LR: 2e-4, cosine schedule, 0.1 warmup
• Epochs: 1 (default)
• Packing: enabled
• max_seq_len: 2048

What the Model Actually Trained On

Due to truncation + packing, most training examples were reduced to:

[truncated system prompt fragment][truncated system prompt fragment][truncated...]

The model spent compute learning to predict system prompt text, not learning the user→assistant mapping.