0.6.0 training session: Oracle Bot, RL combat, Mind's Eye, multilingual pipeline
Major changes from this session: Training: - 0.6.0 training running: 9B on steel141 3090 Ti, 27B on rented H100 NVL - 7,256 merged training examples (up from 3,183) - New training data: failure modes (85), midloop messaging (27), prompt injection defense (29), personality (32), gold from quarantine bank (232), new tool examples (30), claude's own experience (10) - All training data RCON-validated at 100% pass rate - Bake-off: gemma3:27b 66%, qwen3.5:27b 61%, translategemma:27b 56% Oracle Bot (Mind's Eye): - Invisible spectator bot (mineflayer) streams world state via WebSocket - HTML5 Canvas frontend at mind.mortdec.ai - Real-time tool trace visualization with expandable entries - Streaming model tokens during inference - Gateway integration: fire-and-forget POST /trace on every tool call Reinforcement Learning: - Gymnasium environment wrapping mineflayer bot (minecraft_env.py) - PPO training via Stable Baselines3 (10K param policy network) - Behavioral cloning pretraining (97.5% accuracy on expert policy) - Infinite training loop with auto-restart and checkpoint resume - Bot learns combat, survival, navigation from raw experience Bot Army: - 8-soldier marching formation with autonomous combat - Combat bots using mineflayer-pvp, pathfinder, armor-manager - Multilingual prayer bots via translategemma:27b (18 languages) - Frame-based AI architecture: LLM planner + reactive micro-scripts Infrastructure: - Fixed mattpc.sethpc.xyz billing gateway (API key + player list parser) - Billing gateway now tracks all LAN traffic (LAN auto-auth) - Gateway fallback for empty god-mode responses - Updated mortdec.ai landing page Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Seth
@@ -0,0 +1,865 @@
|
||||
# Oracle Bot Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build a live HTML5 viewport ("Mind's Eye") that renders what the Mortdecai AI sees, powered by an invisible spectator bot streaming world state and tool traces to browsers.
|
||||
|
||||
**Architecture:** Single Node.js process (mineflayer + Express + ws) connects to Paper server in spectator mode, receives tool traces from the gateway via POST, and streams world data + traces to browsers via WebSocket. Frontend is a single HTML5 Canvas page.
|
||||
|
||||
**Tech Stack:** Node.js, mineflayer, express, ws, HTML5 Canvas, vanilla JS
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-03-22-oracle-bot-design.md`
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
oracle-bot/ # NEW directory at repo root
|
||||
├── package.json # deps: mineflayer, express, ws
|
||||
├── server.js # ENTRYPOINT — Express + ws + requires bot
|
||||
├── bot.js # mineflayer spectator connection + chunk tracking
|
||||
├── world-state.js # Abstracted world state: blocks, entities, players
|
||||
├── public/
|
||||
│ └── index.html # Single-file frontend (Canvas + JS + CSS)
|
||||
└── oracle-bot.service # systemd unit file for deployment
|
||||
```
|
||||
|
||||
**Modified files:**
|
||||
- `langgraph_gateway.py` (in PaperFork repo) — add fire-and-forget POST to `/trace` after tool calls
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Project Scaffold + Bot Connection
|
||||
|
||||
**Files:**
|
||||
- Create: `oracle-bot/package.json`
|
||||
- Create: `oracle-bot/bot.js`
|
||||
|
||||
- [ ] **Step 1: Create package.json**
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "oracle-bot",
|
||||
"version": "0.1.0",
|
||||
"description": "Mortdecai Mind's Eye — live AI vision viewport",
|
||||
"main": "server.js",
|
||||
"scripts": {
|
||||
"start": "node server.js",
|
||||
"dev": "node server.js --dev"
|
||||
},
|
||||
"dependencies": {
|
||||
"mineflayer": "^4.23.0",
|
||||
"express": "^4.21.0",
|
||||
"ws": "^8.18.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Install dependencies**
|
||||
|
||||
Run: `cd oracle-bot && npm install`
|
||||
Expected: `node_modules/` created, no errors
|
||||
|
||||
- [ ] **Step 3: Write bot.js — mineflayer spectator connection**
|
||||
|
||||
```javascript
|
||||
/**
|
||||
* bot.js — Mineflayer spectator bot for Oracle vision system.
|
||||
*
|
||||
* Connects to Paper server in offline mode. Expects server to set
|
||||
* spectator gamemode on join. Tracks chunk data, entities, players.
|
||||
* Emits events for world-state.js to consume.
|
||||
*/
|
||||
|
||||
const mineflayer = require('mineflayer');
|
||||
const EventEmitter = require('events');
|
||||
|
||||
const DEFAULT_CONFIG = {
|
||||
host: process.env.MC_HOST || '192.168.0.244',
|
||||
port: parseInt(process.env.MC_PORT || '25568', 10),
|
||||
username: process.env.BOT_USERNAME || 'OracleBot',
|
||||
version: '1.21.11',
|
||||
};
|
||||
|
||||
class OracleBot extends EventEmitter {
|
||||
constructor(config = {}) {
|
||||
super();
|
||||
this.config = { ...DEFAULT_CONFIG, ...config };
|
||||
this.bot = null;
|
||||
this.connected = false;
|
||||
this.following = null; // player name we're following
|
||||
this._reconnectDelay = 1000;
|
||||
this._maxReconnectDelay = 30000;
|
||||
}
|
||||
|
||||
connect() {
|
||||
console.log(`[bot] Connecting to ${this.config.host}:${this.config.port} as ${this.config.username}...`);
|
||||
|
||||
this.bot = mineflayer.createBot({
|
||||
host: this.config.host,
|
||||
port: this.config.port,
|
||||
username: this.config.username,
|
||||
auth: 'offline',
|
||||
version: this.config.version,
|
||||
});
|
||||
|
||||
this.bot.on('login', () => {
|
||||
console.log(`[bot] Logged in as ${this.bot.username}`);
|
||||
this.connected = true;
|
||||
this._reconnectDelay = 1000; // reset backoff
|
||||
this.emit('connected');
|
||||
});
|
||||
|
||||
this.bot.on('spawn', () => {
|
||||
const pos = this.bot.entity.position;
|
||||
console.log(`[bot] Spawned at (${pos.x.toFixed(0)}, ${pos.y.toFixed(0)}, ${pos.z.toFixed(0)}) mode=${this.bot.game.gameMode}`);
|
||||
|
||||
// Request spectator if not already
|
||||
if (this.bot.game.gameMode !== 'spectator') {
|
||||
console.log('[bot] Not in spectator mode, requesting...');
|
||||
this.bot.chat('/gamemode spectator');
|
||||
}
|
||||
|
||||
this.emit('spawned');
|
||||
});
|
||||
|
||||
this.bot.on('kicked', (reason) => {
|
||||
console.log(`[bot] Kicked: ${reason}`);
|
||||
this.connected = false;
|
||||
this.emit('disconnected', reason);
|
||||
this._scheduleReconnect();
|
||||
});
|
||||
|
||||
this.bot.on('end', (reason) => {
|
||||
console.log(`[bot] Disconnected: ${reason}`);
|
||||
this.connected = false;
|
||||
this.emit('disconnected', reason);
|
||||
this._scheduleReconnect();
|
||||
});
|
||||
|
||||
this.bot.on('error', (err) => {
|
||||
console.error(`[bot] Error: ${err.message}`);
|
||||
});
|
||||
|
||||
// Track player joins/leaves
|
||||
this.bot.on('playerJoined', (player) => {
|
||||
if (player.username !== this.config.username) {
|
||||
this.emit('playerJoined', player.username);
|
||||
}
|
||||
});
|
||||
|
||||
this.bot.on('playerLeft', (player) => {
|
||||
if (player.username !== this.config.username) {
|
||||
this.emit('playerLeft', player.username);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
_scheduleReconnect() {
|
||||
console.log(`[bot] Reconnecting in ${this._reconnectDelay}ms...`);
|
||||
setTimeout(() => {
|
||||
this._reconnectDelay = Math.min(this._reconnectDelay * 2, this._maxReconnectDelay);
|
||||
this.connect();
|
||||
}, this._reconnectDelay);
|
||||
}
|
||||
|
||||
/** Teleport to a player and follow them. */
|
||||
async followPlayer(playerName) {
|
||||
if (!this.connected || !this.bot) return;
|
||||
const player = this.bot.players[playerName];
|
||||
if (!player || !player.entity) {
|
||||
console.log(`[bot] Cannot find player entity for ${playerName}`);
|
||||
return;
|
||||
}
|
||||
this.following = playerName;
|
||||
const pos = player.entity.position;
|
||||
this.bot.entity.position.set(pos.x, pos.y + 2, pos.z);
|
||||
console.log(`[bot] Following ${playerName} at (${pos.x.toFixed(0)}, ${pos.y.toFixed(0)}, ${pos.z.toFixed(0)})`);
|
||||
}
|
||||
|
||||
/** Get online players (excluding self). */
|
||||
getPlayers() {
|
||||
if (!this.bot) return [];
|
||||
return Object.values(this.bot.players)
|
||||
.filter(p => p.username !== this.config.username)
|
||||
.map(p => ({
|
||||
name: p.username,
|
||||
x: p.entity ? Math.floor(p.entity.position.x) : null,
|
||||
y: p.entity ? Math.floor(p.entity.position.y) : null,
|
||||
z: p.entity ? Math.floor(p.entity.position.z) : null,
|
||||
}));
|
||||
}
|
||||
|
||||
/** Scan blocks in a top-down slice around center at given Y level. */
|
||||
scanArea(centerX, centerY, centerZ, radius = 16) {
|
||||
if (!this.bot) return [];
|
||||
const blocks = [];
|
||||
for (let dx = -radius; dx <= radius; dx++) {
|
||||
for (let dz = -radius; dz <= radius; dz++) {
|
||||
const block = this.bot.blockAt(this.bot.vec3(centerX + dx, centerY, centerZ + dz));
|
||||
if (block && block.name !== 'air') {
|
||||
blocks.push({
|
||||
x: centerX + dx,
|
||||
y: centerY,
|
||||
z: centerZ + dz,
|
||||
type: block.name,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
return blocks;
|
||||
}
|
||||
|
||||
/** Get nearby entities around a position. */
|
||||
getNearbyEntities(centerX, centerY, centerZ, radius = 30) {
|
||||
if (!this.bot) return [];
|
||||
const entities = [];
|
||||
for (const entity of Object.values(this.bot.entities)) {
|
||||
if (entity === this.bot.entity) continue;
|
||||
const dist = entity.position.distanceTo(this.bot.vec3(centerX, centerY, centerZ));
|
||||
if (dist <= radius) {
|
||||
entities.push({
|
||||
type: entity.type === 'player' ? 'player' : (entity.name || entity.type),
|
||||
name: entity.username || null,
|
||||
x: Math.floor(entity.position.x),
|
||||
y: Math.floor(entity.position.y),
|
||||
z: Math.floor(entity.position.z),
|
||||
distance: Math.round(dist),
|
||||
});
|
||||
}
|
||||
}
|
||||
return entities;
|
||||
}
|
||||
|
||||
/** Get server time and weather info. */
|
||||
getWorldInfo() {
|
||||
if (!this.bot) return {};
|
||||
return {
|
||||
time: this.bot.time?.timeOfDay || 0,
|
||||
isRaining: this.bot.isRaining || false,
|
||||
gameMode: this.bot.game?.gameMode || 'unknown',
|
||||
};
|
||||
}
|
||||
|
||||
destroy() {
|
||||
if (this.bot) {
|
||||
this.bot.removeAllListeners();
|
||||
this.bot.end();
|
||||
this.bot = null;
|
||||
}
|
||||
this.connected = false;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = OracleBot;
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Test bot connection manually**
|
||||
|
||||
Run: `cd oracle-bot && node -e "const B = require('./bot'); const b = new B(); b.connect(); b.on('spawned', () => { console.log('Players:', b.getPlayers()); setTimeout(() => { b.destroy(); process.exit(0); }, 3000); });"`
|
||||
|
||||
Expected: Bot connects, logs spawned position, lists online players, exits.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add oracle-bot/package.json oracle-bot/bot.js oracle-bot/package-lock.json
|
||||
git commit -m "feat(oracle): scaffold project + mineflayer spectator bot"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: World State Abstraction
|
||||
|
||||
**Files:**
|
||||
- Create: `oracle-bot/world-state.js`
|
||||
|
||||
- [ ] **Step 1: Write world-state.js**
|
||||
|
||||
```javascript
|
||||
/**
|
||||
* world-state.js — Unified world state for the Oracle vision system.
|
||||
*
|
||||
* Aggregates bot perception (chunks, entities) and gateway traces
|
||||
* into a single state object that can be serialized and streamed.
|
||||
*/
|
||||
|
||||
class WorldState {
|
||||
constructor(bot) {
|
||||
this.bot = bot;
|
||||
this.mode = 'idle'; // 'idle' | 'god' | 'sudo'
|
||||
this.activePlayer = null; // player the AI is currently interacting with
|
||||
this.activeSessionId = null;
|
||||
this.traces = []; // recent tool traces (ring buffer, max 50)
|
||||
this.lastActivity = 0; // timestamp of last trace
|
||||
this.ACTIVE_TIMEOUT = 10000; // ms before reverting to idle
|
||||
this.MAX_TRACES = 50;
|
||||
}
|
||||
|
||||
/** Called when gateway sends a trace event. */
|
||||
onTrace(traceData) {
|
||||
this.lastActivity = Date.now();
|
||||
|
||||
// Switch to active mode
|
||||
if (traceData.mode && traceData.mode !== this.mode) {
|
||||
this.mode = traceData.mode;
|
||||
}
|
||||
if (traceData.player) {
|
||||
this.activePlayer = traceData.player;
|
||||
}
|
||||
if (traceData.session_id) {
|
||||
this.activeSessionId = traceData.session_id;
|
||||
}
|
||||
|
||||
// Store trace
|
||||
this.traces.push({
|
||||
...traceData,
|
||||
ts: Date.now(),
|
||||
});
|
||||
if (this.traces.length > this.MAX_TRACES) {
|
||||
this.traces.shift();
|
||||
}
|
||||
}
|
||||
|
||||
/** Check if we should revert to idle. */
|
||||
checkIdle() {
|
||||
if (this.mode !== 'idle' && Date.now() - this.lastActivity > this.ACTIVE_TIMEOUT) {
|
||||
this.mode = 'idle';
|
||||
this.activeSessionId = null;
|
||||
return true; // changed to idle
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Build a heartbeat message (idle mode, low frequency). */
|
||||
buildHeartbeat() {
|
||||
return {
|
||||
v: 1,
|
||||
type: 'heartbeat',
|
||||
mode: this.mode,
|
||||
activePlayer: this.activePlayer,
|
||||
players: this.bot.getPlayers(),
|
||||
world: this.bot.getWorldInfo(),
|
||||
ts: Date.now(),
|
||||
};
|
||||
}
|
||||
|
||||
/** Build a world snapshot message (active mode, high frequency). */
|
||||
buildWorldSnapshot() {
|
||||
if (!this.activePlayer) return null;
|
||||
|
||||
const players = this.bot.getPlayers();
|
||||
const target = players.find(p => p.name === this.activePlayer);
|
||||
if (!target || target.x === null) return null;
|
||||
|
||||
const blocks = this.bot.scanArea(target.x, target.y, target.z, 16);
|
||||
const entities = this.bot.getNearbyEntities(target.x, target.y, target.z, 30);
|
||||
|
||||
return {
|
||||
v: 1,
|
||||
type: 'world',
|
||||
mode: this.mode,
|
||||
activePlayer: this.activePlayer,
|
||||
center: { x: target.x, y: target.y, z: target.z },
|
||||
blocks,
|
||||
entities,
|
||||
players,
|
||||
ts: Date.now(),
|
||||
};
|
||||
}
|
||||
|
||||
/** Build a trace event message for the browser. */
|
||||
buildTraceEvent(traceData) {
|
||||
return {
|
||||
v: 1,
|
||||
type: 'trace',
|
||||
...traceData,
|
||||
ts: Date.now(),
|
||||
};
|
||||
}
|
||||
|
||||
/** Build a mode change message. */
|
||||
buildModeChange() {
|
||||
return {
|
||||
v: 1,
|
||||
type: 'mode',
|
||||
mode: this.mode,
|
||||
player: this.activePlayer,
|
||||
ts: Date.now(),
|
||||
};
|
||||
}
|
||||
|
||||
/** Build a status message (connected/disconnected). */
|
||||
buildStatus(connected) {
|
||||
return {
|
||||
v: 1,
|
||||
type: 'status',
|
||||
connected,
|
||||
ts: Date.now(),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = WorldState;
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add oracle-bot/world-state.js
|
||||
git commit -m "feat(oracle): world state abstraction layer"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Express + WebSocket Server
|
||||
|
||||
**Files:**
|
||||
- Create: `oracle-bot/server.js`
|
||||
|
||||
- [ ] **Step 1: Write server.js**
|
||||
|
||||
```javascript
|
||||
/**
|
||||
* server.js — Oracle Bot entrypoint.
|
||||
*
|
||||
* Express HTTP server + WebSocket for browser streaming.
|
||||
* Receives tool traces from gateway, merges with bot world data,
|
||||
* streams to connected browsers.
|
||||
*/
|
||||
|
||||
const express = require('express');
|
||||
const http = require('http');
|
||||
const WebSocket = require('ws');
|
||||
const path = require('path');
|
||||
const OracleBot = require('./bot');
|
||||
const WorldState = require('./world-state');
|
||||
|
||||
const PORT = parseInt(process.env.PORT || '3333', 10);
|
||||
const MAX_WS_CONNECTIONS = 100;
|
||||
const MAX_PER_IP = 5;
|
||||
const HEARTBEAT_INTERVAL = 5000; // 5s idle heartbeat
|
||||
const ACTIVE_INTERVAL = 1000; // 1s active world snapshots
|
||||
|
||||
// ── Setup ────────────────────────────────────────────────────
|
||||
|
||||
const app = express();
|
||||
app.use(express.json({ limit: '64kb' }));
|
||||
app.use(express.static(path.join(__dirname, 'public')));
|
||||
|
||||
const server = http.createServer(app);
|
||||
const wss = new WebSocket.Server({ server, path: '/ws' });
|
||||
|
||||
const bot = new OracleBot();
|
||||
const state = new WorldState(bot);
|
||||
|
||||
// Track connections per IP
|
||||
const ipCounts = new Map();
|
||||
|
||||
// ── WebSocket ────────────────────────────────────────────────
|
||||
|
||||
wss.on('connection', (ws, req) => {
|
||||
const ip = req.headers['x-forwarded-for']?.split(',')[0]?.trim() || req.socket.remoteAddress;
|
||||
|
||||
// Rate limit
|
||||
const count = ipCounts.get(ip) || 0;
|
||||
if (wss.clients.size >= MAX_WS_CONNECTIONS || count >= MAX_PER_IP) {
|
||||
ws.close(4029, 'Too many connections');
|
||||
return;
|
||||
}
|
||||
ipCounts.set(ip, count + 1);
|
||||
console.log(`[ws] Client connected from ${ip} (${wss.clients.size} total)`);
|
||||
|
||||
// Send current state on connect
|
||||
ws.send(JSON.stringify(state.buildHeartbeat()));
|
||||
if (state.mode !== 'idle') {
|
||||
ws.send(JSON.stringify(state.buildModeChange()));
|
||||
}
|
||||
|
||||
ws.on('close', () => {
|
||||
const c = ipCounts.get(ip) || 1;
|
||||
if (c <= 1) ipCounts.delete(ip);
|
||||
else ipCounts.set(ip, c - 1);
|
||||
console.log(`[ws] Client disconnected (${wss.clients.size} total)`);
|
||||
});
|
||||
});
|
||||
|
||||
function broadcast(msg) {
|
||||
const data = JSON.stringify(msg);
|
||||
for (const client of wss.clients) {
|
||||
if (client.readyState === WebSocket.OPEN) {
|
||||
client.send(data);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── HTTP Endpoints (internal only) ───────────────────────────
|
||||
|
||||
app.post('/trace', (req, res) => {
|
||||
const trace = req.body;
|
||||
if (!trace || !trace.tool) {
|
||||
return res.status(400).json({ error: 'Missing tool field' });
|
||||
}
|
||||
|
||||
// Update world state
|
||||
const previousMode = state.mode;
|
||||
state.onTrace(trace);
|
||||
|
||||
// If mode changed, broadcast mode change
|
||||
if (state.mode !== previousMode) {
|
||||
broadcast(state.buildModeChange());
|
||||
// Follow the active player
|
||||
if (trace.player) {
|
||||
bot.followPlayer(trace.player);
|
||||
}
|
||||
}
|
||||
|
||||
// Broadcast trace event to browsers
|
||||
broadcast(state.buildTraceEvent(trace));
|
||||
|
||||
// Send a world snapshot (active mode burst)
|
||||
const snapshot = state.buildWorldSnapshot();
|
||||
if (snapshot) {
|
||||
broadcast(snapshot);
|
||||
}
|
||||
|
||||
res.json({ ok: true });
|
||||
});
|
||||
|
||||
app.post('/command', (req, res) => {
|
||||
const { action, target, center, radius } = req.body || {};
|
||||
|
||||
if (action === 'follow' && target) {
|
||||
bot.followPlayer(target);
|
||||
return res.json({ ok: true, action: 'follow', target });
|
||||
}
|
||||
|
||||
if (action === 'scan' && center) {
|
||||
const blocks = bot.scanArea(center.x, center.y, center.z, radius || 16);
|
||||
return res.json({ ok: true, blocks: blocks.length });
|
||||
}
|
||||
|
||||
res.status(400).json({ error: 'Unknown action' });
|
||||
});
|
||||
|
||||
app.get('/health', (req, res) => {
|
||||
res.json({
|
||||
ok: true,
|
||||
botConnected: bot.connected,
|
||||
wsClients: wss.clients.size,
|
||||
mode: state.mode,
|
||||
activePlayer: state.activePlayer,
|
||||
});
|
||||
});
|
||||
|
||||
// ── Periodic Updates ─────────────────────────────────────────
|
||||
|
||||
setInterval(() => {
|
||||
if (wss.clients.size === 0) return;
|
||||
|
||||
// Check idle transition
|
||||
const wentIdle = state.checkIdle();
|
||||
if (wentIdle) {
|
||||
broadcast(state.buildModeChange());
|
||||
}
|
||||
|
||||
if (state.mode === 'idle') {
|
||||
// Low-frequency heartbeat
|
||||
broadcast(state.buildHeartbeat());
|
||||
}
|
||||
}, HEARTBEAT_INTERVAL);
|
||||
|
||||
// Active mode: higher frequency world snapshots
|
||||
setInterval(() => {
|
||||
if (wss.clients.size === 0 || state.mode === 'idle') return;
|
||||
|
||||
const snapshot = state.buildWorldSnapshot();
|
||||
if (snapshot) {
|
||||
broadcast(snapshot);
|
||||
}
|
||||
}, ACTIVE_INTERVAL);
|
||||
|
||||
// ── Bot Events ───────────────────────────────────────────────
|
||||
|
||||
bot.on('connected', () => {
|
||||
console.log('[server] Bot connected to MC server');
|
||||
broadcast(state.buildStatus(true));
|
||||
});
|
||||
|
||||
bot.on('disconnected', (reason) => {
|
||||
console.log(`[server] Bot disconnected: ${reason}`);
|
||||
broadcast(state.buildStatus(false));
|
||||
});
|
||||
|
||||
bot.on('playerJoined', (name) => {
|
||||
console.log(`[server] Player joined: ${name}`);
|
||||
broadcast(state.buildHeartbeat());
|
||||
});
|
||||
|
||||
bot.on('playerLeft', (name) => {
|
||||
console.log(`[server] Player left: ${name}`);
|
||||
broadcast(state.buildHeartbeat());
|
||||
});
|
||||
|
||||
// ── Start ────────────────────────────────────────────────────
|
||||
|
||||
bot.connect();
|
||||
|
||||
server.listen(PORT, () => {
|
||||
console.log(`[server] Oracle Bot listening on port ${PORT}`);
|
||||
console.log(`[server] Frontend: http://localhost:${PORT}`);
|
||||
console.log(`[server] WebSocket: ws://localhost:${PORT}/ws`);
|
||||
console.log(`[server] Health: http://localhost:${PORT}/health`);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Test server starts and health endpoint works**
|
||||
|
||||
Run: `cd oracle-bot && timeout 15 node server.js &; sleep 5; curl -s http://localhost:3333/health; kill %1`
|
||||
Expected: `{"ok":true,"botConnected":true,"wsClients":0,"mode":"idle","activePlayer":null}`
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add oracle-bot/server.js
|
||||
git commit -m "feat(oracle): express + websocket server with trace/command endpoints"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: HTML5 Canvas Frontend
|
||||
|
||||
**Files:**
|
||||
- Create: `oracle-bot/public/index.html`
|
||||
|
||||
- [ ] **Step 1: Write index.html — complete single-file frontend**
|
||||
|
||||
This is the largest file. Single HTML file with embedded CSS and JS. Key sections:
|
||||
- CSS: dark theme, Rajdhani Bold font, Sethian orange accents, three-panel layout
|
||||
- Canvas: 2D top-down tile map, block color mapping, entity rendering
|
||||
- Tool trace panel: scrolling timeline with tool call entries
|
||||
- Status bar: mode indicator, player info, connection status
|
||||
- WebSocket: auto-connect, reconnect on close, handle all message types
|
||||
- Visual modes: idle (muted pulse), god (orange/gold), sudo (blue/green)
|
||||
|
||||
The file will be ~400-500 lines. Key implementation details:
|
||||
|
||||
**Block color map:**
|
||||
```javascript
|
||||
const BLOCK_COLORS = {
|
||||
stone: '#808080', cobblestone: '#6B6B6B', dirt: '#8B6914',
|
||||
grass_block: '#4CAF50', sand: '#F4E4A0', gravel: '#A0A0A0',
|
||||
oak_planks: '#BC8F4F', oak_log: '#6B4226', spruce_planks: '#5C3A1E',
|
||||
water: '#2196F3', lava: '#FF5722', redstone_wire: '#FF0000',
|
||||
diamond_ore: '#4FC3F7', iron_ore: '#D4A574', gold_ore: '#FFD54F',
|
||||
bedrock: '#1A1A1A', obsidian: '#1A0A2E', glass: '#E0F7FA',
|
||||
torch: '#FFEB3B',
|
||||
// default for unknown blocks
|
||||
_default: '#9E9E9E',
|
||||
};
|
||||
```
|
||||
|
||||
**Canvas rendering loop:**
|
||||
- `requestAnimationFrame` for smooth animations
|
||||
- World tiles drawn from `blocks` array, centered on active player
|
||||
- Entities drawn as colored dots with labels
|
||||
- Active scan areas pulse with translucent overlay
|
||||
- Tool traces appear as floating text annotations
|
||||
|
||||
**WebSocket handler:**
|
||||
```javascript
|
||||
const ws = new WebSocket(`ws://${location.host}/ws`);
|
||||
ws.onmessage = (e) => {
|
||||
const msg = JSON.parse(e.data);
|
||||
switch (msg.type) {
|
||||
case 'heartbeat': updateHeartbeat(msg); break;
|
||||
case 'world': updateWorld(msg); break;
|
||||
case 'trace': addTrace(msg); break;
|
||||
case 'mode': updateMode(msg); break;
|
||||
case 'status': updateStatus(msg); break;
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Test frontend loads in browser**
|
||||
|
||||
Run: `cd oracle-bot && node server.js &; sleep 3; echo "Open http://localhost:3333 in browser"; curl -s http://localhost:3333/ | head -5`
|
||||
Expected: HTML page served, title contains "MORTDECAI"
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add oracle-bot/public/index.html
|
||||
git commit -m "feat(oracle): HTML5 Canvas frontend — Mind's Eye viewport"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Gateway Integration
|
||||
|
||||
**Files:**
|
||||
- Modify: `/root/bin/Sethpc-Minecraft-PaperFork/langgraph_gateway.py` (tool loop section, ~line 1318)
|
||||
|
||||
- [ ] **Step 1: Add oracle trace posting to gateway**
|
||||
|
||||
Add a helper function near the top of the file (after imports):
|
||||
|
||||
```python
|
||||
# ── Oracle Bot integration ─────────────────────────────────────────
|
||||
ORACLE_URL = 'http://localhost:3333/trace'
|
||||
|
||||
def _oracle_trace(tool_name, tool_args, result, step, session):
|
||||
"""Fire-and-forget trace event to Oracle Bot."""
|
||||
try:
|
||||
requests.post(ORACLE_URL, json={
|
||||
'tool': tool_name,
|
||||
'input': {k: str(v)[:200] for k, v in (tool_args or {}).items()},
|
||||
'ok': result.get('ok', result.get('success', False)),
|
||||
'step': step,
|
||||
'mode': session.mode,
|
||||
'player': session.player,
|
||||
'session_id': session.session_id,
|
||||
}, timeout=1)
|
||||
except Exception:
|
||||
pass # Oracle is optional, never block the gateway
|
||||
```
|
||||
|
||||
Then in `_model_driven_tool_loop`, after the tool result is captured (after line ~1338 `tool_trace.append(trace_entry)`), add:
|
||||
|
||||
```python
|
||||
_oracle_trace(tool_name, tool_args, result, step, session)
|
||||
```
|
||||
|
||||
Also post on session start/end — in `run_pipeline()` after `_model_driven_tool_loop` returns (after the tool loop result), add:
|
||||
|
||||
```python
|
||||
# Notify Oracle of session end
|
||||
_oracle_trace('_session_end', {'commands': commands, 'message': message}, {'ok': True}, -1, session)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify gateway still starts cleanly**
|
||||
|
||||
Run: `python3 -c "import ast; ast.parse(open('/root/bin/Sethpc-Minecraft-PaperFork/langgraph_gateway.py').read()); print('OK')"`
|
||||
Expected: `OK`
|
||||
|
||||
- [ ] **Step 3: Commit in PaperFork repo**
|
||||
|
||||
```bash
|
||||
cd /root/bin/Sethpc-Minecraft-PaperFork
|
||||
git add langgraph_gateway.py
|
||||
git commit -m "feat: add Oracle Bot trace integration (fire-and-forget)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Deployment — systemd + Caddy
|
||||
|
||||
**Files:**
|
||||
- Create: `oracle-bot/oracle-bot.service`
|
||||
|
||||
- [ ] **Step 1: Write systemd service file**
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Oracle Bot — Mortdecai Mind's Eye
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/opt/oracle-bot
|
||||
ExecStart=/usr/bin/node server.js
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
Environment=PORT=3333
|
||||
Environment=MC_HOST=192.168.0.244
|
||||
Environment=MC_PORT=25568
|
||||
StandardOutput=append:/var/log/oracle-bot.log
|
||||
StandardError=append:/var/log/oracle-bot.log
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write deploy instructions in README**
|
||||
|
||||
Create `oracle-bot/README.md` with:
|
||||
- How to deploy to CT 644
|
||||
- Caddy config for mind.mortdec.ai
|
||||
- How to test locally
|
||||
- How to check health
|
||||
|
||||
- [ ] **Step 3: Add Caddy config snippet**
|
||||
|
||||
For CT 600 Caddyfile:
|
||||
```
|
||||
mind.mortdec.ai {
|
||||
@internal path /trace /command
|
||||
respond @internal 404
|
||||
|
||||
reverse_proxy 192.168.0.244:3333
|
||||
}
|
||||
```
|
||||
|
||||
Note: WebSocket upgrade is handled automatically by Caddy's reverse_proxy.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add oracle-bot/oracle-bot.service oracle-bot/README.md
|
||||
git commit -m "feat(oracle): systemd service + Caddy config + deploy docs"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Integration Test — End to End
|
||||
|
||||
- [ ] **Step 1: Start the bot locally**
|
||||
|
||||
Run: `cd oracle-bot && node server.js`
|
||||
Expected: Bot connects, logs spawn position, server listening on 3333
|
||||
|
||||
- [ ] **Step 2: Open frontend in browser**
|
||||
|
||||
Navigate to `http://localhost:3333`
|
||||
Expected: Dark themed page, "MORTDECAI — MIND'S EYE" title, idle mode, player dots visible
|
||||
|
||||
- [ ] **Step 3: Simulate a trace event**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3333/trace \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"tool":"rcon.execute","input":{"command":"give slingshooter08 minecraft:diamond 16"},"ok":true,"step":0,"mode":"god","player":"slingshooter08","session_id":"test-001"}'
|
||||
```
|
||||
|
||||
Expected: Frontend switches to god mode (orange), trace appears in sidebar, map centers on player
|
||||
|
||||
- [ ] **Step 4: Simulate multiple traces (tool chain)**
|
||||
|
||||
```bash
|
||||
for tool in "world.player_info" "world.nearby_entities" "rcon.execute" "journal.write"; do
|
||||
curl -s -X POST http://localhost:3333/trace \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"tool\":\"$tool\",\"input\":{},\"ok\":true,\"step\":$((RANDOM % 8)),\"mode\":\"god\",\"player\":\"slingshooter08\",\"session_id\":\"test-001\"}"
|
||||
sleep 1
|
||||
done
|
||||
```
|
||||
|
||||
Expected: Each trace appears in the sidebar sequentially, map updates with each event
|
||||
|
||||
- [ ] **Step 5: Verify idle timeout**
|
||||
|
||||
Wait 10 seconds after last trace.
|
||||
Expected: Frontend fades back to idle mode (muted colors)
|
||||
|
||||
- [ ] **Step 6: Test health endpoint**
|
||||
|
||||
Run: `curl -s http://localhost:3333/health | python3 -m json.tool`
|
||||
Expected: JSON with botConnected=true, wsClients=1 (your browser)
|
||||
|
||||
- [ ] **Step 7: Final commit**
|
||||
|
||||
```bash
|
||||
git add -A oracle-bot/
|
||||
git commit -m "feat(oracle): Oracle Bot v0.1.0 — Mind's Eye live viewport"
|
||||
```
|
||||
@@ -0,0 +1,237 @@
|
||||
# Oracle Bot — Mortdecai Mind's Eye
|
||||
|
||||
**Date:** 2026-03-22
|
||||
**Status:** Approved design, pending implementation
|
||||
**Public URL:** `mind.mortdec.ai`
|
||||
|
||||
## Summary
|
||||
|
||||
A live HTML5 viewport that renders what the Mortdecai AI model "sees" during Minecraft server interactions. An invisible spectator bot (mineflayer) maintains real-time world state, the gateway streams tool traces to it, and browsers connect via WebSocket to watch the AI think.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser (mind.mortdec.ai)
|
||||
↕ WebSocket (ws://CT644:3333)
|
||||
Oracle Bot (Node.js, single process)
|
||||
├── MC Client (mineflayer, spectator mode)
|
||||
├── Vision Server (Express + ws, port 3333)
|
||||
├── Trace Receiver (POST /trace from gateway)
|
||||
└── Command API (POST /command, future tool integration)
|
||||
↕ MC Protocol (offline auth)
|
||||
Paper Server (1.21, CT 644:25568 dev)
|
||||
```
|
||||
|
||||
**Approach:** Smart Bot is the Server (Approach 2). One Node.js process handles MC connection, WebSocket streaming, and HTTP endpoints. Designed to evolve into a gateway tool (Approach 3) where the AI controls the bot directly.
|
||||
|
||||
## Bot Core
|
||||
|
||||
### Three roles in one process:
|
||||
|
||||
1. **MC Client** — mineflayer bot in spectator mode. Maintains live chunk cache, entity list, player positions. Username: `OracleBot`. Connects to dev server (port 25568, offline auth).
|
||||
|
||||
2. **Vision Server** — Express + ws. Serves the HTML5 frontend. Streams world state and tool traces to connected browsers via WebSocket on port 3333.
|
||||
|
||||
3. **Trace Receiver** — `POST /trace` endpoint. Gateway calls this (fire-and-forget) on every tool invocation during the model-driven tool loop.
|
||||
|
||||
### Future-ready command API:
|
||||
|
||||
- `POST /command` — accepts instructions: `{action: "follow", target: "player"}`, `{action: "scan", center: {x,y,z}, radius: 20}`
|
||||
- Day one: only `follow` and `scan` implemented
|
||||
- Endpoint exists so the gateway can later call it as a tool (`oracle.scan`, `oracle.look`)
|
||||
|
||||
### Bot behavior:
|
||||
|
||||
- On connect: spectator mode, fly to first online player
|
||||
- Follows the player the AI is currently interacting with (switches on trace events)
|
||||
- On idle: parks at last active player or world spawn
|
||||
|
||||
## Data Flow & States
|
||||
|
||||
### Two modes:
|
||||
|
||||
**Idle Mode (no active trace):**
|
||||
- Bot parks at last active player position
|
||||
- Low-frequency heartbeat to browsers: player list, positions, time, weather
|
||||
- Update rate: ~5 seconds
|
||||
- Frontend: calm ambient view, slow-updating minimap, player dots
|
||||
|
||||
**Active Mode (trace incoming):**
|
||||
- Gateway fires `POST /trace` with tool call data
|
||||
- Bot teleports to relevant player
|
||||
- Scans burst of chunk data around player
|
||||
- High-frequency updates: blocks, entities, tool trace overlay
|
||||
- God mode: dramatic visual (golden glow, Sethian orange accents)
|
||||
- Sudo mode: clinical/technical (grid overlay, command syntax)
|
||||
- Persists 10s after last trace, then fades to idle
|
||||
|
||||
### WebSocket message types:
|
||||
|
||||
```javascript
|
||||
// Heartbeat (idle)
|
||||
{type: "heartbeat", players: [{name, x, y, z}], time: 6000, weather: "clear"}
|
||||
|
||||
// World snapshot (active)
|
||||
{type: "world", center: {x, y, z}, blocks: [{x, y, z, type}], entities: [{type, x, y, z, count}]}
|
||||
|
||||
// Tool trace event (active)
|
||||
{type: "trace", tool: "world.scan_area", input: {...}, result: {...}, step: 2, mode: "god"}
|
||||
|
||||
// Mode change
|
||||
{type: "mode", mode: "god"|"sudo"|"idle", player: "slingshooter08"}
|
||||
```
|
||||
|
||||
## Frontend (HTML5 Canvas)
|
||||
|
||||
### Single page, no build step. Pure HTML5 Canvas + vanilla JS.
|
||||
|
||||
**Layout:**
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┬──────────────────┐
|
||||
│ │ TOOL TRACE │
|
||||
│ WORLD MAP │ │
|
||||
│ (2D top-down tiles) │ [scan_area] ● │
|
||||
│ │ [rcon.exec] ● │
|
||||
│ ○ player dots │ [journal] ● │
|
||||
│ █ blocks colored by type │ │
|
||||
│ ◇ entities │ step 3/8 │
|
||||
│ │ │
|
||||
├─────────────────────────────────┤ │
|
||||
│ STATUS BAR │ │
|
||||
│ Mode: GOD | Player: sling... │ │
|
||||
│ HP: 20 | Pos: (12, -60, 15) │ │
|
||||
└─────────────────────────────────┴──────────────────┘
|
||||
```
|
||||
|
||||
### Visual modes:
|
||||
|
||||
- **Idle:** Dark muted palette, slow pulse animation. Sleeping eye aesthetic.
|
||||
- **God active:** Sethian orange (#D35400), golden particles on commands, dramatic god message text. Blocks glow where AI acts.
|
||||
- **Sudo active:** Cool blue/green terminal aesthetic, monospace overlays, precise grid. Clinical.
|
||||
|
||||
### Block rendering:
|
||||
|
||||
- Each block type → color (stone=gray, dirt=brown, water=blue, redstone=red, air=transparent)
|
||||
- Top-down slice at player Y level (configurable)
|
||||
- Entities as icons/dots with distance rings
|
||||
- Scanned areas pulse/highlight as tool traces arrive ("AI is looking here")
|
||||
|
||||
### Branding:
|
||||
|
||||
- Font: Rajdhani Bold
|
||||
- Primary accent: Sethian orange (#D35400)
|
||||
- Background: dark (#1a1a2e)
|
||||
- Title: "MORTDECAI — MIND'S EYE"
|
||||
- Subtle eye/pyramid motif
|
||||
|
||||
## Security & Resilience
|
||||
|
||||
### Public vs internal endpoints:
|
||||
- **Public (via Caddy):** WebSocket `/ws`, static files `/`, `/index.html`
|
||||
- **Internal only (localhost):** `POST /trace`, `POST /command` — Caddy must NOT proxy these. Gateway calls them on localhost:3333 directly.
|
||||
- WebSocket: max 100 concurrent connections, per-IP cap of 5. Excess connections get 429.
|
||||
|
||||
### Caddy config:
|
||||
```
|
||||
mind.mortdec.ai {
|
||||
reverse_proxy /ws localhost:3333
|
||||
reverse_proxy / localhost:3333 {
|
||||
# Only serve static files and WebSocket, not /trace or /command
|
||||
}
|
||||
@blocked path /trace /command
|
||||
respond @blocked 404
|
||||
}
|
||||
```
|
||||
|
||||
### Chunk loading after teleport:
|
||||
- After bot teleports to a player, wait 2 seconds for chunk packets before scanning
|
||||
- `world-state.js` tracks chunk load events and exposes `awaitChunksLoaded(center, radius, timeoutMs)`
|
||||
- If timeout expires, scan with whatever chunks are loaded (partial data is better than no data)
|
||||
|
||||
### Spectator mode enforcement:
|
||||
- Add to `mc_aigod_paper.py` PlayerJoinEvent: if player name is `OracleBot`, set gamemode spectator before spawn
|
||||
- Fallback: bot self-executes `/gamemode spectator OracleBot` via chat on spawn event
|
||||
|
||||
### Bot reconnection:
|
||||
- On `kicked` or `end` event: exponential backoff reconnect (1s, 2s, 4s, 8s, max 30s)
|
||||
- Broadcast `{type: "status", connected: false}` to all browsers on disconnect
|
||||
- Frontend shows "Bot offline — reconnecting..." overlay with pulse animation
|
||||
- On reconnect: broadcast `{type: "status", connected: true}`, resume normal flow
|
||||
|
||||
### Payload limits:
|
||||
- World snapshots: max 32x32x1 top-down slice (1,024 blocks). Air blocks excluded.
|
||||
- Delta compression: after initial snapshot, only send changed blocks
|
||||
- Max WebSocket frame: 64KB. If payload exceeds, chunk into multiple messages.
|
||||
|
||||
### Multiple simultaneous sessions:
|
||||
- Trace events include `session_id` and `player` fields
|
||||
- Bot follows the most recent trace's player
|
||||
- Frontend tool trace panel shows all active sessions, color-coded by player
|
||||
- If two sessions overlap, traces interleave in the timeline (both visible)
|
||||
|
||||
### Message versioning:
|
||||
- All WebSocket messages include `v: 1` field
|
||||
- Frontend ignores messages with unknown `v` values gracefully
|
||||
|
||||
## Deployment
|
||||
|
||||
**Location:** CT 644 (same container as MC servers + gateway). Lowest latency.
|
||||
|
||||
**Public access:**
|
||||
```
|
||||
mind.mortdec.ai → Caddy (CT 600) → CT 644:3333 (WebSocket upgrade)
|
||||
```
|
||||
|
||||
- No Authelia — fully public
|
||||
- DNS: CNAME to Caddy ingress
|
||||
|
||||
**Gateway integration (minimal):**
|
||||
- One addition to `langgraph_gateway.py`: fire-and-forget POST to `http://localhost:3333/trace` after each tool call
|
||||
- Also POST on session start (mode + player) and session end (final response)
|
||||
- Non-blocking: try/except with 1s timeout. If bot is down, gateway doesn't care.
|
||||
|
||||
**Process management:**
|
||||
- systemd service: `oracle-bot.service`
|
||||
- Auto-restart on crash
|
||||
- Logs: `/var/log/oracle-bot.log`
|
||||
|
||||
**Server-side setup:**
|
||||
- Paper server needs to `/gamemode spectator OracleBot` on join (command block or plugin event)
|
||||
|
||||
## Future Evolution
|
||||
|
||||
### Phase 2: Gateway tool integration
|
||||
- `oracle.scan` tool — model queries bot's chunk cache instead of RCON. Faster, richer.
|
||||
- `oracle.look` tool — bot teleports to coords and returns what it sees.
|
||||
- `POST /command` endpoint (built day one) becomes the tool backend.
|
||||
|
||||
### Phase 3: Multi-bot fleet
|
||||
- Oracle spawns additional bots on command
|
||||
- Model dispatches: `oracle.dispatch({task: "watch", target: "SwiftWolf"})`
|
||||
- All bots feed into same vision server → same frontend
|
||||
|
||||
### Phase 4: Multimodal training capture
|
||||
- Frontend frames captured as screenshots paired with model decisions
|
||||
- Builds (visual_state, model_action) dataset for multimodal fine-tuning
|
||||
- Mind's Eye becomes training data for visual understanding
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Bot + Server:** Node.js, mineflayer, express, ws
|
||||
- **Frontend:** HTML5 Canvas, vanilla JS, WebSocket API
|
||||
- **Integration:** HTTP POST (gateway → bot), WebSocket (bot → browser)
|
||||
- **Deployment:** systemd on CT 644, Caddy reverse proxy
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
oracle-bot/
|
||||
├── package.json # entrypoint: "main": "server.js"
|
||||
├── server.js # ENTRYPOINT — express + ws, requires bot.js, serves frontend
|
||||
├── bot.js # mineflayer connection, spectator, chunk tracking
|
||||
├── world-state.js # abstracted world state (blocks, entities, players)
|
||||
├── public/
|
||||
│ └── index.html # single-file frontend (HTML + Canvas + JS + CSS)
|
||||
└── README.md
|
||||
```
|
||||
Reference in New Issue
Block a user