The AI Harness Audit: Clean Your Setup Before You Upgrade

YouTube/Podcast/Substack · YouTube · Podcast · Read

• AI harnesses (custom instructions, project files, prompts, memory, skills, tools, permissions, checks) accumulate one correction at a time and degrade performance even as underlying models improve — Nate’s own setup had grown to 66 skill routes and 172 instruction files/assets, totalling 18,384 words loaded before reaching any platform-specific guidance.
• Concrete test: adding roughly 5,000 words of extra instructions to Fable 5 made it “think harder” and score better on analysis, but it failed actual delivery two runs out of three; a leaner, compact brief passed all three delivery tests.
• New named tools: “Clean My AI Harness — Claude Edition” and “Codex Edition” — runnable skills that map a user’s accessible AI setup, flag what’s dragging performance, and propose changes for approval.
• Frames a six-rule harness-maintenance discipline: give every instruction a single owner and purpose, and load specialist context on demand rather than keeping it always-resident.
• Practical takeaway: audit and prune accumulated AI configuration before every model upgrade — complexity accumulated through ad hoc fixes silently caps the benefit of better models.