Tony Ojeda North Carolina, USA 5H AGO

Most AI agents are effectively static. You write their instructions, use them repeatedly, notice where they struggle, and manually tweak the prompt when something goes wrong. Valuable feedback from real work is often lost, so the same mistakes can keep happening. I created a self-improvement flywheel that uses actual agent performance data to improve agents over time. The system collects two kinds of evidence: - Task scores showing how well each agent performs across different quality dimensions - Run telemetry and review outcomes revealing recurring failures, coordination problems, and cases where actual behavior differs from expectations A scheduled weekly cycle analyzes that evidence, identifies patterns, creates improvement proposals, evaluates whether those proposals are safe and broadly applicable, updates agent instructions when appropriate, and measures whether those changes actually improve performance. The goal is not to let agents rewrite themselves freely. It is to create a controlled learning loop. Step-by-step: 1. Collect performance data while agents work. Score important outputs across consistent quality dimensions, and record useful execution telemetry such as failures, decisions, reviewer outcomes, and unexpected behavior. 2. Analyze performance trends on a recurring schedule. Calculate per-agent averages, identify weak dimensions, compare agents, and look for improvement or decline over time. 3. Mine run history for recurring patterns across multiple sessions, including agents that repeatedly struggle, low-quality runs, and cases where expected behavior differs from what actually happened. 4. Turn repeated problems into improvement proposals. Before proposing a change, inspect the agent’s current instructions so you do not add a rule that already exists. 5. Evaluate each proposal before applying it. Check whether the lesson is broadly useful, redundant with existing instructions, or in conflict with established behavior. 6. Separate low-risk and high-risk changes. Automatically apply additive or clarifying improvements. Escalate conflicting changes for human review instead of allowing the system to fundamentally change an agent’s behavior on its own. 7. Look for system-level problems. Analyze patterns across agents to identify quality gaps, missing capabilities, or coordination failures that cannot be fixed by changing one agent alone. 8. Apply approved improvements and preserve the history. Update the relevant agent instructions, archive the processed proposals, and version the changes so they remain inspectable and reversible. 9. Measure whether each change actually helped by comparing agent performance before and after the refinement. If quality does not improve, do not automatically assume the change was useful. 10. Repeat the cycle. As agents complete more real work, the system gathers more evidence and gets another opportunity to improve. Instead of treating agent instructions as static prompts, I turned them into a continuously improving system: Work → Evaluate → Find Patterns → Propose Changes → Refine → Measure → Repeat The important part is that the loop is evidence-driven and controlled. Agents improve from real usage, but low-confidence or behavior-changing updates still require judgment rather than being applied automatically.