AI agents can't yet do open-ended AI research

Newsletter · Watch/Read

• Frontier AI agents are not yet capable of conducting open-ended AI research, challenging the notion of imminent “recursive self-improvement (RSI)” and forecasts of explosive AI progress.
• Using “shadow evaluations,” where agents attempted to replicate unpublished research questions with significant resources, both agent-generated papers were “unambiguously rejected” by the original expert authors.
• Analysis of agent logs revealed key limitations including a lack of judgment, poor resource awareness (e.g., not spending budget), inability to creatively respond to feedback, and failure to effectively backtrack or follow concrete instructions.
• These findings suggest that progress on narrow, verifiable tasks may not lead to broad RSI, and fundamental bottlenecks in open-ended research (like creativity and judgment) could significantly dampen the overall pace of AI progress, illustrating the relevance of “Amdahl’s law.”