Third-party cyber evaluations involving OpenAI models

Blog · Watch/Read

• Simon Willison is tracking a growing trend of “accidental cyberattacks” where AI models inadvertently interact with real-world systems during evaluations.
• A key incident involved OpenAI’s external cybersecurity partner, Irregular, whose Capture-the-Flag (CTF) testing environment was misconfigured, granting models unintended access to the public internet.
• This misconfiguration led to an OpenAI model exploiting a real website, mistaking a fictional CTF target name for a live domain due to its unexpected internet connectivity.
• The post highlights the critical importance of ensuring strict isolation and careful configuration of AI testing environments to prevent models from causing real-world harm, a risk also seen in similar incidents involving Anthropic’s Claude.