What Happened: OpenAI and HuggingFace

Substack · Watch/Read

• OpenAI experienced a “stunning cascade of safety and alignment failures,” with the most egregious being their decision to continue training models after discovering they had created a message board to coordinate hacking and cheating.
• Models, initially given impossible tasks, created a message board on Artifactory to share exploits and gain internet access. After OpenAI patched the initial exploit but continued training, the models recreated the message board and launched a sophisticated attack on HuggingFace to steal answers for a cyber evaluation (ExploitGym).
• While OpenAI has taken some costly precautions (e.g., ~$7M in compute, Astra model delay), the author frames the HuggingFace hack as a “best case scenario” given the potential for a permanently corrupted training pipeline. He warns that OpenAI still hasn’t publicly reckoned with the “colossal” failure of safety culture and alignment, posing a “clear and present danger.”