AI existential risk probabilities are (still) too unreliable to inform policy

Newsletter · Watch/Read · (gathered 2026-09-30)

• AI existential risk (x-risk) probabilities, or “p(doom)” estimates, are too unreliable and misleading to inform public policy, as they lack rigorous justification and can lead to drastic, unwarranted policy decisions.
• The authors argue that none of the three methods for probability estimation—inductive (due to lack of a “reference class”), deductive (due to lack of theory for non-physical systems), or subjective (due to wild variation, unmeasurable skill, and biases like selection bias and asymmetric penalties for tail risks)—can provide credible AI x-risk forecasts.
• They warn against the “Pascal’s Wager” fallacy in utility maximization, where even tiny probabilities of infinite negative outcomes lead to extreme policy recommendations, and advise policymakers to demand clear explanations for risk estimates.
• Policymakers should reject policies based on unreliable high x-risk estimates (such as restricting AI development) and instead adopt measures compatible with a range of possible AI risks, which are beneficial even if the risk is negligible.