Quoting Florian Herrengt

Florian Herrengt warns that over-reliance on AI in software development is creating “convoluted” systems that are incomprehensible to human engineers, effectively “removing the middle class of software engineering.”
The scenario illustrates developers blindly trusting AI (e.g., “Claude seems very confident” despite unknown veracity) to fix bugs and explain data origins, leading to a critical loss of fundamental human understanding of the codebase.
The practical warning is that unchecked AI assistance can lead to severe “cognitive debt,” where projects become so complex and layered that no human team member can understand or effectively debug them, resulting in persistent, unfixable issues.