↓Skip to main content
Zeitgeist — a spike by Chris Gathercole
  1. Creators/
  2. Simon Willison — Creator of Datasette; co-creator of Django/

Creepy crawlies

Blog · Watch/Read · (gathered 2026-09-10)

  • Abusive web crawlers, or “creepy crawlies,” are becoming a significant operational burden, consuming disproportionate server resources.
  • Konstantin Ryabitsev from git.kernel.org reports that they spend more CPU cycles rendering commits for scrapers than for all legitimate access (including git clones), with 14 CPU cores across 5 nodes constantly doing this.
  • Simon Willison expresses concern about this “background radiation” of abusive crawlers, particularly for his project Datasette, which serves a vast number of crawlable web pages.