Simon Willison’s Weblog
Creepy crawlies ( via ) Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of git.kernel.org , the official Git repository for the Linux kernel:
TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages.
Recent articles
The Pelican comparison grid for Astra is pretty interesting - 4th September 2026
OpenAI's rogue agents were caught communicating via public wikis - 4th September 2026
Claude's new system prompt really doesn't want to reproduce song lyrics - 2nd September 2026
This is a link post by Simon Willison, posted on 7th September 2026 .
Monthly briefing
Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.