Unfortunately the #OpenStreetMap wiki is very slow today. We are fighting an aggressive web scraper bot. 10,000 of IPs involved. Randomised User-Agent. Ignoring robots.txt #aibot #ddos
Update: Fixed. We've been able to mitigate the bot traffic. #fail2ban
May 29, 2025, 12:16 PM@osm_tech
Argh stupid morons.
@osm_tech I noticed. Thank you for your work!
Fine to stick Anubis or some other system in front of it IMO: it never causes me any problems even on slightly weird (desktop) browsers.
Or some sort of quiet scraped data poisoner made just for stuff ignoring robots.txt, maybe,,,
May 29, 2025, 12:47 PM@osm_tech Like others are saying, implement this: https://github.com/TecharoHQ/anubis
@osm_tech distributed maps n stuff ay
@osm_tech for those who do you need to access the wiki, know there is a cached version made by @luc
https://lgms.nl/p/osm/cached-wiki/?page=Main_Page
Read only, but very fast
#OpenStreetMap
And if you're using a search engine that supports redirects like #Kagi
You can actually automatically have any search result open on said cached version:
https://en.osm.town/@thibaultmol/114462842334898485
May 29, 2025, 1:57 PM
@osm_tech Maybe https://github.com/dgl/haphash could help.
@osm_tech I'm going to resist the urge to say something really unpleasant but I will say this.
It's the open internet that suffers from the VC-backed LLM / AI bubble. They could *not* care less.
"If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality." - Drew DeVault, https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html
@osm_tech@osm.town I am not free of blame having used these machines, so please don't take this as me trying to take some sort of moral high ground. But it's the first time since I started using computers 40 years ago that I have been so disappointed in technological "advancement".
@mvexel This is a whole new level of abuse we're seeing. 32,000+ IPv4 /24 ranges now blocked. Requests being proxied via 2 million+ unique IP addresses. Server ranges and home users. Annoyingly anonymous.
@osm_tech Incredible. Thanks for keeping OSM working for all of us.
@osm_tech @mvexel 👋 Researcher from GreyNoise.io here. Any chance we might be able to collab and get the bad IPs + (if possible timestamps? I realize it's prbly a [residential] proxy service for-hire being used by AI startups, but that'd be a ton of firepower for a malicious++ campaign and my team can work with region CSIRTs to try to make a difference. cc: @Dio9sys
@osm_tech Anubis is the solution
@osm_tech
For situations like this you could have a site full of random text where you could temporarily redirect to. Legitimate users would see that too, but you could hope they'd understand the temporary measure.
From what I understand about how LLMs work, feeding them with large masses of random words could do some serious damage.
@osm_tech If you add an innocent link to the page, style it transparent using CSS, and add it to robots.txt as an exclusion. Then you can temporarily block every IP address that requests it.
Or you can skip adding it to robots.txt if you want to temporarily ban all crawlers.
@carlosefr We have a good regex match on the abuse. 32,000+ IPv4 /24 ranges now blocked. Requests being proxied via 2 million+ unique IP addresses and growing.
@osm_tech is it feasible for you to block the entire AS of the source IP? Moving IPs is easy, moving AS not as much...
@siguza 32,000+ IPv4 /24 ranges now blocked. Requests being proxied via 2 million+ unique IP addresses.
@siguza @osm_tech the scrapers usually rely on “residential proxies”, companies that pay ordinary people to launder their traffic through their home Internet connections (when they’re not botnet operators). There won’t be a single ASN.
I’m afraid Anubis or equivalent is the only real option at this point.
@fazalmajid @siguza Suspicion is the proxies are embedded in a seemly innocent "free / gratis" software. Mostly Brazil at the moment.
Also, like, isn't this open data, where they could've simply asked for it, and gotten it in a less-harmful way?
@MegaMichelle Yes, we even publish daily exports: https://wiki.openstreetmap.org/dump/
Maybe they just wanted to make sure that the dump matches the pages online 😆
Those robots are the new plague. Also they're not paying for the traffic they cause.
@osm_tech I probably missed it explained somewhere, but (given repeated issues), could not the wiki be put behind CDN to help offload those attacks?
@osm_tech they literally just could’ve torrented it
@osm_tech Do you guys want a free copy of my book on DDoS attacks and mitigation? https://buchshop.bod.de/ddos-understanding-real-life-attacks-and-mitigation-strategies-stefan-behte-9783819226212 For now just print, but eBook will be available in 1-2 weeks according to my publisher. 😊 #ddos #ddosattack
@osm_tech Thanks and well done 👍
@osm_tech good work. Thank you
@osm_tech Thank you for your hard work.
May 29, 2025, 7:42 PM@osm_tech I'd love to hear more of a breakdown of what you did, when y'all get a chance! (currently also dealing with this, already using fail2ban)
