You're viewing the frontrange.co public feed.

Federated feed Local feed

Dillo browserdillo@fosstodon.org
Oct 18, 2026, 12:39 PM
Moved our git web server from cgit to stagit to reduce load on the server caused by crawlers, as it only uses static files.
https://git.dillo-browser.org/dillo/
Roughly 99.9% of requests are bots fetching commits and files over and over again at a rate of 50 req/s despite our robots.txt excluding any crawler. Stagit only exposes a small number of files and commits so they are not reachable to bots.
Sadly, many anti-bot protections rely on JS or fingerprinting which come with their own set of problems.
💬 3🔄 3⭐ 0

Replies

rqmrqm@0xde501a7a.net
Oct 18, 2026, 12:54 PM
@dillo@fosstodon.org How about banning the /24 IP range of anyone who makes more than 10 requests 10 seconds?

💬 0🔄 0⭐ 0
Dillo browserdillo@fosstodon.org
Oct 18, 2026, 12:59 PM
@rqm they use residential proxies (i.e. hacked TVs and other rogue devices) so that each IP only makes one request that mimics a random legitimate browser.
💬 0🔄 0⭐ 0
Erik Play2Learnfallbackerik@mastodon.social
Oct 18, 2026, 1:00 PM
@dillo Strange that git crawlers use the http endpoint and not the git endpoint, which may be more efficient. Also does it make sense for a git crawler to ping this often? Even once a day or two days might be fine for most usecases.
💬 0🔄 0⭐ 1
DDDsenslayer@mastodon.social
Oct 18, 2026, 2:33 PM
@fallbackerik @dillo primarily automated crawlers using naive scraping policies and residential proxies (thanks IoT). Potentially believing they can get a "competitive advantage" by avoiding robots.txt and scraping any data not realizing robots.txt and git endpoints were developed for a reason.
https://people.kernel.org/monsieuricon/creepy-crawlies
https://weirdgloop.org/blog/clankers
💬 0🔄 2⭐ 1
オチュー (stop colonialism 🇵🇸🇸🇩🇳🇨 )otyugh@pouet.chapril.org
Oct 18, 2026, 1:18 PM
@dillo
If you can tell they're bots without JS (from useragent or numerous request I guess ?) why not use this ?
(a friend webmaster just blocked IPV4 and 99% of the bot crawl stopped since)
💬 0🔄 0⭐ 0
Dillo browserdillo@fosstodon.org
Oct 18, 2026, 1:19 PM
@otyugh https://fosstodon.org/@dillo/117462032652462765
💬 0🔄 0⭐ 0