Login
You're viewing the mastodon.iow.social public feed.
  • 4 mice in a php filealgernon@come-from.mad-scientist.club
    Oct 23, 2026, 1:01 AM

    @dee

    There's no real way to defend against that except to make the website private and only allow access to those you validate IRL...

    There's a neat trick, though. As far as I observed, the vast majority of those real browsers from residential IP scrapers are all following a crawling queue collected by simpler, easier to identify bots. So if you can poison the queue, you can block an IP after looking at one request path. That's a big save. No need to serve those, no need to do a TLS handshake either.

    I'm trapping identifiable bots in a maze of garbage, where every link carries a poison ID I can identify trivially. The residential proxies virtually disappeared, my firewall takes care of them.

    I still see ~5m requests a day across the few dozen sites I host, but that's a far cry from the 110m1 I've seen before. I can handle a constant hum of 50-60 req/sec, that keeps their queues poisoned.

    It still sucks I have to do this, but the problem can be mitigated, in my experience.

    1. "only" 110m, because my server fell over under the load. ↩︎

    💬 1🔄 0⭐ 0

Replies

  • Andrea Pappacodatachi@infosec.exchange
    Oct 23, 2026, 1:29 PM

    @algernon please tell me more! My local hacklab receives way less requests (for now), yet our Git instance (and sometimes more stuff) is constantly going down :(

    How are you poisoning the queue? How do you even identify that something is building the queue?

    💬 0🔄 0⭐ 0
  • 4 mice in a php filealgernon@come-from.mad-scientist.club
    Oct 24, 2026, 7:58 PM

    @tachi

    please tell me more

    I wrote about my approach at length here, but the TLDR is that I use iocaine to serve identifiable bots garbage, and to firewall the poison-hitters off.

    How are you poisoning the queue?

    I serve identifiable bots an infinite maze of garbage, where every URL has a substring that never appears outside of the maze.

    It doesn't completely get rid of the bots - I need to let some hit the maze to keep the queue poisoning alive, but it drastically reduces the crawlers I need to serve, and the crawlers that even see what's behind this firewall of sorts.

    How do you even identify that something is building the queue?

    I don't need to identify that programmatically. I observed this happen through trial & error, and by analyzing a metric ton of logs.

    Initially, I didn't try to poison the queue, I didn't know there was one. I just served random garbage, with URLs randomly chosen by combining 1-3 words from a wordlist. A few days later, I observed Chromes from residential IPs hitting URLs that were generated for an identifiable bot a few days earlier1. Then I adjusted my generator to include an identifiable substring, and voilá, random bots were hitting that within a few days, and pretty much every Chrome (its always Chrome :P) was hitting the poisoned URLs. Apart from identifiable bots and human traffic, nothing seemed to go for URLs that exist outside of the maze.

    I even tried placing legit links onto pages that are served when hitting a posioned URL, and nothing followed those links.

    1. I checked my logs when said URL was first served. ↩︎

    💬 1🔄 0⭐ 0
  • Andrea Pappacodatachi@infosec.exchange
    Oct 24, 2026, 8:04 PM

    @algernon thanks for the reply! If you don't mind, I'll share this with the hacklab so that we can look at this together.

    Really cool analysis though! Happy to see that we are not the only ones trying to think at something "new" to survive them :)

    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0