Login
You're viewing the front-end.social public feed.

Replies

  • 💬 0🔄 0⭐ 0
  • Marginaliamarginalia@mastodon.social
    Sep 8, 2026, 12:32 PM

    @hvdsomp

    The URL obfuscation and redirect links are an emergency anti-scraping measure to make the search results less palatable to the bots. It's triggered when there's a sustained search rate of more than 5 queries per second, which is like 3x baseline traffic.

    Was seeing upwards of 160 searches per second from bots the other week so I put this in to try to dissuade them.

    💬 2🔄 1⭐ 0
  • 💬 0🔄 0⭐ 0
  • Marginaliamarginalia@mastodon.social
    Sep 8, 2026, 2:53 PM

    @avirr @hvdsomp

    Like for context, I disabled some of the mitigations just now since so many people were popping by. This is how bad it immediately gets when they're off:

    from 2 req/s to 300 req/s in a heartbeat
    💬 0🔄 1⭐ 2
  • herberthvdsomp@w3c.social
    Sep 8, 2026, 4:53 PM

    @marginalia Thanks for the clarification! Makes total sense. And thanks, especially, for creating this search engine. I did a vanity search. Obviously I did. And found a vast collection of results referring to stuff I had long forgotten about. And that don’t show up in any mainstream search engine. Quite amazing. Thank you!

    💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0