Login
You're viewing the functional.cafe public feed.

Replies

  • Aug 7, 2026, 4:32 PM

    @pat_dev I understand wat you mean and agree with your conclusion. But requesting to scrape a website until it cannot respond is not something positive despite the intentions they might have.

    💬 1🔄 0⭐ 0
  • Aug 7, 2026, 4:44 PM

    @Lluis_Revilla Sure, I also didn't say that scraping is good. It's simply how the web works these days (and for the foreseeable future). This applies to every Git instance and website out there. If a site contains code, it simply becomes more interesting for these actors.

    A baseline protection is generally required these days in my opinion, no matter what type of site one is hosting.

    💬 1🔄 0⭐ 0
  • Aug 8, 2026, 9:19 AM

    @pat_dev I thought the baseline was robots.txt and fail2ban... Which was wnough until recently. Why do we need something else now?

    💬 1🔄 0⭐ 0
  • Aug 9, 2026, 2:01 PM

    @Lluis_Revilla robots.txt is close to meaningless these days, most (evil) AI scrapers don't respect it. It's more of a gentlemen-like note at your door, but besides Google and a few friends, most don't care bout this anymore. Surely good to have one compared to none at all but the heavy traffic is not shielded by it.

    fail2ban helps for SSH (which is actually a different topic) as a baseline but takes a bit to be configured efficiently while at the same time preventing the risk from locking yourself out.

    As always it "depends" - on the type of site you're hosting and your overall performance goals. But the scraping protection game definitely became (way) more rough and complex in the last 2 years.

    💬 0🔄 0⭐ 0