(long one, I am sorry)

Reading around the internet I see a lot of people expressing their extreme displeasure at how everything on the internet now seems to show a "verifying your browser" type page, and while this is extremely annoying I assure you from the web sites point of view this is basically existentially required.

Allow me to go into more detail from the point of view of the company that I run, and I'm sure the story is similar in many different places.

On an average day bgp.tools gets about 500~ users or so actively using the website at any given time, and let's assume this results in a request rate of ~10 RPS of dynamic requests (this is not the actual number but it's easier to just provide round benchmark numbers), so basically this excludes CSS/JS/Images/etc.

At the time of writing bgp.tools is "shooting down" about ~40 bot scraper connections _per second_, and the connection attempts the do get through tend to go on to each try and perform ~5 RPS until stopped. To add on top these things continuously do cache misses, which makes sense given at that point is to try and extract as much information from you, for no cost, and as much/fast as possible.

This is very quickly unsustainable.

The problem is that these scrapers are aware that they are unwelcome, and so they will happily do things like: (listed in the order of most commonly done)

A) Use normal browser user agent strings so it's not obvious from basic access.log files

B) Emulate real browser TLS handshakes to evade JA-3 and similar mitigations

C) Use proxies that have lease lines with popular eyeball ISP providers and rented IP space, so that AS blocks do not work

D) Use entire botnets of smart TVs and other compromised devices, mostly sitting behind "normal" networks

E) Use free mobile phone app/games with a "special" SDK as proxies

F) Run entire browser stacks purely to load pages in the most "authentic" way possible

With C+D+E traits combined, IP/Subnet/ASN blocks no longer work, with A+B you cannot easily identify what is what, so this leaves you in a dilemma:

If you run a website which serves information anonymously for free, you have a choice of three options:

1) Put everything behind a log-in barrier (this might destroy your business)

2) Purchase/rent more infrastructure purely for the purpose of serving bots, and at a significant multiplier of your existing infrastructure (this might destroy your business)

3) Use a "verify your browser" step to cut out 90%+ of this mess at the cost of annoying users a bit

(3) is the only really viable option for a lot of people, the pure scale of the scraping problem for some website is just not survivable at most peoples current scales.