New page on seirdy.one: Scrapers I block (and allow), with explanations.

I’ve replaced all the comments in my robots.txt file with a more readable and detailed web page on scrapers I block. It includes info on the multiple blocking-approaches and criteria I use, commonly-blocked scrapers I allow, and more fact-checking than most of the more comprehensive alternatives.

#RobotsTxt #Scrapers #POSSE