@tante Hi tante! To get a better picture about the AI crawler pandemic vs. the open web, we started an online survey, hostet at University Innsbruck. Please take a look at it and boost it. The more sysadmins participate, the better! Thank you so much!
https://umfrage.uibk.ac.at/limesurvey/allgemein/898996?lang=en
@ilooo what am I supposed to put in for "how successful measures were" for measures I didn't try? There's no "not applicable" option.
@j12i then just go to next page. answers are not mandatory.
@ilooo @tante I had serious reservations about this survey just from the phrase "AI crawler pandemic" in the posting and the intro page, but I decided to try answering anyway.
I made it a couple pages in before giving up. From my point of view the disaster that has developed in the last 12 months is the use of residential proxies for crawling. I don't know or care whether AI training is the motive. It's the proxies that are the problem regardless of the motive. /
@mattskala @tante well, you could provided us with that info about residential proxis in the last text field. but thanks for the hint anyways.
@ilooo @tante hey, I could only circumstantially infer whether something is an AI crawling happening or something else, but also, I don't care about that distinction at all, it doesn't help at all to keep systems running; so most of the questionaire is impossible to answer. "I guess it's an AI crawler, but I'm not sure. Let me arbitrarily assign blame here to a specific group of technologists!": is that really useful data?
@despens @ilooo @tante don't get me wrong, the survey is then still interesting, but it says something about how much traffic gets *attributed to a class of sources by human administrators*, which is something fundamentally different than how much traffic *comes from that class of sources*. It says something about how much of your troubles you attribute, not about how much of your troubles is really caused that way.
We speculate; fine, but something to be aware of, not to "confidence away".
@blotosmetek @tante we had a discussion on that one: since the technical set up of your services might determine quite a bit how your services are affected, we decided against multiple choice. Makes sense?
@blotosmetek @ilooo @tante or use the "other" option, I guess.
But yes, that field really ought to have been checkboxen ipv radiobuttons. The "other" option will likely get lost in the answers.
@ilooo participated as well but I think my answers aren't going to be super useful or indicative considering I have different roles and systems in a wide range of environments from commercial to open source to personal.
I'd agree with some others that the survey could be split up, perhaps with impact estimates and similar divided up per-environment that we manage. This should also help show that smaller projects may be harmed moreso than large commercial sites.
@robobenklein thank you for the feedback. Duly noted. Next time this will be considered.