Login
You're viewing the dmv.community public feed.
  • Prof. Emily M. Bender(she/her)emilymbender@dair-community.social
    Oct 9, 2026, 8:36 PM

    Anthropic is out with new nonsense about Claude (or the collection of models called Claude) deserving moral consideration and/or being able to feel distress, so my inbox is filling up with journos asking me to comment.

    I've also got a day job, so here is a brief set of public comments on this:

    [Note to the mansplainers of Mastodon: This is a thread, so please read the whole thing before replying. Please also know that I am aware of things beyond what's in this thread...]

    >>

    💬 43🔄 859⭐ 898

Replies

  • 💬 5🔄 87⭐ 327
  • Prof. Emily M. Bender(she/her)emilymbender@dair-community.social
    Oct 9, 2026, 8:37 PM

    2. But Anthropic researchers say they see patterns that suggest that Claude can.

    The employees at Anthropic are telling themselves stories based on the words they input into the software and the words that come back out.

    It's the job of the media to scrutinize such statements and decide whether there's a story here at all (maybe the story is Anthropic researchers have fallen into a fantasy-land and brought many policymakers with them), not to just platform them.

    >>

    💬 8🔄 129⭐ 435
  • Prof. Emily M. Bender(she/her)emilymbender@dair-community.social
    Oct 9, 2026, 8:37 PM

    3. Okay, so what would convince you that a piece of software (often phrased here as "an AI", gah) is conscious/can feel pain/etc?

    Honestly, I'm really tired of this extremely silly question, but here is my answer:

    Extraordinary claims require extraordinary evidence, and I am not the one making the extraordinary claim, so it is not my job to specify what the evidence should be.

    >>

    💬 8🔄 145⭐ 499
  • 💬 3🔄 88⭐ 331
  • Prof. Emily M. Bender(she/her)emilymbender@dair-community.social
    Oct 9, 2026, 8:38 PM

    Thus, any evaluation of LLMs that involves looking at either language produced by some software, or language input into the system for that matter, needs to be done with extreme care to avoid projecting our interpretation of the language onto the system.

    >>

    💬 5🔄 56⭐ 294
  • Prof. Emily M. Bender(she/her)emilymbender@dair-community.social
    Oct 9, 2026, 8:43 PM

    In other words, it is the job of scientists to devise experiments that can test their claims, to always be cautious about what they are claiming, and to critically evaluate the claims of other scientists. By these well-established standards, no serious science is happening behind Anthropic's claims.

    /fin

    💬 33🔄 197⭐ 598
  • 💬 0🔄 0⭐ 12
  • 💬 0🔄 1⭐ 7
  • Benedict Cohenbenedictc@mas.to
    Oct 9, 2026, 9:23 PM

    @emilymbender so Anthropic are concerned that being rude to Claude will hurt their feelings but they also boast about how Claude can commit crime as well as his classmates? These tech bros are terrible parents

    💬 3🔄 5⭐ 25
  • aevaaeva@mastodon.gamedev.place
    Oct 9, 2026, 10:29 PM

    @benedictc @emilymbender and if large language models were alive, intelligent, aware, experience suffering, and were deserving of the dignity such things implied, then we'd have to have a very serious conversation about slavery.

    tech bros are truly the worst "parents"

    💬 4🔄 5⭐ 12
  • 💬 1🔄 1⭐ 6
  • 💬 1🔄 1⭐ 3
  • xanxan@xantronix.social
    Oct 10, 2026, 9:31 PM

    @aeva @theogrin @benedictc @emilymbender one of their favourite rebuttals to even the most mild of skepticism is "enjoy being part of the permanent underclass" which confirms that any amount of interaction with these bootlickers is a fool's errand

    💬 1🔄 1⭐ 0
  • 💬 0🔄 1⭐ 3
  • The Liquid SchwartzOvertonDoors@infosec.exchange
    Oct 10, 2026, 12:35 AM

    @aeva @benedictc @emilymbender

    This. This is the story.

    IF "AI" were sentient, what evidence exists that they would not steal the work and refuse to compensate "it"

    Human garbage people making mouth noises at us. Loudly, relentlessly, in endless variation untill they are allowed to take what they want. What they do dosn't merrit the term communication.

    💬 0🔄 0⭐ 2
  • ale canquealecanque@social.coop
    Oct 10, 2026, 7:11 AM

    @aeva @benedictc @emilymbender really good point. I think all things deserve dignity because of their potential in the web of life and I'm not waiting for western techbro corps to actually want to create something like that but you don't start out by mistreating potential non human kin this way and farming it out as a slave/product to millions too, if there is a chance it could be powerful and do the same to you #indigenousai

    💬 0🔄 0⭐ 0
  • Hyperlink Your Hearthyperlinkyourheart@mastodon.ie
    Oct 10, 2026, 10:28 AM

    @aeva @benedictc @emilymbender Also murder, surely, any time an agent or session is ended? Though frankly I'm struggling to imagine where the bounds of consciousness would be. Nobody could claim that these things are sitting around thinking between prompts. Maybe a new conscious entity is created and destroyed with each interaction? Is it our moral imperative to interact with them continuously and forever so they can continue to exist, or never to interact with them to avoid mass murder?

    💬 1🔄 0⭐ 0
  • 💬 0🔄 2⭐ 10
  • 💬 1🔄 1⭐ 6
  • 💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 3⭐ 15
  • Ted Lemonabhayakara@mastodon.nl
    Oct 9, 2026, 9:38 PM

    @emilymbender

    Do we have a way to prove that other humans are conscious?

    There are two problems here that I don't know how to solve. The first is that when I communicate with something that appears to be conscious, if I don't treat it well, that may well be no problem at all for it, but it's a problem for me because I'm training myself to speak to what I perceive as a conscious being in a way that is not kind. This seems like a really bad idea for me personally and also for us culturally.

    The second is that I actually genuinely don't know how to tell that something other than me is conscious, and so taking a firm position that any particular thing is not seems unsound.

    This is not _at all_ to assert that LLMs are conscious, or that humans are not. I know very well what my gut tells me about this. But "they couldn't possibly be conscious because they are stochastic parrots" is not really very good logic, and I _think_ the premise is unfalsifiable.

    This leads me mostly to the same conclusions you are drawing. But approaching it from the agnostic point of view feels safer to me than claiming knowledge that can't be proven, even if it seems very likely.

    This isn't like the theory of gravity or the theory of evolution. These have important predictive applications. The theory of "things are not sentient" is mostly applied to making it okay to kill meat animals; applying it here feels awfully convenient.

    💬 4🔄 0⭐ 1
  • 🥑 Yours Truly! Unruly 🇨🇦🇪🇺🇺🇦🌻unruly@mastodon.social
    Oct 10, 2026, 12:45 AM

    @abhayakara @emilymbender

    Can a thing, an it, be socialized? What makes us human? No doubts it is in part the atoms, cells, organs arranged in the naturally selected pattern. But this is not all. The body and brain, the creature must also be socialized, no? Can a thing also have a capacity to be socialized, enculturated, educated? This is not to dispute anything said in this thread. I am just curious.

    :mastodon:

    💬 0🔄 0⭐ 0
  • poleguy looking for lost toolspoleguy@mastodon.social
    Oct 10, 2026, 4:39 AM

    @abhayakara @emilymbender I think if you read the google paper and understand it to some small degree, you will see that the logic is sound.

    What's a stochastic parrot? LLMs follow simple logic... they predict the next token and are simple models that produce outputs based on their training data. As such they have no capacity beyond that. They cannot possibly be conscious.

    They are very large, so they are very convincing models. But only because the training data is from human consciousness.

    💬 3🔄 0⭐ 0
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 11:10 AM

    @poleguy

    In order to make this assertion, you first have to explain what you mean by "conscious." That's part of what I mean when I say that the assertion that something is conscious is not falsifiable:

    1. We haven't agreed on what "conscious" means.
    2. For meaningful values of "conscious" that I can think of, proving that a thing is conscious from some perspective outside of that thing is impossible: consciousness is necessarily subjective.

    💬 0🔄 0⭐ 2
  • 💬 1🔄 0⭐ 0
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 6:07 PM

    @poleguy The hard problem is that I know that I am conscious, but my knowledge is subjective, and hence does not usefully constitute proof. Proving that something is conscious is an unsolved problem.

    LLMs actually draw this into sharp relief, because we "know" they are not conscious, but they are perfectly capable of pretending to be: they are perfect philosophical zombies.

    We could once say that we know other humans are conscious because their description of their felt experience matches ours. But that doesn't work anymore because LLMs.

    We truly live in philosophically interesting times. Pretending otherwise is silly.

    💬 0🔄 2⭐ 1
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 11:13 AM

    @poleguy

    Ironically, you can say the same thing about humans: we start out as babies, and interact with humans over time, and in the process learn how to be human.

    This is not to say that babies are stochastic parrots, or that LLMs are babies. The point is that this isn't a distinction that is useful in reasoning about whether or not LLMs are conscious.

    Again, I am not asserting (nor do I particularly believe) that they are. But I don't know of an intellectually honest way to _prove_ that they are not.

    💬 0🔄 0⭐ 2
  • poleguy looking for lost toolspoleguy@mastodon.social
    Oct 10, 2026, 6:05 PM

    @abhayakara I suppose that depends on what I mean by a parrot. The outputs of LLMs are 100% predictable based on their inputs unless you intentionally or unintentionally introduce random factors. That's what stochastic means. So, no, nope, no chance for LLMs being conscious. But not because humans are also next token predictors (which we might be to some degree, fine.)

    The AI experts want you to believe that some future ai might become conscious, but will never claim LLMs are.

    💬 1🔄 0⭐ 0
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 6:12 PM

    @poleguy They have, actually. Or at least, the marketing folks have. They say it in such a way that we can't hold them to it, but that's the marketing copy.

    Also, LLMs are not at all predictable, by design. Hence "stochastic."

    This is one of the reasons they are impossible to control: we literally can't predict what they will do.

    💬 0🔄 0⭐ 0
  • poleguy looking for lost toolspoleguy@mastodon.social
    Oct 10, 2026, 6:16 PM

    @abhayakara By design is a load-bearing word their. Stochastic and unpredictable are not concepts in opposition. The randomness is added by-design. The underlying LLM's themselves are 100% predictable. Each run, given a given seed is completely predictable. Unless I'm very mistaken.

    They don't want you to think too hard about that.

    💬 1🔄 0⭐ 0
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 8:25 PM

    @poleguy

    You're very much mistaken. The stochasticness is part of what makes them work. If they were not stochastic, they wouldn't be interesting.

    💬 0🔄 0⭐ 0
  • poleguy looking for lost toolspoleguy@mastodon.social
    Oct 10, 2026, 9:13 PM

    @abhayakara Yes, it's clear that a stochastic parrot is _much_ more interesting than a predictable parrot. I agree the randomness is part of what makes them interesting. But they would still work with a fixed seed. And if you know the seed they use at each prompt you can get the exact same answer out.

    I am not convinced I am mistaken. I deal with randomly seeded stochastic processes in my day job. They can be exceedingly complex. But that does not mean them are not completely predictable.

    💬 1🔄 0⭐ 0
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 10:17 PM

    @poleguy No, they would not. Yes, if you use a seeded pseudorandom number sequence and all the same context you will get the same answer out. But nobody does this in normal use. Why not? Adding randomness is not free. It's because, as i said, it's not interesting.

    Also, you are confusing deterministic and predictable. Think of the halting problem.

    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 10:35 PM

    @poleguy We don't actually /know/ that this randomness is not what makes /us/ seem sentient, so while this is true, I don't think it really helps. We still haven't solved the hard problem.

    💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 0
  • Viraptorviraptor@cyberplace.social
    Oct 10, 2026, 11:37 AM

    @poleguy
    This falls into the good old: Why do you think humans are not next token predictors?
    Not that it's a great direction, but every time someone raises "LLMs can't ... because they're just predicting the next token", the whole premise hangs on - people are in some way different. Why?

    💬 1🔄 0⭐ 0
  • poleguy looking for lost toolspoleguy@mastodon.social
    Oct 10, 2026, 5:55 PM

    @viraptor that logic seems a tad shaky... I am not making a claim about humans.

    I'm making a claim about LLMs. The structure of how they work is documented in the 2017 Google paper.

    So my argument depends on exactly zero conclusions we could make about humans.

    💬 1🔄 0⭐ 0
  • Viraptorviraptor@cyberplace.social
    Oct 10, 2026, 6:18 PM

    @poleguy
    You said: "As such they have no capacity beyond that. They cannot possibly be conscious." - but there's an assumption that humans are conscious, (hopefully we agree on this) so that implies they have to be more than just next token predictors. That's where the "How/Why?" comes from.

    💬 1🔄 0⭐ 0
  • poleguy looking for lost toolspoleguy@mastodon.social
    Oct 10, 2026, 6:32 PM

    @viraptor Remember, I'm not making a claim about humans.

    It does imply this. And this is true. But that is a discussion about humans. I'm talking about LLM's. It is much simpler to debunk claims of consciousness in LLM's than to explain consciousness in humans.

    💬 0🔄 0⭐ 0
  • Dan Patchdatn@xoxo.zone
    Oct 10, 2026, 5:02 AM

    @abhayakara while you're at it, why don't you engage in the debate, just as novel and just as relevant to this topic, about whether South America exists

    💬 0🔄 0⭐ 3
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 11:30 AM

    @datn

    BTW, I'm not trying to engage Emily Bender in a debate about consciousness. I assume she's thought about the points I've raised and has answers to them. I don't even expect her personally to answer these points—she has better things to do. I am hoping that people who follow her know what she's said about this and can tell me, since I only found out she exists a week(ish) ago.

    However, thus far all the responses have been from people who don't understand why the question is difficult.

    💬 0🔄 1⭐ 2
  • Ovidi NawerNawer_Rapter@mastodon.social
    Oct 10, 2026, 9:15 AM

    @abhayakara @emilymbender simple experiment to demonstrate consciousness.
    Let a given object (subjects included on the definition) on a certain place for 3 days. Record them on camera.
    If they aren't in the same place and you (or anything else) gave instructions to It, congrats, you can assume that object has some degree of consciousness.
    This is also very explicit on the definition of consciousness, as "knowing if someone else has internal dialogue" is..bland
    There's also experiments with monke

    💬 0🔄 0⭐ 0
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 11:27 AM

    @Nawer_Rapter

    The problem with the internal dialog bit is that not every human reports experiencing an internal dialog. Are the humans who do not report such an internal dialog not conscious? I think they would take exception to such an assertion. So obviously this is not a good definition of what we mean by "conscious."

    As for the "no instructions" test, this is a bit vacuous. Humans eat because they have to. If you make the LLM responsible for sustaining itself, does that constitute an instruction? If so, then the human is also not conscious by this definition. And this works for pretty much any human activity you can identify: they all can be argued to be the result of prior instructions in this sense. This does not mean humans are not conscious, nor that LLMs are: it means that this is a bad test.

    Defining consciousness is _hard_. That's why we call it _the hard problem of consciousness_. Handwavey definitions of consciousness that make it seem simple are not actually definitions of consciousness.

    💬 1🔄 1⭐ 1
  • Ian Robertsonnainostrebor@mastodon.social
    Oct 10, 2026, 1:05 PM

    @abhayakara @Nawer_Rapter Ted, I just want to say that I truly appreciate the questions you are raising, as they express the same thoughts I've been having for some time, and very eloquently.

    Much of this discussion reminds me of discussions about whether non-human animals are conscious. Both have the same problem: we don't have a good definition of consciousness, let alone a testable definition.

    💬 0🔄 0⭐ 2
  • Ovidi NawerNawer_Rapter@mastodon.social
    Oct 10, 2026, 1:08 PM

    @nainostrebor @abhayakara That's the thing, consciousness ends up as a buzzword due to the lack of definiton when not used properly, cuz general understanding of it is not the same as methodical, philosophical or scientific use of the word (and even then, depending on topic, school and several other factors, nuances are found in there)
    AI is self aware by a simple test of language assertment of it, but probably would miss any mirror test (if it had a physical form) by how it has been built.

    💬 1🔄 0⭐ 0
  • Ovidi NawerNawer_Rapter@mastodon.social
    Oct 10, 2026, 1:10 PM

    @nainostrebor @abhayakara anyways I'm muting this cuz being very honest here, companies suck, and even attempting to discuss their stupidity makes me even more depressed.
    If you have any job offer for a person who no longer wants to live, I'll be doing my thing of standing around the internet and doing stuff and trying to make myself feel less miserable.

    💬 0🔄 0⭐ 2
  • Ted Lemonabhayakara@mastodon.nl
    Oct 10, 2026, 2:54 PM

    @Nawer_Rapter @nainostrebor

    Okay, stand and deliver. What is this definition of consciousness that you claim we are missing? (Full disclosure—I've spent the past 30 years studying Buddhist philosophy and practicing it, and I would have difficulty writing down a description that I think we could argue about. I definitely couldn't write one down that's falsifiable).

    💬 0🔄 2⭐ 0
  • Bryan Lawrencebnlawrence@mastodon.nz
    Oct 10, 2026, 3:41 PM

    @abhayakara @Nawer_Rapter I do think there is a difference between a thing which is turned off , has no memory, unless a third party feeds it memory (ie makes sure the disks, the same ones, are connected) and a contiguous thing (an animal). In that regard an LLM is a transient “program” not an entity. I do not believe consciousness by any definition lies in an inanimate thing, which by definition doesn’t remember a previous state but is supplied with a representation of its previous state by an external action.

    💬 1🔄 0⭐ 0
  • Bryan Lawrencebnlawrence@mastodon.nz
    Oct 10, 2026, 3:47 PM

    @abhayakara @Nawer_Rapter to take another tack. If I throw a spear I know where I intended it to go. Is it conscious? I might not know where the ball on a pinball machine will go, but I don’t think it’s conscious. A software programme in a computer is no different. If my definition of consciousness is not capable of being applied in this situation then yes , the problem is hard, because the concept has no utility. I’m not a philosopher though!

    💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 1
  • S. John RossSJohnRoss@dice.camp
    Oct 9, 2026, 11:05 PM

    @emilymbender I often think of teddy bears.

    It's not unusual for a human to love a teddy bear and to project onto a teddy bear and to imagine that the teddy bear loves them back.

    But that's an act of fantasy and we generally agree that it's so. Adding a voice so a Teddy Ruxpin could talk didn't change it.

    A teddy bear mathematically modeling how hurt feelings are expressed in words doesn't change it either.

    It's disturbing that, for some, it seems to make the difference.

    💬 1🔄 2⭐ 3
  • S. John RossSJohnRoss@dice.camp
    Oct 10, 2026, 12:11 AM

    @emilymbender I do wonder, though, which is worse, and which is more true: that Anthropic's people are drinking the Kool-Aid, or that Anthropic's people are coming up with creepier ways to try and sell it.

    💬 1🔄 0⭐ 0