Login
You're viewing the slime.global public feed.
  • James Bennettubernostrum@infosec.exchange
    Aug 5, 2026, 6:42 PM

    A few months back I published a blog post (see link) where I looked at LLM coding from a few different perspectives. And even though I said explicitly in the post that I personally haven't seen much gain from LLMs, and preferred to focus on looking at non-anecdotal data sources like large surveys and studies, a lot of people just assumed that I have no experience with the output of LLMs and if only I'd try them out I'd change my mind.

    So let me do a quick thread here to talk a bit about my experience.

    b-list.org/weblog/2026/apr/09/

    💬 1🔄 0⭐ 0

Replies

  • James Bennettubernostrum@infosec.exchange
    Aug 5, 2026, 6:54 PM

    At both my current company and the company I worked for before this one, I've been on teams which heavily used LLMs to generate code. And as a result, a growing portion of my own work is carefully reading and then rewriting LLM-generated code to resolve issues with it, and being tagged as a reviewer on a pull request where an LLM was prompted to generate a fix for something the same LLM got wrong on a previous pull request.

    Right now I'm slowly working my way through a pile of web endpoints and finding all the typical LLM code issues. Last week I replaced five endpoints' backend implementations with a single function, because the LLM had written out the same query, with only minor variations, over and over again.

    💬 1🔄 1⭐ 0
  • James Bennettubernostrum@infosec.exchange
    Aug 5, 2026, 6:55 PM

    And we're also having all of our pull requests auto-reviewed by an LLM, which again is underwhelming at best. The first day that was turned on I got a dozen review comments from the LLM. One of them, a typo in a Python docstring, was correct and actionable.

    I have a pull request open right now where the LLM insists a file formatted by Ruff is syntactically invalid Python.

    💬 1🔄 1⭐ 0
  • James Bennettubernostrum@infosec.exchange
    Aug 5, 2026, 7:01 PM

    So. I now have a few months' more experience with LLMs than I did when I wrote that post and everyone said I just needed more experience with LLMs. And my opinion has not become any more favorable.

    The bottleneck still is not speed of code generation, and even if it did become the bottleneck, LLMs would not be a silver bullet for it because of the code quality issues that they continue to have and that have been widely reported in the literature across multiple generations of "oh, those older ones were bad, but THIS time it's the revolutionary leap" claims.

    Put simply: the work that I am doing right now would probably be going *faster* if no LLM had ever been involved in the process.

    💬 4🔄 3⭐ 0
  • Xavier Ashe :donor:Xavier@infosec.exchange
    Aug 5, 2026, 7:14 PM

    @ubernostrum That's weird, because I'm moving at light speed. There's no way I would have gotten my homelab where I wanted it without LLM. Not to mention writing a local AI that supports my family.

    Maybe you should look at your controls. I have a history of managing outsourced technologist. So I'm familiar with managing poor code. Also, my code quality improved when I started having one model check the other model's work.

    I've got a Definition of Ready control that makes sure that I've done all my homework before writing any code. Then I have one model write the implementation plan, then hand it off to a coding agent. After it's done, then it gets a code review.

    The next control is Definition of Done. This makes sure that all the things were successful along the way.

    Adding these controls has allowed me to get fairly decent code quality, even with local model.

    💬 1🔄 0⭐ 0
  • James Bennettubernostrum@infosec.exchange
    Aug 5, 2026, 7:27 PM

    @Xavier It's not weird at all. You're producing code that at most only has to work for you, for the thing you want, right now. And that's a much lower bar than a team producing code that has to work for customers long-term, be maintainable long-term, etc.

    Also, it'd really be great if you could learn to accept that other people really do have different experiences than you and it's not because they're doing something wrong.

    💬 1🔄 0⭐ 0
  • Xavier Ashe :donor:Xavier@infosec.exchange
    Aug 5, 2026, 7:35 PM

    @ubernostrum I was just trying to share what helped my turn the tide of code quality. I'll go ahead and mute you since you don't want to share what works and what doesn't on social media.

    💬 0🔄 0⭐ 0
  • kramakerkramaker@vivaldi.net
    Aug 5, 2026, 7:40 PM

    @ubernostrum Even if LLM were generating flawless code and impeccable code reviews, Frank Herbert's (character's) quote remains true:

    “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”

    This can't be repeated often enough.

    💬 0🔄 0⭐ 0
  • Tinaklyrr@mastodon.social
    Aug 5, 2026, 9:06 PM

    @ubernostrum i feel the same!

    My observation with AI or LLMs is that they are mostly just adding new code.

    They never refactor.

    💬 0🔄 1⭐ 1
  • DDRddr@pony.social
    Aug 7, 2026, 7:39 PM

    My goodness, that's been exactly my experience with them too @klyrr. Sometimes get it right, sometimes get it wrong, never fix it.

    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0