Login
You're viewing the f.duriansoftware.com public feed.

Replies

  • 💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 6:44 AM

    i literally can't write these up fast enough, the followup research to confirm and write MWEs and rule out other mitigations takes forever, but in the meantime they'll roll the client code again and in the diff there will be yet another "hmm wait that's not how it should work"

    💬 2🔄 0⭐ 1
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 19, 2026, 7:23 PM

    Something that is extremely awesome about LLM code is its propensity to make endless fallback chains to rescue its fragmented and unreliable logic. The LLM in each pass wants the current thing to succeed, and will rig up hacks that give it trivial test passes. This has the wonderful byproduct of making things fail in the wrong direction in surprising cases - failure is not a smooth outcome, you can't just fail something once and conclude the route is blocked off. you have to fuzz and fail everything in all the different ways, because for some reason one of those failures will engage an entirely different mechanism that rolls over weirdly. This is aided by being given the client source with revealing LLM comments, and having access to the server binary on a system rich with tokens and time for brute force disassembly

    💬 2🔄 3⭐ 0
  • 💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 19, 2026, 8:05 PM

    LLM code constantly dares to ask the question "how hard could you make it to have a constant." In this case, "how on earth can we set the page title to "Muse""

    export const LOGGED_OUT_PRODUCT_NAME = "Muse"; export const LOGGED_IN_PRODUCT_NAME = "Muse"; const LEGACY_PRODUCT_FULL_NAME = "Project Hatch"; const LEGACY_PRODUCT_SHORT_NAME_PATTERN = /\bHatch\b/g; export type ProductBrandNames = { readonly fullName: string; readonly shortName: string; }; function createProductBrand(productName: string) { return { fullName: productName, shortName: productName, shortNameLowercase: productName.toLowerCase(), assistantFallbackName: productName, } as const; } export const LOGGED_OUT_PRODUCT_BRAND = createProductBrand( LOGGED_OUT_PRODUCT_NAME, ); export const LOGGED_IN_PRODUCT_BRAND = createProductBrand( LOGGED_IN_PRODUCT_NAME, ); /** Existing signed-in surfaces use this authenticated-product alias. */ export const PRODUCT_BRAND = LOGGED_IN_PRODUCT_BRAND;
    import { LOGGED_IN_PRODUCT_BRAND } from "@/_lib/productBranding"; /** Browser-only product name revealed after the NDA has been accepted. */ export const POST_NDA_BROWSER_TITLE = LOGGED_IN_PRODUCT_BRAND.fullName; /** The product mark that goes with {@link POST_NDA_BROWSER_TITLE}. */ export const POST_NDA_FAVICON_SVG_URL = "/images/favicon/app-squiggle.svg";
    return () => { unsub_0(); document.removeEventListener("visibilitychange", onReturnToTab); window.removeEventListener("focus", onReturnToTab); closePendingApprovalBrowserNotification(); document.title = POST_NDA_BROWSER_TITLE; setFaviconBadge(false); }; }, []);
    💬 4🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 2🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 7:48 AM

    @Viss
    I'm trying to prioritize by "actually dangerous" vs "dangerous by design" and things in between, but that also takes me awhile to chase down if stuff is currently exploitable or not

    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 7:52 AM

    @Viss
    There are so many undocumented mechanisms in this thing on the client and server side that I am convinced are just Dev hacks left in place and forgotten about. They aren't surfaced to the agent unless it tries to hack itself, but once it finds them its like "oh we can just do this, its in the strings or the client source just not hooked up to a skill or ui" and all the undocumented mechanisms are just like... Gold.

    💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Olafmrkaiser@mastodon.social
    Oct 19, 2026, 8:12 PM

    @jonny I dunno, Jonny. Reading LLM code makes me feel as if I had a stroke. This here is no exception :toot:

    💬 0🔄 0⭐ 0
  • Omnivorous Narratoromnivorous_narrator@mastodon.social
    Oct 19, 2026, 8:16 PM

    @jonny more generally - a lot LLM code is so inconsistent re. level of abstraction, almost hilariously so. “hacks held together by duct tape” mashed together with “enterprise Java Beans”

    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • RojCowlesRojCowles@techhub.social
    Oct 20, 2026, 12:49 PM

    @jonny

    Ooof!

    Urban legend at a company I worked at was that years ago a well meaning developer added a nullptr check to fix a rare crash case in one of the applications.

    Turns out the crash was all that stood in the way of another undiscovered bug that effectively did rm -rf /* when triggered.

    To say that customers were unamused might be a charming understatement.

    The thought that these lash ups of nested LLMs and under the covers shell commands, AKA coding assistants, can be generating so many statistically probable code changes in production systems that they are un-reviewable by competent developers is frankly terrifying.

    💬 0🔄 0⭐ 0