Login
You're viewing the discuss.systems public feed.

Replies

  • 💬 1🔄 1⭐ 0
  • 💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 6:44 AM

    i literally can't write these up fast enough, the followup research to confirm and write MWEs and rule out other mitigations takes forever, but in the meantime they'll roll the client code again and in the diff there will be yet another "hmm wait that's not how it should work"

    💬 2🔄 1⭐ 12
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 19, 2026, 7:23 PM

    Something that is extremely awesome about LLM code is its propensity to make endless fallback chains to rescue its fragmented and unreliable logic. The LLM in each pass wants the current thing to succeed, and will rig up hacks that give it trivial test passes. This has the wonderful byproduct of making things fail in the wrong direction in surprising cases - failure is not a smooth outcome, you can't just fail something once and conclude the route is blocked off. you have to fuzz and fail everything in all the different ways, because for some reason one of those failures will engage an entirely different mechanism that rolls over weirdly. This is aided by being given the client source with revealing LLM comments, and having access to the server binary on a system rich with tokens and time for brute force disassembly

    💬 2🔄 16⭐ 38
  • 💬 1🔄 0⭐ 7
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 19, 2026, 8:05 PM

    LLM code constantly dares to ask the question "how hard could you make it to have a constant." In this case, "how on earth can we set the page title to "Muse""

    export const LOGGED_OUT_PRODUCT_NAME = "Muse"; export const LOGGED_IN_PRODUCT_NAME = "Muse"; const LEGACY_PRODUCT_FULL_NAME = "Project Hatch"; const LEGACY_PRODUCT_SHORT_NAME_PATTERN = /\bHatch\b/g; export type ProductBrandNames = { readonly fullName: string; readonly shortName: string; }; function createProductBrand(productName: string) { return { fullName: productName, shortName: productName, shortNameLowercase: productName.toLowerCase(), assistantFallbackName: productName, } as const; } export const LOGGED_OUT_PRODUCT_BRAND = createProductBrand( LOGGED_OUT_PRODUCT_NAME, ); export const LOGGED_IN_PRODUCT_BRAND = createProductBrand( LOGGED_IN_PRODUCT_NAME, ); /** Existing signed-in surfaces use this authenticated-product alias. */ export const PRODUCT_BRAND = LOGGED_IN_PRODUCT_BRAND;
    import { LOGGED_IN_PRODUCT_BRAND } from "@/_lib/productBranding"; /** Browser-only product name revealed after the NDA has been accepted. */ export const POST_NDA_BROWSER_TITLE = LOGGED_IN_PRODUCT_BRAND.fullName; /** The product mark that goes with {@link POST_NDA_BROWSER_TITLE}. */ export const POST_NDA_FAVICON_SVG_URL = "/images/favicon/app-squiggle.svg";
    return () => { unsub_0(); document.removeEventListener("visibilitychange", onReturnToTab); window.removeEventListener("focus", onReturnToTab); closePendingApprovalBrowserNotification(); document.title = POST_NDA_BROWSER_TITLE; setFaviconBadge(false); }; }, []);
    💬 7🔄 7⭐ 26
  • Plaidplaidtron3000@jorts.horse
    Oct 17, 2026, 4:56 PM

    @jonny begging the stochastic bullshit machine to please don’t create porn of people omg

    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 2
  • 💬 2🔄 0⭐ 1
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 7:48 AM

    @Viss
    I'm trying to prioritize by "actually dangerous" vs "dangerous by design" and things in between, but that also takes me awhile to chase down if stuff is currently exploitable or not

    💬 0🔄 0⭐ 2
  • 💬 1🔄 1⭐ 1
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 7:52 AM

    @Viss
    There are so many undocumented mechanisms in this thing on the client and server side that I am convinced are just Dev hacks left in place and forgotten about. They aren't surfaced to the agent unless it tries to hack itself, but once it finds them its like "oh we can just do this, its in the strings or the client source just not hooked up to a skill or ui" and all the undocumented mechanisms are just like... Gold.

    💬 1🔄 1⭐ 2
  • 💬 1🔄 0⭐ 2
  • 💬 0🔄 0⭐ 0
  • Olafmrkaiser@mastodon.social
    Oct 19, 2026, 8:12 PM

    @jonny I dunno, Jonny. Reading LLM code makes me feel as if I had a stroke. This here is no exception :toot:

    💬 0🔄 1⭐ 8
  • Omnivorous Narratoromnivorous_narrator@mastodon.social
    Oct 19, 2026, 8:16 PM

    @jonny more generally - a lot LLM code is so inconsistent re. level of abstraction, almost hilariously so. “hacks held together by duct tape” mashed together with “enterprise Java Beans”

    💬 1🔄 0⭐ 3
  • Omnivorous Narratoromnivorous_narrator@mastodon.social
    Oct 19, 2026, 8:17 PM

    @jonny like, sometimes even fractally

    occasionally I see this happen in written code but it usually takes like, 20 years of tech debt and on-and-off time crunch

    💬 0🔄 0⭐ 1
  • Mathaetaesmathaetaes@infosec.exchange
    Oct 19, 2026, 8:16 PM

    @jonny I've seen code bases like this. Usually in a utility class written by some fresh-out-of-college new hire who has never heard of YAGNI.

    💬 0🔄 0⭐ 3
  • 💬 1🔄 0⭐ 1
  • 💬 1🔄 0⭐ 3
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 2
  • 💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 1
  • 💬 1🔄 0⭐ 2
  • 💬 1🔄 0⭐ 4
  • 💬 0🔄 0⭐ 3
  • 💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 1
  • Dianealienghic@timeloop.cafe
    Oct 19, 2026, 7:59 PM

    @jonny

    Your descriptions make me think AI generated code is like an evolved system with lots of duplication then modification of functionality. Its very far from an engineered system that can be reliably reasoned about

    💬 0🔄 1⭐ 5
  • 💬 0🔄 0⭐ 0
  • lizzzzardlizzard@social.tchncs.de
    Oct 18, 2026, 8:20 AM

    @jonny this is what used to be the difference between computers and people: computers used to just do exactly what you told them to, and that was *useful*

    💬 0🔄 0⭐ 0