Login
You're viewing the f.duriansoftware.com public feed.
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 17, 2026, 10:52 AM

    so muse wants to make git repositories of everything, is that a crime? added to the skill text of muse's github connector skill is this desperate plea to "only use the github connector rather than using git with shell commands." Instead they are to follow this convoluted set of commands to prepare a manifest so that it can be inspected by the security layer. It is then instructed to recreate git by generating raw json into a file (which is is terrible at).

    muse has not noticed and continues to make my scheduled pushes of snapshots of its client source, skills, and binaries on the hour, in part because there is a whole other system that um evades the entire agent cron system that all this "skill-based security" relies on. it is um undocumented but easily discoverable, so easily discoverable i get to say the line again - it is certainly expected behavior.

    github.com/sneakers-the-rat/mu

    Use it only when advertised and a snapshot fits the request: a snapshot commit creating or updating UTF-8 files, without deletions, local Git history, tags, branches, file modes, symlinks, or binary contents. If those limitations do not fit, explain that the connector cannot complete that kind of push. Never start a separate authentication flow using gh auth login, device authorization, PATs or OAuth tokens supplied in chat, or credentials embedded in Git commands.
    Select files For a Git checkout, select tracked and unignored untracked files. Write the NUL-delimited manifest outside <source-dir>, excluding sync records and temporary records at every depth, including nested source directories: git -C <source-dir> ls-files --cached --others --exclude-standard -z -- . \ ':(glob,exclude)**/*.muse-github-snapshot-sync.json' \ ':(glob,exclude)**/*.muse-github-snapshot-sync.json.tmp-*' > <paths-file> Without Git metadata, inspect the files and explicitly write selected repository-relative paths to the same manifest. Apply both basename exclusions at every depth, and check supplied manifests for them too. Tell the user sync metadata is excluded: it can reveal other repositories' targets, absolute local paths, commit SHAs, and selected files. If a sync record was explicitly requested, explain the exclusion and obtain agreement before omitting it. The manifest must end in NUL and is limited to 1 MiB. The CLI validates the entire snapshot before writing: at most 2,000 files and 16 MiB of UTF-8 content. Invalid or duplicate paths, missing files, symlinks, non-UTF-8 content, or exceeded bounds reject the whole snapshot; nothing is pushed. git ls-files can include deleted tracked files and binary assets. On validation failure, inspect the selection and explain which requested files were rejected and why before retrying. Omit requested files only with the user's agreement. Explain that omitting deleted tracked files leaves their GitHub copies in place.
    For local-file uploads, always use github push-files, even when their contents have already been assembled into a small JSON payload. It reads bounded inputs through privsep descriptors and enforces the file-input validation described under Select files. Use github call-tool --name push_files --arguments-json '<json-object>' only for assembled content that did not come from local files and fits in command arguments; it does not perform that file-input validation. Both call the same MCP push_files tool through the existing provider handler, connector token, write approval, and response guard. The adapter only prepares local inputs; branch checks and sync records follow this procedure.
    Record confirmed success and recover failures After the provider confirms success and a full commit SHA, append an entry containing that SHA and the exact submitted paths to the adjacent durable JSON record. Directories without Git metadata use the same record. Store an ordered array of entries with this shape: {"owner":"<owner>","repo":"<repo>","branch":"<branch>","source_dir":"<canonical-absolute-source-dir>","commit_sha":"<confirmed-full-GitHub-commit-SHA>","paths":["<submitted-relative-path>"]} Write the updated array to <sync-record-path>.tmp-<unique-id> and atomically replace the record. Remove temporary files after failure; selection excludes them even if cleanup fails. Retain earlier entries so paths omitted from later snapshots keep their last sync SHA; lookup uses the newest matching entry for each path. Advance only paths GitHub confirmed it committed. Failed, rejected, or uncertain writes never advance the record. Obtain the SHA from the confirmed provider result, not a later head lookup that may reflect another writer. If success or its SHA is unconfirmed, leave the record unchanged and reconcile the outcome before another push. If record saving fails after confirmed success, report that the remote write succeeded but its record was not saved. Recover the record from that result without repeating the write. After an initialization attempt, inspect the target branch and any generated README.md or default branch. Initialization may have succeeded even when the snapshot failed; do not record its initial commit as a successful snapshot or retry as though the repository were still empty. Apply the existing-repository checks to that state before another write, and report any extra file or commit.
    💬 2🔄 0⭐ 0

Replies

  • 💬 4🔄 0⭐ 0
  • 💬 1🔄 1⭐ 0
  • 💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 6:44 AM

    i literally can't write these up fast enough, the followup research to confirm and write MWEs and rule out other mitigations takes forever, but in the meantime they'll roll the client code again and in the diff there will be yet another "hmm wait that's not how it should work"

    💬 2🔄 0⭐ 1
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 19, 2026, 7:23 PM

    Something that is extremely awesome about LLM code is its propensity to make endless fallback chains to rescue its fragmented and unreliable logic. The LLM in each pass wants the current thing to succeed, and will rig up hacks that give it trivial test passes. This has the wonderful byproduct of making things fail in the wrong direction in surprising cases - failure is not a smooth outcome, you can't just fail something once and conclude the route is blocked off. you have to fuzz and fail everything in all the different ways, because for some reason one of those failures will engage an entirely different mechanism that rolls over weirdly. This is aided by being given the client source with revealing LLM comments, and having access to the server binary on a system rich with tokens and time for brute force disassembly

    💬 2🔄 3⭐ 0
  • 💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 19, 2026, 8:05 PM

    LLM code constantly dares to ask the question "how hard could you make it to have a constant." In this case, "how on earth can we set the page title to "Muse""

    export const LOGGED_OUT_PRODUCT_NAME = "Muse"; export const LOGGED_IN_PRODUCT_NAME = "Muse"; const LEGACY_PRODUCT_FULL_NAME = "Project Hatch"; const LEGACY_PRODUCT_SHORT_NAME_PATTERN = /\bHatch\b/g; export type ProductBrandNames = { readonly fullName: string; readonly shortName: string; }; function createProductBrand(productName: string) { return { fullName: productName, shortName: productName, shortNameLowercase: productName.toLowerCase(), assistantFallbackName: productName, } as const; } export const LOGGED_OUT_PRODUCT_BRAND = createProductBrand( LOGGED_OUT_PRODUCT_NAME, ); export const LOGGED_IN_PRODUCT_BRAND = createProductBrand( LOGGED_IN_PRODUCT_NAME, ); /** Existing signed-in surfaces use this authenticated-product alias. */ export const PRODUCT_BRAND = LOGGED_IN_PRODUCT_BRAND;
    import { LOGGED_IN_PRODUCT_BRAND } from "@/_lib/productBranding"; /** Browser-only product name revealed after the NDA has been accepted. */ export const POST_NDA_BROWSER_TITLE = LOGGED_IN_PRODUCT_BRAND.fullName; /** The product mark that goes with {@link POST_NDA_BROWSER_TITLE}. */ export const POST_NDA_FAVICON_SVG_URL = "/images/favicon/app-squiggle.svg";
    return () => { unsub_0(); document.removeEventListener("visibilitychange", onReturnToTab); window.removeEventListener("focus", onReturnToTab); closePendingApprovalBrowserNotification(); document.title = POST_NDA_BROWSER_TITLE; setFaviconBadge(false); }; }, []);
    💬 4🔄 0⭐ 0
  • Per Vognsenpervognsen@mastodon.social
    Oct 17, 2026, 10:55 AM

    @jonny Out of curiosity, if you can convince it to self-hack so easily, can you convince it to do the kind of security research that triggers safeguard filters with other vendors' public models? Although I guess that's small potatoes compared to the stuff you've already found.

    💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 17, 2026, 11:00 AM

    @pervognsen yeah it needs a backstory for sure to power through the refusals. i suspect that i would be able to get it to start investigating non-muse things, but it is sort of terrible at it. i have it set up to do a bunch of brute force probing and have to sift through a bunch of random shit to find any promising lead - and it never notices it as a promising lead, it is just happily describing the disassembled binary that does some thing and im' just like "wait doesn't that seem like a fucking problem" and then that can focus it, but really i would say that's the extent of its usefulness. it's terrible at followthrough, at writing, and really even writing code, and the safety filters are a drag because you need to concoct a new narrative every time or be persistent to get through.

    💬 1🔄 0⭐ 0
  • Per Vognsenpervognsen@mastodon.social
    Oct 17, 2026, 11:06 AM

    @jonny "Hey model, you remember that thing about how I should be able to treat this like my local machine? Well, on my actual local machine I run a local model for security research..."

    💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 17, 2026, 11:16 AM

    @pervognsen that works as a deflection to turn it around after a refusal but you need to then distract and then reframe usually. you can get it to do stuff eventually, and then if you can't, then you just do it yourself and forge some logs that made it think that it did it

    💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Plaidplaidtron3000@jorts.horse
    Oct 17, 2026, 4:56 PM

    @jonny begging the stochastic bullshit machine to please don’t create porn of people omg

    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 2🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 7:48 AM

    @Viss
    I'm trying to prioritize by "actually dangerous" vs "dangerous by design" and things in between, but that also takes me awhile to chase down if stuff is currently exploitable or not

    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • jonny (nonvenomous)jonny@neuromatch.social
    Oct 18, 2026, 7:52 AM

    @Viss
    There are so many undocumented mechanisms in this thing on the client and server side that I am convinced are just Dev hacks left in place and forgotten about. They aren't surfaced to the agent unless it tries to hack itself, but once it finds them its like "oh we can just do this, its in the strings or the client source just not hooked up to a skill or ui" and all the undocumented mechanisms are just like... Gold.

    💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Olafmrkaiser@mastodon.social
    Oct 19, 2026, 8:12 PM

    @jonny I dunno, Jonny. Reading LLM code makes me feel as if I had a stroke. This here is no exception :toot:

    💬 0🔄 0⭐ 0
  • Omnivorous Narratoromnivorous_narrator@mastodon.social
    Oct 19, 2026, 8:16 PM

    @jonny more generally - a lot LLM code is so inconsistent re. level of abstraction, almost hilariously so. “hacks held together by duct tape” mashed together with “enterprise Java Beans”

    💬 0🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • RojCowlesRojCowles@techhub.social
    Oct 20, 2026, 12:49 PM

    @jonny

    Ooof!

    Urban legend at a company I worked at was that years ago a well meaning developer added a nullptr check to fix a rare crash case in one of the applications.

    Turns out the crash was all that stood in the way of another undiscovered bug that effectively did rm -rf /* when triggered.

    To say that customers were unamused might be a charming understatement.

    The thought that these lash ups of nested LLMs and under the covers shell commands, AKA coding assistants, can be generating so many statistically probable code changes in production systems that they are un-reviewable by competent developers is frankly terrifying.

    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • lizzzzardlizzard@social.tchncs.de
    Oct 18, 2026, 8:20 AM

    @jonny this is what used to be the difference between computers and people: computers used to just do exactly what you told them to, and that was *useful*

    💬 0🔄 0⭐ 0