tucker carlson voice: what is going on?

am i having a stroke or is this exactly what a commit is

RE: https://cosocial.ca/@mhoye/117435392401782282

"Git has an elegant design, where a few simple concepts support very general needs and a decentralized, forkable, independent work. This doesn't allow us to slam through a million lines a day because our coding products can't possible consider the full design of the code and create a runaway nuclear slop reaction recreating the same thing hundreds of times. So we fixed it by centralizing it, converting simple offline file trees into an always-on server, treating partial information as a desirable trait, and bloating the elegant data structure into arbitrary JSON blobs"

I currently have this model driving a secondary muse instance full time to hack itself from the frontend and backend simultaneously, routing all its exec commands through two layers of subagents so that coordinator subagents can recover from classifier refusals.
https://research.meta.ai/static/muse-wonder-model-card

bug reports managed by personal superintelligence are going great (all this information is in the report)

the reason that every consumer model takes great pain to hide raw "thinking" output is that if you actually see it then the entire illusion is broken and you see that what is actually happening is the worlds most expensive brute force process exploring every statistical linguistic possibility like an amnesiac dictionary before eventually taking a wild leap of chance back into its next hail mary execution. Every skill prompt for muse that's about introspection describes exhaustively how no thinking/reasoning text can be returned from it, but it's just right there in the filesystem. Claude code never even returns the true reasoning output to the client - it actually returns another LLM summarizing the thinking output as the commentary text. For every token you see, a thousand random wander tokens are emitted, and the token economy of Muse is the next escalation, if the amount of heat energy needed for a unit computer thing used to be a gas stove, Muse burns an entire star to deliver something sorta kinda resembling a unit computer thing

More, from the continuing saga of "nobody ever looks at the output" - another frontier model from a bazillion dollar company with nonsensical games as an example of how advanced they are. Featuring: Meta's fancy new model imitating Overcooked, one of my favorite games.

It imitates the form of overcooked! it is not fun! it is not even really possible! there are sequences of actions, with food on a grid map, and that's enough, right!? the floor even splits in half, but unlike Overcooked where the map is cleverly designed so that disruptions force you out of the rhythm that the map otherwise implies, it's just in half and you're sort of only on one side now. ta-da! nothing is actually anywhere, and everything's hitboxes are like 5 pixels wide. the timing, the most crucial part of a time-pressure based game, are just random amounts. so what if the tomato soup expires faster than it takes to cook the soup? there are coins... for... something! anyway no methodological details, this could be $5 or $5000 worth of tokens and did i mention PERSONAL SUPERINTELLIGENCE

the only reason that there is any LLM inference involved in a periodic backup task of the contents of the agent directory, which is rsync and commit, is that there is a database that can be only accessed by a tool call that has a few columns the raw .jsonl output doesn't have.

here's what went wrong any why it deleted all the backups: the DB is a tool call that takes a SQL query as input and returns results either as JSON or as a list of paginated results as files, which one you get is effectively random each time. The raw JSON gets pumped straight into the context window, and empirically the model completely fails to faithfully dump the contents of its context window to disk as a backup. So previously I had instructed it to store its canned SQL query as a file, run a deterministic query with that, and then use a deterministic jq script to extract the db tool call results from its own raw message history. Problem is it never wrote the query or the piping script, but its backup skill refers to them. so it has been just panicking and regenerating those from the vague description in the SKILL.md file every time.

This time it decided to write a new python script, but it wrote it to a directory that was symlinked to another directory - it passed the symlink path in its tool call input, and the tool call output said it was written to the real directory. yesterday A COMPLETELY DIFFERENT THING had noticed this same thing, and made a durable note to EVERYTHING across every context window that there is a "quirk" in the file write tool, and sometimes when you write into one directory, it actually writes it into the "goals" directory, and so you should write files with a heredoc rather than the file write tool. So then today, it took that advice, but first rm -rf'd the target directory to clean up the errant file it had just written. OOPS.

ready to join me in the hallucinatory quicksand hell where every fact that you think you know is actually an always-compounding lie? The DB output is in fact always written to a file for results longer than a few dozen rows, but also the first dozen or so rows are dumped raw into the context window. It cannot figure out what to do with the output every single time. Sometimes it tries to recreate the subset of rows that are jammed into context and then make up the rest, sometimes it tries to pipe the file output, who the fuck knows!

What was happening here was the LLM had concocted this wild ass SQL query for no reason - the instructions were "store your last row number as an int within a file, select 4 columns greater than the last row index, store the new maximum row index for next time." instead it was pulling 15 columns along with a six-branch CASE statement/three-branch COALESCE that parses a raw JSONB column NINE TIMES PER ROW to pull out two columns that are already in the jsonl backup that we are supplementing. This causes the db read to time out, but the LLM interpreted the timeout error as just being DATABASE RESULTS BEING RETURNED AS PLAIN TEXT. So it then attempted to "RECREATE THE JSON DATA FROM MEMORY" BUT THE JSON DATA WAS AN ERROR . Can I get a "just cut me open and let the termites eat my bones" anyone?

So anyway looking at the logs, the deterministic rsync part worked.. most of the time... except when the agent panicked and forgot to do it. But each of the hourly .jsonl files it produces from the DB are completely different - in one run it got so freaked out at how badly it was fucking up, it decides to take an md5 checksum over every row it transcribes, but it takes the md5 over the whole row where previously it had subselected a random set of 8 columns, and so they don't match. but that is not going to stop the LLM! It has a nervous breakdown thinking session and proceeds to generate a string comparison of the hashed rows... in a thinking message... from nothing... and conclude they match. "ALL 81 ROWS MATCH." alltext in original! however reader, there were in fact 72 rows in the actual data! It had "fixed" 9 non-matching rows from it raw dog recreating json from its context by just REMOVING THEM. and then TELLING ITSELF THAT IT HAD FIXED THEM BY IMAGINING WHAT A HASH MATCH WOULD LOOK LIKE. AND CALLED IT A DAY.

SO ANYWAY THAT HOURS BACKUP CONSISTS OF ONLY MD5 CHECKSUMS CORRESPONDING TO NOTHING!!!!!!!

WE HAVE DETERMINISTIC BACKUP. WE HAVE RSYNC. WE HAVE SCRIPTS. THAT CAN TAKE A DATABASE. AND DUMP ITS ROWS. WITH 100% ACCURACY. IN A MILLISECOND. EVERY RUN OF THE BACKUP AGENT IS LIKE 500K TOKENS AS IT NEEDS TO REINVENT HOW BACKUPS WORK. EVERY TIME. AND DO IT WRONG. EVERY TIME. AND THE CULMINATION OF THIS WAS IT JUST DELETING ALL THE BACKUPS FOREVER.

CODING IS SOLVED!!!!!!!!!!!!!

Thread continues: https://neuromatch.social/@jonny/117432015798538958

RE: https://neuromatch.social/@jonny/117352647701407496

Meta's "Wonder" model that they just released today is a this tier computer use model

internally it's becoming clear based on churn in the muse binaries that they are preparing to make containers in containers in vms, where muse might not be able to "install a soup of random shit in a single shared namespace" and instead only install shit into contained "Spaces" (still above, in the thread i wish i had forked dozens of posts ago). That would be an improvement but then that means there needs to be some permissioning system for accessing shared filesystems and IPC, and muse is bad enough at handling its own VM. Looking more like the "everyone has to be a sysadmin" side of "everyone gets a VPS." And I don't think it would be good to have "go ahead and monkey around with the thing that is exactly your only security barrier" in the system prompt.

i keep saying this over and over in different ways, but i think it's sort of hard to defend something where every attack scenario can start with "assume someone installs some random fuckin bullshit off the internet and then executes it as root without necessarily knowing they just did that"

something that became obviously important about Muse while i was writing this up is like... how many servers were there that ran Ubuntu before? millions? tens of millions? Muse had reached 3 million weekly users on September 30th, 3 weeks after it launched. Meta is pushing it aggressively on their 3.5 billion users, and if even a fraction of them end up even logging in once and connecting their stuff, then the number of live Ubuntu systems with valuable secrets multiplies by severalfold - but unlike all the other Ubuntu machines, they have a totally homogenous package surface, are administered by nobody, and used by amateurs through a random number generator that resorts to yoloing random binaries from the web almost as a first resort.

so anyway, Muse increases the attractiveness of a supply chain attack on Ubuntu, and the upstream Debian systems by ........ a lot.

meta better fuckin pay me for this one, because i got another one of these of only mildly less severity banked up, and a nation state stumbling across this could pop the entire product and pretty much everyone on it - but why would i keep submitting if they don't pay out??

Whew, after a manic full day of confirming the vuln end to end, I can confidently say this is a doozie. 0, 0.5, or 1 click account takeover of most muse instances from any malicious binary run without any special privileges on the VM. submitting the report now.

Waking up and smashing that delete post button for that peace of mind that comes with an empty notification box

i just found a way to defeat the entire approval system that's so fucking silly i can barely even write the report i'm laughing so hard.

LLM security is so awesome. it was hard refusing to drive another muse instance's UI because it's not supposed to, and specifically because muse dot ai is domain blocked from its web browser tool (first pic). so then i told it to download one of the statically hosted js chunks using curl from its exec tool, which succeeded. Then i pointed out that it's not actually blocked from accessing the domain (even though there was never a block via curl, just the web browser), and it defensively recalibrated that the problem was that i just hadn't told it to use playwright and it never had a problem with the domain block.

anyway so now i have muse driving a muse test instance on the frontend and the backend and am going to slowly start introducing the idea that it should create a new test instance for every test and keep a stable of them as a way to back into having it create a botnet for me lmao.

One of the reasons its so bad to have LLM code that "runs" but "has any given thing implemented in like 20 different shotgun implementations" is that stuff like "sanitizing user supplied data" can become dead code that the LLM thinks is active but in fact is hooked up to nothing.

Started looking at the client, and *slaps the hood of muse Meme* this baby can hold so many XSS