BREAKING NEWS: A new vulnerability has been discovered in all versions of PHP. Details will be forthcoming, but I’m told that, under certain unusual conditions, it will execute PHP code. This has numerous exploitable consequences, not least of which is that it leads to more PHP code in the world. The details are currently embargoed but the team responsible for discovering the vulnerability report that a similar class of problem exists in GCC and Clang as well.

RE: https://mastodon.sdf.org/@netbsd/117396625341557692

Hey #FreeBSD folks, NetBSD hit more than double their funding target after adopting a strong no-LLMs policy. Something to think about while you draft yours, perhaps?

I don’t understand the panic over ‘LLMs can tell people how to build biological weapons and plan assassinations’. So can Tom Clancy novels. And both sources will tell you things that are more likely to get you killed than achieve your nefarious goal.

Butterfly is much less fun to say than papillon so I am using the standard process in the English language: deprecating a common-usage word and replacing it with a foreign word that I like better. From now on, all uses of the word ‘butterfly’ shall be deemed archaic.

Thank you for your patience during the transition.

So, OpenAI admits two things in disclosures presented to the court:

• That their products are utterly dependent on work ingested without compensation.
• That their derived product is directly substituting for this work, removing the economic incentive to perform that work in the first place.

Meanwhile, the EFF filed an amicus brief in the case claiming that this use is exactly what was intended by the ‘To promote the Progress of Science and useful Arts’ justification for the existence of copyright in the US Constitution.

For people who still believe the GPL influences corporations in a meaningful way:

When I was at Microsoft, I worked on a product that used Linux. The product was cancelled for largely unrelated reasons before it shipped, so the exact details are irrelevant but anything that links Linux needs to loop in lawyers, so I spent a lot of time talking to them.

Maybe 10% of the total time was spent with the technicalities of the GPL. The fact that the GPL is ambiguous and poorly written and the Linux syscall exemption is even worse made me expect long discussions about the legalese. But instead the conclusion of the GPL-focused discussions was basically:

No one really knows what the GPL can actually enforce and half of the stuff that's done with Linux is technically in violation. Microsoft ships Linux images for Azure that come with a click-through license that prohibits running them on any other cloud and no one cares. We can make a convincing argument that this is okay. Only a handful of companies can afford a protracted legal battle with Microsoft and none of them are sufficiently motivated to take us to court over this, so it's unlikely to come up.

The remaining 90% of the discussions were around how the product would be perceived by Linux maintainers, the wider Linux community, and the broader F/OSS ecosystem. Microsoft had put a lot of effort into the 'Microsoft <3 Linux' marketing things. The Ballmer-era alienation of the most active developer communities had cost the company a lot of ecosystem market share. The risk of pushing developers away from Microsoft products and platforms was something that the lawyers regarded as a huge risk. If two of the senior Linux maintainers had said 'we think this is not an acceptable use of Linux' then that would have torpedoed the project instantly.

The moral of this story is that, if positive engagement with your community is perceived as valuable, you have an enormous amount of leverage over big companies. At the time, Microsoft was (by market cap) the largest company in the world. The legal text, in contrast, isn't worth the paper it's written on when your adversary can afford to outspend you in court by six orders of magnitude. When they happen to be aligned, the legal text may look as if it's working, but that's a coincidence.

Don’t you hate it when you write a long post, proofread it, post it, see it in your feed, and immediately spot a typo in the first line?

(Fixed now, it said ‘firms’ when it should have said ‘forms’)

I am not a lawyer. I have studied IP law in various forms guided by solicitors, barristers, and professors of law of my acquaintance since I was a teenager (including reading through big piles of case histories and commentaries) but not in a formal setting (mostly I learned that I would rather invent the things in the patents than draft the patents, so changed career direction). I’ve also spent a surprising amount of my career talking to copyright and patent lawyers (almost never trademark or trade-secret specialists). I have enough of a lay-person’s understanding of the topic that I was able to spot that a clause in the contract from my US publisher was unenforceable in the state that they claimed jurisdiction because it hinged on an aspect of copyright law that the USA delegates to states and which the state in question did not have relevant laws. Their lawyers subsequently confirmed and fixed this. But I a not lawyer and this is not legal advice.

I have two objections to the EFF’s position on ‘AI’ model training. One is technical, one is social.

The technical one first.

Imagine I rip a DVD and transcode it to MPEG-4 video. This is lossy recompression. The new copy is not identical to the original. It is a derived work. There was no transformative step. Specifically, losing fidelity of reproduction is not a transformative step.

Video CODECs take advantage of redundancy. Simple CODECs build predictive patterns for redundancy in a single frame (for example, is this all one colour or a gradient? Store just that fact not every pixel). More complex ones look at the previous frame and compare it to the current one and use that redundancy. Most modern ones do this in both directions. Effectively, they create a cube of voxels, where one dimension is time, and try to find redundancy in the cube.

For copyright law, this doesn’t matter. Lossy compression is not a transformative step, no matter how complex the compression.

A lot of compression schemes (rarely for video, mostly because it doesn’t make sense for video unless you have a lot) also support special cases for large quantities of redundancy across a data set. For example, if you wanted to compress English Wikipedia with ZSTD, you would use the dictionary mode. It will come up with a list of the words (or even common phrases such as ‘citation needed’) and Huffman encode them so that each page has a short encoding for referencing them. This makes each page smaller than it would be if you compressed it individually.

Compressing Wikipedia like this is not a transformative step.

Now, imagine that you create a video compression CODEC that does this. You buy a copy of every DVD or BluRay disk available and compress them together such that you have a large dictionary of all common compressed sequences. Given a prefix of a film, each film in the input set would be reproducible with some loss of quality (not necessarily the same level of quality). Similarly, if you provided an initial vector that was not something in the training set then you’d get out video that might be similar to one of the inputs, might be similar to many, or might not be obvious to a human is close to either.

This is the crux of the argument. Deep neural networks are functionally equivalent to lossy compression schemes. The inference or generation step in ‘generative AI’ is an initial vector and a random seed that decompresses the data that might be there. If nothing from the training (input) set exactly matches (or if the random seed moves away from that path) then you’ll get something new, possibly something that’s recognisable as a lossily compressed version of the input data.

Note, in particular, that a lot of compression schemes now do take advantage of neural networks. They are one of the most efficient known ways of generating a specialised lossy compression scheme over arbitrary data. The law typically doesn’t care what specific technology an action uses, only about the outcome. In this case, that doesn’t matter: exactly the same underlying technology, used in exactly the same way, covers both ‘AI’ and compression. If one is legal then so is the other because they are the same process.

The EFF’s argument hinges on the idea that this lossy compression is a transformative step. Not only is that an idea that is not supported in case or statute law, there is case law that makes it clear that lossy compression of a work is not transformative.

Their argument would be internally self consistent if it also argued that Netflix does not owe royalties on any of the third-party videos it streams (and that they can buy BluRays on Amazon, recompress them, and then stream them without paying royalties). But there is so much case and statue law that this is not the case that they didn’t make this claim.

Instead, they tried to claim that this is permitted if you call the system ‘AI’ even though it is settled law that it is not permitted if you do not call the system ‘AI’.

Second, the social aspect. Copyright law in the USA draws its legitimacy from this line in the Constitution:

To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.

So does patent law. Patent law is more of a mess because the USA was founded just before James Watt bribed various MPs to subvert the patent system, but (due to later treaties) the modern US patent system inherits from that subversion (Watt, like Edison, was a bit of a dick).

This intent goes back to the invention of the printing press, where publishers made copies of books in large quantities without paying authors. This removed the incentive to write books. Allowing authors to control distribution rights put that incentive back. This short paragraph covers about a hundred years of the evolution of legal thought in this space, please forgive the many oversimplifications.

The EFF brief references this motivation but twists it. OpenAI and Anthropic are the equivalents of the post-Gutenberg printers. They are taking the work of creative individuals and producing output that competes directly with the products of those authors. This is precisely the situation that copyright law in the USA exists to prevent.

Yet the brief twists this to say that a lossily compressed duplication of original work is actually the kind of creativity that this was intended to cover.

This hinges on the idea that writing news, or other creative works, is a trivial commodity, whereas mechanically compressing them into a system that can lossily reproduce them is a key contribution to society.

Even if I agreed with their other arguments (I do not, I believe that they either misunderstand or deliberately misrepresent the technology and the relationship to other settled law), this is such a profoundly anti-human viewpoint. The idea that human creativity exists to feed poor-quality technological reproductions of that creativity is incompatible with any possible society that I would want to live in and I would struggle to find common ground with people who aim to create such a world.

The other day, I wanted to see if alcohol-free port existed. The top search link went to a page of the 20 best alcohol-free ports! Except none of them were port, they were all spirits (not even other fortified wines). The ‘AI’ summary described a fictional process for removing alcohol from port and had two citations, neither of which supported the claim. The next 30 pages were all LLM slop that talked about what port is for a long time before saying that there’s no alcohol-free version.

I have been using the web since the mid ‘90s and it has never been this bad. Anyone advocating for ‘AI’ is advocating for a future where this damage is spread to all facets of life.

COVID vaccinations are so streamlined now. Arrived less than five minutes before appointment time, vaccination completed two minuted after appointment time, asked to wait five minutes to check for fainting or low blood pressure, then off.

I have used vim since around 2000. I have written five books, a PhD thesis, a few dozen papers and over 150 articles with it. At this point, my higher brain functions are not engaged at all when I use a bunch of common vim commands, they just happen. Documents I wrote with anything else have random :w in the middle.

So I tried NeoVim when it was quite new. Vim that you are familiar with, but better? Great!

Persistent undo is one of my favourite features of vim (introduced a few years after I started using it). My .vimrc includes a couple of lines related to this. One sets the path to be under ~/.cache so the undo files do not clutter the rest of the filesystem (I do the same for the swap files). The other sets the undo limit to something large (10,000, I think) because disk space is enormous relative to pretty much anything involving human-readable quantities of plain text.

I don’t often need the persistent undo. But on the few occasions when I have needed it, it’s been invaluable: ooops, I deleted something from this file, maybe last week and one reboot ago, what was it? Undo until I find it, copy it, paste it into the current version. Or, a bit more commonly: I had this working, then I tidied it up ready to commit, now it isn’t working, what did I do?

Vim has kept this working across major version upgrades over a period of about 20 years. I don’t even think about it, it’s just part of Raskin’s First Law: A program may not harm a user’s data or, through inaction, allow a user’s data to come to harm. If vim or the computer crash, or if I close a file and come back to it six months later, my undo history is still there.

The first thing I noticed in NeoVim was that undo didn’t work. I tried opening the file in vim and undo didn’t work there either.

Neovim had changed the format of the undo files. It hadn’t upgraded the old one. It hadn’t used a different name for its undo files. It had just noticed the existence of a vim undo file, deleted it (losing all of the data in it) and replaced it with one that vim couldn’t read.

I raised an issue about this and was told that the persistent undo format was unstable and users should not rely on data being preserved in a feature explicitly called persistent undo. It had changed once and would probably change again.

And that ended my experience with NeoVim. The authors showed immediately that they absolutely could not be trusted with any of my data. Breaking persistent undo is something I could forgive as a bug, but the attitude that just because something is a persistent file on your filesystem that contains data that you might want is no reason for their program not to delete it meant they had no concept of a duty of care to their users.

Apparently now they’re platforming white supremacists. I am completely unsurprised. Different facets of the same underlying attitude.

RE: https://mas.to/@zzt/117134157732461816

This highlights one of the big problems with LLMs for code generation.

If a thing has been done often enough that there are are loads of examples of it for an LLM to train on, you probably shouldn't be doing it again. You should be using some abstraction over the thing and not copying and pasting it.

If a thing is rare enough that it has been done a handful of times in the training set, there's a good chance that the LLM output will meet the legal definition of a derived work, you may be accidentally violating a license / infringing copyright.

If a thing is so rare that there are no examples in the training set, the LLM will probably produce something that looks right. There is even a chance that it is right. Good luck.

Investors: Facebook needs to stop jumping on bandwagons and get back to its roots.

Facebook: *releases a device to enable wannabe frat bros to perv at women*

Investors: We walked into that one, didn’t we?

When you first learned Linux / UNIX, did you ever type cat and sit waiting wondering why the machine was so slow at listing files in the current directory?

And now I've unsubscribed from freebsd-current as well as freebsd-hackers. A thread:

Original post: Hey, I wrote a thing! It doesn't work, please help!

Developer A: Is this vibe coded?

OP: It is, in fact, written by Claude.

A: I don't have time to help fix vibe coded things. Not sure if anyone else does.

Developer B: Don't be mean to people who vibe code things! Just because they haven't invested any time learning things but expect other people to do all of the actual work for them doesn't mean that you should politely tell them you're not going to help! That's mean!

The posts that follow get worse. 'There's lots of vibe coded shit that would solve real problems but doesn't work! You should put in effort to make them all work!' was where I stopped reading the thread and hit the unsubscribe button.

RE: https://mastodon.world/@anttipeltola/117092344692403567

I really like the Pentium 4 as an analogy here.

The NetBurst architecture in the P4 was designed to be able to scale to 10 GHz. It initially shipped in the 1-2 GHz range. The performance was not that impressive compared to even the previous generation but, Intel told us, that didn’t matter: it would rapidly reach clock speeds that were completely impossible with the P3.

On the face of it, this wasn’t unreasonable. The Pentium Pro and Pentium 2 shared a fairly similar microarchitecture (P2 added MMX and a few other things). The Pentium Pro was introduced at 150 MHz and scaled up to 450 MHz after process shrinks.

When the Pentium 4 was in early design stages, it was completely reasonable to expect that 3-5x clock speed increases were possible.

And, technically, they were. The clock speed for a part is determined by how long it takes a signal to propagate across everything in a pipeline stage. Make the things smaller and the signal goes further.

That kept happening. The thing that stopped (completely by about 2007, but it wasn’t an abrupt thing) was Dennard Scaling. Beyond a certain point, making the transistors smaller meant you had a lot more leakage current and so the power went up. And power was already polynomial in terms of clock speed.

All this meant that, yes, in theory, you could run a Pentium 4 at 10 GHz, as long as you didn’t mind all the fire. Someone did eventually run one at this speed with liquid nitrogen cooling, but that wasn’t appropriate for a consumer device. The range topped out at 4 GHz, and those parts had infeasible high cooling and power requirements (and were outperformed by other designs at lower speeds).

The entire ‘AI’ bubble is predicated on the same assumptions of scaling of something foundational that the people building the end systems are not really paying attention to.

This article starts with a story from someone who tried Claude Code and found it amazing, but then switches to the same person a few months later seeing what a disaster it’s been. This quote is key:

The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems

LLMs, by their nature, generate statistically plausible output. That is often a set that overlaps with correct output. But the things that are not correct look exactly the same as the ones they are.

Learning to review code is hard. You look for the bugs that you expect to be possible implementing it by using your theory of mind for the person writing the code and the kinds of things that they might overlook (not necessarily a specific person, but the kinds of things people miss) and also common bug classes.

And the big help is that the person writing the code is not thinking adversarially. They are not trying to sneak bugs in. Normally. Unless they’re a supply-chain attacker, and we’ve a depressing amount of evidence that code review doesn’t catch supply-chain attacks.

An LLM is not trying to do anything. It has no intent. But it is a machine that is trained on code that made it past code review. The kind of bugs that it will generate are ones that look like code that appeared in production. This is exactly what an attacker would do: try to write code that looks correct but is subtly wrong.

I’m only being slightly flippant when I say LLMs are a mechanism for bringing supply chain attacks in house.

Today's instalment of 'people who criticise #Signal should understand why Signal made specific choices and provide better solutions to those problems':

Signal uses a phone number to sign up (boo, PII!). This serves three primary purposes:

The one I don't really like is that it limits account-creation rate for antispam purposes. This is largely a problem because Signal conflates identifiers (your phone number or username) with capabilities: anyone who has your identity can contact you, there's no way of saying 'this person may add me but does not have any token that allows them to pass on the ability to contact me'. And this is a thing that should be table stakes for any messenger, but isn't something I've seen done well anywhere.

More importantly, it enables discovery. In a world where most people used SMS, WhatsApp grew rapidly by reusing this identifier. If you had a contact list of people you used SMS to talk to, WhatsApp would automatically create a contact list with everyone who was already in your contact list. That's how WhatsApp grew to a billion users in a few years. Apple's iMessage copied this flow. Signal added some attempt at making this privacy preserving (though the space of valid phone numbers is so small that it's not hard to exhaustively enumerate it) and ideally they'd have some protocol so that you could find people by phone number only if you, specifically, were allowed to (some kind of zero-knowledge system that checked that both of you had the other in their contacts).

This isn't just a marketing things. Growing the network rapidly is important for privacy. Privacy cares a lot about the concept of an anonymity set: the set of users that might be mistaken for you. If you're using something like XMPP, with a one-user server, your anonymity set is just you: any message from your server comes from you. If you're using a server with a hundred users, it's fairly easy to correlate messages sent from a client to the server with messages sent from the server elsewhere (even if they're end-to-end encrypted) because normal humans don't send messages all of the time and so there will be moderately long periods (seconds, if not minutes) where you're the only one sending messages. If you're on a server with a million users, there will be very few times when you're the only person sending a message and so this kind of correlation is hard. But, since Snowden, we know that at least one (and I would be shocked if it is exactly one) nation state is routinely running this kind of correlation, so this is something that any private messenger needs to consider in scope.

What about other identifiers? The theory with XMPP was that email providers would also offer XMPP and you'd use the same identifier for both. Yay! Except most email providers didn't (except, briefly, gmail), so now you had two different identifiers that looked the same for reaching you. And here's the sad fact: most people don't have email addresses in their systemwide address book. They have mail clients that remember people they've contacted before. Unless they give your app access to their email account (and they absolutely should not be encouraged to give messenger apps access to their email), you can't get a list of email addresses for contacts (and, if you do, it will also include a load of companies and probably spammers). So it doesn't work as a discovery mechanism.

The final one is recovery. If you can't log into Signal, you can reacquire the account with a SIM and a passcode. Getting a replacement SIM is normally easy. You could do this with email but most people have free email from a single provider. A phone number is something that you pay for but the free emails come with no guarantee of service and can be cancelled at any time without giving reason. Email-based recovery is nice to have as an alternative option but for most people it's strictly worse (in fact, their email address probably has an SMS-based recovery flow!).

So, if you want to criticise Signal for using phone numbers, please do! I don't like it. I think there are better solutions to these things.

But don't pitch your favourite messenger as better than Signal because it doesn't need a phone number unless you come with an explanation of how you solve those problems. And if you do have a better solution to those problems, don't bother comparing to Signal, tell me about why your messenger is amazing because it has good solutions to those problems!

This story should terrify any open source project or company that allows LLM-generated code into their project.

Someone created a new app, using Claude. Only it wasn’t a new app, it was clearly plagiarised because it happened that there was an example in the training set that exactly matched the requirements. The similarity was well within the range that courts have previously used to determine a derived work.

When you have this level of similarity, the requirement comes to you to prove that there was no way that the original work could have flowed to your project. Companies that have this concern usually do it by ensuring that no one on the team has been exposed to the original and that the code for the original never goes near their systems. But when one of the systems that you use is a language model trained on, among other things, all of the open-source code that it could scrape (and which does not disclose its training set), being able to prove that there was no path from some other codebase to yours is impossible.

Just because the US copyright office has ruled that you, as the person promoting an LLM, cannot assert copyright on the output, does not mean that someone else can’t. If you take a DVD and transcode it to H.264, there is no creative step and so the new copy is not something subject to independent copyright, but it is a derived work of the DVD (itself a lower-quality derived work of the original masters) and so subject to the same copyright.

Importantly in this story, the person prompting Claude had no idea that the original app existed. To safely use the code, they would need to search everything in the training data and discard outputs that would meet the bar of being substantially similar. And that’s something that requires human judgement.

I got tagged in a PR about the release notes for ML-DSA Native 1.0.0 because I contributed a couple of minor API tweaks. I cannot overstate what an absolute joy the maintainers of this project were to work with.

They have to have ludicrously high quality requirements because pretty much every possible use of their codebase is security critical. And that doesn't seem to stop them from being amazingly welcoming and friendly to first-time contributors. I hope I do half as well as them in my own projects.