I've enjoyed the how-fast-is-`cat` benchmark for a long time for terminals, but I am sad to say its done. We've finished it. Libghostty processes plain text data faster than both the macOS and Linux kernel can feed it through a pty fd read. It can't get any faster without kernel changes. 😮‍💨

That doesn’t mean we process ALL data faster than a kernel read. Only plain text (both ascii and unicode with full grapheme clustering, btw).

There are still slow OSCs, especially those that have side effects (e.g. clipboard ops). So processing faster than the current can write still has benefits: when we get stalled on slow sequences, we can catch up through the buffer faster on text runs.

libghostty has always been fast, but it's also been getting significantly faster. Here are IO throughput measurements comparing Ghostty 1.3.1, 1.4.0, and main as of this morning. We'll be backporting most of these to a 1.4.1 release soon.

Here are some highlights of the changes since 1.4:

- Ghostty 1.4 had a fast-path for printing runs of plain text. But the fast-path only applied to limited cases. We now support mixed width characters and hyperlinks in the fast path.

- Our input path always widened the pty data to codepoints (8-bit to 32-bit) as part of UTF-8 decode, which also incurred a memory copy. We now have a SIMD fast path to detect pure-ASCII and we process the run direct from the read buffer (no copy, narrower data).

- New memory pages use most recent two pages actual utilization to pre-size themselves accordingly. This sped up styles, hyperlinks etc.

- When a VM page runs out of capacity for a certain type of data (e.g. hyperlinks, styles), we now project the usage so far to guess what you'll need, then shrink available rows to avoid a realloc. Before we'd naively double. This caused a classic "laddering" problem.

- We recognized "erase" being a slow path for us, and it's a common sequence, so we use another bit in row headers to track blank (we had space).

- Styles are now processed once per SGR. Previously, a sequence like "ESC[1;38;2;r;g;bm" would apply each style as it came in (1, then 38, then 2, etc.). We now apply all attributes at once and write to the cursor atomically.

- Styles are now packed into 128-bit values and use aarch64 and x86_64-specific assembly for minimizing instructions to use it.

- On x86_64, we found that LLVM was lowering some row resets (hot path) to a 4-byte store followed by an 8-byte load which couldn't be store-forwarded. This caused an almost 10% (!!!) slowdown on scroll operations only on x86_64. The fix was to change our reset to build the full header in a register then do a single memmove per row.

Net results are what you see in the graph, but written out main to 1.4: ASCII 1.5x, heavy styling 1.5x, styled Unicode 1.8x, random UTF-8 1.4x, random OSC 8 2.7x, mixed styled text with links 4.8x, cursor/erase 16x, hyperlink-dense text 5.6x.

Been working hard on Rex server scalability. In short: we were able to run 10,000 active terminals simultaneously with 3.8 GB memory usage while maintaining responsiveness. We didn't try more yet, we want to improve some numbers before going bigger, but the goal is 100,000.

This wasn't free and to get to this point required a lot of very fun engineering challenges! Here are just a couple:

- Linux fork preallocates enough fd table space for the highest open fd at fork time to the nearest power of 2. For terminals beyond ~5000, that was 512KB per terminal process JUST for a fd table that had ~4 fds. We now use a trampoline written in C to unshare the fd table so it shrinks back down to fitting the minimum fds.

- We previously kept all active (non-snapshot) terminals blocked on pty read, but that takes up an OS thread which at this scale uses a prohibitively high amount of RAM. We now move all quiet pty fds (2 seconds without read activity) to a shared epoll thread.

I didn't compare to any other multiplexers at this time. I'm not particularly interested at this state to have relative numbers against others, because the enemy I'm trying to beat is myself. We'll continue to improve these numbers, these are still alpha-quality builds.

The status of the program status specification I shared yesterday in alphabetical order: integrated in Amp, Factoryai, TUIOS, libghostty. Verbal support and in-progress PRs by Claude Code, cmux, Codex, Herdr, OpenCode, Pi. Requests in to many others. Not bad for 24 hours!

I've published a new, general-purpose terminal specification that gives any program a way to tell the terminal what it's doing: idle, working, waiting, finished, or failed, and why. It is easy to implement on both sides. I've written more about why: https://mitchellh.com/writing/program-status-osc7501

The problem this solves is generic and applies in compelling ways to all sorts of use cases (e.g. package managers, build tools), but it really rears a hideous head when it comes to AI and the "agentic inbox" problem.

I found over 250 different agent orchestrators (with various goals) that all individually implement a heuristic based approach to detecting whether things like Claude Code are working, blocked, done, etc.

If you look at Herdr’s commit history, you can see around 10 compatibility changes to detect Claude Code within just the past 3 months. (This isn't a dig at Herdr! They're doing great work with what they have, and everyone else is doing the same thing).

Heuristics are not the way. And over-indexing on agentic use cases isn't the way, because this problem is really interesting for other tools too: Homebrew, Terraform, Cargo, etc. etc.

Proprietary protocols and out of band APIs are not the way. They have an O(N) integration problem and require extra work to work over SSH or within local VMs/containers.

The program status specification (OSC 7501) solves this in a reliable, deterministic way that is general-purpose and can be easily adopted by the entire industry.

Read more: https://mitchellh.com/writing/program-status-osc7501

Terminal programs bias to dark modes and explicit colors are hostile to light modes and themes. We now detect programs when they request colors too out of whack from your theme, apply some Oklab magic and hue rotation to match your theme. Harmonious themes, everywhere, always.

New week new demo! Today I want to showcase the CLI to control the Superlogical multiplexer. The CLI is an extremely powerful tool for automation and hooking into other tools like editors, agentic coding tools, and more. And, it's super fun!

https://youtu.be/9fcfDF8SBnc

Everything you can do in the GUI, you can do via the CLI. Actually, you can do more in the CLI. You can create sessions, run commands, make splits, move things around, change focus, simulate any input device (key/mouse/etc.), wait on scripts, and more.

There are a couple unique parts of Rex I'm showing off here too: our streaming event system that any client can hook into (or script). And the terminal-specific API to retrieve information like a JSON object of the running process.

I hope this demo lets your imagination run free of where you might slot something like this into your existing tools.

A quick look into remote persistent sessions with Superlogical! Not just remote sessions, but some functionality like remote directory lists, too. Superlogical can actually be a full SSH replacement (I don't run SSH on my servers anymore), and I talk about that briefly, too.

https://www.youtube.com/watch?v=PdwTjSBW6Y8

RE: https://hachyderm.io/@mitchellh/117202903279918618

Guess which one is Rust btw. 🤣

(The point is the language isn't a free lunch for performance. Rust could be just as small as any other here. It isn't a given. Fundamentals and system design trump everything else. Language wars are stupid as hell.)

RE: https://hachyderm.io/@mitchellh/117202903279918618

A video explaining a handful of the tricks we use to optimize for memory within the Superlogical server: in-memory compression, moving file descriptors between event-driven and blocked threads dynamically, full terminal state binary snapshot/restore in microseconds, and more!

https://www.youtube.com/watch?v=T5gV6anSt-4

The Superlogical server is built to run anywhere and everywhere and serve a level of scale (terminals+clients) unmatched w/ prior solutions. So let's start talking about that, starting with memory usage and comparisons to other popular multiplexers.

I want to start by talking high level: why does terminal/client scale matter? This matters because we're seeing unprecedented terminal usage and client attachment mainly due to AI. They like to spawn more sessions than we've ever seen, and they are all their own clients. So we have to optimize it. Even if you don't use AI/agents, this just makes the software better for humans, too.

Okay, onto the results. First thing to notice is that while tmux's empty states are excellent, Superlogical makes it up at scale and there is an early breakpoint with busy terminals and clients where we use significantly less memory than tmux or anyone else.

Zellij simply does very poorly. There is no other way to put it. Its default out of the box empty footprint is high. And its per-client attachment is simply unscalable. The best I can guess (I haven't talked to them)is that it was designed in its current form for small single-human low-terminal-count scale on relatively beefy servers.

The per-client attachment for Zellij was so unbelievable I had to dig into why. Out of the box (zero config) Zellij has a tab bar and status bar plugin. These are wasm-plugins and each tab gets an instantiation. Each instantiation is its own wasm runtime. Huge memory explosion. I also noticed the memory was never reclaimed when I closed the tabs (even when waiting for minutes for memory to settle) but didn't dig into that more.

Anyways, credit where credit is due tmux has excellent memory usage for empty states. The tmux project has always been great and they've been really receptive to the bugs I've found via Ghostty or the protocols we've championed. Much respect.

I'm going to talk more about how we achieve this in a follow-up video.

I can't tell you the number of people over my career that have said "I tried learning tmux" First problem: the fact you have to "learn it" at all. The world's best multiplexer is the one you don't have to learn. It just works. That's what we're building (and then some 😜).

Quick Superlogical demo to end the week: https://www.youtube.com/watch?v=XyxShCZrGiw

This demo is just of the basic functionality, but I want to highlight just how FAST it is. There's a WHOLE lot more functionality already present and coming but I wanted to keep the demo short. But seriously, check out that speed.

I'm showing the macOS app here and I get asked a lot so let me just say up front: its not macOS only. We're still figuring out exactly what platforms will be stable enough for the initial public release, but we're working on others. In particular, the web interface is very functional but I'll demo that another time (or someone else will).

This is also all showing off the architecture I previously talked about for terminal multiplexing which is significantly different from other mainstream multiplexers: this is all a custom binary protocol where the server is maintaining N replica distributed terminal state machines. It works!

Tons of hard work going into this by the team. Hope we can bring you something you can use soon. ❤️

The Ghostty memory usage memes are dead with the upcoming 1.4 release, much to the dismay of internet trolls everywhere. ~10x less memory for a visible window. ~450x lower for a non-visible window. Very reasonable compared to other terminals with 10 tabs.

This was the result of a lot of incremental hard work, but there were two big wins. One I talked about before which was scrollback compression. I want to note that these charts do NOT include scrollback, so with scrollback the results are more dramatic!

The other big win is that we free almost all GPU resources when a window is fully occluded (covered by another window, minimized, etc.). We triple-buffer our screens so for large screens this has major savings.

Those account for a large chunk of the improvements, but we also had dozens of incremental 1% wins here, a few KB there, etc. throughout the 1.4 cycle. And there are more coming!

I noted before that Ghostty's memory waste was all in the app layer and we can see it here. libghostty has always been very tight in comparison and I've shown comparisons to other embedded terminal libraries elsewhere.

And importantly, the charts below use the same grid sizes with a very small +/-2% difference due to differences in how cell metrics are calculated between terminals.

OSC parsing in libghostty was our last non-SIMD optimized VT stream path. OSCs are usually small so it just never showed up on benchmarks. But now that we are potentially passing megabytes of data (Kitty clipboard), it got the SIMD treatment. 🫡

I still read and write code myself and love it. But, I think this is a perfect example of how AI assistance can practically help. Here is how I had Fable+Sol work together on the aforementioned validation set:

1. Sol put together the harness.
2. Fable + Sol (two separate agents) in parallel would write test cases and output them on disk in their own folders. These two are just in a ralph loop.
3. Sol + Fable (reversed) with an adversarial prompt would judge the others work by picking up the changes on disk that step 2 wrote. They would determine how accurate/worthwhile it is to keep. They'd put it in another folder.
4. Sol finally woke up for changes to this final folder and would determine if its a dup or not and then add it to the final repo.

This happened mostly while I wasn't at the computer. Then, when I got back to the computer, I'd manually pick up the bug reports, validate them myself, and either fix them myself or kick off new agents manually.

Finally, at the end, I re-ran both agents once against the full test suite to verify what I saw myself: everything passed, all images look the same. I verified this manually, but had them double check for me because why not.

Common question, what did this cost me: two subscriptions, each $200/month each, so $400 total. I didn't utilize them to their full limits and I use them for other tasks too so I don't have the exact quantified cost of this specific set of tasks.

RE: https://hachyderm.io/@mitchellh/117135178412268410

Part of the LLM-assisted work here is that Fable+Sol put together a test suite of over 400 cases that are bare shell scripts plus a harness that runs Kitty and Ghostty GUIs and captures their full pty stream AND screenshots (multiple, for animations).

For the pty stream we assert byte equality. So Ghostty's success and error messages and field order of responses (its k=v) directly matches Kitty byte for byte.

For screenshots, we can't do pixel-identical comparisons because the way Kitty and Ghostty calculate grids and do alpha blending doesn't match. But, LLMs are pretty good now at "do they look the same" PLUS I went through all the screenshots myself.

Super helpful AI assist. It allowed me to focus a lot of my brain energy and time on reading the spec, reading/writing the implementation, considering the right shape of things, while I had a couple very good interns in the background doing work like this.

Ghostty (and libghostty!) now has a 100% complete implementation of the Kitty Graphics Protocol. We support every feature, every option. We are the first terminal beyond Kitty to do this.

Most terminals stop at basic image placements, then give up on the long tail of other features. Ghostty covers all of them:

- Unicode placeholders (so it works in tmux)
- Relative placements (so images can be positioned relative to other images)
- Animation
- Scroll region handling (so images scroll properly with terminal scroll regions)
- All transport mediums: shared memory, temp files, etc.
- etc...

All of these features don't impact steady-state use (CPU or memory, besides raw code size). And for libghostty consumers, you can use the compile-time features flags to disable them completely.

I read the spec myself a hundred times. I read Kitty's source code to validate behaviors (our implementation is markedly different, so no GPL violation).

I also utilized agents, primarily for validation. I spun Fable and Sol in a loop for 3 weeks (pausing only when their dedicated subs exhausted limits) with a prompt to find mismatches between the spec, Kitty's reference implementation, and Ghostty.

And now we're done. Ghostty is the most compliant Kitty Graphics implementation (ignoring Kitty itself of course). And, this is all in libghostty, so now everyone can be perfectly compliant.

I've moved on to optimizing libghostty Wasm runtime memory usage and I am learning a lot! The first discovery: Zig's `std.heap.MemoryPool` is very bad for Wasm! We replaced it with a custom Wasm-specific pool that lowered libghostty's required terminal memory by 75% without any measurable throughput impact.

Some background: `std.heap.MemoryPool` is a very high performance memory pool implementation for identically sized items directly in the Zig standard library. For non-wasm targets, its very good! We use it a lot!

But there is a major problem due to the interaction of two separate behaviors: first, the Zig std memory pool grows by 1.5x whenever it has no free space. To grow, it asks the backing allocator, which is usually general-purpose. On Wasm targets, that general-purpose allocator is the BrkAllocator which grows by power-of-two for big allocation slots (what we were hitting).

On native (non-wasm) targets, this just gets increasingly large virtual memory allocations, which is fine! Good, even! Virtual memory mappings are cheap and the total mapped memory isn't resident/physical until it's used. Allocation syscalls are expensive and its better to get too much.

But wasm is not like that. Every linear memory growth is immediately measured, zeroed, wired memory. Second, memory growth doesn't require a "syscall" in the same sense/overhead of the word.

The result is you get really big linear memory growth in WebAssembly and it looks/feels bad in web inspector and saves absolutely nothing on future IO.

We replaced it with a very simple pool (~20 lines of effective code) that grows the linear memory by exactly 1 item size when it needs it and maintains a module-global free list (assumes single-threaded Wasm module) so same-item pools are shared globally.

PR here: https://github.com/ghostty-org/ghostty/pull/13865

cc @andrewrk I think you'll like this.

For Wayland people, one of Ghosttys maintainers made a libghostty based Wayland terminal. It starts up faster, has lower input latency, lower memory usage, and higher IO throughput than foot. If you don’t want the heavy GTK overhead of Ghostty this is the way. Libghostty is the way. https://github.com/rockorager/monstar