Two of my banks recently implemented #passkeys, and the contrast between them is stark.

I went to log in to one and they wouldn’t let me in until I “verified my identity” using a text message code to a phone number they apparently got from some data broker or by uploading my government ID and a video (obviously not happening). When I eventually got their support to let me in, the site forced passkey setup with no options at all. There’s no way to set up more than one passkey, no way to revoke access from the one I set up, and no way to log in via password anymore. They made every single decision in the most incorrect way possible. Utter garbage. Unfortunately, they’re the only option available to me for the particular account I have, so I’m stuck with them.

Meanwhile, Wealthfront reopened a support ticket I filed years ago requesting passkey support to let me know they had added it. I logged in with a password, and the site offered to set up a passkey. In my user profile’s authentication page, there’s a section listing all my passkeys (and app-specific password names, etc.). It allows me to specify friendly names to track which is which, add more, and remove keys I no longer control. It’s pretty much perfect!

#passkey #webauthn

New Spiderweb remaster of Geneforge 3 launched on Kickstarter about seven hours ago, and it’s already at 130% of the goal. Nice!

https://www.kickstarter.com/projects/spiderwebsoftware/geneforge-3-detonation-cult-classic-indie-rpg-remade

Looking at the Saleae Logic MSO. It uses USB Type-C ports for the digital probes, which decidedly *DO NOT* speak USB. Don’t like that one little bit.

At least they seem to run it over a (proprietary) Type-C alternate mode, so hopefully it shouldn’t blow up a real USB port if you plug it in.

Was discussing Git with some coworkers new to the concept of version control, and I had a flash of Glengarry Glen Ross combined with that awful LLM-plagiarized diagram Microsoft published.

CBM. C, continvoucly. B, be. M, morging. Continvoucly be morging!

@dgar Yeah, but KITT was actually smart and actually autonomous. I can't think of a single time when it locked Michael inside and burst into flames.

Made myself a nice bowl of yellow for dinner. Recipe in alt text.

About to hard-wire a dashcam for a friend, and I started by researching both the vehicle and all the hardware we’re planning to add. This uses a type of fuse tap I haven’t used before, but it’s basically the same as other fuse taps. While confirming that, though, I have run across a *ridiculous* amount of misinformation about fuses and fuse taps.

First, let’s talk about why cars have fuses at all. Wires in a car are specified to carry a particular amount of current safely. If the accessory at the other end of the wire breaks, it might create a short circuit, allowing the accessory to draw far more power than it ordinarily does. The higher current causes the wire powering that accessory to heat up, running the risk of an electrical fire (incidentally, this is an overload, as in the wire is carrying more load than it is rated for). Fuses prevent this situation. Once the accessory starts drawing too much power for the circuit, the fuse blows and there’s no more circuit. They don’t protect accessories—they protect the rest of the car from an accessory which has already failed.

Many (but not all!) electrical circuits in cars get one pole from the car’s frame or chassis, and the other pole is a wire from the accessory to a fusebox. The fusebox routes the wire to one end of a connector for the fuse. The other end of the fuse connector goes to a bus bar in the fusebox. The bus bar is a big, continuous piece of metal which can usually carry hundreds of amps (since it’s a part of all of the circuits handled by the fusebox).

Now, back to the taps. In general, fuse taps replace an existing fuse with two fuses with independent outputs. One output goes right back where the original fuse was, and you plug the original fuse into a slot on the tap. The other output has a whole separate fuse. Three pins: one common, one fused with the original fuse, one fused for the new circuit. Simple, right?

Where this gets complicated is having a common pin makes them directional. Lots of people are apparently confused about just what this means. In short, if you install the tap in one direction, you get two independent circuits. Flip is around in the same socket, and *both* paths depend on the current through the original fuse.

This is a little like the mechanism behind the 1981 Hyatt Regency walkway collapse in which a particular set of nuts were bearing twice as much load as they did in the design. Fortunately, installing fuse taps backwards can’t directly be deadly, it just might blow the original fuse more readily than you expect. With how big a hassle some cars make it to get to the fusebox (in my friend’s car, the interior fusebox is under the carpet in the passenger footwell), this can be seriously annoying.

The right direction to install the tap depends on how the fusebox is wired. You may have heard of “negative-frame” or “positive-frame” cars. Those approximately refer to which side of the 12v battery is connected to the frame. The fusebox generally has the other pole on its bus bar. Either negative frame with positive fusebox or positive frame with negative fusebox works, you just have to connect the correct pole from the accessory to the wire to the fusebox. Most cars are negative-frame, so you have a big +12v bus in the fusebox, fuses leading to wires, wires going to accessories’ +12v input, then the accessories’ ground input going to the frame or chassis.

The tap’s common pin is functionally a tiny bus bar, and it should be connected to the fusebox’s bus bar. In a negative-frame car, when you plug the tap in with the new accessory’s fuse but without the original fuse you’re replacing, the output for the new accessory should be +12v relative to the frame of the car.

It seems appropriate that in spite of the tarp, we have a photo of workers cleaning the P off of the Kennedy Center.

It’s more than a little weird that Apple decided to demonstrate their new LLM-centric features with a demo application centered around origami, a hobby rather famously being wrecked by LLMs generating impossible models and garbage directions.

Just had yet another call in which my team and I figured out a problem in an environment we have no access to. It took over 30 minutes to get the other company to even look where we knew the problem was. They had dismissed it using indirect evidence from a tool which, we discovered, wasn’t telling them quite what they thought it was.

I just had a brief conversation involving how reads work under #ZFS, how flash-based SSDs actually perform, and how this means cache devices are only rarely useful. Thought I would elaborate a bit and share more broadly.

First, a quick description of ARC, the Adaptive Replacement Cache. It’s the ZFS read caching layer. It involves an index and two data pools. One pool stores most recently used data. The other pool stores most frequently used data, even if it wasn’t used recently. Any time data is read, the system checks the ARC index to see if the data is cached. If it is, the copy in RAM is returned. Otherwise, the storage is read. For most workloads, this combination works *really* well. The data you request is almost always in RAM, and only a small percentage of requests need to hit the much slower storage.

ZFS cache devices are often called L2ARC. They don’t really use ARC logic, though. Instead, cache devices are populated by data which is about to fall out of ARC. This is a constant trickle of data (usually about 10 MB/s) being written to the cache device. An index of what data is on the cache device is also stored in RAM. The amount of RAM spent on the index depends on the record size (100 GB of small records costs more to index than 100 GB of big records). Now, when data is requested, the system checks the ARC index, then the cache index, then the storage. If the cache index says the cache device has the data, it’s read from there rather than from the storage.

Everybody who has dealt with spinning disks knows they aren’t very good at reading and writing at the same time. The heads have to seek all over the disk. Unfortunately, SSDs have a very similar problem. If you give them an all-read or all-write workload, they’re really fast, but even a 10% mix and performance craters. Tom’s Hardware did an excellent test of an Optane drive a few years ago ( https://www.tomshardware.com/reviews/intel-optane-ssd-900p-3d-xpoint,5292-2.html )which demonstrated this problem. I’ve attached one of the more interesting performance graphs.

Note that at 90% read workload, performance of the non-Optane drives drops to less than half of what it is at 100% read. ZFS cache vdevs have a constant low-level write workload. They’ll never be above 90% read workload.

Meanwhile, the devices used to provide capacity to a ZFS pool only receive write workload when a transaction group closes. By default, this is every five seconds, but it can be less if the transaction group fills early. The rest of the time, there’s no writing. This leaves them free for 100% read workload for seconds at a time, even under heavy write load.

The result of this is that to provide benefit, a cache device has to be significantly faster than the pool devices. With SATA SSDs providing capacity, even an NVMe SSD is probably too slow to be a good cache device.

I wish to join the bunnies.

I shared this most recent story to remind myself of it because this week has been anything but fun troubleshooting. Just slog after slog. I get brought into some issue, and can tell in under an hour it’s not a problem in any of the gear anyone on my team owns or can access. Four hours later, we finally manage to lead the other team to the problem in their environment. One in another section of my company, one in a customer’s environment, and two so far in *vendors’* environments. Why am I the one finding problems in our vendors’ environments?

Then this F5 alarm fire breaks out.

And my NTP story for anybody who has somehow missed it:

https://infosec.exchange/@bob_zim/111862834586135218

Had another minor troubleshooting adventure recently! Not quite as exciting or far-ranging as my NTP story, but it was still fun.

I get pulled into a call. Something happened over the weekend, and I’m getting too much information from too many people. They’re asking me about load balancers, WAN circuits, a bunch of firewalls, and more. Not sure yet what actually happened. I have them slow down a bit and give me one thing which is affected. I confirm the path, and it goes through one firewall, over a WAN link to another datacenter, through another firewall, then to the server. No load balancers! It also skipped several other firewalls people had been asking about.

I ask for more problematic traffic flows. Eventually, I find an even simpler one: in one interface on the second firewall and out another interface on that same firewall. No load balancers. No routers. Just switches and one firewall.

I start packet captures on all the relevant interfaces. While waiting for a report of issues between those endpoints, we realize something interesting. You see, a bunch of separate teams have all been troubleshooting issues on their own. They’ve been keeping track of the issue timings separately. When we get enough together, we see that while the affected flows are rarely the same, *something* is happening every 20 minutes like clockwork.

The owner of the systems I’m watching reports they’ve had a traffic issue, so I kill the captures and pull them down. After a few seconds in the client-side capture, I’ve found the problem. Sure enough, some traffic just goes missing. Client sends some data, server doesn’t respond for several seconds. I look at the capture on the server-side interface. It’s different. The server sends the packet which was missing on the client-side capture. And the server-side capture doesn’t have some packets the client sent. So the firewall is definitely dropping or severely delaying some traffic. What gives?

Shortly before the issue is expected to occur, I start a debug which gives me a brief message about every packet the firewall drops along with a timestamp. I wait a few minutes, issues are reported, issues go away, and I kill it.

Copy it down, open it up, and I start by just scrolling through. The timestamps zip by, then I start seeing a lot of the same second. Then a *LOT* of the next second. I let it run for about five minutes, and 95% of the file has timestamps from a 15 second span. Huh.

This particular firewall brand works like a load balancer and a bunch of tiny firewalls in a box. It has worker processes which actually handle applying the rules, inspecting the data, and so on, and it has dispatcher processes which pick which traffic goes to which worker. A lot of the drops are problems enqueueing packets for a particular worker. And it’s all the same worker.

I look for that worker in the debug, and I find about 200k drops, all in the span of 15 seconds. None of the drops are for flows reporting problems, though. I do a little RegEx witchery to get just the source IP, destination IP, and destination port, then sort the result and pipe it to ‘uniq -c’ to get a feel for heavy hitters. Lots of tuples which appear one or two times, then two which appear 98,000 times each. It’s UDP 514 traffic from one client to a few destinations which were decommissioned some time ago. So this client is basically DoSing the firewall worker process, which prevents it from handling the other flows which are directed to it. The firewall has around 65 of these workers, so that explains why the issue is so sporadic! It only affects whatever ~1.5% of traffic is on the affected worker, and that changes all the time.

UDP port 514 is used for syslog. While waiting on someone from the team which owns the client sending all this traffic, I run another capture. I limit it to 500 packets, and I look for any traffic from that client to the two destinations it is trying to reach. It runs for a few minutes, then stops. I pull down the file and open it up.

Sure enough, it’s syslog traffic. It’s all from a sendmail process complaining it has no more space available in the filesystem which contains the mail spool. It can’t move messages around or accept any more mail.

The team which owns it joins and confirms they don’t need it, because they never actually finished setting it up. They approve shutting it off. 20 minutes later, no reports of problems! An hour later, and it’s officially declared resolved.

Lessons learned (really, reinforced):
• Document, document, document! Troubleshooting is a science, and that means you should follow the scientific method. Document your observations, tests you made, and what the tests told you. Document changes to your variables.
• Isolate a simple part of the issue and focus on that. If you find an even simpler version, change your focus to that simpler version. Once you learn what’s going on there, you can usually generalize it to the broader problem.
• Set up monitoring for your stuff early. Ideally before you set up any services or cronjobs on it.
• Decommission stuff you don’t actually use.
• It can be really hard to tell when an issue is just your problem versus when it’s something broader. If you’re dealing with weird, sporadic issues, record the timing and see if anybody else in similar teams has weird, sporadic issues. See if they mesh in any way.