I learned something interesting while out for a walk at night recently. I was walking across a lawn and noticed all these little sparkles catching the light from my flashlight. Thought maybe sprinkler droplets which hadn’t evaporated yet.
I looked closer, and nope! Spider eyes! This lawn had a *lot* of wolf spiders. Very cool.
My goodness, the amount of cargo-culted nonsense in IP networking is ridiculous. Currently dealing with yet another person who insists the router ID needs to be an address on the router and it should be a loopback. Absolutely incorrect, otherwise you would never be able to run dynamic routing on an IPv6-only system!
Starting to think we need a Networking for Network Administrators book. @mwl
I guess my next tool is technically going to be a set of low-profile Allen wrenches. The car’s an Audi C6/4F, and it also needs a new bushing for the snub mount (connects the front of the engine/transmission package to the front of the frame). The OEM bushing is some kind of foam like the shock bump stops which totally disintegrates by 30k miles. Metal-on-metal makes for rough shifting.
There’s very little clearance above the bolts on the engine side of the mount, and my socket bit set won’t fit.
Just finished two hours of Pilates (replacing the rear shocks on a car I’ve never done suspension work on before). I am exhausted and filthy, but it worked out!
Again, I’m incredibly glad I have a nice impact wrench. It made super quick work of the wheels. The lower bolt through the shock was brutal, but it gave after a few seconds.
I think my next tool is going to be a power ratchet. The wheel well liners involved a ton of T25 screws at awkward angles.
I think the idea behind vibrating the phone when the call connects is to let you hold your phone away from your head while it’s ringing. The haptics tell you when to hold it to your ear.
I still can’t stand it and I’m baffled why this would default to on.
@mcc Some time ago, you mentioned trying to find a way to disable a haptic vibration on your phone when someone picked up a call. iPhones do that too now. While I understood your aversion in the abstract, I can say I also *loathe* it.
At least it can be turned off on iOS. Settings > Apps > Phone > Haptics.
I just had a brief conversation involving how reads work under #ZFS, how flash-based SSDs actually perform, and how this means cache devices are only rarely useful. Thought I would elaborate a bit and share more broadly.
First, a quick description of ARC, the Adaptive Replacement Cache. It’s the ZFS read caching layer. It involves an index and two data pools. One pool stores most recently used data. The other pool stores most frequently used data, even if it wasn’t used recently. Any time data is read, the system checks the ARC index to see if the data is cached. If it is, the copy in RAM is returned. Otherwise, the storage is read. For most workloads, this combination works *really* well. The data you request is almost always in RAM, and only a small percentage of requests need to hit the much slower storage.
ZFS cache devices are often called L2ARC. They don’t really use ARC logic, though. Instead, cache devices are populated by data which is about to fall out of ARC. This is a constant trickle of data (usually about 10 MB/s) being written to the cache device. An index of what data is on the cache device is also stored in RAM. The amount of RAM spent on the index depends on the record size (100 GB of small records costs more to index than 100 GB of big records). Now, when data is requested, the system checks the ARC index, then the cache index, then the storage. If the cache index says the cache device has the data, it’s read from there rather than from the storage.
Everybody who has dealt with spinning disks knows they aren’t very good at reading and writing at the same time. The heads have to seek all over the disk. Unfortunately, SSDs have a very similar problem. If you give them an all-read or all-write workload, they’re really fast, but even a 10% mix and performance craters. Tom’s Hardware did an excellent test of an Optane drive a few years ago ( https://www.tomshardware.com/reviews/intel-optane-ssd-900p-3d-xpoint,5292-2.html )which demonstrated this problem. I’ve attached one of the more interesting performance graphs.
Note that at 90% read workload, performance of the non-Optane drives drops to less than half of what it is at 100% read. ZFS cache vdevs have a constant low-level write workload. They’ll never be above 90% read workload.
Meanwhile, the devices used to provide capacity to a ZFS pool only receive write workload when a transaction group closes. By default, this is every five seconds, but it can be less if the transaction group fills early. The rest of the time, there’s no writing. This leaves them free for 100% read workload for seconds at a time, even under heavy write load.
The result of this is that to provide benefit, a cache device has to be significantly faster than the pool devices. With SATA SSDs providing capacity, even an NVMe SSD is probably too slow to be a good cache device.
@neurovagrant And if you’re in the mood for sci-fi which makes you sad, Elizabeth Bonesteel’s Central Corps series and her standalone novel Arkhangelsk are all amazing. To me, they hurt like watching Arrival.
Arkhangelsk is mid-future. It’s set on a colony descended from a generation ship which left Earth at the height of a war they were certain would end humanity. They landed hundreds of years ago, and haven’t heard from anybody else. Then one of their observatories sees an object they’ve been tracking change course.
Central Corps is more like a mix of Star Trek and The Expanse. Far future, more strife than Star Trek, but less militaristic than The Expanse.
@neurovagrant I’ve been reading some Elizabeth Bear lately.
Jenny Casey - near-future sci-fi involving the discovery of alien technology on Mars. Cybernetics, AI, a new FTL engine, political intrigue, weird first contact.
Jacob’s Ladder - far-future sci-fi which feels like high fantasy at the start. It’s a little hard to discuss without spoilers. I enjoyed them a lot.
White Space - far-future, inspired by classic pacifist sci-fi like Sector General and Culture. The first book follows interstellar tow truck drivers, the second follows an interstellar ambulance crew.
I also just finished reading all of the Diving Universe books which are out. Far-future stories mostly focused on a salvage diver who explores old wrecks for historical significance rather than pillaging them. Has strong militaristic elements, but those feel more Star Trek rather than Expanse.
I wish to join the bunnies.
I shared this most recent story to remind myself of it because this week has been anything but fun troubleshooting. Just slog after slog. I get brought into some issue, and can tell in under an hour it’s not a problem in any of the gear anyone on my team owns or can access. Four hours later, we finally manage to lead the other team to the problem in their environment. One in another section of my company, one in a customer’s environment, and two so far in *vendors’* environments. Why am I the one finding problems in our vendors’ environments?
Then this F5 alarm fire breaks out.
And my NTP story for anybody who has somehow missed it:
Had another minor troubleshooting adventure recently! Not quite as exciting or far-ranging as my NTP story, but it was still fun.
I get pulled into a call. Something happened over the weekend, and I’m getting too much information from too many people. They’re asking me about load balancers, WAN circuits, a bunch of firewalls, and more. Not sure yet what actually happened. I have them slow down a bit and give me one thing which is affected. I confirm the path, and it goes through one firewall, over a WAN link to another datacenter, through another firewall, then to the server. No load balancers! It also skipped several other firewalls people had been asking about.
I ask for more problematic traffic flows. Eventually, I find an even simpler one: in one interface on the second firewall and out another interface on that same firewall. No load balancers. No routers. Just switches and one firewall.
I start packet captures on all the relevant interfaces. While waiting for a report of issues between those endpoints, we realize something interesting. You see, a bunch of separate teams have all been troubleshooting issues on their own. They’ve been keeping track of the issue timings separately. When we get enough together, we see that while the affected flows are rarely the same, *something* is happening every 20 minutes like clockwork.
The owner of the systems I’m watching reports they’ve had a traffic issue, so I kill the captures and pull them down. After a few seconds in the client-side capture, I’ve found the problem. Sure enough, some traffic just goes missing. Client sends some data, server doesn’t respond for several seconds. I look at the capture on the server-side interface. It’s different. The server sends the packet which was missing on the client-side capture. And the server-side capture doesn’t have some packets the client sent. So the firewall is definitely dropping or severely delaying some traffic. What gives?
Shortly before the issue is expected to occur, I start a debug which gives me a brief message about every packet the firewall drops along with a timestamp. I wait a few minutes, issues are reported, issues go away, and I kill it.
Copy it down, open it up, and I start by just scrolling through. The timestamps zip by, then I start seeing a lot of the same second. Then a *LOT* of the next second. I let it run for about five minutes, and 95% of the file has timestamps from a 15 second span. Huh.
This particular firewall brand works like a load balancer and a bunch of tiny firewalls in a box. It has worker processes which actually handle applying the rules, inspecting the data, and so on, and it has dispatcher processes which pick which traffic goes to which worker. A lot of the drops are problems enqueueing packets for a particular worker. And it’s all the same worker.
I look for that worker in the debug, and I find about 200k drops, all in the span of 15 seconds. None of the drops are for flows reporting problems, though. I do a little RegEx witchery to get just the source IP, destination IP, and destination port, then sort the result and pipe it to ‘uniq -c’ to get a feel for heavy hitters. Lots of tuples which appear one or two times, then two which appear 98,000 times each. It’s UDP 514 traffic from one client to a few destinations which were decommissioned some time ago. So this client is basically DoSing the firewall worker process, which prevents it from handling the other flows which are directed to it. The firewall has around 65 of these workers, so that explains why the issue is so sporadic! It only affects whatever ~1.5% of traffic is on the affected worker, and that changes all the time.
UDP port 514 is used for syslog. While waiting on someone from the team which owns the client sending all this traffic, I run another capture. I limit it to 500 packets, and I look for any traffic from that client to the two destinations it is trying to reach. It runs for a few minutes, then stops. I pull down the file and open it up.
Sure enough, it’s syslog traffic. It’s all from a sendmail process complaining it has no more space available in the filesystem which contains the mail spool. It can’t move messages around or accept any more mail.
The team which owns it joins and confirms they don’t need it, because they never actually finished setting it up. They approve shutting it off. 20 minutes later, no reports of problems! An hour later, and it’s officially declared resolved.
Lessons learned (really, reinforced):
• Document, document, document! Troubleshooting is a science, and that means you should follow the scientific method. Document your observations, tests you made, and what the tests told you. Document changes to your variables.
• Isolate a simple part of the issue and focus on that. If you find an even simpler version, change your focus to that simpler version. Once you learn what’s going on there, you can usually generalize it to the broader problem.
• Set up monitoring for your stuff early. Ideally before you set up any services or cronjobs on it.
• Decommission stuff you don’t actually use.
• It can be really hard to tell when an issue is just your problem versus when it’s something broader. If you’re dealing with weird, sporadic issues, record the timing and see if anybody else in similar teams has weird, sporadic issues. See if they mesh in any way.
If you haven’t seen the movie Primer, you should. It’s short, and really good.
Out for a drive today, and went past this place. Tempted to drop in and try rewriting some recent history.
When COVID started, I stopped going to the gym. My office closed for a while, then closed entirely, so I also stopped walking ~24 flights of stairs per day. As a result, I’m woefully out of shape.
To help address this, I’ve picked up rucking (walking/hiking/jogging with added weight). Just did two miles with ~29.5kg on my back at 17’05 per mile. Slowly improving, and my own weight is slowly trending down. Now I just need to keep it up.
It’s hard getting good photos of this dog because he’s so big and so fast. You try to line up a shot, and suddenly there’s a huge dog right there with his mouth around your wrist.
Visited some friends over the weekend. We had a bonfire for the solstice, then I got to spend a few hours hanging out with a high-percentage wolfdog grabbing my arms and demanding belly rubs.
Techbros are still learning a lesson Mattel shared with the world in 1983.
Everybody talks about how sad Jurassic Bark (Futurama s04e07, the episode with Fry’s dog) is, but the end of Game of Tones (s07e23) hits me *much* harder.
I’m not yet through the second season of Shrinking, but it consistently hits like a ton of bricks. Everyone involved in that show is amazing.