"Disconnect our fridges from the internet — NOW" is kind of the simplest policy to implement, yet many companies and hospitals failed to do just that for months, so these fridges keep getting manipulated by script kiddies.
That's the story.
"Disconnect our fridges from the internet — NOW" is kind of the simplest policy to implement, yet many companies and hospitals failed to do just that for months, so these fridges keep getting manipulated by script kiddies.
That's the story.
Here's the simple emergency action plan:
"Until we have a proven and tested way to make sure that control messages that can change the temperature setting of our fridges can only be sent from known good sources, we block all control messages, either in transit or at the device. If this means disconnecting and taking sensor data by hand for the time being, so be it."
@jwildeboer But how do you disconnect that WLAN cable?! 🙀
@jwildeboer I'm visiting my parents this week.
One of their first technical requests was asking if I could get their new fridge connected so they'd get filter reminders.
We had a talk.
@jwildeboer "we don't have the money for a firewall in that location but we need connectivity for the legally mandated temperature check protocol and the documentation for it"
Now what?
@Laird_Dave @jwildeboer in our shop we walk around writing down the temps for the morning using eyes, pen and paper. Legally mandated ✅
Hackable (bribable), Hmmm, how much are we talking.....😆
@mrshsaxon @jwildeboer one shop != an entire chain, one chain != each and every retailer
Especially in retail margins are slim and many, many owners, managers and others try to cut costs wherever possible, thereby generating (often unnoticed) risks.
@Laird_Dave And they got their first wake up call a few weeks ago. A second one now. If they keep on ignoring the problem, they will be next and face far higher costs to fix it and in addition a big hit to their reputation. That's called risk calculation. Make the wrong decision and you risk going out of business. @mrshsaxon
@jwildeboer @mrshsaxon I agree. However, there are a lot of things that are not reasonably fast and easy taken off the grid.
Where this is possible, it would be absolutely moronic to not implement it as a stop-gap-measure at least until something better can be implemented.
For other applications - they're up the creek without a paddle...
@Laird_Dave A stop-gap measure could be to configure the fridges to only send sensor data (that's what they need to collect to satisfy regulations) and ignore/block all control messages that can change the temperature and other settings. An even more simpler temporary solution would be to disconnect completely until a solution is ready to be deployed and take the measurements by hand for now. Solutions are not the problem, the (un)willingness to act is ...
@jwildeboer But why are they failing to implement such simple policy? Can it be that they dependent on such fridges to be connected?
PS: Are those fridges really on the internet? At least from the experience I made with retailer, they are behind a VPN (that hundred of underpaid support workers in different countries have access to)
@jwildeboer There are different stories....
@jurjen_heeck I know them all. I was there, talking with customers, when the whole Siemens PLC story unfolded and networks were redesigned to mitigate the risks. That was some 5-7 years ago. Good CISOs learned from that. So no, I have zero sympathy for the cheapskates that decided to ignore the whole issue because they convinced themselves that no one would ever care about their exposed fridges ...
@jwildeboer @jurjen_heeck The same goes for printers and advanced photocopiers IMO. Looks bland, innocuous, but should be shot when making the wrong noise.
@koen_hufkens Yep. Buy a used office print station, extract the hard drive or SSD inside and find tons of stored print files with loads of confidential information because removing or at the very least wiping the drive isn't part of the decommissioning process ;) @jurjen_heeck
@jwildeboer @jurjen_heeck For that reason probably still #1 targets for many national security agencies (in both physical and remote compromises).
@jwildeboer
Die Preisetiketten an den Regalen sind auch digital und werden zentral gesteuert. Die Kassensysteme und Scanner auch.
Kühlschränke sind das geringste Problem.
@jwildeboer starts to sound cheap compared with all the food that was canned, right?
@goncalor It should be part of any decent operations plan. You should have plans you regularly test to be prepared for emergencies ranging from "no connectivity at all", "connectivity is unreliable", "connectivity is limited to 1 hour per day" etc.
@jwildeboer simple to write, much harder to actually do in practice
@dequbed I work with customers who not only plan emergency procedures, they actually test them in practice at random times. It's called "being responsible" :)
@jwildeboer True, but do those emergency procedures include a sudden hundred-fold multiplication of workload within strict and very short deadlines?
@dequbed How about an airport having to deal with a sudden and complete outage of one of their datacenters? At a random time? This is a real scenario that they test at least twice a year. By going through that, they have identified bottlenecks and unexpected problems that they fixed. Experience beats theory, is their mantra.
@jwildeboer I don't think that's comparable. For one, airports mostly deal in heavily redundant systems (backup radars for example) which can run in parallel (which a fridge obviously can't) and secondly airports have the option of diverting and grounding all flights. You can not stop food distribution.
@jwildeboer I don't disagree that having a plan here is important. But I think it's a bit presumptive to just go "take measurements by hand". That's replacing a highly efficient and entirely automated system with a completely manual one taking a *lot* of time. Even if noting measurements were instant, you have to go to every fridge, in a full warehouse of them. Just moving the human around will take a significant amount of time.
@dequbed It's the final boss of an emergency plan. The stage you hope to never reach. But if all else fails, you need to have such a plan. Of course it is far better to have mitigation plans that work without such catastrophic disruptions. Maybe we will see more of that after all of this. That's my hope.
@jwildeboer honestly, in this situation I would seriously consider the emergency plan of "disconnect fridges from the internet, eat the fine for not being able to provide temperature tracking and tell people to thoroughly cook their food" instead.
@dequbed @jwildeboer A simple firewall in front of the network - something any router shipped in the last decades - would have blocked random script kiddies from doing this.
I'm not one for victim blaming. But for this to be actually exploitable, one had to embody the dictionary definition of "gross negligence".
@larsmb @jwildeboer Its easy to say that from the outside, yes. I'm not saying the different food distributors and hospitals aren't at fault here, but consider: If it truly where as simple and trivial as you're saying it would have been done. There were real-world reasons beyond just greed and gross negligence at play here, and I feel like you're ignoring those reasons.
@dequbed I look at the results, not at the theory. Fridges are still being attacked and manipulated, weeks after the first stories hit the public. The responsibility to act on these risks lies at the companies that operate them. I can only observe how they mitigate those risks. I am not responsible or involved. But I can use this to share possible approaches. Nothing more. @larsmb
@jwildeboer @larsmb I fully agree with you here. There is a responsibility here that isn't being fulfilled and it's a critical one at that. I'm just noting that the particular emergency plan you brought up does ignore the practical reality of the situation as well.
@jwildeboer The key problem as always was to get customers to actually apply those updates and put the new practices into place.
Or even update systems that are so old that they're out of the limited maintenance cycle. (The CRA et al are going to help with that, but not retroactively for the huge footprint that already exists ...)
GenAI is great at finding vulnerabilities, but it seems to not help at all with fixing the actual problems in the industry 🙂
@jwildeboer or, in other words, extremely basic / standard considerations for ANY piece of software.
@jwildeboer seemingly these fridges were not only connected to the Internet but many of them were still using the default user name "admin" and password the current date. Presumably they also routed through a local network so why wasn't admin access, if not just local to the network, restricted at the router to a secure VPN from wherever they were managed from?
@jwildeboer
There are a lot of apologists for why various infrastructure both is and "should be" connected to the Internet, even among IT experts, and it'd all bullshit. What no one wants you to know I'd that the real reason they connect everything to the Internet is because they're f***ing.cheapskate bastards.
1/
@jwildeboer because they need the insurance and monitoring/alerting for legal requirements
@eckes You did not read my first reply to my post, I guess? https://social.wildeboer.net/@jwildeboer/117591203640974728
@jwildeboer I would just ask the vendor to fix it, much less work
@eckes The vendor did. The customers didn't install the updates nor did they change the default password ...
@jwildeboer well, why would you expect them the. To have night shifts to read thermometers…