Login
You're viewing the mastodonapp.uk public feed.

Replies

  • Hanno Zullahzulla@infosec.exchange
    Oct 31, 2026, 11:15 AM

    It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting may have on the global electricity grid.

    💬 13🔄 23⭐ 92
  • 💬 1🔄 9⭐ 48
  • 💬 0🔄 1⭐ 0
  • Veronica Olsen 🕷️veronica@mastodon.online
    Oct 31, 2026, 11:01 AM

    @hzulla Similar problem for servers and clusters with on the hour spikes in compute. It was a big issue in my previous job. Especially around 14:00 when most data sources we used had been updated.

    💬 1🔄 1⭐ 23
  • Violetamethyst@hachyderm.io
    Oct 31, 2026, 12:37 PM

    @veronica @hzulla

    Yeah, I know at my last place we had to add deliberate a deliberate couple minutes of jitter to user-scheduled jobs because clients would all choose the same time!

    💬 1🔄 2⭐ 2
  • Veronica Olsen 🕷️veronica@mastodon.online
    Oct 31, 2026, 1:31 PM

    @amethyst We had the additional fun result that since our VMs ground to a halt at 14:00, people moved to scheduled CI jobs instead, so then between 14 and 16 each day, no one could get a PR through because all the runners were queued up for the next 2h. 😅

    💬 0🔄 1⭐ 3
  • benbenjamineskola@hachyderm.io
    Oct 31, 2026, 11:14 AM

    @hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.

    I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.

    💬 4🔄 0⭐ 2
  • 💬 0🔄 0⭐ 4
  • 💬 2🔄 0⭐ 4
  • 💬 1🔄 1⭐ 1
  • 💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Marcus Müllerfunkylab@mastodon.social
    Oct 31, 2026, 2:04 PM

    @benjamineskola @hzulla I think that premise, "cron would wake up on the minute mark" is incorrect: you'd randomize the time it sleeps, in both positive and negative directions Compare how systemd-timers sleep; they already have a well-configurable randomization of wake times, exactly to combat loading a shared resource with a large number is simultaneously triggering machines.

    💬 1🔄 0⭐ 0
  • benbenjamineskola@hachyderm.io
    Oct 31, 2026, 2:09 PM

    @funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)

    Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.

    @hzulla

    💬 2🔄 0⭐ 1
  • Marcus Müllerfunkylab@mastodon.social
    Oct 31, 2026, 2:17 PM

    @benjamineskola @hzulla agreed. Waking up happens hundreds to thousands of times per second to a modern application processor running an OS like Linux, Windows, *BSD anyways. The extra cost of drawing a pseudorandom or even random number and going back to sleep would be negligible to impossible to measure, even at large scales.

    💬 2🔄 0⭐ 0
  • benbenjamineskola@hachyderm.io
    Oct 31, 2026, 2:22 PM

    @funkylab there is a related problem, which is when hourly/daily jobs get run all at the same minute across a large number of machines — updates for example. The issue there is either network traffic or load on the update servers though; and spreading it out doesn’t (and isn’t intended) to reduce the overall load, just to reduce the size of the spikes.

    So I guess we also have to consider whether the power spikes are an issue in themselves (if they’re even measurable), or whether we’re more concerned with overall power consumption.

    @hzulla

    💬 2🔄 0⭐ 0
  • Marcus Müllerfunkylab@mastodon.social
    Oct 31, 2026, 2:29 PM

    @benjamineskola @hzulla well they certainly are if whatever the machines are doing is suddenly power intense, and if the machines are a significant consumer in the synchronous grid. (Which my guess would be is something data center operators need to negotiate when getting a grid operator to connect up their DC; they need to buy cable capacity and build transformers for Max load, not just median load)
    At a residential PCs scale, wouldn't worry about it much. English tea kettles & halftime breaks.

    💬 1🔄 0⭐ 0
  • EQeq@mas.to
    Oct 31, 2026, 8:45 PM

    @funkylab @benjamineskola @hzulla When I first started with computers we had gotten our hands on some old harddrives that were about 5 cm thick. They had some jumper settings for how many seconds to delay spinup on boot. Since they were pretty heavy the engines would draw too much power if you start two harddrives at the same time and kill the power to the computer. I think the new hdd's we bought when we had money had the same thing for a few years.

    My cron jobs are still all running a few minutes off the hour where I can thanks to this experience. Some have randomized wait times between starts.

    💬 0🔄 0⭐ 0
  • 💬 2🔄 2⭐ 0
  • benbenjamineskola@hachyderm.io
    Oct 31, 2026, 6:46 PM

    @johntimaeus oh yeah, I don’t mean to say it’s a new or unsolved issue, but rather something that’s been a consideration in running scheduled jobs for a long time by now.

    @funkylab @hzulla

    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Hanno Zullahzulla@infosec.exchange
    Oct 31, 2026, 2:25 PM

    @funkylab @benjamineskola My guess it's not cron/systems waking up on specific times being the problem, but maintainers setting up larger jobs (such as backups) starting at the minute mark. Too many things globally are set to run *something* at xx:xx:00 and this seems to add up to a measurable strain on the grid, thanks to synchronized system clocks. The researchers point out that this strain is not a giant problem yet, but it can be measured globally.

    💬 1🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Gabriele Sveltogabrielesvelto@mas.to
    Oct 31, 2026, 5:51 PM

    @benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.

    💬 2🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • James 🌈💜shaknais@mastodon.social
    Nov 1, 2026, 3:21 AM

    @DamonHD

    RANDOM_DELAY is part of the POSIX standard, so I actually would be very, very surprised, if they don't.

    Cronie, anachon, and even systemd supports it.

    💬 1🔄 0⭐ 0
  • DamonHDDamonHD@mastodon.social
    Nov 1, 2026, 8:22 AM

    @shaknais Even if it were available, it seems to randomise by whole minutes, so not potentially easing the problem in this article.

    "The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable."

    My current server cron is Vixie, which does not support this feature.

    💬 0🔄 0⭐ 0
  • 💬 0🔄 1⭐ 0
  • DerPumu (he/him)DerPumu@mstdn.games
    Oct 31, 2026, 6:50 PM

    @benjamineskola @hzulla depends. Is cron checking the time the power spike or the actual job itself? I'd expect the on-the-minute bump of cron setting a random delay to be small compared to all the busy work smeared across the rest of the minute.

    💬 0🔄 0⭐ 0
  • Tim Ward ⭐🇪🇺🔶 #FBPETimWardCam@c.im
    Oct 31, 2026, 11:36 AM

    @hzulla One does however wonder about the effect of switching domestic scale consumption/generation on the half hour when (in the UK) the electricity price changes. Obvs you want to switch your battery from fully charging to fully discharging exactly on the half hour because that makes you the most money ... but what if everybody does this?

    💬 1🔄 0⭐ 0
  • 💬 1🔄 0⭐ 0
  • 💬 2🔄 0⭐ 0
  • DamonHDDamonHD@mastodon.social
    Oct 31, 2026, 12:34 PM

    @TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.

    I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.

    💬 1🔄 0⭐ 1
  • 💬 0🔄 1⭐ 0
  • 💬 0🔄 2⭐ 3
  • 💬 1🔄 1⭐ 6
  • patrislav ♾️ #RIPNatenompatrislav@chaos.social
    Oct 31, 2026, 12:10 PM

    @jane namely RandomizedDelaySec in systemd.timer.

    It was probably intended to avoid CPU and network congestion, but it’s interesting that we also have to think about power spikes.

    @hzulla

    💬 0🔄 0⭐ 4
  • George Bgbargoud@masto.nyc
    Oct 31, 2026, 1:08 PM

    @hzulla

    At google, the internal cron service has a "@" option in the crontab which effectively hashes the name and uses that to pick a value for the entry so it's stable for a given job but globally distributed.

    So instead of `0 0 0 * * *` for running a job daily at midnight, you would do `@ @ 0 * * *` to run it at a randomly chosen but stable time between midnight and 1 am.

    💬 1🔄 1⭐ 5
  • 💬 1🔄 0⭐ 0
  • George Bgbargoud@masto.nyc
    Oct 31, 2026, 7:11 PM

    @DamonHD @hzulla

    I might have the order wrong but I'm pretty sure I put @ for seconds and minutes and 0 for hours which would start at a random but stable minute and second within the given hour

    💬 1🔄 0⭐ 0
  • DamonHDDamonHD@mastodon.social
    Oct 31, 2026, 7:49 PM

    @gbargoud @hzulla we may be talking across one another here, but cron only goes does to minute resolution AFAIK; seconds can't be specified in any version that I've seen or used.

    💬 1🔄 0⭐ 0
  • George Bgbargoud@masto.nyc
    Oct 31, 2026, 8:15 PM

    @DamonHD @hzulla

    Oh I must be misremembering that part then. It's been a while since I had to write a crontab and I always have to consult the comment at the top and then forget everything once I exit.

    💬 1🔄 1⭐ 0
  • 💬 0🔄 0⭐ 0
  • markATLeagle@mastodon.online
    Oct 31, 2026, 1:43 PM

    @hzulla being in some of the very early hyperspace AI datacenter space, we were informed by the power company that they were going to disconnect us for drops and spikes that were significant enough to harm the turbines. Its mitigated now, but at that time when a massive jobs stops all gpus at once, then another job comes in right behind it and starts them all again it could literally break the generation

    💬 1🔄 0⭐ 1
  • 💬 0🔄 0⭐ 0
  • datenwolfdatenwolf@chaos.social
    Oct 31, 2026, 2:22 PM

    @hzulla

    Uhh...

    I certainly do. For the longest time I wondered what would happen if one deployed a malware that

    - synchronized clocks by NTP
    - shut down all affected systems at precisely the same time on the evening (so that power plants are ramped down quicker)
    - and at the time where normally the evening dip would turn upward suddenly turn on all those systems again (ACPI RTC wakeup) and launched some really CPU & GPU heavy load.

    💬 1🔄 0⭐ 0
  • 💬 1🔄 1⭐ 0
  • 💬 0🔄 1⭐ 0
  • Ryek Darkener_RyekDarkener_@mastodon.social
    Oct 31, 2026, 4:13 PM

    @hzulla

    The interesting question is if this randomisation might become necessary in case more and mor datacenters run more and more cron-jobs or other jobs historically timed at fixed hours?

    💬 0🔄 0⭐ 0
  • Kurt Nelsonkurt@nelson.fun
    Oct 31, 2026, 4:15 PM

    @hzulla in the early Android days, cell network operators were confused by sudden stampedes on a regular schedule. Turns out, all phones were checking for OTAs roughly at the same time. Google didn't notice because to them it was noise and expected.

    💬 0🔄 1⭐ 1
  • maya_b 🇨🇦maya_b@hachyderm.io
    Oct 31, 2026, 4:40 PM

    @hzulla

    UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And non of that non-British 1kW kettle nonsense, proper British kettles are 3kW!

    💬 3🔄 3⭐ 0
  • PsyMar aka Lady Estradiolpsymar@unstable.systems
    Oct 31, 2026, 8:58 PM

    @maya_b And this is why most americans don't bother with kettles -- I don't know if our outlets can even supply 3kW -- and just heat up a cup of water in the microwave instead

    💬 0🔄 0⭐ 0
  • maya_b 🇨🇦maya_b@hachyderm.io
    Oct 31, 2026, 8:59 PM

    @psymar

    13A on 240V is doable in most of Europe, but generally all the kettles I've seen outside the UK are only 1kw - or a hairdryer or microwave's worth of power.

    💬 1🔄 0⭐ 0
  • 💬 2🔄 0⭐ 0
  • 💬 2🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0