Hello everyone,
All the services we offer are down and unreachable. We are currently investigating the cause. For the time being, it appears that all three servers are without power.
We do not expect the issue to be resolved soon. We are waiting for a fix from the datacenter operator and will monitor the situation in order to restore service as soon as possible after power returns.
Currently, one server is with power again, but the others remain unreachable. Our investigation shows that network equipment is also affected.
We have recovered services after power returned, leading to a total downtime of about 7 hours.
We apologize for the impact and we are in touch with our datacenter operator to discuss the incident.
@codebergstatus I hope it's not because of the heat.
Jun 27, 2026, 10:34 PM@codebergstatus Could possibly be the thunderstorms heading throughout Europe?
Jun 27, 2026, 10:34 PM@codebergstatus Best of luck getting to the bottom of the issue! (And thanks for running an awesome community service!)
@codebergstatus@social.anoxinon.de
Goddamnit.
@codebergstatus
Dann geh' ich mal ins Bett …
Aber ihr habt ja noch lange Zeit, bis eure Uptime auch nur in die Nähe von GitHub … sinkt.
Viel Erfolg auch von mir bei der Fehlersuche.
Gute Nacht.
@codebergstatus #opshugs ! Fingers crossed.
well, on the bright side, if your services were down and reachable that would just be weird
Jun 27, 2026, 11:04 PM@codebergstatus I think I found the culprit behind the power outage. Hope you get everything detangled and back online soon!
@codebergstatus my bad
Jun 27, 2026, 11:09 PM@codebergstatus oops I had a bad feeling about my last commit...
Someone found the perfect socket to plug the vacuum cleaner into. #illgetmycoat
@codebergstatus Let me guess, the datacenter is testing its power failover. Can you tell I've been through that before?
@codebergstatus thanks for the information and your services. #hugops for all working on the issue.
A site that big runs on only three servers?? I'm impressed!
I mean, I knew #Forgejo was efficient—I run an instance myself—but I didn't know it was *that* efficient.
Jun 28, 2026, 5:30 AM@argv_minus_one @codebergstatus
I'm running it on an N200 for me and my friends and it had less problems than many other services takes up almost nothing of the limited resources and still serves pages real fast; it is really just well optimised (and not written in JS :neocat: )
What's an N200? The Intel CPU?
Jun 28, 2026, 7:12 AM@argv_minus_one
Yes, it pulls like nothing in terms of power which is why I wanted a NUC with it for my little homelab so it doesn't get to expensive.
Neat, so, like a laptop CPU?
Jun 28, 2026, 7:36 AM@argv_minus_one
Yes it isn't socketet, but it is much slower than most laptop CPUs but also more power efficient. It is basically what a Raspberry Pi with x86 would be; performance wise somewhere between a Pi and an average laptop processor. It just pulls like ~4W in my config + some watts for SSD und HDD as well as Mainboard and USB stuff totalling 11W with a peak of like 14W.
The N200 only has like 4 cores with each a single thread but that is fine as long as I don't compile anything on it and for Forgejo I just get the precompiled binaries :3
@argv_minus_one @codebergstatus that's what you get when competent developers write software using something that's not #ruby, #python or #javascript but instead opt to use a fast language.
Jun 27, 2026, 11:23 PM@codebergstatus #hugOps
Jun 27, 2026, 11:23 PM@codebergstatus It's relatable. Sometimes I too am unresponsive when I'm without power.
<3
@codebergstatus
sorry everyone, I just wanted to register but that might have been like the last mint leaf.
It might be not very helpful but I obtained an 500 Internal Server Error when I followed the activation link and only shortly afterwards, the connection could not be established anymore.
Might just be a result of a backend service going down slightly earlier than the frontend.
@mauk_y @codebergstatus It's quite brave of you to admit to this, given the context ☺️
Is this the part where we wildly speculate on the cause of the power outage?
I'll start: aliens abducted the entire power plant! They were all like, “yoink this is mine now.”
Jun 28, 2026, 12:26 AM@argv_minus_one maybe it was a thunderstorm or the heatwave. People need to understand that climate change is here to stay and that what we took for granted can be taken away in seconds...
@codebergstatus
Jun 28, 2026, 1:06 AM@argv_minus_one @codebergstatus
Portable AC plugged into their server power strip… 😂
Yep, that'll trip a breaker or two.
It's probably Donald Trump's fault.
I'm not sure how he could cause a power outage at a data center in Europe, but he probably did somehow.
@argv_minus_one@mastodon.sdf.org @codebergstatus@social.anoxinon.de
Jun 28, 2026, 5:33 AM@danjones000 @argv_minus_one @codebergstatus
The power wanted to get even further away from him and started migrating out of Germany to the penguin lands southwards, since current politics think foreigners* must leave the country the power found leaving was quite simple for it.
*Since the EU is sharing power between countries I am making up that these were Electrons from somewhere else /silly
Jun 28, 2026, 12:47 AM
@codebergstatus Oof, that's tough, it happens. Sending HugOps and hoping the servers get well soon!
@felurx @codebergstatus What a marvellous term! I couldn't think how to express myself and that is absolutely perfect.
Also sending HugOps 🥰
@pabryan Hehe yeah, I think it's very cute :D
Not sure when I first heard it, maybe during the DENIC outage
Jun 28, 2026, 5:35 AM@codebergstatus
I think it's mostly up again :3
At least the status page is getting greener :3
Yay
Jun 28, 2026, 5:35 AM@codebergstatus
Just slept the time Codeberg was down → good problem solving; sleep whilst the problem exists :3
Jun 28, 2026, 6:22 AM@Wolkensteine @codebergstatus Yeah, except for Forgejo Actions, everything is showing green: https://status.codeberg.eu/status/codeberg 🎉
Seeing some positive sings of recovery for my website right now.
Thank you for all your work.
Jun 28, 2026, 5:51 AM@codebergstatus last time I saw a data centre have a power cut was when someone dropped a spanner in Redbus, and it was not just a power cut but a bridged phase which fried a lot of kit. Data centres don’t normally have power outages!
Best of luck.
@codebergstatus You'll probably won't see this Codeberg staff but thanks for the effort of keeping it up! Keep up the good work guys. <3
@micdan @codebergstatus yeah, I can only agree to that:
I only started to actively use codeberg recently and was doing stuff tonight. Suddenly, codeberg didn't work anymore, so I checked the status page and mastodon. I genuinely didn't think there would be any reaction so quickly in the night. But there it was: a new toot, made 5 min ago (with information on what happened!).
I took it as a sign to go to bed, so at least I got some sleep out of this outage.
Jun 28, 2026, 6:35 AM@codebergstatus thanks for all your efforts! 💪
@codebergstatus Thank you for working so hard!
@codebergstatus thanks! Big DevHugs 🥰
@codebergstatus It's great to see this level of transparency and I look forward to a potential post-incident report.
Welcome back #Codeberg!
Do keep us updated, would you? I'd certainly like to hear what happened, out of curiosity if nothing else.
Jun 28, 2026, 7:15 AM@codebergstatus Great news !
I'll be able to take advantage of the return to normal temperatures to get some work done this Sunday.
Thanks for everything you do.
@codebergstatus You are the best!
@codebergstatus TX for the transparency and for your hard work.
Jun 28, 2026, 8:42 AM@codebergstatus Maybe (just speculating) it's because the electical grid can't keep up with the demand because of the heat? Had local blackouts last week, but lasted 2h in total. Anyway, thanks for hosting my mirrors 👍
@codebergstatus that’s honestly a really amazing recovery time for a power outage! Thanks for your hard work, came back up by the time I woke up!
Jun 28, 2026, 9:36 AM@codebergstatus A 7 hour outage due to power delivery failure to the servers is really a bad thing. I hope there will be a postmortem on how this was possible, and, more importantly, which steps are being taken to make sure this doesn't happen again. Anyway, thank you all for getting things working again!
Jun 28, 2026, 6:14 PM@codebergstatus thanks to all the folks on the infra team, you all are awesome! 💜
