Login
You're viewing the functional.cafe public feed.
  • Nov 14, 2026, 7:28 PM

    These where interesting few days. In short. we have done dry-run repair of the filesystem and we are quite certain we will manage to restore everything. There is literally just few causalities (one email, few Nextcloud file previews and few Nextcloud uploads). If everything works out, tomorrow we are back to normal operations \o/ The process of getting to that place was very delicate, stressful but educational.. Getting to that place was not easy. Sorry if the below part makes little sense to most.

    During the initial checks after the backup was done, we knew there is an issue with unreadable 4KiB block in the filesystem. This was the result of the fact that we initially been rebuilding the first failed disk from its pair which then, while rebuilding, started throwing read media errors. This left us with rebuilt array with a missing part on the disk. After replacing the second broken disk and rebuilding again, the array was brought back to optimal state but we knew we had a missing area on the disk (we rebuilt the secondary disk with the same missing area as both of them were a pair in raid10). However, we did managed to mount it read-only and we’ve moved everything we could in the past days. That was a big win already. Once we had the backup we started approaching the repair.

    Running xfs_repair -n on the storage filesystem (read-only) was erroring out when the repair approached the missing part of the filesystem. Knowing the block size and it’s position we knew that zeroing that area could possibly fix the issue. But before doing so on a live system (point of no return), we wanted to play safe and test it first. The idea was to create a device mapper ,linear mapping over the broken filesystem, and replace the affected 4KB block with zeros. That mapper was set to read-only as well. On top of that we created a persistent writable snapshot of 500GB COW file on a separate volume. This file was used to write all the changes without touching the affected filesystem, since any attempt to replay requires write space,. With the snapshot mounted read-write, we have managed to complete the dirty journal recovery. With that out of the way, seeing that the journal was cleaned up, it was time to try the repair. With the area of the unreadable file system written with zeros now, we tried running xfs_repair -n against that snapshot.

    This approach provided us with possibility to see whether we will be able to recover the filesystem without damaging it. No data during this operation is written to the filesystem but rather to the COW file snapshot. Once we’ve confirmed that the dry run (-n) worked, we have run real repair on the overlayed snapshot. After about 4 hours (so over 8 when counting the dry run) we have managed to successfully mount the file system and are now checking if everything is fine.

    Tomorrow we will attempt a real repair and will try to bring all services back to normal online state!

    At this moment we are talking about tiny bit of lost data (one email, one missing commit-graph on Forgejo repo, few Nextcloud uploads and few Nextcloud image previous) no more then 33MB total. Looks like we were very lucky. And I really mean very lucky. Something we will be addressing in coming weeks/months cause it’s a perfect wake up call for us as the core team but also as the whole project, platform and community.

    So things are looking good but of course, we still need to wait to see it all end well. It has been a tense weeks for me but I hope by tomorrow this will all be over.

    💬 3🔄 4⭐ 0

Replies

  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • Nov 14, 2026, 11:22 PM

    @disroot thanks for the update, hopefully you can learn lessons for this for the future, so Disroot is better prepared for the potential future issues! It's been a massive downtime, which, to be fair, reduced my trust in the project, but I hope everything recovers

    Don't forget to regularly communicate! There has been many days without an update or updates that feel like "tomorrow/in a few days it's fixed" when it's not.

    But it was a tough situation... thanks for your work!

    💬 0🔄 0⭐ 1
  • Nov 15, 2026, 10:36 AM

    @nollo Totally understandable. It will be a hard work to rebuild the trust. Specially that the improvements will require substantial up-front expense (in hardware) as well as monthly upkeep costs.

    💬 0🔄 0⭐ 0