A Better MvM

State of the Server

Hi everyone. Sorry for the extended server downtime this month. It's about time I gave an explanation on what happened...

The short one-paragraph summary, is that a bunch of data got corrupted; I thought it was going to be recoverable for a while; but my data recovery efforts ultimately turned out to be futile; and then I had to decide what to do with the server, given this bad situation, combined with a number of other contributing factors. And I've decided that, at least for the time being, I'm going to shut down the server. I do hope to eventually bring it back, potentially better than ever, after a while; but no guarantees. We'll see what happens.

Alright... details. On January 5, I was doing software updates on the server, and since I was going to have to reboot it anyway, I figured it would be a good opportunity to make a full backup of the server's SSD contents. Unfortunately (and rather ironically), in the process of preparing to make the backup image file for the main filesystem containing the game server files, I did a routine preparatory resize operation to reduce the amount of data the backup would take... and then Btrfs, ostensibly a "stable" "production-ready" filesystem, started to bug out like crazy for no reason; it literally tried to rearrange my data into places that did not exist on the SSD. (Online filesystem resize is supposedly a completely stable feature of Btrfs.[btrfs.wiki.kernel.org])

As soon as I discovered that a problem was occurring, I forced a shutdown of the system, and copied the raw partition contents as soon as I could. Now, despite my disdain for Btrfs's apparent penchant for randomly corrupting data despite supposedly being a "mature" filesystem that has existed for approximately 13 years, I actually have quite a bit of nuts-and-bolts knowledge about how it works under the hood. (I actually wrote a working read-only Btrfs implementation for Windows quite a few years back.) And based on my initial recovery efforts, it appeared that perhaps Btrfs had simply messed up its structures in such a way that it had moved data around to where it wasn't expecting it to be, but that I could manually fix it, and so I spent a couple of weeks attempting to do data recovery. Eventually I discovered that I was wrong, and that giant chunks of the metadata B-tree were just completely missing, and were not present in any part of the partition image even when I did a comprehensive scan.

So, all of the data pertaining to the game server instances was lost. The last backups of that data that I have are, rather embarrassingly, about 1 to 1.5 years old. I do still have e.g. the source code to my own mods; however, things like server config files, scripts, custom content, and many other things were lost, and would have to be painstakingly re-figured-out and put back into place to be able to get back up to speed based on recovering the old backups.

And unfortunately, right now, I've found that I just do not have the motivation and willpower to be able to put in the effort to fix everything. (Personal life, stress, blah blah blah.) But I also don't like the idea of shutting down the server forever either.

So that's why my current plan is to shut down for now; give myself some time; and then, perhaps later this year, if I am feeling up to it, I will likely make an attempt to bring the server back up to working status.

One of the things that I've always disliked about the game servers I've hosted, is that I was never really able to set things up in a way that made it very easy for anyone other than myself (e.g. other admins) to upload/install/modify custom content. A lot of this has to do with the way I originally set up the server back in ~2014-2015, as a little test project. Back at that time, I didn't anticipate the need to make delegating these tasks easy, and so no particular effort went into facilitating that in the way I built up the system.

When the server later became popular, I was able to band-aid some custom content uploading functionality onto the existing setup; but it was never as good as it really needed to be. A lot of things were really fragile: adding or updating many kinds of custom content (most stuff other than easy things like pop files) just was never really easily doable by the other admins because it required personal intervention from me to make sure that nothing would break. And this frankly just made those things a massive nuisance to deal with. (Maps, custom wave icons, and custom upgrade files were among the types of files that I never came up with a good delegation system for.)

So, I'm hopeful that after taking some time off, I'll be able to get the server going again; and this time, build things from the ground up with the explicit goal to make it easy for any admin to install or update custom content without requiring intervention from me. We'll see how that goes.

I'm sorry to everyone about this turn of events. Believe me, it's been a real punch to the gut for myself as well.

For now, the server will be on hiatus. And when I'm back to feeling up to it, the server will make a return, hopefully in the relatively near future.

If anyone wants to take a peek at what the server kernel log looked like, I uploaded an annotated copy of it here[gist.github.com]. It's not pretty.