Monday at about 4.30 CET the hosting provider of my host got hacked at the root level which resulted in all of my websites going down, until manually being restored to an old backup Wednesday morning at about 4am CET. 

Just to clarify this before getting started.. while I purchase my hosting of Skytoaster, this specific server is run by AltusHost. If the following post is too long for your liking: I still recommend Skytoaster, just please do not, ever, trust a host that doesn't keep backups. And keep backups more often than I did.

What happened?

According to the official statement of AltusHost posted on Facebook, their version of WHMCS (which seems shady to say the least) a hacker gained root access to their billing panel and started removing servers, which lasted about 10 minutes. While they're statement says 5PM CET, I was working on a site on that server at around 4.30PM when it started showing connectivity issues, so my guess is that they've rounded up the time.

When a server gets deleted, I can imagine that a site goes down, and when there's a lot of sites to restore that can take time. While annoyed I could not finish the site I was working on, I think I'm quite forgiving when it comes to people doing stuff I know little about and when that goes wrong. I assume there are a lot of people all over the globe working on it who are just as frustrated (if not more), so let them do their thing. Spent the night hanging out with friends so tried not too worry about it too much.

At Tuesday morning 2am CET I received an update from Skytoaster on the situation, mainly letting me know that AltusHost was working on getting servers back online and to explain the situation. At that point it wasn't clear yet if my server was one of the removed ones or not.

And yes, it was.

A few hours later Patrick came back with a confirmation that it indeed was. We had been provided a new (empty) server which we could use to put up some sort of notice or restore from a backup of our own. The backup server, which was apparently with Altus as well, had not yet been located and the fate of that was unknown. That morning I worked with Patrick to put up a brief explanation on my websites, and started to upload backup data to try and get the site back up. Right now all articles are back up, no comments though.

I guess I'm a bit naive when it comes to servers, as I only tend to take backups when making major changes such as my relaunch back in December. That's the last one I could find on my computer, even though I recall clicking cPanel's "backup" button a week or two ago, I never downloaded it from the server after it was done generating. After all, I've been promised 7 days of backups on server-level, so in the event something goes seriously wrong with the server, that would be more easily restored than something I download manually per user on the VPS. Not a big deal, until it is.

Here's where it gets nasty.

Now, the real problems only became evident late Tuesday evening. The backup had been uploading and was almost ready when I managed to get hold off a technical support member who told me "we have no backups for our main hardware node".

I'm not sure about you, but I don't run a hosting company nor do I have the wish or expertise to start one. And I do realize in the end it's the responsibility of the user to backup what they create. But what I don't understand is how on earth someone can have the gut to run a hosting company without any backups of servers you're selling. When something goes wrong, and I do think eventually something goes wrong with any server as there's just so much things to worry about, you will want to be able of either re-routing requests to a different mirrored server or be able of restoring from the backup made last night... right? As a hosting company you would want the certainty that in case something goes wrong, you can keep your clients satisfied with a quick solution right? Well, apparently AltusHost doesn't.

Restoration Phase

After hearing this news that there's no chance of the server being restored from their end, roughly 30 hours after witnessing the first symptoms, it kinda gave me no reason but looking at getting my site back up one way or another. And this is where Patrick reminded me why I hosted with SkyToaster again as he had been working through my local backups and after getting at least my personal sites back to the way they were on December 22nd, he has been helping me manually restoring my blog articles and other websites from Google Cache in a time period of over 4 hours. Thanks Patrick!

Never, ever, ever again.

Somewhere in between, he also set up a new US based backup server (unaffiliated with the current provider) which automatically writes a nightly, weekly and monthly backup of all users on my VPS which I can access directly as well. While they made a mistake in working with Altus for their Dutch VPS, he's been working day and night (literally) to prevent this from ever being possible again.

Now what?

I'm still a great fan of Skytoasters' service. Patrick (and Sal too for that matter) are easy enough to get hold off and respond promptly when needed. They're all human, who made a mistake when signing up with Altus, but up to now all my hosting (shared and VPS) has been great. We've got most of the stuff back thanks to the hard work by Patrick, which once again proves to me they are passionate about what they do, and willing to step up when things go wrong. I'll be working with Sal & Patrick over the next weeks to find out where to move from here (they and Altus aren't off the hook yet!), but getting back up and running is priority number one.

Oh, and I'm setting up automatic downloading of backups to various locations for my own sites as well now.

Question from Twitter: "What does Skytoaster have to do with AltusHost?"

Nothing, except that the VPS I bought from them was physically hosted with AltusHost (and I've been aware of it not being one of their own servers from the get go). While Skytoaster didn't get everything right in this case, I consider most of the blame to be with Altus who knowingly risks their clients' data and, in retrospect, seems to be a bit of a shady business in general.