Avris

A big chunk of my free time in the last week has been consumed by work on restoring a bunch of my projects after my hosting provider, OVH, completely removed my server and backups. Here's what happened.

TL;DR

This post ended up longer than I was hoping for, so if you don't wanna read the whole thing, here's an executive summary:

• OVH has made me start a new account for each country I've lived in; it's annoying to keep track of, so I'm starting to clean it all up as well as to slowly move my projects a better competitor.
• I requested removal of one of those accounts, clearly marking which one, and they did remove it… and then removed another account too 🤦 The one where my server and backups were located and paid for in advance.
• It's been over a week and support still hasn't helped me a bit.
• My planned move to Hetzner got expedited; and I've decided to also make a bunch of improvements to my projects along the way: to update language versions, dependencies, licenses, etc., to discontinue some, and to make my setup more resilient, easy to restore and less reliant on a single company in the future. So it's been infuriating, but with silver lining.
• Unfortunately, I also lost some databases in the process – causing one project to be prematurely discontinued, and a few others to lose some less important but cool stats…
• Update: After 17 days I received a call from the claims department: they took responsibility for the incident, apologised, confirmed that the data is not recoverable, and offered me 136,38€ (six months worth of the cost of the service) – which would be a reasonable compensation for the trouble, but is a laughable compensation for the unrecoverable loss of data.

• Why would OVH do that? How did they handle the situation?
• What could I have done better?
• Silver linings
• Databases lost
• Receipts

Why would OVH do that? How did they handle the situation?

I've been using OVH to host my VPS and register my domains for ages now. They used to be good enough, but most importantly cheaper than everyone else. Now, however, I can get better machines and better service for less. I was recommended a German company called Hetzner and I've been planning to move there, and procrastinating on that plan, since, *checks notes*, October 2022 😅

I've actually been using Hetzner for a while for a different project, ever since I moved Pronouns.page to its separate setup (see: Migrating Pronouns.page to a new server – a success story) – and I can only recommend them!

Which is not the case for OVH at all… While the price difference is not terrible – I could live with this specification for that money – there are so many other things about OVH that kept annoying me greatly. Their dashboard is slow, buggy and has terrible user experience. Your account is bound to a country in which you created it, meaning that me, a person who moved from Poland to Germany and then from Germany to Netherlands, ended up with three separate accounts – and scary automated emails warning me that if my whois information is proved to be inaccurate they will have to take away my domains. That information includes the physical address and phone number, which are bound to a country, so it won't let me update them without making me create a brand new account.

Okay, then just create a new account and move the domains and servers there, how hard can that be? Well, hard… The procedure to move a single domain, at least at the time I did that, required separately changing three contacts (iirc: administrative, billing and technical), confirming each of them via a code from the email sent to the old owner, then another email sent to the new owner. Multiply that by, idk like ten domains?, and you're stuck for hours with like sixty confirmation codes between two browsers, two email accounts, waiting for all emails to arrive and verifying which identifier is which. Rather than just letting me say: “hey, btw, I live in the Netherlands now” 🤦‍ They seem to have an affinity to spamming people with long plain-text emails – to a point one starts just glancing at them briefly and throwing them in archive without giving them too much thought.

Anyways… In early June 2024 the status of my accounts is:

• a Polish account that hasn't been used in like 5 years,
• a German account that only has a VPS, which I'm planning to just move directly to Hetzner,
• a Dutch account that hosts a bunch of domains.

So as a person who would like to finally get the ball moving on the Hetzner move I've decided to at least clean up what I can: instead of three NIC's (account identifiers) in three countries connected to three email addresses, three password manager entries and three OTP manager entries, I could have two of each – and some more motivation to slowly but surely start moving everything. I tried looking for a “remove account” button, but to no avail. After a bunch of digging, I finally managed to find where to request the removal, via the GDPR procedure. And I did – while logged in to the Polish account, specifying the NIC of the Polish account in the form, and then again in the text of the request. Over the next week I got a few emails that they're working on it, and I thought that was it.

Oh how wrong I was. One day my husband let me know that Generator is down – so I tried SSH-ing to the VPS; it didn't let me, so I logged in to the OVH dashboard to debug with their in-browser console or to just reset the server: only to see that… there is no server.

I confirmed that the Polish account is indeed entirely gone, but the German one, while still existing, is completely empty, except for a history of payments and a support ticket. A ticket that I opened from the Polish account, written in Polish, specifying the Polish account's NIC – and their reply, also in Polish, that the German account is being removed.

I opened a new ticket bringing the situation to their attention. I even got over my social anxiety around phone calls and called their customer support – after all the sooner they react, the better the chance that the data is recoverable. Surprisingly, they didn't leave me on hold for ages or pass me through lengthy automated process, I actually got to talk to a real person within minutes. They gathered the info, they could see my tickets in the system, they told me they'll forward it to the appropriate department and that I should expect someone to contact me within hours.

Fast forward a week (and an angry nudge in the ticketing system) – still zero reaction.

What could I have done better?

I could've moved to Hetzner 1,5 year ago. Oh honey, I so wish I had. But that's not an effort for one afternoon. There used to be 52 (!) projects hosted there along with 5 third-party web-based apps, and a bunch of dependencies. Some of that was using PHP 7.4, some PHP 8.3, some Node 18, some Node 20, some were just statically generated pages. If I were to move it over, I'd rather upgrade to newer version instead of supporting separate ones. And I'm a busy enby. I wrote earlier that I was “procrastinating” on the move, but that's not really accurate – I was damn busy with a bunch of other things, like Pronouns.page (which luckily was on a separate server already), or idk, one of the 153 repos I seem to have on my Gitlab (some of them private, so they won't show up under the link). So a proper move simply hasn't been a priority. Until OVH made it my priority.

I could've made the project easier to restore. Keep the configuration and a library of system packages in a repo, Ansible-style, or whatnot. Yeah, well, that was the plan for the Hetzner move 😅

And yes, I can hear hypothetical people screaming. Yes, I should've had backups. Well, I did. I had a script that was supposed to zip the most important data and store them on S3, but it broke and I haven't found time to debug the issue… I gave it low prio because I was planning to abandon the server anyway, and more crucially because I also had separate backups of the most important thing – databases. Other things I can restore with some effort, but databases would just be lost. The thing is, what I used for those backups is… OVH's built-in automatic backups of MySQL databases. I wish I could still see any indication in the dashboard that those backups were ever there, cause my anxiety whispers in my ear that my memory might be misleading me – but I can't, because the whole account is wiped.

Anyways… the thing is: I know I could've handled a bunch of things better. But my biggest mistake was trusting that OVH will handle my account appropriately. For heaven's sake, that server was even paid for for another month an a half – why would you wipe customer's infrastructure that's paid for, actively running, and part of a different account than the one that made the GDPR request?

Until now, I simply disliked OVH as a customer. How I actively distrust them, and I doubt that will ever change. I could never trust them again with any data or infrastructure.

Silver linings

I had a list of 57 projects/apps/hosts/(sub)domains running on the old server. Now that list has just 25 items. The remaining ones are those that either got popular, or didn't at all but I really like them and they aren't that much effort to maintain. I'm following Tom Scott's advice to create a lot of stuff and increase the odds of some of them succeeding – but that also means I need to be ready for a lot of what I do to fail. Killing a project is tough, even if no one, including myself, even uses it anymore. But this whole situation made me tear off that band-aid.

A half-baked idea that I decided to publish way too soon? Poof. A framework that helped me learn a lot about PHP, software and software architecture, but that also makes less sense to use than something more established? Poof. (Well, the repo is still public, but no need to keep the website up). An app that stopped working after Elon Musk broke Twitter's API (unless I pay a bazillion euros)? Gone. A legacy version of a page that I kept up just in case? Bye!

It's not easy to say goodbye to them, but it helps with the overwhelm. I no longer maintain a terrifying number of apps I can't keep track of – it's just two dozen that I actually care about.

There's a bunch of things that I wanted to fix or improve in each and every one of those projects, but doing that to twenty something of them is way more manageable than to fifty something. So here we go: I upgraded Node, PHP, Symfony and a bunch of dependencies to the latest versions for all of my apps – and to the same version across the whole server. I renamed the master branch to main in those older projects that still used the outdated name. I changed my name in the READMEs and footers to reflect my new legal name ( it's official! my name is andrea vos now 🥳) wherever it still needed an update. I changed the license of my projects to OQL wherever it was still MIT. I switched from Yarn to pnpm. Stuff like that.

When upgrading Cake I also added better RTL support by upgrading Bootstrap and improving keyboard control logic. For OQL I made the dark mode look nicer. For Heartbeat I added a way to compress older data to reduce database size and refactored the config so that it's more readable and allows for async calls (like API calls or caching). Deployer got a bunch of improvements that I'll release soon as v2.0.

The server configuration as a whole got better too. I applied a bunch of the same improvements that I had earlier made for Pronouns.page ( Migrating Pronouns.page to a new server – a success story) – including a switch from Apache to Nginx and organising configs better. My notes from that move were super helpful and made this one easier and faster – but this time I wanted to make the process even better, and turn them into a nice little repository of scripts and configs that can be used to really boost the process if I ever need to do it again (or, more likely, use it as a template for future projects).

And, of course, setting up proper backups in multiple locations (all far away from OVH) get a way higher priority on my list after this incident. The new setup is still in the works, but it will be a great improvement.

Databases lost

The biggest by far was naked-adventure.eu – a website that crowdsources information on nudist beaches and establishments. On one hand, so much of mine and other people's work suddenly got wiped from OVH's infrastructure, but on another I'm actually a bit relieved… It's the kind of page that makes the most sense on a big scale – after all, just under 200 places in our database is almost nothing on the global scale; what's the point of even visiting the page if you're statistically unlikely to find what you're looking for? And for me personally, over the years naturism stopped being a new exciting thing that I love, and became just a regular part of my life. The enthusiasm and motivation to run this page faded, while overwhelm from other projects and other aspects of my life grew. I was falling behind on moderation, never gotten to implementing some features I planned for it, I didn't have ideas or resources to advertise the platform and to grow it… So while it's sad to see it go, the project had been on life support for a while already. I might resurrect it one day, with some new twist, but for now, rest in peace.

I also had some pages that collected stats about people's responses in a quiz, like CoreValues. The pages are back up, but the stats are gone.

My husband's blog got wiped too – but he has local copies of all texts, and he hasn't published there in a while anyway, so not that much harm.

I also lost website analytics from Plausible. But to be honest, I haven't really checked them in a while – I just let my websites run, without obsessing about the numbers. I still set up a new instance on the new server and I'll be collecting anonymised analytics, but I won't cry about that particular spilled milk.

Mental health impact

Receipts