A big chunk of my free time in the last week has been consumed by work on restoring a bunch of my projects after my hosting provider, OVH, completely removed my server and backups. Here's what happened.
TL;DR
This post ended up longer than I was hoping for, so if you don't wanna read the whole thing, here's an executive summary:
- OVH has made me start a new account for each country I've lived in; it's annoying to keep track of, so I'm starting to clean it all up as well as to slowly move my projects a better competitor.
- I requested removal of one of those accounts, clearly marking which one, and they did remove it…
and then removed another account too
The one where my server and backups were located and paid for in advance.
- It's been over a week and support still hasn't helped me a bit.
- My planned move to Hetzner got expedited; and I've decided to also make a bunch of improvements to my projects along the way: to update language versions, dependencies, licenses, etc., to discontinue some, and to make my setup more resilient, easy to restore and less reliant on a single company in the future. So it's been infuriating, but with silver lining.
- Unfortunately, I also lost some databases in the process – causing one project to be prematurely discontinued, and a few others to lose some less important but cool stats…
- Update: After 17 days I received a call from the claims department: they took responsibility for the incident, apologised, confirmed that the data is not recoverable, and offered me 136,38€ (six months worth of the cost of the service) – which would be a reasonable compensation for the trouble, but is a laughable compensation for the unrecoverable loss of data.
- Why would OVH do that? How did they handle the situation?
- What could I have done better?
- Silver linings
- Databases lost
- Receipts
Why would OVH do that? How did they handle the situation?
I've been using OVH to host my VPS and register my domains for ages now. They used to be good enough,
but most importantly cheaper than everyone else. Now, however, I can get better machines
and better service for less. I was recommended a German company called Hetzner
and I've been planning to move there, and procrastinating on that plan, since, *checks notes*, October 2022
Company | Model | CPU | RAM | Disk space | Bandwidth | Price |
---|---|---|---|---|---|---|
OVH | VPS 2018 SSD 3 | 2 vCores | 8 GB | 80 GB + 50 GB extension | Unlimited (with fair use policy) | 22.73€/month |
Hetzner | CX42 | 8 vCores | 16 GB | 160 GB | 20 TB | 19.84€/month |
I've actually been using Hetzner for a while for a different project, ever since I moved Pronouns.page to its separate setup (see: Migrating Pronouns.page to a new server – a success story) – and I can only recommend them!
Which is not the case for OVH at all… While the price difference is not terrible – I could live with this specification for that money – there are so many other things about OVH that kept annoying me greatly. Their dashboard is slow, buggy and has terrible user experience. Your account is bound to a country in which you created it, meaning that me, a person who moved from Poland to Germany and then from Germany to Netherlands, ended up with three separate accounts – and scary automated emails warning me that if my whois information is proved to be inaccurate they will have to take away my domains. That information includes the physical address and phone number, which are bound to a country, so it won't let me update them without making me create a brand new account.
Okay, then just create a new account and move the domains and servers there, how hard can that be?
Well, hard… The procedure to move a single domain, at least at the time I did that, required separately
changing three contacts (iirc: administrative, billing and technical), confirming each of them via a code
from the email sent to the old owner, then another email sent to the new owner.
Multiply that by, idk like ten domains?, and you're stuck for hours with like sixty confirmation codes
between two browsers, two email accounts, waiting for all emails to arrive and verifying which
identifier is which. Rather than just letting me say: “hey, btw, I live in the Netherlands now”
They seem to have an affinity to spamming people with long plain-text emails –
to a point one starts just glancing at them briefly and throwing them in archive without giving them too much thought.
Anyways… In early June 2024 the status of my accounts is:
- a Polish account that hasn't been used in like 5 years,
- a German account that only has a VPS, which I'm planning to just move directly to Hetzner,
- a Dutch account that hosts a bunch of domains.
So as a person who would like to finally get the ball moving on the Hetzner move I've decided to at least clean up what I can: instead of three NIC's (account identifiers) in three countries connected to three email addresses, three password manager entries and three OTP manager entries, I could have two of each – and some more motivation to slowly but surely start moving everything. I tried looking for a “remove account” button, but to no avail. After a bunch of digging, I finally managed to find where to request the removal, via the GDPR procedure. And I did – while logged in to the Polish account, specifying the NIC of the Polish account in the form, and then again in the text of the request. Over the next week I got a few emails that they're working on it, and I thought that was it.
Oh how wrong I was. One day my husband let me know that Generator is down – so I tried SSH-ing to the VPS; it didn't let me, so I logged in to the OVH dashboard to debug with their in-browser console or to just reset the server: only to see that… there is no server.
I confirmed that the Polish account is indeed entirely gone, but the German one, while still existing, is completely empty, except for a history of payments and a support ticket. A ticket that I opened from the Polish account, written in Polish, specifying the Polish account's NIC – and their reply, also in Polish, that the German account is being removed.
I opened a new ticket bringing the situation to their attention. I even got over my social anxiety around phone calls and called their customer support – after all the sooner they react, the better the chance that the data is recoverable. Surprisingly, they didn't leave me on hold for ages or pass me through lengthy automated process, I actually got to talk to a real person within minutes. They gathered the info, they could see my tickets in the system, they told me they'll forward it to the appropriate department and that I should expect someone to contact me within hours.
Fast forward a week (and an angry nudge in the ticketing system) – still zero reaction.
What could I have done better?
I think the reason why someone removed the German account along with the Polish one is that while the account I was logged into when requesting removal, the NIC I put in the form, the content and the language of my message all pointed to the Polish account, one item did not: the form asked me for an email address and I mistakenly put in the one associated with the German account.
Which is an easy mistake to make when you're being forced to manage three separate accounts that could've easily been one. But more importantly: they didn't check if the email address entered actually belongs to the person making the GDPR request (eg. by them being logged in, by the user sending them an email from that account, by requiring a confirmation code by email); in this case both accounts happened to belong to the same person, but what if I put in someone else's email? Would someone be able to bulldoze a competitor's entire infrastructure by simply knowing what email they used for their OVH account? That's absurd!
Admittedly, they did notify me by email that my account is being removed. Among a spam of various emails,
in a weird mix of Polish and German text, and in which the account to be removed is referenced solely
by a NIC: a random string of numbers that I need to either remember or check my notes each time to know which is which.
I think they also tried to tell me that my VPS is gonna be removed along the account,
but the wording is absolutely terrible: “referencja do usługi na Twoim koncie wymaga niezbędnych operacji”,
which translates to “a reference to a service in your account requires necessary operations”
Quite a long way from what I'd expect to be the best practice:
a big, bold “hey, your account XYZ still has a VPS that's running and paid for, are you sure?!”
I could've moved to Hetzner 1,5 year ago. Oh honey, I so wish I had. But that's not an effort for one afternoon. There used to be 52 (!) projects hosted there along with 5 third-party web-based apps, and a bunch of dependencies. Some of that was using PHP 7.4, some PHP 8.3, some Node 18, some Node 20, some were just statically generated pages. If I were to move it over, I'd rather upgrade to newer version instead of supporting separate ones. And I'm a busy enby. I wrote earlier that I was “procrastinating” on the move, but that's not really accurate – I was damn busy with a bunch of other things, like Pronouns.page (which luckily was on a separate server already), or idk, one of the 153 repos I seem to have on my Gitlab (some of them private, so they won't show up under the link). So a proper move simply hasn't been a priority. Until OVH made it my priority.
I could've made the project easier to restore. Keep the configuration and a library of system packages in a repo,
Ansible-style, or whatnot. Yeah, well, that was the plan for the Hetzner move
And yes, I can hear hypothetical people screaming. Yes, I should've had backups. Well, I did. I had a script that was supposed to zip the most important data and store them on S3, but it broke and I haven't found time to debug the issue… I gave it low prio because I was planning to abandon the server anyway, and more crucially because I also had separate backups of the most important thing – databases. Other things I can restore with some effort, but databases would just be lost. The thing is, what I used for those backups is… OVH's built-in automatic backups of MySQL databases. I wish I could still see any indication in the dashboard that those backups were ever there, cause my anxiety whispers in my ear that my memory might be misleading me – but I can't, because the whole account is wiped.
Anyways… the thing is: I know I could've handled a bunch of things better. But my biggest mistake was trusting that OVH will handle my account appropriately. For heaven's sake, that server was even paid for for another month an a half – why would you wipe customer's infrastructure that's paid for, actively running, and part of a different account than the one that made the GDPR request?
Until now, I simply disliked OVH as a customer. How I actively distrust them, and I doubt that will ever change. I could never trust them again with any data or infrastructure.
Silver linings
But… it's not all shit. I actually managed to find a lot of silver linings in the whole situation. Obviously, OVH forced me to finally prioritise the changes I've been planning to do for over a year – so I finally got most of them done, yay. I guess, thank you? But other things on my list are stuff that I would've done anyway, eventually… Still, I like to think about the positives, so here we go.
I moved to a faster and cheaper machine. Yay!
My IP is finally fully private and protected by Cloudflare's proxy. I've had the old server way before I even knew that it's a thing – so someone somewhere dug it up and used it to DDoS me, circumventing Cloudflare's protection. Ironically, they were queerphobes targeting Pronouns.page, but they didn't know that this project has had its own separate server for a while now, and its IP is still secret. Now, so is Avris's.
I had a list of 57 projects/apps/hosts/(sub)domains running on the old server. Now that list has just 25 items. The remaining ones are those that either got popular, or didn't at all but I really like them and they aren't that much effort to maintain. I'm following Tom Scott's advice to create a lot of stuff and increase the odds of some of them succeeding – but that also means I need to be ready for a lot of what I do to fail. Killing a project is tough, even if no one, including myself, even uses it anymore. But this whole situation made me tear off that band-aid.
A half-baked idea that I decided to publish way too soon? Poof. A framework that helped me learn a lot about PHP, software and software architecture, but that also makes less sense to use than something more established? Poof. (Well, the repo is still public, but no need to keep the website up). An app that stopped working after Elon Musk broke Twitter's API (unless I pay a bazillion euros)? Gone. A legacy version of a page that I kept up just in case? Bye!
It's not easy to say goodbye to them, but it helps with the overwhelm. I no longer maintain a terrifying number of apps I can't keep track of – it's just two dozen that I actually care about.
There's a bunch of things that I wanted to fix or improve in each and every one of those projects,
but doing that to twenty something of them is way more manageable than to fifty something.
So here we go: I upgraded Node, PHP, Symfony and a bunch of dependencies to the latest versions for all of my apps –
and to the same version across the whole server. I renamed the master
branch to main
in those older
projects that still used the outdated name. I changed my name in the READMEs and footers
to reflect my new legal name ( it's official! my name is andrea vos now ) wherever it still needed an update.
I changed the license of my projects to OQL wherever it was still MIT.
I switched from Yarn to pnpm. Stuff like that.
When upgrading Cake I also added better RTL support by upgrading Bootstrap and improving keyboard control logic. For OQL I made the dark mode look nicer. For Heartbeat I added a way to compress older data to reduce database size and refactored the config so that it's more readable and allows for async calls (like API calls or caching). Deployer got a bunch of improvements that I'll release soon as v2.0.
The server configuration as a whole got better too. I applied a bunch of the same improvements that I had earlier made for Pronouns.page ( Migrating Pronouns.page to a new server – a success story) – including a switch from Apache to Nginx and organising configs better. My notes from that move were super helpful and made this one easier and faster – but this time I wanted to make the process even better, and turn them into a nice little repository of scripts and configs that can be used to really boost the process if I ever need to do it again (or, more likely, use it as a template for future projects).
And, of course, setting up proper backups in multiple locations (all far away from OVH) get a way higher priority on my list after this incident. The new setup is still in the works, but it will be a great improvement.
Databases lost
Luckily, most of my projects don't really need a database – they're either landing pages for libraries, or tools that encode data in the URL, or something like that. Restoring them was just a matter of spending time and effort on configuring everything again. But for those that do need to store data generated by users, the loss is unfortunately bigger…
The biggest by far was naked-adventure.eu – a website that crowdsources information on nudist beaches and establishments. On one hand, so much of mine and other people's work suddenly got wiped from OVH's infrastructure, but on another I'm actually a bit relieved… It's the kind of page that makes the most sense on a big scale – after all, just under 200 places in our database is almost nothing on the global scale; what's the point of even visiting the page if you're statistically unlikely to find what you're looking for? And for me personally, over the years naturism stopped being a new exciting thing that I love, and became just a regular part of my life. The enthusiasm and motivation to run this page faded, while overwhelm from other projects and other aspects of my life grew. I was falling behind on moderation, never gotten to implementing some features I planned for it, I didn't have ideas or resources to advertise the platform and to grow it… So while it's sad to see it go, the project had been on life support for a while already. I might resurrect it one day, with some new twist, but for now, rest in peace.
I also had some pages that collected stats about people's responses in a quiz, like CoreValues. The pages are back up, but the stats are gone.
My husband's blog got wiped too – but he has local copies of all texts, and he hasn't published there in a while anyway, so not that much harm.
I also lost website analytics from Plausible. But to be honest, I haven't really checked them in a while – I just let my websites run, without obsessing about the numbers. I still set up a new instance on the new server and I'll be collecting anonymised analytics, but I won't cry about that particular spilled milk.
Mental health impact
Speaking of crying, though… I just wanna take a moment to acknowledge the impact that the whole situation has had on my mental health.
I'm really proud of how I mostly handled the issue. I kept calm, focused on solutions, on positives, on opportunities. I managed to focus on fixing one thing at a time, to prioritise not only some projects against others, but also the whole fix against my personal needs – all of which are stuff that I've been struggling with. I gave myself time to deal with the situation properly, as opposed to panicking and frantically trying to fix everything with a hammer and a scotch tape, as fast as I can – which would've been my response a few years, if not months, ago. There was no need to rush that much: no one got angry with me, I just got a few messages from friends and strangers asking if I know that this or that tool is down; I lost some traffic, but it's not like I was monetising it anyway; none of those projects was an essential, irreplaceable, urgently needed part of anyone's live (I hope).
But there are limits to what my head can take. While calm on the outside, I was extremely irritable for the first two days or so, any minor inconvenience could get me over the edge. I had two major breakdowns – but luckily also two amazing partners that helped me through them.
Basically: I needed time to grieve. I lost a lot of data, a lot of work, my plans for the upcoming week got turned upside-down. One can't just brush off stuff like this.
Anyways… I'm fine now. But I think sharing my experience might help someone one day, so I wanted to it.
Receipts
Context: pa13■■■■ = Polish account; pa60■■■■ = German account