When it comes to DevOps, I'm just the “dev”. I write code, but I'd rather have someone else worry about making sure it keeps running as intended. I manage my personal VPS, I manage some servers at work, but I wouldn't call myself an expert in that area at all. So I'm super proud of myself and how well it went when I migrated a big project to a new machine 😊 The downtime was just 15 minutes! Here's the story, if you're interested.
I bought the server on Wednesday and started setting it up, completely independently of the old machine. I created a setup that from the outside was indistinguishable from the old one, except for using an older database backup. Until Saturday morning they were running simultaneously – the DNS were pointing to the old IP, but my local /etc/hosts
to the new one.
I picked Saturday morning, because mornings are when our traffic is lowest, and that day I'm free and can focus on the migration, even if something goes wrong and takes more time than expected.
An important part of the plan was taking notes – every important command I had run, I documented for myself. In case anything goes wrong and I have to start over, or in case I wanted to later migrate my other projects to Hetzner too (and I do), I'd have a recepe basically ready.
The old configs were… meh. Long, repetitive and messy. Just switching from Apache to nginx simplified them massively, but I went further and extracted common parts for all domains and subdomains to make them reusable. Setting up everything for a new language version used to be a half-an-hour-or-so long process – now it takes me a few minutes.
There's two things that were really annoying for me to figure out: analytics and monitoring. Those things are expensive when your traffic is as big as ours. Corporations with such traffic can afford it easily, but we're not a business, we have no considerable income other than donations and an arc.io widget. We need to self-host those things, then.
For traffic analytics, I've recently switched from Matomo to Plausible. As much as Matomo is better than Google Analytics from the privicy perspective (and it's a lot better), it's still really heavy and has way more features than I'll probably ever need. I needed to pay 11€/m extra for a separate database server just for Matomo to store its logs. Plausible, on the other hand, is exactly what I need. So neat!
For monitoring… We used to have one, hosted on AWS Lambda, but it kept causing trouble that I had no time to fix, and in the end I disabled it. Monitoring is quite hard, because it needs to run on a dedicated and super reliable infrastructure – we can't monitor ourselves after all; if the server is down, so is the monitor that's supposed to let us know. I found out a really cool tool that keeps blowing my mind with its ingenuity: Upptime. It runs entirely on GitHub pages and actions, totally for free (as long as you keep it public).
We have multiple node servers running for different language versions. Each needs to run at some port and then nginx needs to be configured as a reverse proxy to that specific port. Before it was really tediuos, I'd just enumerate the ports starting at 3001 and for each new version I'd first need to sift through configs to find what the current max value was. I kept looking up which port was related to which domain. Annoying shit.
In the new setup, I wanted to have some single source of truth for the domain-port pairs, but it turned out to be near impossible (at least for my skills). Nginx is strict about its configs being static. I couldn't find an easy way to pass data about those ports to it.
Ultimately, I settled for a compromise system: I unambiguously map each version to a port by convering each letter of its ISO code to its index in the English alphabet. For example for Japanese, ja
, the port would be 31001
because “j” is the 10th letter of the alphabet, and “a” is the 1st. Not a perfect system, but it's gonna simplify my flow a lot.
I wanted to switch from supervisor to pm2 for its nicer interface and cool features, but for some reason it kept dropping a significant percentage of requests at (seemingly) random. I couldn't figure out the root cause, so I gave up. That switch wasn't worth that much of my time, I reverted back to supervisor.
A huge issue was that… no emails were getting sent from the new server. Every SMTP request would just time out. I tried everything I could think of, I figured I must've misconfigured something. I asked the team for help, but we couldn't find a root cause. But then I got this random hypothesis – what if Hetzner is blocking port 465 regardless of our firewall settings? A quick seach – turns out they do! Ugh… I get their reasoning, seems like a perfectly reasonable approach. Switching ports worked. I just wish they somehow gave notice more visibly, not just an unexplained timeout, so that I wouldn't waste so much time figuring it out.
I struggled with moving Plausible too… It runs in Docker containers – but my knowlege of Docker is pretty limited. I spent way too much time trying to pg_dump
and pg_restore
my way from one container to host to another host via scp
to another container, only to finally succeed and then… realise that the Postgres database is just half of the story. The main chunk of data resides in ClickHouse. Instead of struggling with the whole thing again, I took a different approach: backing up and restoring entire Docker volumes. It worked like a charm! Here are the commands, if you're interested:
# old host
docker run -v plausible_db-data:/volume --rm loomchild/volume-backup backup - > ./db-data.tar.bz2
docker run -v plausible_event-data:/volume --rm loomchild/volume-backup backup - > ./event-data.tar.bz2
scp ./db-data.tar.bz2 pp:/home/admin/www/stats.pronouns.page
scp ./event-data.tar.bz2 pp:/home/admin/www/stats.pronouns.page
# new host
cat ./db-data.tar.bz2 | sudo docker run -i -v statspronounspage_db-data:/volume --rm loomchild/volume-backup restore -f -
cat ./event-data.tar.bz2 | sudo docker run -i -v statspronounspage_event-data:/volume --rm loomchild/volume-backup restore -f -
When everything was ready, on Friday evening, I announced the upcoming maintanance and waited. I had all the commands ready in a notepad.
When the time came, I just… stopped both servers, moved the database, moved plausible volumes, started the new server, and then updated the DNS entries. Simple as that.
It was tons of work over a couple of evenings, but good preparation made wonders: it only took a quarter to actually switch. Considering how ops is not my forte at all, I'm so proud of having accomplished such a smooth transition 🥰
For quite a while my VPS was misconfigured – any HTTP requests it got but couldn’t assign to a vhost, it redirected to the main website, avris.it. I didn’t think it would be a big deal, until I recently found out that my post Ungoogling is indexed by Google under https://askara.avris.it/blog/ungoogling
This subdomain hadn’t existed for a long time already, my server doesn’t serve a certificate for it anymore, but it requires HSTS, so browsers end up showing users a scary error message.
I had to do something about it.
First of all, users have to see anything other than a security warning. I need a wildcard certificate.
Fortunately, Let’s Encrypt offers them now, and it’s totally free. I just followed an instruction to obtain one, and then configured Apache to serve the /www/default
directory with the *.avris.it
certificate for all requests that don’t fit to any vhost.
Once the users can see the website, I can show them things. Ideally, just a 404 with an information that it’s not a valid domain, and a suggestion where they might have wanted to go (same request string, but with the base domain). Easy.
Btw, I used Water.css, a ridiculously simple CSS framework – I just added two lines, no classes, and the page already looks way better!
But that doesn’t solve the root cause: bots are confused about which domains they should be using. They don’t care whether my certificate is working or not, they don’t understand the message I left there for the users.
They need a proper HTTP 301 -- Moved permanently
. So I had to add a simple recognition whether I’m serving a bot or a user, and adjust the response for each of them.
So, here’s what I ended up with:
DirectoryIndex index.php
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_URI}::$1 ^(/.+)/(.*)::\2$
RewriteRule ^(.*) - [E=BASE:%1]
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^index\.php(/(.*)|$) %{ENV:BASE}/$2 [R=301,L]
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule .? - [L]
RewriteRule .? %{ENV:BASE}/index.php [L]
</IfModule>
<IfModule !mod_rewrite.c>
<IfModule mod_alias.c>
RedirectMatch 302 ^/$ /index.php/
</IfModule>
</IfModule>
<?php
function isBrowser($ua): bool
{
if (!$ua) {
return false;
}
$isProbablyBot = (bool) preg_match('#bot|crawler|baiduspider|80legs|ia_archiver|voyager|curl|wget|yahoo! slurp|mediapartners-google|facebookexternalhit|twitterbot|whatsapp|php|python#i', mb_strtolower($ua));
$isProbablyBrowser = (bool) preg_match('#mozilla|msie|gecko|firefox|edge|opera|safari|netscape|konqueror|android#i', mb_strtolower($ua));
return $isProbablyBrowser || !$isProbablyBot;
}
$url = 'https://avris.it' . $_SERVER['REQUEST_URI'];
if (!isBrowser($_SERVER['HTTP_USER_AGENT'] ?? null)) {
http_response_code(301);
header('Location: ' . $url);
die;
}
http_response_code(404);
echo <<<HTML
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>404 – Not found</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" href="https://avris.it/assetic/gfx/favicon.png" />
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/kognise/water.css@latest/dist/light.min.css">
</head>
<body>
<h1>
404 – Not found
</h1>
<p>
This is not a valid subdomain.
</p>
<p>
Did you mean <a href="$url">$url</a> ?
</p>
<hr/>
<p>
<small>
You'll be redirected there in <span id="countdown">15</span> seconds anyway... 🤷
</small>
<p>
<script>
const \$el = document.getElementById('countdown');
seconds = 15;
setInterval(_ => {
if (seconds === 0) {
window.location.href = '$url';
return;
}
seconds--;
\$el.innerHTML = seconds;
}, 1000);
</script>
</body>
</html>
HTML;
]]>Może nie napiszę tu nic ciekawego, ale po prostu muszę się pochwalić :D Kupiłem sobie wirtualny serwer prywatny. Czyściutki Linux na nim, do zainstalowania i skonfigurowania wszystko: Apache, PHP, MySQL, SFTP, domeny, maile i cała masa innych pierdół. W wielu z nich grzebałem pierwszy raz w życiu. Ale chyba dogrzebałem się do wszystkiego co trzeba, bo wygląda na śmigające ślicznie.
Wreszcie mogę deployować po ludzku, z gitem i composerem, pisać w nowszej wersji PHP, skonfigurować sobie, co mi się tylko żywnie podoba, nie robić ręcznie eksportów rozkładu na Busa, któremu nie starczało pamięci na współdzielonym hostingu... Miodzio! :3
]]>