{"tag":"hetzner","articles":{"blog\/technology\/pronouns-page-migration":{"key":"blog\/technology\/pronouns-page-migration","type":"article","published":true,"meta":{"createdAt":"2022-08-13T10:27:16+02:00","publishedAt":"2022-08-13T10:27:16+02:00","group":null,"category":"blog","subcategory":"technology","slug":"pronouns-page-migration"},"content":{"en":{"slug":"pronouns-page-migration","title":"Migrating Pronouns.page to a new server \u2013 a success story","intro":"\u003Cfigure\u003E\n                \u003Cnoscript\u003E\n                    \u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_small.png\u0022 alt=\u0022Terminal with `ls -lah \/etc\/nginx\/sites-enabled\/` executed on the new server\u0022 class=\u0022border-bottom\u0022 width=\u0022480\u0022 height=\u0022273.17773788151\u0022\u003E                \n                \u003C\/noscript\u003E\n                \u003Cspan class=\u0022hide-noscript\u0022\u003E\u003Cimg src=\u0022data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAACQAAAAUCAYAAADlep81AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAHjElEQVRIiT2Uy28kVxnFf\/dVVd3V7Xbbbj9mPA9lwgx5KORBkGADkVBYsWHDgj0S\/xMSyg4JiSXZICEkpAwRRAkkkSbJTDzjGbvtdr+7Hrdu3XtZOGJ3Ft\/ifEfnd8Rvf\/ebuDfapw0z8v4hril48Op7yO4hn60eENYnrJ\/8lV5SoE2PuraU8yly9FPc3s+4dahZF4EffT\/h6ekZcfOYd1\/f5V\/\/\/pDpeMb+\/iEnpx\/z4vyUpm1pW08vPcRFyfxqzPHRAcv1nBenV\/S3+sjj27sYAwRJr284uDlApyt++NZr\/PIt6BZ\/49bBJaO9c5Qf44tT2rbFzT8nSQJXyxap4OGnT9m8eMh+Hz788C98\/eWnvHTnJheTR0gNWzsddkbbpGmf8dkljx99SfCOrR3JerVhZz\/n8HgL8cGf\/hAfP\/oM79d0ejvs7r2E0iXvvvE+g6TlopqiI0ipqasI0dC2noYuTg5RJqV1DdhLMt2SaiiqBSEEunmXGD3aKFzraNuAEIrFbM4Hf\/w9g70u2aClrgSLyw0v3b2B7u9lbC9Tri4v6I0Cs1nJrePX+MdHf2a+tNRNTWNLtO8QgyQEkEIRkTi1Q2E9Irb4ekaaaIx0rFdzhIQQA1JGhFQ45zFaI4TAh0iaZ1hbcv5lSZoa5lclNOfo8eQjekMN8ojFZIlSNVfTzylWkk8efsNqWRFji0ki3Wyb9bpksamwMSWYAUoItEmQCIgt0k4QOIrNkqJaUJYWJRU+BCBCBG0099+8i0kU0+mcLNUs50vyziF6Pq7Z2x9weJgSrCNET7GCr796zmJeMhh0mc3mRAKDgcBYQW9vh57uEpHkFHR6QyBjvZjg0wFVMWddrIFImqT40CKRxBiIRBrXoGRL2wjSRJHnCYQhQgnkfLrh\/OyKYt1wfHePZyeX7Ix6ON9gbU1RVCijaKwnRk+SKoQIOGGp7JRiNUWFKdFO6aiSTuLodhVaR5TSJJkhSRPSzCCVIEk1Uko6\/QHKaLrdLZLegAdvvEpnu48UMmE4GCKFpFhZ3nv\/Lfr9Lu\/\/4m1u3RnRzTOMUoQg8D4CAZMAtDjvCd4hgkcLi1ICKUAKiXMtbeuwlaWxjmJd4mxLY1uMMQyGQ97+8TtsDbYZ7HSpqpI8z9GziyWvvHyHTh7QRtK4AmUMp08XiChITEJNTZJqXBMoqwa5lSFiiypWZHkfrQ3GaKwVnJ2cM7m6wvtA8IHgI5GISQyNdYgoCS6gXMn0cgyh4ezxhsn4goPjA+T+1i4yJFxdFIwvllSlp9PNyDoK2ziKoqQsK7RSpJlBxEi5LkiznIqUxWJxnYaPaC3Y2ztgZ2cP7wEk2iiUkogAqU7o5zkCgWscD\/\/+T87HY6bjCSB4\/uQ5crQ\/JOsIOmmXjuhBVEjlefD6DbYGXdI0wdYNIgR6iWE4yOl2UlxjESrBR0FjW2zlcA20rcfojLybE2LEtYEQoPUB17YsVisqW\/Of\/z5GJwmL2YrGO4SUDHYH6M4g4cnJOWcnlxzdHZHlCTQSdEGInjQz1LUleMf+\/pAkkUgfSI0iz6BZt1jbkmUZTVPTWItvW5bzFSFeo+7bgBACpSXee4SExWRFt58htGS4t40tGqqNRU+vKuqN5+jOiNWsQqqEJgmsi4rzyQRtNPu3B0ibsFkXCOEJIeKaAsqSfi8hxIr5wuK9Zz7bsF5vEAhEEN+tsyDGa2PBB6QSDPa20MZQbWomz2ZY5zA6Qe8e5WxWghzJbq\/HctNwdrrEaM35ySWuaZEi0skMna5hvbHEAN1eh+WiJARP3kuRSuJsy3pT09gW7z2Na\/5vAiDNEqQQxBBZnJXXj7UtPnrqqqHTydCLieP27veYTqfcOBhRzL5gmAQeffsVSkZMxyAEGKOp6oCSCdIIWteitcB7CVFgqxbvA1JKtJJsD7ao64rGNTSNp7EOax2DrT4hBPK8Q6dvWC0qBBo\/9FSrGj0\/eY4bj5mu1yyutpGpoHKOu\/fu8vEnT2gaD1yjK5UmxpYQI0SJENcURRQq6UPrkH4DwtOGljRLSRKN1Q1GS6xtKYqCLMu4\/\/IDIoIxY549fcaDB\/dJb2n00e0hmyIw6kk6WwPybp\/FckpvsI3SGhMlUqnvYg\/XOkSQCUprhJToJMP5DK389Y3weB\/wPhKRmCS5LrVSNE2DlJKTp89ompbdnV3u338FISRSpuhWSnYPc4oNEB2XkwsObgxIsoxf\/frnNFWLFAlNY1EmYssGZbZxrSGkR2j3BdnOT6jbXUy4oF4+wlYT5rMLlFTUNazWNVeTSxKtGA1zhMr59vQF3V4P1zpenDwFBKP9fbT3grpuCd6jM0OeSDrdjBhhNDrAWUldlQihSIymrjWrlSXtDunffBMZ7hHEAQedDsPOIdVin8nlI5z1LFdrfvD2K3QzxcXFjLKqsGXJarHg8GBAXTckMuX2zXd49M0TJpcXyBs39xEqkA\/3qW2FMR2E3EapXUKbsNmU33UowYcuIUjqusQ1NWVRE+jh3RJvp0gRqMorvvnqMT4qujmsFjOsrTi+dUyWKo5uDullPQ4OhmiTEmQgUjDYSrh37w7\/A2YLLsjoRLMdAAAAAElFTkSuQmCC\u0022 data-src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_small.png\u0022 alt=\u0022Terminal with `ls -lah \/etc\/nginx\/sites-enabled\/` executed on the new server\u0022 class=\u0022border-bottom\u0022 width=\u0022480\u0022 height=\u0022273.17773788151\u0022\u003E\u003C\/span\u003E\n                \n            \u003C\/figure\u003E\u003C\/p\u003E\n\u003Cp\u003EWhen it comes to \u003Ca href=\u0022https:\/\/en.wikipedia.org\/wiki\/DevOps\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E DevOps\u003C\/a\u003E, I\u0027m just the \u201cdev\u201d.\nI write code, but I\u0027d rather have someone else worry about making sure it keeps running as intended.\nI manage my personal VPS, I manage some servers at work, but I wouldn\u0027t call myself an expert in that area at all.\nSo I\u0027m super proud of myself and how well it went when I migrated a big project to a new machine \ud83d\ude0a\nThe downtime was just 15 minutes! Here\u0027s the story, if you\u0027re interested.\u003C\/p\u003E\u003Csvg xmlns=\u0022http:\/\/www.w3.org\/2000\/svg\u0022 style=\u0022display: none;\u0022\u003E\u003C\/svg\u003E","content":"\u003Cfigure\u003E\n                \u003Cnoscript\u003E\n                    \u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_big.png\u0022 alt=\u0022Terminal with `ls -lah \/etc\/nginx\/sites-enabled\/` executed on the new server\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022546.35547576302\u0022\u003E                \n                \u003C\/noscript\u003E\n                \u003Cspan class=\u0022hide-noscript\u0022\u003E\u003Cimg src=\u0022data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAACQAAAAUCAYAAADlep81AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAHjElEQVRIiT2Uy28kVxnFf\/dVVd3V7Xbbbj9mPA9lwgx5KORBkGADkVBYsWHDgj0S\/xMSyg4JiSXZICEkpAwRRAkkkSbJTDzjGbvtdr+7Hrdu3XtZOGJ3Ft\/ifEfnd8Rvf\/ebuDfapw0z8v4hril48Op7yO4hn60eENYnrJ\/8lV5SoE2PuraU8yly9FPc3s+4dahZF4EffT\/h6ekZcfOYd1\/f5V\/\/\/pDpeMb+\/iEnpx\/z4vyUpm1pW08vPcRFyfxqzPHRAcv1nBenV\/S3+sjj27sYAwRJr284uDlApyt++NZr\/PIt6BZ\/49bBJaO9c5Qf44tT2rbFzT8nSQJXyxap4OGnT9m8eMh+Hz788C98\/eWnvHTnJheTR0gNWzsddkbbpGmf8dkljx99SfCOrR3JerVhZz\/n8HgL8cGf\/hAfP\/oM79d0ejvs7r2E0iXvvvE+g6TlopqiI0ipqasI0dC2noYuTg5RJqV1DdhLMt2SaiiqBSEEunmXGD3aKFzraNuAEIrFbM4Hf\/w9g70u2aClrgSLyw0v3b2B7u9lbC9Tri4v6I0Cs1nJrePX+MdHf2a+tNRNTWNLtO8QgyQEkEIRkTi1Q2E9Irb4ekaaaIx0rFdzhIQQA1JGhFQ45zFaI4TAh0iaZ1hbcv5lSZoa5lclNOfo8eQjekMN8ojFZIlSNVfTzylWkk8efsNqWRFji0ki3Wyb9bpksamwMSWYAUoItEmQCIgt0k4QOIrNkqJaUJYWJRU+BCBCBG0099+8i0kU0+mcLNUs50vyziF6Pq7Z2x9weJgSrCNET7GCr796zmJeMhh0mc3mRAKDgcBYQW9vh57uEpHkFHR6QyBjvZjg0wFVMWddrIFImqT40CKRxBiIRBrXoGRL2wjSRJHnCYQhQgnkfLrh\/OyKYt1wfHePZyeX7Ix6ON9gbU1RVCijaKwnRk+SKoQIOGGp7JRiNUWFKdFO6aiSTuLodhVaR5TSJJkhSRPSzCCVIEk1Uko6\/QHKaLrdLZLegAdvvEpnu48UMmE4GCKFpFhZ3nv\/Lfr9Lu\/\/4m1u3RnRzTOMUoQg8D4CAZMAtDjvCd4hgkcLi1ICKUAKiXMtbeuwlaWxjmJd4mxLY1uMMQyGQ97+8TtsDbYZ7HSpqpI8z9GziyWvvHyHTh7QRtK4AmUMp08XiChITEJNTZJqXBMoqwa5lSFiiypWZHkfrQ3GaKwVnJ2cM7m6wvtA8IHgI5GISQyNdYgoCS6gXMn0cgyh4ezxhsn4goPjA+T+1i4yJFxdFIwvllSlp9PNyDoK2ziKoqQsK7RSpJlBxEi5LkiznIqUxWJxnYaPaC3Y2ztgZ2cP7wEk2iiUkogAqU7o5zkCgWscD\/\/+T87HY6bjCSB4\/uQ5crQ\/JOsIOmmXjuhBVEjlefD6DbYGXdI0wdYNIgR6iWE4yOl2UlxjESrBR0FjW2zlcA20rcfojLybE2LEtYEQoPUB17YsVisqW\/Of\/z5GJwmL2YrGO4SUDHYH6M4g4cnJOWcnlxzdHZHlCTQSdEGInjQz1LUleMf+\/pAkkUgfSI0iz6BZt1jbkmUZTVPTWItvW5bzFSFeo+7bgBACpSXee4SExWRFt58htGS4t40tGqqNRU+vKuqN5+jOiNWsQqqEJgmsi4rzyQRtNPu3B0ibsFkXCOEJIeKaAsqSfi8hxIr5wuK9Zz7bsF5vEAhEEN+tsyDGa2PBB6QSDPa20MZQbWomz2ZY5zA6Qe8e5WxWghzJbq\/HctNwdrrEaM35ySWuaZEi0skMna5hvbHEAN1eh+WiJARP3kuRSuJsy3pT09gW7z2Na\/5vAiDNEqQQxBBZnJXXj7UtPnrqqqHTydCLieP27veYTqfcOBhRzL5gmAQeffsVSkZMxyAEGKOp6oCSCdIIWteitcB7CVFgqxbvA1JKtJJsD7ao64rGNTSNp7EOax2DrT4hBPK8Q6dvWC0qBBo\/9FSrGj0\/eY4bj5mu1yyutpGpoHKOu\/fu8vEnT2gaD1yjK5UmxpYQI0SJENcURRQq6UPrkH4DwtOGljRLSRKN1Q1GS6xtKYqCLMu4\/\/IDIoIxY549fcaDB\/dJb2n00e0hmyIw6kk6WwPybp\/FckpvsI3SGhMlUqnvYg\/XOkSQCUprhJToJMP5DK389Y3weB\/wPhKRmCS5LrVSNE2DlJKTp89ompbdnV3u338FISRSpuhWSnYPc4oNEB2XkwsObgxIsoxf\/frnNFWLFAlNY1EmYssGZbZxrSGkR2j3BdnOT6jbXUy4oF4+wlYT5rMLlFTUNazWNVeTSxKtGA1zhMr59vQF3V4P1zpenDwFBKP9fbT3grpuCd6jM0OeSDrdjBhhNDrAWUldlQihSIymrjWrlSXtDunffBMZ7hHEAQedDsPOIdVin8nlI5z1LFdrfvD2K3QzxcXFjLKqsGXJarHg8GBAXTckMuX2zXd49M0TJpcXyBs39xEqkA\/3qW2FMR2E3EapXUKbsNmU33UowYcuIUjqusQ1NWVRE+jh3RJvp0gRqMorvvnqMT4qujmsFjOsrTi+dUyWKo5uDullPQ4OhmiTEmQgUjDYSrh37w7\/A2YLLsjoRLMdAAAAAElFTkSuQmCC\u0022 data-src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_big.png\u0022 alt=\u0022Terminal with `ls -lah \/etc\/nginx\/sites-enabled\/` executed on the new server\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022546.35547576302\u0022\u003E\u003C\/span\u003E\n                \n            \u003C\/figure\u003E\u003C\/p\u003E\n\u003Cp\u003EWhen it comes to \u003Ca href=\u0022https:\/\/en.wikipedia.org\/wiki\/DevOps\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E DevOps\u003C\/a\u003E, I\u0027m just the \u201cdev\u201d.\nI write code, but I\u0027d rather have someone else worry about making sure it keeps running as intended.\nI manage my personal VPS, I manage some servers at work, but I wouldn\u0027t call myself an expert in that area at all.\nSo I\u0027m super proud of myself and how well it went when I migrated a big project to a new machine \ud83d\ude0a\nThe downtime was just 15 minutes! Here\u0027s the story, if you\u0027re interested.\u003C\/p\u003E\n\u003Ch3\u003EWhy move?\u003C\/h3\u003E\n\u003Cul\u003E\n\u003Cli\u003EPronouns.page gets lots of traffic:\n\u003Ca href=\u0022https:\/\/stats.pronouns.page\/en.pronouns.page?period=30d\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E around 4 million pageviews a month\u003C\/a\u003E\nfor the English version alone. And on top of that, as an unapologetically queer website, we\u0027re a target of regular DDoS attacks.\nThe previous server didn\u0027t handle it well. We need an upgrade.\u003C\/li\u003E\n\u003Cli\u003EPronouns.page used to run on my personal VPS together with all my other pet projects.\nThey might not need many resources in comparison\n(damn, PP has outgrown everything else, including my wildest dreams \ud83d\ude05),\nbut it\u0027s still gonna be useful to give PP dedicated resources.\u003C\/li\u003E\n\u003Cli\u003EI used to be a bottleneck when it comes to fixing any issues on the server.\nNow that my personal stuff is on a different server, multiple people can have SSH access to the PP instance\nand intervene when necessary.\u003C\/li\u003E\n\u003Cli\u003EWe moved from OVH to Hetzner, getting a way better deal.\nInstead of 21.5\u20ac\/m\u003Csup\u003E[1]\u003C\/sup\u003E for 2 vCPU, 8GB RAM and 130GB drive\nwe\u0027ll now pay 27.25\u20ac\/m for 8 vCPU, 16GB RAM and 240GB drive.\u003Csup\u003E[2]\u003C\/sup\u003E\u003C\/li\u003E\n\u003Cli\u003EThe cost can now be charged directly to the \u003Ca href=\u0022https:\/\/en.pronouns.page\/team\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E NLC\u003C\/a\u003E account,\nI won\u0027t have to manage reimbusements (well, as soon as I figure out AWS migration).\u003C\/li\u003E\n\u003Cli\u003EI could use the opportunity to migrate to a new stack,\nwhich would\u0027ve been hard to do on a running server with tens of active projects.\u003C\/li\u003E\n\u003C\/ul\u003E\n\u003Ch3\u003EThe plan\u003C\/h3\u003E\n\u003Cp\u003EI bought the server on Wednesday and started setting it up, completely independently of the old machine.\nI created a setup that from the outside was indistinguishable from the old one, except for using an older database backup.\nUntil Saturday morning they were running simultaneously \u2013 the DNS were pointing to the old IP,\nbut my local \u003Ccode\u003E\/etc\/hosts\u003C\/code\u003E to the new one.\u003C\/p\u003E\n\u003Cp\u003EI picked Saturday morning, because mornings are when our traffic is lowest,\nand that day I\u0027m free and can focus on the migration,\neven if something goes wrong and takes more time than expected.\u003C\/p\u003E\n\u003Cp\u003EAn important part of the plan was taking notes \u2013 every important command I had run, I documented for myself.\nIn case anything goes wrong and I have to start over, or in case I wanted to later migrate my other projects to Hetzner too\n(and I do), I\u0027d have a recepe basically ready.\u003C\/p\u003E\n\u003Ch3\u003ECool things\u003C\/h3\u003E\n\u003Cp\u003EThe old configs were\u2026 meh. Long, repetitive and messy.\nJust switching from \u003Ca href=\u0022https:\/\/apache.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Apache\u003C\/a\u003E to \u003Ca href=\u0022https:\/\/nginx.org\/en\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E nginx\u003C\/a\u003E simplified them massively,\nbut I went further and extracted common parts for all domains and subdomains to make them reusable.\nSetting up everything for a new language version used to be a half-an-hour-or-so long process \u2013 now it takes me a few minutes.\u003C\/p\u003E\n\u003Cp\u003EThere\u0027s two things that were really annoying for me to figure out: analytics and monitoring.\nThose things are \u003Cem\u003Eexpensive\u003C\/em\u003E when your traffic is as big as ours.\nCorporations with such traffic can afford it easily, but we\u0027re not a business,\nwe have no considerable income other than donations and an \u003Ca href=\u0022https:\/\/arc.io\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E arc.io\u003C\/a\u003E widget.\nWe need to self-host those things, then.\u003C\/p\u003E\n\u003Cp\u003EFor traffic analytics, I\u0027ve recently switched from \u003Ca href=\u0022https:\/\/matomo.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Matomo\u003C\/a\u003E to \u003Ca href=\u0022https:\/\/plausible.io\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Plausible\u003C\/a\u003E.\nAs much as Matomo is better than Google Analytics from the privicy perspective\n(and it\u0027s \u003Cem\u003Ea lot\u003C\/em\u003E better), it\u0027s still really heavy and has way more features than I\u0027ll probably ever need.\nI needed to pay 11\u20ac\/m extra for a separate database server just for Matomo to store its logs.\nPlausible, on the other hand, is \u003Cem\u003Eexactly\u003C\/em\u003E what I need. So neat!\u003C\/p\u003E\n\u003Cp\u003EFor monitoring\u2026 We used to have one, hosted on AWS Lambda, but it kept causing trouble\nthat I had no time to fix, and in the end I disabled it.\nMonitoring is quite hard, because it needs to run on a dedicated and super reliable infrastructure\n\u2013 we can\u0027t monitor ourselves after all; if the server is down, so is the monitor that\u0027s supposed to let us know.\nI found out a really cool tool that keeps blowing my mind with its ingenuity:\n\u003Ca href=\u0022https:\/\/upptime.js.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Upptime\u003C\/a\u003E. It runs entirely on GitHub pages and actions,\ntotally for free (as long as you keep it public).\u003C\/p\u003E\n\u003Ch3\u003EIssues\u003C\/h3\u003E\n\u003Cp\u003EWe have multiple node servers running for different language versions.\nEach needs to run at some port and then nginx needs to be configured as a reverse proxy to that specific port.\nBefore it was really tediuos, I\u0027d just enumerate the ports starting at 3001 and for each new version\nI\u0027d first need to sift through configs to find what the current max value was.\nI kept looking up which port was related to which domain. Annoying shit.\u003C\/p\u003E\n\u003Cp\u003EIn the new setup, I wanted to have some single source of truth for the domain-port pairs,\nbut it turned out to be near impossible (at least for my skills). Nginx is strict about\nits configs being static. I couldn\u0027t find an easy way to pass data about those ports to it.\u003C\/p\u003E\n\u003Cp\u003EUltimately, I settled for a compromise system: I unambiguously map each version to a port\nby convering each letter of its ISO code to its index in the English alphabet.\nFor example for Japanese, \u003Ccode\u003Eja\u003C\/code\u003E, the port would be \u003Ccode\u003E31001\u003C\/code\u003E because \u201cj\u201d is the 10\u003Csup\u003Eth\u003C\/sup\u003E letter of the alphabet,\nand \u201ca\u201d is the 1\u003Csup\u003Est\u003C\/sup\u003E. Not a perfect system, but it\u0027s gonna simplify my flow a lot.\u003C\/p\u003E\n\u003Cp\u003EI wanted to switch from \u003Ca href=\u0022http:\/\/supervisord.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E supervisor\u003C\/a\u003E to \u003Ca href=\u0022https:\/\/pm2.io\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E pm2\u003C\/a\u003E\nfor its nicer interface and cool features, but for some reason it kept dropping a significant percentage of requests at (seemingly) random.\nI couldn\u0027t figure out the root cause, so I gave up. That switch wasn\u0027t worth that much of my time, I reverted back to supervisor.\u003C\/p\u003E\n\u003Cp\u003EA huge issue was that\u2026 no emails were getting sent from the new server.\nEvery SMTP request would just time out. I tried everything I could think of,\nI figured I must\u0027ve misconfigured something. I asked the team for help, but we couldn\u0027t find a root cause.\nBut then I got this random hypothesis \u2013 what if Hetzner is blocking port 465 regardless of our firewall settings?\nA quick seach \u2013 \u003Ca href=\u0022https:\/\/docs.hetzner.com\/cloud\/servers\/faq\/#why-can-i-not-send-any-mails-from-my-server\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E turns out they do\u003C\/a\u003E!\nUgh\u2026 I get their reasoning, seems like a perfectly reasonable approach. Switching ports worked.\nI just wish they somehow gave notice more visibly, not just an unexplained timeout,\nso that I wouldn\u0027t waste so much time figuring it out.\u003C\/p\u003E\n\u003Cp\u003EI struggled with moving Plausible too\u2026 It runs in Docker containers \u2013 but my knowlege of Docker is pretty limited.\nI spent way too much time trying to \u003Ccode\u003Epg_dump\u003C\/code\u003E and \u003Ccode\u003Epg_restore\u003C\/code\u003E my way from one container to host to another host via \u003Ccode\u003Escp\u003C\/code\u003E\nto another container, only to finally succeed and then\u2026 realise that the Postgres database is just half of the story.\nThe main chunk of data resides in \u003Ca href=\u0022https:\/\/clickhouse.com\/clickhouse\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E ClickHouse\u003C\/a\u003E.\nInstead of struggling with the whole thing again, I took a different approach: backing up and restoring entire Docker volumes.\nIt worked like a charm! Here are the commands, if you\u0027re interested:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode class=\u0022hljs php border\u0022\u003E\u003Cspan class=\u0022hljs-comment\u0022\u003E# old host\u003C\/span\u003E\ndocker run -v plausible_db-data:\/volume --rm loomchild\/volume-backup backup - \u0026gt; .\/db-data.tar.bz2\ndocker run -v plausible_event-data:\/volume --rm loomchild\/volume-backup backup - \u0026gt; .\/event-data.tar.bz2\n\nscp .\/db-data.tar.bz2 pp:\/home\/admin\/www\/stats.pronouns.page\nscp .\/event-data.tar.bz2 pp:\/home\/admin\/www\/stats.pronouns.page\n\n\u003Cspan class=\u0022hljs-comment\u0022\u003E# new host\u003C\/span\u003E\ncat .\/db-data.tar.bz2 | sudo docker run -i -v statspronounspage_db-data:\/volume --rm loomchild\/volume-backup restore -f -\ncat .\/event-data.tar.bz2 | sudo docker run -i -v statspronounspage_event-data:\/volume --rm loomchild\/volume-backup restore -f -\n\u003C\/code\u003E\u003C\/pre\u003E\n\u003Ch3\u003EThe switch\u003C\/h3\u003E\n\u003Cp\u003EWhen everything was ready, on Friday evening, I announced the upcoming maintanance and waited.\nI had all the commands ready in a notepad.\u003C\/p\u003E\n\u003Cp\u003EWhen the time came, I just\u2026 stopped both servers, moved the database, moved plausible volumes,\nstarted the new server, and then updated the DNS entries. Simple as that.\u003C\/p\u003E\n\u003Cp\u003EIt was tons of work over a couple of evenings, but good preparation made wonders:\nit only took a quarter to actually switch.\nConsidering how ops is not my forte at all,\nI\u0027m so proud of having accomplished such a smooth transition \ud83e\udd70\u003C\/p\u003E\n\u003Cfigure\u003E\n                \u003Cnoscript\u003E\n                    \u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-cpu_big.png\u0022 alt=\u0022Screenshot of a CPU usage graph, dropping from ~50% to ~20%\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022538.96006028636\u0022\u003E                \n                \u003C\/noscript\u003E\n                \u003Cspan class=\u0022hide-noscript\u0022\u003E\u003Cimg src=\u0022data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAACQAAAAUCAYAAADlep81AAAACXBIWXMAAA7EAAAOxAGVKw4bAAACy0lEQVRIic2VzW7bRhCAv9klKcmRJaVC\/oAWRQ9FEyA99ZSnyHPmDXoreuw9QJs2aRIngt1KjCmapkQuudMDRVpynABJHKcDLLiD2Z9vZmeGcjBb6DLNWRcOY4T94YCTbEWWFzz46XvCwHKVItoIzjnCMOwM68LR74Xv2foZYEQaIABVRUSuFOAiCdpJWZZEUfQlWRARzBcluED+d0\/WRcg5h6ryanmKV0W3xvPjbEf\/XGMHqKhqAP5cpPz69xG1Kk\/fnLDIC9J1uePF6zQnKyt+eXZInBcAJGt3KRHqknq+cvyV5Pz2dMYkDJilOQdxyjUjOGCWnDKKAv7JVsxPVuwNeiyXGS8XKbcnQ9Ky4t7NMT\/cGNOzH5+aoqpaVRXPj2IePT7gjhUEyCrPwArBJq9KBFWPAWptQhuZxnbkPH0BYw3Tr0Y8\/PE7hlHwzkvfCdNWWRAEgBDUzYUC7AemgwGIUHoihCL0jXQwALdCwyQwjATcccphkn0QyKtl3s07N6y17NuPq7Lzu2rvP6hiT13Vre+AvPfsfcLbb0vh6q5qWqm8UqvSs4bKK16V5bpkutcjL1zXdjogVX8pMAC\/z5eMhgPm2YrAWm4P+xwkGbmr+fb6kH9PVmSFYzKImC1znhy+4UWccu\/m5AzImMv7q8eLhJ8XCUaEEhj3QvKyQgT+eNY8p1ElEDCbp5rXnqmwDXR5Xfra1lk9wJeOPoACre1cjo026dIB7YWG+19PURRjGqP6Tfc0BkXx3iNIo6videOtvK179agqRgwicqYbgwDeK4pijUVR1Hum4\/2zPpQkyQ6xiOB9k1eqirW2a\/EispO0rd5+W5u1FmstdV3vrDt\/Tyvj8Rgpy1LjOG483JRru+i8Xtd14+HmYO891toOvouASAfRRrt1rtW3zwKYTqdEUUTw+vCYeXzKaDgAEaqq5v7dby5OjiuQ\/wAsXdPKiHvexAAAAABJRU5ErkJggg==\u0022 data-src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-cpu_big.png\u0022 alt=\u0022Screenshot of a CPU usage graph, dropping from ~50% to ~20%\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022538.96006028636\u0022\u003E\u003C\/span\u003E\n                \n            \u003C\/figure\u003E\u003C\/p\u003E\n\u003Cfigure\u003E\n                \u003Cnoscript\u003E\n                    \u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-ram_big.png\u0022 alt=\u0022Screenshot of a RAM usage graph, dropping from almost 100% to under 40%\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022537.31343283582\u0022\u003E                \n                \u003C\/noscript\u003E\n                \u003Cspan class=\u0022hide-noscript\u0022\u003E\u003Cimg src=\u0022data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAACQAAAAUCAYAAADlep81AAAACXBIWXMAAA7EAAAOxAGVKw4bAAACyUlEQVRIic2VW29bRRDHf7t7LnZuTpOoLQKqSqBSLi9IvPI5eOiX6wvfAokH4BEhIQjqQyWUVC2kru04ts9td6YPx+fYJg5Kmogw0pFmdndm\/\/ufyzGzrNBfD4+w1rCz3WUyLfjs0ft0Owm3IUZEVUSoqhKMYzrLudPbxBjz34MxBqOqCiCifPvTb4Q0YS92bHRTjDWMZgWVD3zz1SMaiNPSEzvLYFbw7O8hexspmQ88PxnRix1fP37Adhq\/E6ioUSpf4fOCWelRCbwBzrwQW4MFsspzmlccvnxDVZS8mmQMzjK2UCoF6yxOhNwaxg\/vs5VEF9\/6LwwtvGqiEJF2aTuy9RZwPJzw47NjqqwAVRTYswYwJM0pW3P4\/fNXPPnyoysDArCNkiR12MqH88iBH\/44QrKc1EBqDR17cY0NB2N+OX5d18QVPlhKmfe+fuecqXPIy\/LSha4ijKfZhbEuEmPMgiGZO3etXX\/4CoF9EFQuD+bFeNbqi5TFdVd03fXbXeY1dtlU9Sf5asq89yvFfF2x1GMk94HUWcogJG6V+VFe0okczpqaUdVFl0VRRFmWNwYoMoYgwu9\/Dfn4YIej4QRnLWlkeXhni8OTEf2znPs7XfY2O5QhMMgK9jc6i8Goqjz97uebA5XETLxwb3eTOIoYZznOGHY3OvzZPyUWIQQh6aaEylP5wKcP7i4Nxqq6MTAAvqzoAKeD8cp6Np6SznVrQPMCC6QGtPSsb6lblBaQc+42cbTyv2NoqYY8X3x4ACz+Z3Y+JM\/ZqjBvU2MMqtq27XXsg93tustUldFoRAj1f6w5BJxzFpF2iDXry2KMaeM453DOtQ9ajvtPH1Wl1+vVDPX7fbz37UZzaXPZOru5sDm\/jlERaYeuqrb7jc9y3e7v7xNFEZGI8vJkxnRW8N69XfKiIgTh808+WGGiecmyve7V72o3sd8C0UCqwVDzC1oAAAAASUVORK5CYII=\u0022 data-src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-ram_big.png\u0022 alt=\u0022Screenshot of a RAM usage graph, dropping from almost 100% to under 40%\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022537.31343283582\u0022\u003E\u003C\/span\u003E\n                \n            \u003C\/figure\u003E\u003C\/p\u003E\n\u003Cfigure\u003E\n                \u003Cnoscript\u003E\n                    \u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-network_big.png\u0022 alt=\u0022Screenshot of a network activity usage graph\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022529.55538809344\u0022\u003E                \n                \u003C\/noscript\u003E\n                \u003Cspan class=\u0022hide-noscript\u0022\u003E\u003Cimg src=\u0022data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAACQAAAATCAYAAAD4f6+NAAAACXBIWXMAAA7EAAAOxAGVKw4bAAACQElEQVRIicWV224TMRCGv7H3lE0iCi0ScMcF7\/8gPAOVeoMKbRq6UbIne4aLsEvaUtomkTrSSh7bMzv+\/NuWq0Vl5xc\/UTPen865Wa45eVPy5fMHXsPEzGy3o+8DIkKS+CeDLVSYgSRzqibiBaa5R0T2LigB2Gw2qCrT6XQs5F6d\/zTVgMZI13k6TfAC1tfkkympf3lRInKXkJkdtLrBvt92fHqT7RX7gNBzyDxlfdC98ojItqCyLIHjEYrG3nkSgLquiTEejVBUO4zQZDIBjkfIOICQmVHX9VE1pHYAobZtj64hkP0JFUVB0zSEEI5GyA4hBFAUxZjoOIRepiHtrnHZGfDnlDVNQ4yRsixfRUPaV0h6iojgALIsI89zLqseEUENgkLVRNadcttEFuuAGmw6pQ2GIRjC+aIFhGUdt1e\/CIOGnvthYSSaAMQYiTHy7brlsuqo6yWSzNj0Qpl6skRogxIVJknACdS9omRMc8fXixUDkDSRF2vINIxySQDSNMV7z5k\/51e15mO5Bskgi5i47dXrHPgECQuQhLeimD9B+hXmpiAGrgQN3HRzLq7ekXohRKWPRp44eoUyhaiQJ8KyDjhxdKsbqH4wLYq\/j+tqtaJt2zsrc84RYxzbIoKqjuMi8oDEMGfYAufcGDP0DTH3\/fl8vtVQ0zS0bfvgZKgqzjnMDFUdEw\/tIdHu2LBdMcYx325eEcE5N84VEbz3TCYTiqLYblnf9+PzsbuaXSKP+bt6GYr\/n\/\/YP2azGQC\/AdK2jZUbYES7AAAAAElFTkSuQmCC\u0022 data-src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-network_big.png\u0022 alt=\u0022Screenshot of a network activity usage graph\u0022 class=\u0022border\u0022 width=\u0022960\u0022 height=\u0022529.55538809344\u0022\u003E\u003C\/span\u003E\n                \n            \u003C\/figure\u003E\u003C\/p\u003E\n\u003Chr \/\u003E\n\u003Cul\u003E\n\u003Cli\u003E\u003Csup\u003E[1]\u003C\/sup\u003E \u003Csmall\u003EOr more precisely: 80% of the old price (based on traffic ratio),\nwas paid by \u003Ca href=\u0022https:\/\/en.pronouns.page\/team\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E NLC\u003C\/a\u003E\nand 20% by me \u2013 now that it\u0027s just my personal server again I\u0027m gonna pay 100% \ud83d\ude05\u003C\/small\u003E\u003C\/li\u003E\n\u003C\/ul\u003E\n\u003Cul\u003E\n\u003Cli\u003E\u003Csup\u003E[2]\u003C\/sup\u003E \u003Csmall\u003EThat\u0027s just VPS costs, AWS bill is a whole other story.\u003C\/small\u003E\u003C\/li\u003E\n\u003C\/ul\u003E\u003Csvg xmlns=\u0022http:\/\/www.w3.org\/2000\/svg\u0022 style=\u0022display: none;\u0022\u003E\u003C\/svg\u003E","tags":["devops","server","nginx","apache","pm2","supervisor","ovh","hetzner","node"],"hasMore":true,"image":"https:\/\/avris.it\/image\/pronouns-page-migration_small.png","introLite":"\u003Cfigure\u003E\u003Ca href=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_big.png\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_mini.png\u0022 alt=\u0022Terminal with `ls -lah \/etc\/nginx\/sites-enabled\/` executed on the new server\u0022 width=\u0022240\u0022 height=\u0022136.58886894075\u0022 loading=\u0022lazy\u0022\u003E\u003C\/a\u003E\u003C\/figure\u003E\u003C\/p\u003E\n\u003Cp\u003EWhen it comes to \u003Ca href=\u0022https:\/\/en.wikipedia.org\/wiki\/DevOps\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E DevOps\u003C\/a\u003E, I\u0027m just the \u201cdev\u201d.\nI write code, but I\u0027d rather have someone else worry about making sure it keeps running as intended.\nI manage my personal VPS, I manage some servers at work, but I wouldn\u0027t call myself an expert in that area at all.\nSo I\u0027m super proud of myself and how well it went when I migrated a big project to a new machine \ud83d\ude0a\nThe downtime was just 15 minutes! Here\u0027s the story, if you\u0027re interested.\u003C\/p\u003E","contentLite":"\u003Cfigure\u003E\u003Ca href=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_big.png\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration_mini.png\u0022 alt=\u0022Terminal with `ls -lah \/etc\/nginx\/sites-enabled\/` executed on the new server\u0022 width=\u0022240\u0022 height=\u0022136.58886894075\u0022 loading=\u0022lazy\u0022\u003E\u003C\/a\u003E\u003C\/figure\u003E\u003C\/p\u003E\n\u003Cp\u003EWhen it comes to \u003Ca href=\u0022https:\/\/en.wikipedia.org\/wiki\/DevOps\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E DevOps\u003C\/a\u003E, I\u0027m just the \u201cdev\u201d.\nI write code, but I\u0027d rather have someone else worry about making sure it keeps running as intended.\nI manage my personal VPS, I manage some servers at work, but I wouldn\u0027t call myself an expert in that area at all.\nSo I\u0027m super proud of myself and how well it went when I migrated a big project to a new machine \ud83d\ude0a\nThe downtime was just 15 minutes! Here\u0027s the story, if you\u0027re interested.\u003C\/p\u003E\n\u003Ch3\u003EWhy move?\u003C\/h3\u003E\n\u003Cul\u003E\n\u003Cli\u003EPronouns.page gets lots of traffic:\n\u003Ca href=\u0022https:\/\/stats.pronouns.page\/en.pronouns.page?period=30d\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E around 4 million pageviews a month\u003C\/a\u003E\nfor the English version alone. And on top of that, as an unapologetically queer website, we\u0027re a target of regular DDoS attacks.\nThe previous server didn\u0027t handle it well. We need an upgrade.\u003C\/li\u003E\n\u003Cli\u003EPronouns.page used to run on my personal VPS together with all my other pet projects.\nThey might not need many resources in comparison\n(damn, PP has outgrown everything else, including my wildest dreams \ud83d\ude05),\nbut it\u0027s still gonna be useful to give PP dedicated resources.\u003C\/li\u003E\n\u003Cli\u003EI used to be a bottleneck when it comes to fixing any issues on the server.\nNow that my personal stuff is on a different server, multiple people can have SSH access to the PP instance\nand intervene when necessary.\u003C\/li\u003E\n\u003Cli\u003EWe moved from OVH to Hetzner, getting a way better deal.\nInstead of 21.5\u20ac\/m\u003Csup\u003E[1]\u003C\/sup\u003E for 2 vCPU, 8GB RAM and 130GB drive\nwe\u0027ll now pay 27.25\u20ac\/m for 8 vCPU, 16GB RAM and 240GB drive.\u003Csup\u003E[2]\u003C\/sup\u003E\u003C\/li\u003E\n\u003Cli\u003EThe cost can now be charged directly to the \u003Ca href=\u0022https:\/\/en.pronouns.page\/team\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E NLC\u003C\/a\u003E account,\nI won\u0027t have to manage reimbusements (well, as soon as I figure out AWS migration).\u003C\/li\u003E\n\u003Cli\u003EI could use the opportunity to migrate to a new stack,\nwhich would\u0027ve been hard to do on a running server with tens of active projects.\u003C\/li\u003E\n\u003C\/ul\u003E\n\u003Ch3\u003EThe plan\u003C\/h3\u003E\n\u003Cp\u003EI bought the server on Wednesday and started setting it up, completely independently of the old machine.\nI created a setup that from the outside was indistinguishable from the old one, except for using an older database backup.\nUntil Saturday morning they were running simultaneously \u2013 the DNS were pointing to the old IP,\nbut my local \u003Ccode\u003E\/etc\/hosts\u003C\/code\u003E to the new one.\u003C\/p\u003E\n\u003Cp\u003EI picked Saturday morning, because mornings are when our traffic is lowest,\nand that day I\u0027m free and can focus on the migration,\neven if something goes wrong and takes more time than expected.\u003C\/p\u003E\n\u003Cp\u003EAn important part of the plan was taking notes \u2013 every important command I had run, I documented for myself.\nIn case anything goes wrong and I have to start over, or in case I wanted to later migrate my other projects to Hetzner too\n(and I do), I\u0027d have a recepe basically ready.\u003C\/p\u003E\n\u003Ch3\u003ECool things\u003C\/h3\u003E\n\u003Cp\u003EThe old configs were\u2026 meh. Long, repetitive and messy.\nJust switching from \u003Ca href=\u0022https:\/\/apache.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Apache\u003C\/a\u003E to \u003Ca href=\u0022https:\/\/nginx.org\/en\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E nginx\u003C\/a\u003E simplified them massively,\nbut I went further and extracted common parts for all domains and subdomains to make them reusable.\nSetting up everything for a new language version used to be a half-an-hour-or-so long process \u2013 now it takes me a few minutes.\u003C\/p\u003E\n\u003Cp\u003EThere\u0027s two things that were really annoying for me to figure out: analytics and monitoring.\nThose things are \u003Cem\u003Eexpensive\u003C\/em\u003E when your traffic is as big as ours.\nCorporations with such traffic can afford it easily, but we\u0027re not a business,\nwe have no considerable income other than donations and an \u003Ca href=\u0022https:\/\/arc.io\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E arc.io\u003C\/a\u003E widget.\nWe need to self-host those things, then.\u003C\/p\u003E\n\u003Cp\u003EFor traffic analytics, I\u0027ve recently switched from \u003Ca href=\u0022https:\/\/matomo.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Matomo\u003C\/a\u003E to \u003Ca href=\u0022https:\/\/plausible.io\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Plausible\u003C\/a\u003E.\nAs much as Matomo is better than Google Analytics from the privicy perspective\n(and it\u0027s \u003Cem\u003Ea lot\u003C\/em\u003E better), it\u0027s still really heavy and has way more features than I\u0027ll probably ever need.\nI needed to pay 11\u20ac\/m extra for a separate database server just for Matomo to store its logs.\nPlausible, on the other hand, is \u003Cem\u003Eexactly\u003C\/em\u003E what I need. So neat!\u003C\/p\u003E\n\u003Cp\u003EFor monitoring\u2026 We used to have one, hosted on AWS Lambda, but it kept causing trouble\nthat I had no time to fix, and in the end I disabled it.\nMonitoring is quite hard, because it needs to run on a dedicated and super reliable infrastructure\n\u2013 we can\u0027t monitor ourselves after all; if the server is down, so is the monitor that\u0027s supposed to let us know.\nI found out a really cool tool that keeps blowing my mind with its ingenuity:\n\u003Ca href=\u0022https:\/\/upptime.js.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Upptime\u003C\/a\u003E. It runs entirely on GitHub pages and actions,\ntotally for free (as long as you keep it public).\u003C\/p\u003E\n\u003Ch3\u003EIssues\u003C\/h3\u003E\n\u003Cp\u003EWe have multiple node servers running for different language versions.\nEach needs to run at some port and then nginx needs to be configured as a reverse proxy to that specific port.\nBefore it was really tediuos, I\u0027d just enumerate the ports starting at 3001 and for each new version\nI\u0027d first need to sift through configs to find what the current max value was.\nI kept looking up which port was related to which domain. Annoying shit.\u003C\/p\u003E\n\u003Cp\u003EIn the new setup, I wanted to have some single source of truth for the domain-port pairs,\nbut it turned out to be near impossible (at least for my skills). Nginx is strict about\nits configs being static. I couldn\u0027t find an easy way to pass data about those ports to it.\u003C\/p\u003E\n\u003Cp\u003EUltimately, I settled for a compromise system: I unambiguously map each version to a port\nby convering each letter of its ISO code to its index in the English alphabet.\nFor example for Japanese, \u003Ccode\u003Eja\u003C\/code\u003E, the port would be \u003Ccode\u003E31001\u003C\/code\u003E because \u201cj\u201d is the 10\u003Csup\u003Eth\u003C\/sup\u003E letter of the alphabet,\nand \u201ca\u201d is the 1\u003Csup\u003Est\u003C\/sup\u003E. Not a perfect system, but it\u0027s gonna simplify my flow a lot.\u003C\/p\u003E\n\u003Cp\u003EI wanted to switch from \u003Ca href=\u0022http:\/\/supervisord.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E supervisor\u003C\/a\u003E to \u003Ca href=\u0022https:\/\/pm2.io\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E pm2\u003C\/a\u003E\nfor its nicer interface and cool features, but for some reason it kept dropping a significant percentage of requests at (seemingly) random.\nI couldn\u0027t figure out the root cause, so I gave up. That switch wasn\u0027t worth that much of my time, I reverted back to supervisor.\u003C\/p\u003E\n\u003Cp\u003EA huge issue was that\u2026 no emails were getting sent from the new server.\nEvery SMTP request would just time out. I tried everything I could think of,\nI figured I must\u0027ve misconfigured something. I asked the team for help, but we couldn\u0027t find a root cause.\nBut then I got this random hypothesis \u2013 what if Hetzner is blocking port 465 regardless of our firewall settings?\nA quick seach \u2013 \u003Ca href=\u0022https:\/\/docs.hetzner.com\/cloud\/servers\/faq\/#why-can-i-not-send-any-mails-from-my-server\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E turns out they do\u003C\/a\u003E!\nUgh\u2026 I get their reasoning, seems like a perfectly reasonable approach. Switching ports worked.\nI just wish they somehow gave notice more visibly, not just an unexplained timeout,\nso that I wouldn\u0027t waste so much time figuring it out.\u003C\/p\u003E\n\u003Cp\u003EI struggled with moving Plausible too\u2026 It runs in Docker containers \u2013 but my knowlege of Docker is pretty limited.\nI spent way too much time trying to \u003Ccode\u003Epg_dump\u003C\/code\u003E and \u003Ccode\u003Epg_restore\u003C\/code\u003E my way from one container to host to another host via \u003Ccode\u003Escp\u003C\/code\u003E\nto another container, only to finally succeed and then\u2026 realise that the Postgres database is just half of the story.\nThe main chunk of data resides in \u003Ca href=\u0022https:\/\/clickhouse.com\/clickhouse\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E ClickHouse\u003C\/a\u003E.\nInstead of struggling with the whole thing again, I took a different approach: backing up and restoring entire Docker volumes.\nIt worked like a charm! Here are the commands, if you\u0027re interested:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode\u003E# old host\ndocker run -v plausible_db-data:\/volume --rm loomchild\/volume-backup backup - \u0026gt; .\/db-data.tar.bz2\ndocker run -v plausible_event-data:\/volume --rm loomchild\/volume-backup backup - \u0026gt; .\/event-data.tar.bz2\n\nscp .\/db-data.tar.bz2 pp:\/home\/admin\/www\/stats.pronouns.page\nscp .\/event-data.tar.bz2 pp:\/home\/admin\/www\/stats.pronouns.page\n\n# new host\ncat .\/db-data.tar.bz2 | sudo docker run -i -v statspronounspage_db-data:\/volume --rm loomchild\/volume-backup restore -f -\ncat .\/event-data.tar.bz2 | sudo docker run -i -v statspronounspage_event-data:\/volume --rm loomchild\/volume-backup restore -f -\u003C\/code\u003E\u003C\/pre\u003E\n\u003Ch3\u003EThe switch\u003C\/h3\u003E\n\u003Cp\u003EWhen everything was ready, on Friday evening, I announced the upcoming maintanance and waited.\nI had all the commands ready in a notepad.\u003C\/p\u003E\n\u003Cp\u003EWhen the time came, I just\u2026 stopped both servers, moved the database, moved plausible volumes,\nstarted the new server, and then updated the DNS entries. Simple as that.\u003C\/p\u003E\n\u003Cp\u003EIt was tons of work over a couple of evenings, but good preparation made wonders:\nit only took a quarter to actually switch.\nConsidering how ops is not my forte at all,\nI\u0027m so proud of having accomplished such a smooth transition \ud83e\udd70\u003C\/p\u003E\n\u003Cfigure\u003E\u003Ca href=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-cpu_big.png\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-cpu_mini.png\u0022 alt=\u0022Screenshot of a CPU usage graph, dropping from ~50% to ~20%\u0022 width=\u0022240\u0022 height=\u0022134.74001507159\u0022 loading=\u0022lazy\u0022\u003E\u003C\/a\u003E\u003C\/figure\u003E\u003C\/p\u003E\n\u003Cfigure\u003E\u003Ca href=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-ram_big.png\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-ram_mini.png\u0022 alt=\u0022Screenshot of a RAM usage graph, dropping from almost 100% to under 40%\u0022 width=\u0022240\u0022 height=\u0022134.32835820896\u0022 loading=\u0022lazy\u0022\u003E\u003C\/a\u003E\u003C\/figure\u003E\u003C\/p\u003E\n\u003Cfigure\u003E\u003Ca href=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-network_big.png\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Cimg src=\u0022https:\/\/avris.it\/image\/pronouns-page-migration-old-network_mini.png\u0022 alt=\u0022Screenshot of a network activity usage graph\u0022 width=\u0022240\u0022 height=\u0022132.38884702336\u0022 loading=\u0022lazy\u0022\u003E\u003C\/a\u003E\u003C\/figure\u003E\u003C\/p\u003E\n\u003Chr \/\u003E\n\u003Cul\u003E\n\u003Cli\u003E\u003Csup\u003E[1]\u003C\/sup\u003E \u003Csmall\u003EOr more precisely: 80% of the old price (based on traffic ratio),\nwas paid by \u003Ca href=\u0022https:\/\/en.pronouns.page\/team\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E NLC\u003C\/a\u003E\nand 20% by me \u2013 now that it\u0027s just my personal server again I\u0027m gonna pay 100% \ud83d\ude05\u003C\/small\u003E\u003C\/li\u003E\n\u003C\/ul\u003E\n\u003Cul\u003E\n\u003Cli\u003E\u003Csup\u003E[2]\u003C\/sup\u003E \u003Csmall\u003EThat\u0027s just VPS costs, AWS bill is a whole other story.\u003C\/small\u003E\u003C\/li\u003E\n\u003C\/ul\u003E","words":1432,"readTime":6,"lang":"en"}}}}}