RationalWiki:Server migration 2019
From RationalWiki
Contents
Old setup[edit]
- cache1, with nginx (for SSL termination) and Varnish
- apache3, with everything else
New setup[edit]
- db1: MariaDB
- search1: ElasticSearch
- apache4: Apache, memcached, images, postfix. Images will now be in volume storage.
- cache2, same as cache1 except smaller (4GB)
- backup1: private backups
Migration plan[edit]
- backup1
-
Create with large volume storage
-
- Images
-
Add images1 volume to apache3, formatted btrfs -
Initial rsync -
Stop MW from writing to the old directory. Set up a read-only bind mount of the old directory, tell MW to use this as its upload directory. -
Final rsync -
Switch apache conf to use the new location -
Switch MW conf to the new location -
Set up cron job to send btrfs snapshots of images1 volume to backup1, using btrfs-sxbackup
-
- Search
-
Create search1 -
Install ElasticSearch 1.7.6 -
Stop MW job queue by commenting out cron job -
Copy ElasticSearch data from apache3 to search1 -
Start ElasticSearch on search1 -
Reconfigure MW to use search1 -
Restart MW job queue
-
- db1
-
Create db1 -
Enable binlog on apache3. -
Dump apache3 using mysqldump, single transaction etc., into db1. Also copy to backup1. -
Enable binlog on db1 (for point-in-time recovery from backup1). -
Start replication on db1 with apache3 as the master. Confirm that it is replicating properly and has zero lag. -
Switch MW to read-only mode -
Set read_only on apache3 -
Stop replication -
Reconfigure MW to use db1 as its DB server, and turn off read-only mode -
Set up a cron job to copy db1 binlogs to backup1.
-
- apache4
-
Create apache4, install dependencies, MW -
Add images2 volume, restore from backup1 -
On apache3 remount images as read-only. Make images snapshot and sync to apache4 -
Switch backend in Varnish -
Install btrfs backup cron job -
Archive apache3 log files -
Delete apache3
-
- cache2
-
Create cache2 -
Copy configuration from cache1 -
Start nginx and varnish and test -
Configure MW to send purges to cache2 and accept XFF headers from it -
Switch DNS -
Wait ~2 days -
Delete cache1
-
Firewall[edit]
Unlike the equivalent AWS feature, "private" networking turns out to mean a /16 network shared with everyone in your Linode datacentre. This is not very useful for access control. Instead, everything now has a ufw firewall allowing access to all our public server IPs. The private network is blocked, just like the remainder of the public network.