RationalWiki:Server migration 2019

Old setup

 * cache1, with nginx (for SSL termination) and Varnish
 * apache3, with everything else

New setup

 * db1: MariaDB
 * search1: ElasticSearch
 * apache4: Apache, memcached, images, postfix. Images will now be in volume storage.
 * cache2, same as cache1 except smaller (4GB)
 * backup1: private backups

Migration plan

 * backup1
 * Create with large volume storage
 * Images
 * Add images1 volume to apache3, formatted btrfs
 * Initial rsync
 * Stop MW from writing to the old directory. Set up a read-only bind mount of the old directory, tell MW to use this as its upload directory.
 * Final rsync
 * Switch apache conf to use the new location
 * Switch MW conf to the new location
 * Set up cron job to send btrfs snapshots of images1 volume to backup1, using btrfs-sxbackup
 * Search
 * Create search1
 * Install ElasticSearch 1.7.6
 * Stop MW job queue by commenting out cron job
 * Copy ElasticSearch data from apache3 to search1
 * Start ElasticSearch on search1
 * Reconfigure MW to use search1
 * Restart MW job queue
 * db1
 * Create db1
 * Enable binlog on apache3.
 * Dump apache3 using mysqldump, single transaction etc., into db1. Also copy to backup1.
 * Enable binlog on db1 (for point-in-time recovery from backup1).
 * Start replication on db1 with apache3 as the master. Confirm that it is replicating properly and has zero lag.
 * Switch MW to read-only mode
 * Set read_only on apache3
 * Stop replication
 * Reconfigure MW to use db1 as its DB server, and turn off read-only mode
 * Set up a cron job to copy db1 binlogs to backup1.
 * apache4
 * Create apache4, install dependencies, MW
 * Add images2 volume, restore from backup1
 * On apache3 remount images as read-only. Make images snapshot and sync to apache4
 * Switch backend in Varnish
 * Install btrfs backup cron job
 * Archive apache3 log files
 * Delete apache3
 * cache2
 * Create cache2
 * Copy configuration from cache1
 * Start nginx and varnish and test
 * Configure MW to send purges to cache2 and accept XFF headers from it
 * Switch DNS
 * Wait ~2 days
 * Delete cache1

Firewall
Unlike the equivalent AWS feature, "private" networking turns out to mean a /16 network shared with everyone in your Linode datacentre. This is not very useful for access control. Instead, everything now has a ufw firewall allowing access to all our public server IPs. The private network is blocked, just like the remainder of the public network.