RationalWiki:Server migration 2019

Old setup

• cache1, with nginx (for SSL termination) and Varnish
• apache3, with everything else

New setup

• db1: MariaDB
• search1: ElasticSearch
• apache4: Apache, memcached, images, postfix. Images will now be in volume storage.
• cache2, same as cache1 except smaller (4GB)
• backup1: private backups

Migration plan

• backup1
  • Create with large volume storage
• Images
  • Add images1 volume to apache3, formatted btrfs
  • Initial rsync
  • Stop MW from writing to the old directory. Set up a read-only bind mount of the old directory, tell MW to use this as its upload directory.
  • Final rsync
  • Switch apache conf to use the new location
  • Switch MW conf to the new location
  • Set up cron job to send btrfs snapshots of images1 volume to backup1, using btrfs-sxbackup
• Search
  • Create search1
  • Install ElasticSearch 1.7.6
  • Stop MW job queue by commenting out cron job
  • Copy ElasticSearch data from apache3 to search1
  • Start ElasticSearch on search1
  • Reconfigure MW to use search1
  • Restart MW job queue
• db1
  • Create db1
  • Enable binlog on apache3.
  • Dump apache3 using mysqldump, single transaction etc., into db1. Also copy to backup1.
  • Enable binlog on db1 (for point-in-time recovery from backup1).
  • Start replication on db1 with apache3 as the master. Confirm that it is replicating properly and has zero lag.
  • Switch MW to read-only mode
  • Set read_only on apache3
  • Stop replication
  • Reconfigure MW to use db1 as its DB server, and turn off read-only mode
  • Set up a cron job to copy db1 binlogs to backup1.
• apache4
  • Create apache4, install dependencies, MW
  • Add images2 volume, restore from backup1
  • On apache3 remount images as read-only. Make images snapshot and sync to apache4
  • Switch backend in Varnish
  • Install btrfs backup cron job
  • Archive apache3 log files
  • Delete apache3
• cache2
  • Create cache2
  • Copy configuration from cache1
  • Start nginx and varnish and test
  • Configure MW to send purges to cache2 and accept XFF headers from it
  • Switch DNS
  • Wait ~2 days
  • Delete cache1

Firewall

Unlike the equivalent AWS feature, "private" networking turns out to mean a /16 network shared with everyone in your Linode datacentre. This is not very useful for access control. Instead, everything now has a ufw firewall allowing access to all our public server IPs. The private network is blocked, just like the remainder of the public network.