RationalWiki:Server migration 2019

From RationalWiki
Jump to navigation Jump to search

Old setup[edit]

  • cache1, with nginx (for SSL termination) and Varnish
  • apache3, with everything else

New setup[edit]

  • db1: MariaDB
  • search1: ElasticSearch
  • apache4: Apache, memcached, images, postfix. Images will now be in volume storage.
  • cache2, same as cache1 except smaller (4GB)
  • backup1: private backups

Migration plan[edit]

  • backup1
    • Create with large volume storage
  • Images
    • Add images1 volume to apache3, formatted btrfs
    • Initial rsync
    • Stop MW from writing to the old directory. Set up a read-only bind mount of the old directory, tell MW to use this as its upload directory.
    • Final rsync
    • Switch apache conf to use the new location
    • Switch MW conf to the new location
    • Set up cron job to send btrfs snapshots of images1 volume to backup1, using btrfs-sxbackup
  • Search
    • Create search1
    • Install ElasticSearch 1.7.6
    • Stop MW job queue by commenting out cron job
    • Copy ElasticSearch data from apache3 to search1
    • Start ElasticSearch on search1
    • Reconfigure MW to use search1
    • Restart MW job queue
  • db1
    • Create db1
    • Enable binlog on apache3.
    • Dump apache3 using mysqldump, single transaction etc., into db1. Also copy to backup1.
    • Enable binlog on db1 (for point-in-time recovery from backup1).
    • Start replication on db1 with apache3 as the master. Confirm that it is replicating properly and has zero lag.
    • Switch MW to read-only mode
    • Set read_only on apache3
    • Stop replication
    • Reconfigure MW to use db1 as its DB server, and turn off read-only mode
    • Set up a cron job to copy db1 binlogs to backup1.
  • apache4
    • Create apache4, install dependencies, MW
    • Add images2 volume, restore from backup1
    • On apache3 remount images as read-only. Make images snapshot and sync to apache4
    • Switch backend in Varnish
    • Install btrfs backup cron job
    • Archive apache3 log files
    • Delete apache3
  • cache2
    • Create cache2
    • Copy configuration from cache1
    • Start nginx and varnish and test
    • Configure MW to send purges to cache2 and accept XFF headers from it
    • Switch DNS
    • Wait ~2 days
    • Delete cache1


Unlike the equivalent AWS feature, "private" networking turns out to mean a /16 network shared with everyone in your Linode datacentre. This is not very useful for access control. Instead, everything now has a ufw firewall allowing access to all our public server IPs. The private network is blocked, just like the remainder of the public network.