Running system updates (apt)

live-gateways-X

Both live-gateways are used in the DNS so there will be downtime while rebooting. But that is usually very fast (less than 30 seconds). Upgrade and reboot one machine first and then wait a few hours before doing the second to make sure the upgraded one works properly.

• sudo apt update
• sudo apt upgrade
• sudo apt autoremove
• sudo reboot

live-dockerswarm-X

It's recommended to update the swarm leader last so that the leader only have to change once.

• Change haproxy state to MAINT
  • This will stop all traffic to the proxy on that machine
• [on swarm manager]: docker node update --availability drain live-dockerswarm-X
  • Services will migrate to other nodes. Ensure cluster is stable before proceeding.
• sudo apt update
• sudo apt upgrade
• sudo apt autoremove
• sudo reboot
• Ensure the machine starts and is back in the swarm: [on swarm manager]: docker node ls
• [on swarm manager]: docker node update --availability active live-dockerswarm-X
  • The proxy service (nginx) will start automatically
  • Other services will not move back automatically. If the swarm needs to be rebalanced you can do so by restarting specific services (docker service update --force [name])
• Change haproxy state to READY

live-solrcloud-X

• [on swarm manager]: docker node update --availability drain live-solrcloud-X
  • Solr and Zookeeper will go down since they are tied to a specific node. The Solr cloud will still work fine on two nodes.
  • Other services will migrate to other nodes. Ensure cluster is stable before proceeding.
• sudo apt update
• sudo apt upgrade
• sudo apt autoremove
• sudo reboot
• Ensure the machine starts and is back in the swarm [on swarm manager]: docker node ls
• [on swarm manager]: docker node update --availability active live-solrcloud-X
  • Solr and Zookeeper will start automatically
  • Other services will not move back automatically
• You may need to rebalance the collection leaders with this command: curl -X POST "http://live-solrcloud-1:8973/solr/admin/collections?action=REBALANCELEADERS&collection=biocache"

live-nfsserver-1

All swarm services that use the NFS on live-nfsserver-1 should be stopped when the server is rebooted. There are scripts in /data/apps/maintenance on the swarm managers for stopping and starting these services.

• sudo apt update
• [on swarm manager]:
  1. ./stop-services.sh
  2. ./stop-dbs.sh
• sudo apt upgrade
• sudo apt autoremove
• sudo reboot
• [on swarm manager]:
  1. ./start-dbs.sh
  2. ./start-services-1.sh
  3. ./start-services-2.sh
  4. ./start-services-3.sh
  5. ./start-services-4.sh

live-mail-1

Mailserver will be down while rebooting.

• sudo apt update
• sudo apt upgrade
• sudo apt autoremove
• sudo reboot

live-monitor-1

Monitoring and analytics stacks will be down while rebooting.

• sudo apt update
• sudo apt upgrade
• sudo apt autoremove
• sudo reboot