[Upgrade Ubuntu] de_prod

[dacook]

• Slack thread: https://openfoodnetwork.slack.com/archives/CG7NJ966B/p1730697881850899

Checklist based on general guide https://github.com/openfoodfoundation/ofn-install/wiki/Migrating-a-Production-Server

1. Setting up the new server

• Check old server config for any additional services to be aware of (eg /etc/nginx/sites-available). Document any necessary steps for migration.
• Hosting: provision new server with Ubuntu 20
• DNS: add temporary domain (eg prod2.openfoodnetwork.org)

config

• Add temporary name to inventory/hosts
• Review host_vars/x/config.yml, clean up if needed
  • Make a copy for the temp hostname, add temp domain to bottom of certbot_domains
• Review ofn-secrets:x_prod/secrets.yml, clean up if needed
  • Change to shared bugsnag projects
  • Don't bother making a copy of this one

setup

Enable passthrough on current server to allow new server to generate a certificate:

• ansible-playbook playbooks/letsencrypt_proxy.yml -l x_prod -e "proxy_target=<ip>"

Then setup new server. Ensure you have the correct secrets (current secrets are usually fine).
ansible-playbook -l x_prod2 -e "@../ofn-secrets/x_prod/secrets.yml playbooks/

• setup.yml -e ansible_user=ubuntu
• provision.yml
• deploy.yml
• db_integrations (Permit DB access for n8n, Metabase)

initial migration

• Ensure sidekiq is disabled, to avoid creating subscription orders when data is loaded:
  sudo systemctl stop sidekiq && sudo systemctl disable sidekiq
• Setup direct ssh access for ofn-admin and openfoodnetwork as per guide

ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/

• db_transfer.yml
• transfer_assets.yml

2. Testing

• test reboot
• send test mail (/admin/mail_methods/edit).
• local file, eg terms of service file: /admin/terms_of_service_files
• shop catalogue display correctly, with images, add to cart, begin checkout, login
• note: check cookies if login won't work
• Check integrations
  • Payments (check Stripe connect status /admin/stripe_connect_settings/edit)
  • New Relic
  • Bugsnag

3. Migration

preparation

• Reset database on new server, to avoid any migration issues due to being out of sync
  bin/rake db:reset (You will need to confirm. Make sure you're on the new server!)
• deploy.yml -l x_prod2 -e "git_version=vX.Y.Z" matching version with current prod
• old server: make a tiny data change to verify later (eg add . in meta description /admin/general_settings/edit)

switchover: old server

• 🚧 ansible-playbook playbooks/maintenance_mode.yml -l x_prod
• sudo systemctl stop sidekiq redis-jobs puma
• Transfer /var/lib/redis-jobs/dump.rdb to new server (see guide)
• ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/db_transfer.yml ~3min
• sudo systemctl stop postgres (ensure other integrations no longer touch it)
• ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/transfer_assets.yml just in case

switchover: new server

• sudo systemctl restart puma; sudo systemctl start sidekiq redis-jobs
• cd ~/apps/openfoodnetwork/current; bin/rails runner -e production "Rails.cache.clear" (or migrate redis-cache/dump.rdb also)
• ⏭️ ansible-playbook -l x_prod playbooks/temporary_proxy.yml -e 'proxy_target=<new_ip>' redirect traffic to new prod
  • Note: this doesn't include webservices, and doesn't handle images. So it's a very short-term trick, if at all.
  • Use a hosts file entry to test a direct connection
• Check there are no alarm bells, eg:
  • ~/apps/openfoodnetwork/current/logs/production.log and sidekiq.log
  • tiny data change is present. undo it.
  • shopfront and checkout looks good
  • upload a product image
• Update DNS to point to new server
• get confirmation from local team

4. Cleanup (after 48hrs)

• check server access logs to verify no traffic
• shut down the old server, cancel old VPS
• remove DNS for temporary subdomain
• make sure the entries in ofn-install are up to date: remove the temporary entry made for the migration, and set the new IP address.
• validate that provision.yml still works. This will rename x-prod2 to x-prod
• check metabase sync if required: https://data.openfoodnetwork.org.uk/admin/databases/
• check n8n
• check backups are functioning
• Update documentation:
  • https://github.com/openfoodfoundation/ofn-install/wiki/Current-servers#uk (copy note from staging about redis version pinned)
  • This migration guide if necessary

Rollback plan

• If an error occurs before the temporary proxy is active, and can't be resolved quickly, then restore service back to current server
• If an error occurs after proxy is active, users may have interacted with the new server (eg made payments.
  • if serious, consider putting into maintenance mode (and stop sidekiq) to avoid further changes
  • otherwise seek to resolve issue in-place.

[dacook]

Awaiting access to DNS provider (https://openfoodnetwork.slack.com/archives/CG7NJ966B/p1730697881850899)

[dacook]

✅ 🚀 Switched over, all looks good to me!