[DEVOPS] Setup fresh stage server on Azure

[kshitijrajsharma]

Stage server should be pointed to this URL : https://export-staging.hotosm.org/ , Currently it is broken and we are trying to setup fresh with raw-data-api

[eternaltyro]

Standard instance on Azure with the following configuration.

Instance capacity:

Root disk: 32GB (mounted on /)
Data disk: 200 GB (mounted to /data)

Applications to be installed:

0. Ubuntu 22.04 LTS
1. PostgreSQL server (in-situ)
2. Redis server (in-situ)
3. Python == 3.6.10 (installed using conda or PyPy)
4. Node.js == 16.14

Monitoring:

1. Jumpcloud agent
2. Newrelic infra agent

Configuration

• RAW_DATA_API_URL = https://raw-data-api-demo.hotosm.org/
• HDX_Site = demo

Sensitive strings:

1. Mailgun SMTP credentials
2. PostgreSQL credentials
3. OAuth1 client ID and secret

Database configuration

• CREATE DATABASE exports
• CREATE EXTENSION postgis
• Add exports specific database user

Configuring OAuth app

Create OAuth1 application in OSM with Read preferences in scope. Redirect URI = https://export-stage.hotosm.org/authorized

Other packages

• Some java packages are required for Garmin exports.
• osmium - for updating the planet file

Scheduled jobs

• cleanup.py and schedule.py must be running as cron services (or systemd.timer) . Configuration and schedule to be copied from production

Building frontend

cd ui
npm run dist
cd ..
python manage.py collectstatic

[eternaltyro]

Added supporting systemd files and corresponding timers and env files in this PR: #495

[eternaltyro]

Export tool requires a specific (and unfortunately unsupported) version of Python to function. Old Python versions can be managed using miniconda.

$ curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda-setup.sh
$ chmod +x miniconda-setup.sh && ./miniconda-setup.sh
$ export PATH=$PATH:/home/user/miniconda3/bin
$ conda config --set auto_activate_base false
$ conda create --name exports python=3.6.10 rasterio gdal rtree
$ conda activate exports 
(exports) $ python --version
Python 3.6.10 :: Anaconda, Inc.

[eternaltyro]

Suggested changes to env vars:

Via review to PR: #498
And by issue: #499

[eternaltyro]

Discovered that data is being shipped to Sentry using an old outdated python library called Raven. Requested removal: #500