-
Notifications
You must be signed in to change notification settings - Fork 12
Deployment
Note: these are internal instructions for deployment on the CERN OpenStack infrastructure.
Therefore, unless you're deploying HEPData.net, you may safely ignore this document.
It's good practice to check that the tests pass locally before pushing to GitHub. You first need to create a local database called hepdata_test, then the command can be found in the .travis.yaml file:
py.test tests -k 'not tests/e2e'
You can then push to the master branch on GitHub. The local tests can interfere with the index, so you might need to run hepdata utils reindex locally after running the tests to recover your original records. Check that the tests pass on Travis for the master branch on GitHub.
The production branch is used for deployment. To synchronise with the master branch, repeat the following steps locally:
git rebase master production
git push
git checkout master
First ssh to aiadm.cern.ch, then to root@hepdata-builder1.
workon hepdata
cdvirtualenv scripts
fab update
fab build
fab -u <user_name> -I deploy
You can also stop and start as well as deploy.
There are some bugs that are difficult or impossible to reproduce locally, therefore debugging online is necessary. This can be done by editing the source code directly on the web servers. Any changes will be overwritten when the code is deployed. First ssh to aiadm.cern.ch, then to root@hepdata-web1 or root@hepdata-web2. The source code is in the /opt/hepdata/src/hepdata directory. Files need to be edited on both machines hepdata-web1 and hepdata-web2, then the updated .pyc files can be generated from the edited .py files using supervisorctl restart gunicorn, before taking some action from a web browser directed at the hepdata.net site. Log files can be found in the /var/log/supervisor directory.
Celery is used for tasks such as sending emails and registering DOIs with DataCite. The log files can be found as /var/log/supervisor/celery-worker.err on hepdata-task1 or hepdata-task2. Generally tasks are completed within at most a few seconds of being received. If tasks are received but are held in the queue, you might need to restart the Celery daemon using supervisorctl restart celeryd on hepdata-task1 or hepdata-task2.
The hepdata.net site can occasionally go down or tables fail to load, and redeploying doesn't fix the problem. It might be necessary to restart the EOS daemon on hepdata-web1 and hepdata-web2 with service eosd restart.
Check the quota with eos root://eospublic.cern.ch quota /eos/hepdata on any of the hepdata-* machines. Note that there is a quota on both disk space (currently up to a maximum of 5 TB) and number of files (currently up to a maximum of 2 million).
Secrets such as passwords can be accessed from aiadm.cern.ch via the tbag command (see man tbag or tbag -h for help). Here are some basic commands:
# Show all keys for the hepdata hostgroup:
tbag showkeys --hg hepdata
# Show the secret with key MY_PASSWORD:
tbag show MY_PASSWORD --hg hepdata
# Set the secret with key MY_PASSWORD:
tbag set MY_PASSWORD --hg hepdata
Service status can only be monitored from inside the CERN network, e.g. by starting Firefox from lxplus.cern.ch:
ssh lxplus.cern.ch firefox https://hepdata-monitor1.cern.ch/nagios/cgi-bin/status.cgi
The user name (MONITOR_USER) and password (MONITOR_PASS) can be obtained using tbag as described above.
The converter software runs as a Docker container currently deployed on a CERN OpenStack cluster called hepdata-converter-2018 created on 3rd January 2018. See some documentation at http://clouddocs.web.cern.ch/clouddocs/containers/ . To gain access to the cluster, log in to https://openstack.cern.ch with your personal CERN account and select the "GS HepData critical power" project. (Clusters can be created via the web interface at https://openstack.cern.ch/project/clusters .) Select "API Access", then "DOWNLOAD OPENSTACK RC FILE", then "OPENSTACK RC FILE (IDENTITY API V3)". SSH to lxplus-cloud.cern.ch and copy the file, then do e.g. source hepdata-openrc.sh and enter your CERN password. Then do openstack coe cluster config hepdata-converter-2018 > env.sh. These steps only need to be done once.
Subsequently, just log in to lxplus-cloud.cern.ch and do . env.sh to set up the Docker environment, then you can use normal Docker commands to manage the container, e.g.
docker ps
docker pull hepdata/hepdata-converter-ws
docker run --restart=always -d --name=hepdata_converter -p 0.0.0.0:80:5000 hepdata/hepdata-converter-ws hepdata-converter-ws
docker restart db1ec55cb69e
docker stop db1ec55cb69e
docker rm db1ec55cb69e
The Docker container will be accessible within CERN from the URL http://188.184.65.191 . This URL is specified in the config.py file of the main HEPData code.