Deployment 2021-1.0 checklist

[mrakitin]

Current collection workstations

xf17id1-ws1  # interchangeable machine (can be used for DAMA tests if needed)
xf17id1-ws4  # interchangeable machine (can be used for DAMA tests if needed)
xf17id1-ws5  # interchangeable machine (can be used for DAMA tests if needed)
xf17id1-ca1  # collection server serving Jupyter notebooks, running via manage-ioc

Previsit

• Enable Azure-Pipelines CI
• Make sure that the current latest environments are pushed to all the bl machines
• Skim the IPython startup files in profile_collection
  • Check if it's safe to start bsui remotely (does it touch any hardware?) - use git grep .put or git grep caput
• Check for outstanding PRs and issues on beamline repos
• Branch https://github.com/NSLS-II/playbooks for the purpose of updating the FMX current_env_tag in the production file (https://github.com/NSLS-II/playbooks/pull/430)

Housekeeping

• Update /opt/conda/.condarc and conda with ansible

  $ ansible-playbook -i production install-base-conda.yml --limit=FMX -kK -vvvv

• Add BL staff to the BL GitHub organization as owners
• Work with BL staff to commit any un-committed changes to their profiles
• Tag the profile_collection repository as-found as 2020C2.1 and push to GitHub

  $ git tag 2020C2.1 -m "Before 2021C1.0 deployment"

  Note: tagged https://github.com/NSLS-II-FMX/profile_collection/releases/tag/2020C2.1 with the changes from Deployment 2020-2 #8 (that code in the PR branch was used during the previous cycle and was in fact their 'master'). The follow-up updates will be in a separate PR.
• Discuss with BL staff which conda envs they want to keep/delete and perform the cleanup
• Check/update the beamline's workstation inventory with the BL staff (in https://github.com/NSLS-II/playbooks/blob/master/production)
• Ask beamline staff about the beamline unix group they want to use for bluesky logs. The members of the group can be checked with the following command:

  $ getent group <beamline-group>

  Add this group name to the collection vars in the inventory file (see this example).
• In ~/.bashrc, if necessary, update the logging environment variables to use directory /var/log/bluesky/... and add umask 0002.

  umask u=rwx,g=rwx,o=rx  # 0002
  export BLUESKY_LOG_FILE=/var/log/bluesky/bluesky.log
  export BLUESKY_IPYTHON_LOG_FILE=/var/log/bluesky/bluesky_ipython.log

  and (if needed) create that directory with the following permissions and ownership:

  $ sudo mkdir /var/log/bluesky
  $ sudo chown -Rv xf17id1:<beamline-group> /var/log/bluesky
  $ sudo chmod -Rv g+rws /var/log/bluesky

  Remember to source ~/.bashrc.
• Remove explicit setting of ophyd logging level from the first startup script
• Double-check that if there is an open PR removing handlers, it is merged (and tested).
• Update vendored copy of PersistentDict to bug-fixed version (see the updated gist snippet: https://gist.github.com/jklynch/a4366b8900ec0c03883403455ae711b2).
• Make sure all repos have the BSD-3-Clause licenses (see Add a BSD-3-Clause license NSLS-II-SRX/profile_collection#16 as an example)

Test deployment

• Run profile against new environment

  $ BS_ENV=collection-2021-1.0 bsui

  It may be necessary to disable jedi in ipython. Add this to the ipython command line arguments inbsui:

  --IPCompleter.use_jedi=False

• Update the profile as needed to run
• Test databroker v2

  from databroker import catalog
  cat = catalog['fmx']
  cat[-1].primary.read()

• Create a Jupyter kernel for the new conda environment in xf17id1-ca1:/usr/share/jupyter/kernels:

  $ cd /usr/share/jupyter/kernels-available/
  $ sudo cp -r collection-2020-2.0rc8 collection-2021-1.0
  # edit the kernel file to have the `collection-2021-1.0` conda environment in the path and update the description
  $ vi collection-2021-1.0/kernel.json 
  $ cd /usr/share/jupyter/kernels
  $ sudo ln -s ../kernels-available/collection-2021-1.0 .

• Check the new kernel is available:

  $ jupyter kernelspec list

• Update /epics/iocs/notebook/notebooks/services/st.cmd as needed (update the conda activate ... command with the new conda environment)
• Check the status of the Jupyter server (restart if anything was changed in the st.cmd file):

  $ sudo systemctl status softioc-notebook
  $ sudo systemctl restart softioc-notebook  # optional

• Commit and push the latest versions of st.cmd and kernel.json to:
  • https://github.com/NSLS-II-FMX/services
  • https://github.com/NSLS-II-FMX/kernels
• Run acceptance tests (if any) and run a representative set of scans for the beamline against the new kernel in a Jupyter notebook.

Finish

• Tag the updated profile as 2021C1.0

  $ git tag 2021C1.0 -m "After 2021C1.0 deployment"

• Open and merge a PR against https://github.com/NSLS-II/playbooks setting current_env_tag="2021-1.0" for the FMX beamline
• Update bsui via ansible to use the new environment (http://nsls-ii.github.io/deployment/upgrading-beamlines.html#change-bsui-s-default-environment)

[mrakitin]

...@xf17id1-ca1:~$ cat /usr/share/jupyter/kernels/collection-2021-1.0/kernel.json
{
  "argv": [
    "/opt/conda_envs/collection-2021-1.0/bin/python",
    "-m",
    "ipykernel",
    "-f",
    "{connection_file}",
    "--profile=collection",
    "--log-level=DEBUG"
  ],
  "display_name": "Data Collection (2021-1.0)"
}

[mrakitin]

Syncs:

• NSLS-II-FMX/services@899a553
• NSLS-II-FMX/kernels@20092ad

[mrakitin]

@mfuchsbnl, @danielballan, I do not know what was going on at the beamline, but I was able to run the scan with the latest kernel with the collection-2021-1.0 conda environment:

Please note, I created a separate kernel so that it does not interfere with the production kernels, and used a PR branch named deploy-2021-1.0 from #12 for my tests.

Is it something to do with the browser on the beamline? Can you try Google Chrome or a newer version of Firefox?

[mrakitin]

(/opt/conda_envs/collection-2021-1.0) mrakitin@xf17id1-ca1:~$ jupyter kernelspec list
Available kernels:
  python3                         /opt/conda_envs/collection-2021-1.0/share/jupyter/kernels/python3
  collection-17q2.0               /usr/share/jupyter/kernels/collection-17Q2.0
  collection-17q3.0               /usr/share/jupyter/kernels/collection-17Q3.0
  collection-2019-2.0             /usr/share/jupyter/kernels/collection-2019-2.0
  collection-2019-3.0             /usr/share/jupyter/kernels/collection-2019-3.0
  collection-2020-2.0rc8          /usr/share/jupyter/kernels/collection-2020-2.0rc8
  collection-2021-1.0             /usr/share/jupyter/kernels/collection-2021-1.0
  collection-2021-1.0-mrakitin    /usr/share/jupyter/kernels/collection-2021-1.0-mrakitin

[mrakitin]

That was successful with collection-2020-2.0rc8 too: