Merge pull request #118 from epics-containers/april-update

April update

.devcontainer/devcontainer.json

@@ -9,6 +9,7 @@
         // Allow X11 apps to run inside the container
         "DISPLAY": "${localEnv:DISPLAY}"
     },
+    "remoteUser": "${localEnv:EC_REMOTE_USER}",
     "customizations": {
         "vscode": {
             // Set *default* container specific settings.json values on container create.
@@ -33,14 +34,16 @@
             "upgradePackages": false
         }
     },
+    // You can place any outside of the container before-launch commands here
+    "initializeCommand": "bash .devcontainer/initializeCommand ${devcontainerId}",
+    // One time global setup commands inside the container
+    "postCreateCommand": "bash .devcontainer/postCreateCommand ${devcontainerId}",
     "runArgs": [
         // Allow the container to access the host X11 display and EPICS CA
         "--net=host",
         // Make sure SELinux does not disable with access to host filesystems like tmp
         "--security-opt=label=disable"
     ],
     // Mount the parent as /workspaces so we can pip install peers as editable
-    "workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind",
-    // After the container is created, install the python project in editable form
-    "postCreateCommand": "pip install $([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e '.[dev]' && pre-commit install"
+    "workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind"
 }

.devcontainer/initializeCommand

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+# custom initialization goes here - runs outside of the dev container
+# just before the container is launched but after the container is created
+
+echo "devcontainerID ${1}"

.devcontainer/postCreateCommand

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+# Custom initialization goes here if needed.
+# Runs inside the dev container after the container is created
+
+################################################################################
+# When using docker we will not be root inside the container
+# the following steps are then required
+################################################################################
+
+if [[ $USER != "root" ]] ; then
+    # make sure the non-root user can access the virtual environment
+    sudo chown -R ${USER}:${USER} /venv
+fi
+
+################################################################################
+# Shell customizations
+################################################################################
+
+# pick a theme that does not cause completion corruption in zsh
+sed -i $HOME/.zshrc -e 's/ZSH_THEME="devcontainers"/ZSH_THEME="lukerandall"/'
+
+# allow personalization of all devcontainers in this subdirectory
+# by placing a .devcontainer_rc file in the workspace root
+if [[ -f /workspaces/.devcontainer_rc ]] ; then
+    source /workspaces/.devcontainer_rc
+fi
+
+#After the container is created, install the python project in editable form
+pip install $([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e '.[dev]'
+pre-commit install

docs/conf.py

@@ -177,5 +177,5 @@
 html_show_copyright = False
 
 # Logo
-html_logo = "images/dls-logo.svg"
+html_logo = "images/k8s-epics2.ico"
 html_favicon = html_logo

docs/explanations/changes.md

@@ -0,0 +1,88 @@
+Change Management Manifesto
+===========================
+
+Pledge
+------
+
+This page represents a pledge to control breaking changes for users of `epics-containers`. The framework is still under development and there still may be breaking changes in future updates. However we now have a mechanism in place to allow users to adopt the framework, take advantage of the current features and then accept future updates in a controlled fashion.
+
+From version 3.4.0 onwards we will endeavour to make changes in a controlled fashion that obeys SemVer 2.0.0 rules. We will also try to ensure that it is possible to apply updates in a gradual manner and not require a blanket update.
+
+The [](../reference/changelog.md) will give details of any things to be aware of between versions, including minor version updates.
+
+Dependency Matrix
+-----------------
+
+Users of the framework will develop two kinds of repository:
+
+|     |     |
+| --- | --- |
+| Beamline repo | A beamline or other grouping of IOC instance descriptions |
+| Generic IOC repo | A definition of a generic IOC container image for a particular class of device |
+
+Both of these types of repository are initially created using a copier template. The copier template version will have a SemVer version number that determines which component versions it is compatible with.
+
+For a discussion on how to update your projects to the latest version of the templates see [](../how-to/copier_update).
+
+Any breaking changes to the framework will be made in a new Major version of the framework and hence a new Major version of the copier templates.
+
+The following diagram shows the set of components that are involved in the framework and the relationships between them. The dependencies between these components are not a strict API and the diagram attempts to highlight the features of each component that affect other components.
+
+The diagram may be used by developers of the framework to plan how changes will be made and to ensure that breaking changes are made in a controlled manner.
+
+Users of the framework are only concerned with the top two boxes and these are always updated on an as-needed basis via the copier templates.
+
+:::{figure} ../images/dependency_matrix.png
+`epics-containers` dependency matrix
+:::
+
+All `ec` SemVer components will always have their major version bumped simultaneously. Likewise for `ibek` SemVer components. These are at versions 3.4.0 and 2.0.0 respectively at the time of writing.
+
+
+Updating user projects
+----------------------
+
+A repository that was originally created using a copier template can be updated to a new version using the following command (assumes you have an active python venv with copier installed):
+
+```bash
+copier update -r VERSION_NUMBER --trust .
+```
+
+You can supply the VERSION_NUMBER of the template you want or omit the -r option to get the latest released version.
+
+This will update your project in place. You should then inspect the changes using git (the source control pane in vscode is excellent for this purpose) and commit them to your repository.
+
+When a beamline repository is updated, it is still possible to deploy old versions of its IOC instances, even with a major version difference. That is because the deploy mechanism makes a temporary clone of the beamline repository and deploys the instance described in that version.
+
+User Project Versioning
+-----------------------
+
+The documentation has recommended using DateVer for beamline repos and generic IOC repos. This is because SemVer is not really applicable to these. However, DateVer is not required and you are free to use any scheme you wish for these repositories.
+
+It is easy to determine which template version and thus which `ec` SemVer version your repository was last updated from. Inspect the file `.copier_answers.yml` in the root of your repository. This file contains the version of the template that was used to create the repository in the field `_commit`.
+
+
+Types of Changes
+----------------
+
+Changes to the framework are likely to be initiated in one the places described under the following headings. As far as possible such changes will be backwards compatible going forward, and if they are not then a major version release will be made.
+
+### ibek
+
+
+Changes to the CLI commands inside of the container build/runtime are initiated in the `ibek` python module that lives inside every generic IOC. This can affect the support module build recipes in `ibek-support` and potentially the Dockerfile in Generic IOC projects.
+
+### ec-helm-charts
+
+Changes here affect how IOCs and other services are deployed into Kubernetes. These would likely affect beamline repositories as they contain the versions of Helm Charts used to deploy their instances. Potentially changes to these Charts may require an update to the edge-services-cli to support new features.
+
+### ioc-template
+
+The Generic IOC template is well established and stable. However, each time a new target architecture is added, this will need updates to the CI. We will be supporting Windows and ARM targets in future. These changes should certainly be backwards compatible and not affect existing projects.
+
+
+### ibek-support
+
+`ibek-support` is a unique project in that it is a submodule of all Generic IOCs. It is expected that there will be constant change to this module as new support modules are added. However, such changes will almost entirely be adding new folders and not affect existing generic IOCs. We encourage users to fork this repository, add their own support modules and submit PRs back to the original so that a wide range of support modules can be shared (a branch rather than fork is preferred for internal developers).
+
+If there is a need to change the CLI that ibek-support uses, then a new version of `ibek` will be released. Only generic IOCs that have been updated to pick up the new version of `ibek` would be able to use these changes. Because older generic IOCs will retain the old commit of the 'ibek-support' submodule, they will not be affected by the changes until they are updated.

docs/explanations/introduction.md

@@ -39,7 +39,7 @@ forks of these repositories.
 
 #### Generic IOCs and instances
 
-An important principal of the approach presented here is that an IOC container image represents a 'Generic' IOC. The Generic IOC image is used for all IOC instances that connect to a given class of device. For example the Generic IOC image here: [ghcr.io/epics-containers/ioc-adaravis-linux-runtime:2024.2.2 ](https://github.com/epics-containers/ioc-adaravis/pkgs/container/ioc-adaravis-linux-runtime) uses the AreaDetector driver ADAravis to connect to GigE cameras.
+An important principal of the approach presented here is that an IOC container image represents a 'Generic' IOC. The Generic IOC image is used for all IOC instances that connect to a given class of device. For example the Generic IOC image here: [ghcr.io/epics-containers/ioc-adaravisruntime:2024.2.2 ](https://github.com/epics-containers/ioc-adaravis/pkgs/container/ioc-adaravisruntime) uses the AreaDetector driver ADAravis to connect to GigE cameras.
 
 An IOC instance runs in a container runtime by loading two things:
 

docs/how-to/builder2ibek.md

@@ -6,7 +6,7 @@ builder beamline to epics-containers. i.e. those whose beamlines
 have a BLxxY-BUILDER project.
 :::
 
-TODO: this page is WIP and will be updated by Feb 2024.
+TODO: this page is WIP and will be updated by May 2024.
 
 `builder2ibek` is a tool to convert DLS builder XML to ibek instance YAML.
 It is for working with converting IOC instances to epics-containers.

docs/how-to/builder2ibek.support.md

@@ -6,7 +6,7 @@ a DLS support module with builder support into an epics-containers
 Generic IOC. i.e. support modules that have an `etc/builder.py` file.
 :::
 
-TODO: this page is WIP and will be updated by Feb 2024.
+TODO: this page is WIP and will be updated by May 2024.
 
 `builder2ibek.support` is a tool to convert DLS builder support modules
 into ibek support YAML for the `ibek-support` repository.

docs/how-to/copier_update.md

@@ -0,0 +1,96 @@
+Updating User Repositories with Copier
+======================================
+
+In the tutorials we created an example beamline repo and also an example generic IOC. In both cases we used the copier tool to create these projects from a template.
+
+This tool can also be used to merge in the latest changes to the framework into your existing projects. This is done by running the `copier update` command in the root of your project.
+
+Below are some details on how to use this tool.
+
+Introduction
+------------
+
+copier is a python package. To make use of it you require an activated python virtual environment with copier installed. If you don't already have one of these then the following commands will set one up for you:
+
+```bash
+python3 -m venv venv
+source venv/bin/activate
+pip install copier
+```
+
+NOTE: generic IOCs with a given major version number template should work with beamline repos with the same major version number template. When updating a beamline repo to a new major version, you may also need to update the generic IOCs it references. If this is the case it will be noted in [](../reference/changelog.md). Having said this, the two types of repo are reasonably well decoupled and we would aim to avoid this necessity. In which case update the beamline repo first and then the generic IOCs as and when new features are required.
+
+Updating a Beamline Repository
+------------------------------
+
+A beamline (or other grouping) repository is a collection of IOC instances and services that are deployed together. The beamline repository is created using the `ec-services-template` copier template. See [](../tutorials/create_beamline.md) for the tutorial on how to create a new beamline repository.
+
+To update your beamline repository to the latest version of the templates you should run the following command in the root of your repository:
+
+```bash
+copier update -r VERSION_NUMBER --trust .
+```
+
+You can supply the VERSION_NUMBER of the template you want or omit the `-r` option to get the latest released version.
+
+This will update your project in place. You should then inspect the changes using git (the source control pane in vscode is excellent for this purpose) and commit them to your repository. Once you are happy with the changes you can test them by re-deploying some of your IOC instances. When everything is working you can push the changes to your repository.
+
+The template comes with an example IOC called xxxx-ea-test-01. You are free to delete this if you don't want it. However, we recommend keeping that IOC as it is a good reference for what changes might be needed in your own IOC instances.
+
+For example, in version 3.4.0 the `ec-services-template` changed the way that the configmap is created for each IOC instance. This added a soft-link **templates** folder that points at **../../include/ioc/templates**. Looking at what changes happened in the example IOC will help you to understand what changes you might need to make in your own IOC instances. Copier migrations will attempt to make these changes for you but it is recommended to check that they have been done correctly.
+
+
+Updating a Generic IOC Repository
+---------------------------------
+
+To see details of how to initially create a generic IOC repository see {any}`create_generic_ioc`.
+
+To update your generic IOC repository to the latest version of the templates you should run the following command in the root of your repository:
+
+```bash
+copier update -r VERSION_NUMBER --trust .
+```
+
+Typically the only file that the user will have changed is the Dockerfile and typically merges will work well. It is still a good idea to validate the changes to the repo before committing them.
+
+When Update fails
+-----------------
+
+Copier cannot do the 'update' command if any of the following is true:
+
+- Your repo pre-dates the use of copier
+- The copier template version that you last used is no longer available.
+
+In this case you can still do an update using the 'copy' command - you just need to be more careful with the merge.
+
+For the generic IOC case:
+
+```bash
+copier copy gh:epics-containers/ioc-template --trust .
+```
+
+For the beamline case:
+
+```bash
+copier copy gh:epics-containers/ec-services-template --trust .
+```
+
+In both cases you should select Y for each notice of a conflict, then resolve those conflicts in your editor.
+
+Using copy will mean that you get asked the template questions again, but copier is smart enough to supply your previous answers as defaults (if your repo predates copier then you will need to answer all the questions for the first time).
+
+The difference from 'update' is that you will need to do the merge yourself. For example in a generic IOC you will want to restore your support module instantiations in the middle of the Dockerfile, whereas the top and bottom of the Dockerfile should be updated by the template. Using vscode's SOURCE CONTROL pane is useful for this. The following image is an example of a Dockerfile merge:
+
+:::{figure} ../images/dockerfile_merge.png
+merging a Dockerfile after a copy copy
+:::
+
+In the above example the **copier copy** has made many changes to the repo. Because the only user supplied change is the list of support modules in the middle of the Dockerfile, the merge is simple. Using vscode you just need to click button indicated by the red arrow, save your changes, then:
+
+```bash
+# from the root of the repo
+git add .
+git commit -m'update to template version 3.4.0'
+git push
+```
+

docs/how-to/debug.md

@@ -41,7 +41,7 @@ Are you sure ? [y/N]: y
 <font color="#5F8787">docker container create --name busybox -v bl01t-ea-test-02_config:/copyto busybox</font>
 <font color="#5F8787">docker cp /home/giles/tutorial/bl01t/services/bl01t-ea-test-02/config/ioc.yaml busybox:copyto</font>
 <font color="#5F8787">docker rm -f busybox</font>
-<font color="#5F8787">docker run -dit --net host --restart unless-stopped -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02  --name bl01t-ea-test-02 ghcr.io/epics-containers/ioc-adsimdetector-linux-runtime:2024.2.2</font>
+<font color="#5F8787">docker run -dit --net host --restart unless-stopped -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02  --name bl01t-ea-test-02 ghcr.io/epics-containers/ioc-adsimdetectorruntime:2024.2.2</font>
 76c2834dac805780b3329af91c332abb90fb2692a510c11b888b82e48f60b44f
 <font color="#5F8787">docker ps -f name=bl01t-ea-test-02 --format &apos;{{.Names}}&apos;</font>
 </pre>
@@ -59,7 +59,7 @@ Now you can tell `ec` to stop the IOC instance and then run it in a way that you
 
 ```bash
 ec stop bl01t-ea-test-02
-docker run --entrypoint bash -it --net host -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02  --name bl01t-ea-test-02-debug ghcr.io/epics-containers/ioc-adsimdetector-linux-runtime:2024.2.2
+docker run --entrypoint bash -it --net host -l is_IOC=true -l version=2024.2.17-b8.30 -v bl01t-ea-test-02_config:/epics/ioc/config/ -e IOC_NAME=bl01t-ea-test-02  --name bl01t-ea-test-02-debug ghcr.io/epics-containers/ioc-adsimdetectorruntime:2024.2.2
 ```
 
 You should now be in a shell inside the container. You can look at the files and run the IOC instance manually to see what the error is. You can re-run the IOC instance multiple times and you can even install your favourite editor or debugging tools.

docs/how-to/own_tools.md

@@ -17,5 +17,6 @@ epics-containers has been tested with
 - vscode
 - Github Codespaces
 
-TODO: add instructions for using other editors. Potentially we could enhance
-the epics-containers-cli to support other editors with minimal effort.
+If you prefer console based editors like vim or emacs, then you will get the best results by launching the development containers defined in the epics-containers using the devcontainer CLI as described here <https://containers.dev/supporting#devcontainer-cli>.
+
+In addition you could install your editor inside the developer container by adding an apt-install command into the `epics-containers` user personalization file. See [details here](https://github.com/epics-containers/epics-containers.github.io/blob/3a87e808e1c7983430a30c5a4dcd0d0661895d60/.devcontainer/postCreateCommand#L23-L27)