deploy: fadb811

4.1.0b1/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file records the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 49cf32c131dbf95a72edbf85cac039e2
+tags: 645f666f9bcd5a90fca523b33c5a78b7

4.1.0b1/_sources/explanations.md.txt

@@ -0,0 +1,10 @@
+# Explanations
+
+Explanations of how it works and why it works that way.
+
+```{toctree}
+:maxdepth: 1
+:glob:
+
+explanations/*
+```

4.1.0b1/_sources/explanations/changes.md.txt

@@ -0,0 +1,88 @@
+Change Management Manifesto
+===========================
+
+Pledge
+------
+
+This page represents a pledge to control breaking changes for users of `epics-containers`. The framework is still under development and there still may be breaking changes in future updates. However we now have a mechanism in place to allow users to adopt the framework, take advantage of the current features and then accept future updates in a controlled fashion.
+
+From version 3.4.0 onwards we will make changes in a controlled fashion that obeys SemVer 2.0.0 rules. We will also ensure that it is possible to apply updates in a gradual manner and not require a blanket update.
+
+The [](../reference/changelog.md) will give details of any things to be aware of between versions, including minor version updates.
+
+Dependency Matrix
+-----------------
+
+Users of the framework will develop two kinds of repository:
+
+|     |     |
+| --- | --- |
+| Beamline repo | A beamline or other grouping of IOC instance descriptions |
+| Generic IOC repo | A definition of a generic IOC container image for a particular class of device |
+
+Both of these types of repository are initially created using a copier template. The copier template version will have a SemVer version number that determines which component versions it is compatible with.
+
+For a discussion on how to update your projects to the latest version of the templates see [](../how-to/copier_update).
+
+Any breaking changes to the framework will be made in a new Major version of the framework and hence a new Major version of the copier templates.
+
+The following diagram shows the set of components that are involved in the framework and the relationships between them. The dependencies between these components are not a strict API and the diagram attempts to highlight the features of each component that affect other components.
+
+The diagram may be used by developers of the framework to plan how changes will be made and to ensure that breaking changes are made in a controlled manner.
+
+Users of the framework are only concerned with the top two boxes and these are always updated on an as-needed basis via the copier templates.
+
+:::{figure} ../images/dependency_matrix.png
+`epics-containers` dependency matrix
+:::
+
+All `ec` SemVer components will always have their major version bumped simultaneously. Likewise for `ibek` SemVer components. These are at versions 3.4.0 and 2.0.0 respectively at the time of writing.
+
+
+Updating user projects
+----------------------
+
+A repository that was originally created using a copier template can be updated to a new version using the following command (assumes you have an active python venv with copier installed):
+
+```bash
+copier update -r VERSION_NUMBER --trust .
+```
+
+You can supply the VERSION_NUMBER of the template you want or omit the -r option to get the latest released version.
+
+This will update your project in place. You should then inspect the changes using git (the source control pane in vscode is excellent for this purpose) and commit them to your repository.
+
+When a beamline repository is updated, it is still possible to deploy old versions of its IOC instances, even with a major version difference. That is because the deploy mechanism makes a temporary clone of the beamline repository and deploys the instance described in that version.
+
+User Project Versioning
+-----------------------
+
+The documentation has recommended using DateVer for beamline repos and generic IOC repos. This is because SemVer is not really applicable to these. However, DateVer is not required and you are free to use any scheme you wish for these repositories.
+
+It is easy to determine which template version and thus which `ec` SemVer version your repository was last updated from. Inspect the file `.copier_answers.yml` in the root of your repository. This file contains the version of the template that was used to create the repository in the field `_commit`.
+
+
+Types of Changes
+----------------
+
+Changes to the framework are likely to be initiated in one of the places described under the following headings. As far as possible such changes will be backwards compatible going forward, and if they are not then a major version release will be made.
+
+### ibek
+
+
+Changes to the CLI commands inside of the container build/runtime are initiated in the `ibek` python module that lives inside every generic IOC. This can affect the support module build recipes in `ibek-support` and potentially the Dockerfile in Generic IOC projects.
+
+### ec-helm-charts
+
+Changes here affect how IOCs and other services are deployed into Kubernetes. These would likely affect beamline repositories as they contain the versions of Helm Charts used to deploy their instances. Potentially changes to these Charts may require an update to the edge-services-cli to support new features.
+
+### ioc-template
+
+The Generic IOC template is well established and stable. However, each time a new target architecture is added, this will need updates to the CI. We will be supporting Windows and ARM targets in future. These changes should certainly be backwards compatible and not affect existing projects.
+
+
+### ibek-support
+
+`ibek-support` is a unique project in that it is a submodule of all Generic IOCs. It is expected that there will be constant change to this module as new support modules are added. However, such changes will almost entirely be adding new folders and not affect existing generic IOCs. We encourage users to fork this repository, add their own support modules and submit PRs back to the original so that a wide range of support modules can be shared (a branch rather than fork is preferred for internal developers).
+
+If there is a need to change the CLI that ibek-support uses, then a new version of `ibek` will be released. Only generic IOCs that have been updated to pick up the new version of `ibek` would be able to use these changes. Because older generic IOCs will retain the old commit of the 'ibek-support' submodule, they will not be affected by the changes until they are updated.

4.1.0b1/_sources/explanations/decisions.md.txt

@@ -0,0 +1,12 @@
+# Architectural Decision Records
+
+Architectural decisions are made throughout a project's lifetime. As a way of keeping track of these decisions, we record these decisions in Architecture Decision Records (ADRs) listed below.
+
+```{toctree}
+:glob: true
+:maxdepth: 1
+
+decisions/*
+```
+
+For more information on ADRs see this [blog by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions).

4.1.0b1/_sources/explanations/decisions/0001-record-architecture-decisions.md.txt

@@ -0,0 +1,18 @@
+# 1. Record architecture decisions
+
+## Status
+
+Accepted
+
+## Context
+
+We need to record the architectural decisions made on this project.
+
+## Decision
+
+We will use Architecture Decision Records, as [described by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions).
+
+## Consequences
+
+See Michael Nygard's article, linked above. To create new ADRs we will copy and
+paste from existing ones.

4.1.0b1/_sources/explanations/decisions/0002-switched-to-python-copier-template.md.txt

@@ -0,0 +1,28 @@
+# 2. Adopt python-copier-template for project structure
+
+## Status
+
+Accepted
+
+## Context
+
+We should use the following [python-copier-template](https://github.com/DiamondLightSource/python-copier-template).
+The template will ensure consistency in developer
+environments and package management.
+
+## Decision
+
+We have switched to using the template.
+
+## Consequences
+
+This module will use a fixed set of tools as developed in `python-copier-template`
+and can pull from this template to update the packaging to the latest techniques.
+
+As such, the developer environment may have changed, the following could be
+different:
+
+- linting
+- formatting
+- pip venv setup
+- CI/CD

4.1.0b1/_sources/explanations/decisions/0003-use-substitution-files.md.txt

@@ -0,0 +1,48 @@
+# 3. Use of substitution files to generate EPICS Databases
+
+Date: 2023-11-30
+
+## Status
+
+Accepted
+
+## Context
+
+There are two proposals for how EPICS Databases should be generated:
+
+1. At IOC startup `ibek` should generate a substitution file that describes the
+   required Databases.
+
+   The IOC instance yaml combined with the definitions from support module yaml
+   controls what the generated substitution file will look like.
+
+   `ibek` will then execute `msi` to generate the Databases from the
+   substitution file.
+
+2. The dbLoadRecord calls in the startup script will pass all macro substitutions
+   in-line. Removing the need for a substitution file.
+
+## Decision
+
+Proposal 1 is accepted.
+
+Some template files such as those in the `pmac` support module use the
+following pattern:
+
+```
+substitute "P=$(PMAC):, M=CS$(CS):M1, ADDR=1, DESC=CS Motor A"
+include "pmacDirectMotor.template"
+```
+
+This pattern is supported by msi but not by the EPICS dbLoadRecord command which
+does not recognise the `substitute` command.
+
+## Consequences
+
+An extra file `ioc.subst` is seen in the runtime directory. In reality this
+is easier to read than a full Database file. So can be useful for debugging.
+
+Finally those developers who are unable to use `ibek yaml` for some reason can
+supply their own substitution file and ibek will expand it at runtime. This is
+much more compact that supplying a full Database file and important due to the
+1MB limit on K8S ConfigMaps.

4.1.0b1/_sources/explanations/decisions/0004-autosave-req-files.md.txt

@@ -0,0 +1,39 @@
+# 4. How to configure autosave for IOCs
+
+Date: 2023-11-30
+
+## Status
+
+Accepted
+
+## Context
+
+There is a choice of supplying the list of PVs to autosave by:
+
+- adding info tags to the Database Templates
+- supplying a raw req file with list of PVs to autosave
+
+## Decision
+
+We will go with req files for the following reasons:
+
+- <https://epics.anl.gov/tech-talk/2019/msg01600.php>
+- adding info tags would require upstream changes to most support modules
+- default req files are already supplied in many support modules
+- req files are in common use and many facilities may already have their own
+  req files for support modules.
+
+We expect to autogenerate the list of PVs to autosave from the IOC's. We could
+therefore generate a Database override file which adds info tags. But it is
+simpler to just generate a req file.
+
+The mechanism for using req files is that defaults will come from the support
+module or from the generic IOC if the support module does not supply a req file.
+
+Then override files can exist at the beamline level and / or at the IOC
+instance level. These will simply take the form of a req file with the same
+name as the one it is overriding.
+
+## Consequences
+
+Everything is nice and simple.

4.1.0b1/_sources/explanations/decisions/0005-python-scripting.md.txt

@@ -0,0 +1,39 @@
+# 5. Use Python for scripting inside and outside containers
+
+Date: 2022-11-30
+
+## Status
+
+Accepted
+
+## Context
+
+Inside the container, we use the `ibek` tool for scripting. Outside we
+use `ec` from `epics-containers-cli`.
+
+Much of what these tools do is
+call command line tools like `docker`, `helm`, `kubectl`, compilers,
+etc. This seems like a natural fit for bash scripts.
+
+These features were originally implemented in bash but were converted to
+python for the following reasons:
+
+- python provides for much richer command line arguments
+- it is also much easier to provide rich help
+- managing errors is vastly improved with exception handling
+- the unit testing framework allows good test coverage in continuous integration
+- complex string handling is a common requirement and is much easier in python
+- there is a clear versioning strategy and packages can be installed with pip,
+  meaning that you can check which version of the script you are running and
+  report bugs against a specific version
+- the code is much easier to read and maintain
+- because the packages can be pip installed they can be used in CI and inside
+  multiple containers without having to copy the scripts around
+
+## Decision
+
+We always prefer Python and keep bash scripts to a minimum
+
+## Consequences
+
+Scripting is much easier to maintain and is more reliable.

4.1.0b1/_sources/explanations/docs-structure.md.txt

@@ -0,0 +1,17 @@
+# About the documentation
+
+> {material-regular}`format_quote;2em`
+>
+> The Grand Unified Theory of Documentation
+>
+> <p class="attribution">-David Laing</p>
+
+There is a secret that needs to be understood in order to write good software
+documentation: there isn't one thing called *documentation*, there are four.
+
+They are: *tutorials*, *how-to guides*, *technical reference* and *explanation*.
+They represent four different purposes or functions, and require four different
+approaches to their creation. Understanding the implications of this will help
+improve most documentation - often immensely.
+
+[More information on this topic.](https://documentation.divio.com)

4.1.0b1/_sources/explanations/epics_protocols.md.txt

@@ -0,0 +1,66 @@
+# EPICS Network Protocols in Containers
+
+When EPICS IOCs run in containers, Channel Access or PVAcess protocols must be made available to clients. There are some challenges around this that are discussed in this page.
+
+
+## Approaches to Network Protocols
+
+To get clients and servers connected we can use 3 approaches:
+
+1. Run IOC containers in **Host Network**:
+    - This is the approach that DLS has adopted for IOCs running in Kubernetes.
+    - The container uses the host network stack.
+    - This looks identical to running the IOC on the host machine as far as clients are concerned.
+    - See a discussion of the reasoning here: [](./net_protocols.md)
+    - This reduces the isolation of the container from the host so additional security measures may be needed.
+2. Use **Port Mapping**:
+    - This approach is used in the developer containers defined by [ioc-template](https://github.com/epics-containers/example-services)
+    - The container runs in a container network.
+    - The necessary ports are mapped from the host network to the container network.
+    - VSCode can do this port mapping automatically when it detects processes binding to ports.
+    - This approach is good for local development and running tutorials as the mapping can be made to localhost only and PVs can be isolated to the developer's machine.
+3. Run the clients in the **same container network** as the IOCs:
+    - This approach is used in [example-services](https://github.com/epics-containers/example-services).
+    - **example-services** runs a PVA and a CA gateway in the same container network as the IOCs.
+      - The gateways use Port Mapping to give access to their own clients.
+      - The gateways can use any ports and UDP broadcast to communicate with the IOCs.
+    - If your client is a GUI app, like phoebus, then this may not work as it can be difficult to do X11 forwarding into a rootless container network.
+
+## General Observations
+
+Using Host Network or the same container network for client and host is compatible with both PVA and CA protocols.
+
+For podman and docker networks this is true even for UDP broadcast.
+
+For the majority of Kubernetes CNI's the broadcast does not work across pods. It is quite possible that broadcast within pods would work as this is equivalent to 'same container network'. However this would make management of large numbers of IOCs far more of a manual task.
+
+
+## Channel Access
+
+Specification <https://docs.epics-controls.org/en/latest/internal/ca_protocol.html>.
+
+Experiments with Channel Access servers running in containers reveal:
+- Port Mapping works for CA including UDP broadcast.
+- But UDP broadcast only works if the container does not remap the port to a different number inside the container.
+- Using CA Name Server works for Port Mapping
+
+
+## PV Access
+
+Specification <https://docs.epics-controls.org/en/latest/pv-access/Protocol-Messages.html>.
+
+Experimentation with PV Access servers running in containers reveal:
+- Port Mapping works for PVA always fails because PVA servers open a new random port for each circuit and this is not NAT friendly.
+- Using EPICS_PVA_NAME_SERVERS works for Port Mapping.
+- But the client and server must both be PVXS
+- To talk to a non PVXS server, a pvagw running in the same container network may be used.
+
+## Code
+
+The following bash scripts can be run to test the assertions made above:
+
+```{literalinclude} ../demo/channel_access_tests.sh
+```
+
+```{literalinclude} ../demo/pv_access_tests.sh
+```