Chaos test the cluster

[stan-dot]

explore cluster resilience testing

options considered:

• https://www.gremlin.com/kubernetes
• https://litmuschaos.io
• https://github.com/chaostoolkit/chaostoolkit
• https://chaos-mesh.org/docs/production-installation-using-helm/

[DiamondJoseph]

Whatever decisions and learning come out of this are going to be applicable to the other beamline repositories

[stan-dot]

note: this tool is more suited to aws and isn't that well-developed
https://chaostoolkit.org/reference/tutorials/containerising/

[stan-dot]

potential alternatives

• https://chaos-mesh.org/docs/production-installation-using-helm/

[stan-dot]

raised a chaos-mesh ticket with the cloud team about the namespace creation

[stan-dot]

moved to looking into a more restricted tool - https://github.com/asobti/kube-monkey

[stan-dot]

with a helm chart https://github.com/asobti/kube-monkey/blob/master/helm/kubemonkey/README.md

[stan-dot]

working on it

[stan-dot]

tracking the cloud aspect here https://jira.diamond.ac.uk/servicedesk/customer/portal/2/SCHD-6072

[stan-dot]

might use litmus https://github.com/litmuschaos/litmus

[stan-dot]

also this relevant https://confluence.diamond.ac.uk/display/SSCC/Graduate+project+idea%3A+Chaos+Engineering+for+the+Workflow+Engine

[stan-dot]

Workflow Controller: The Argo Workflow Controller responsible for the creation of Chaos Experiments using the Chaos Experiment CR.

and argo isn't quite ready yet? might need to delay this
https://docs.litmuschaos.io/docs/architecture/chaos-execution-plane

[stan-dot]

• might need to fork chaos hub to deploy own experiments
• might need to add DNS and the ingress

https://docs.litmuschaos.io/docs/concepts/chaoshub

[stan-dot]

this is not on the critical path, deprioritizing