Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(deployment): Use a different key for sealed secrets data #2447

Merged
merged 1 commit into from
Aug 22, 2024

Conversation

theosanderson
Copy link
Member

@theosanderson theosanderson commented Aug 17, 2024

I just spent a loooong time debugging a confusing issue on Pathoplexus. It manifested as the database sealed secret that couldn't be extracted as it appeared to be corrupt. It was very confusing because I could see the sealed secrets I had configured in values yaml and they hadn't changed. It turned out that what had happened was that we added a new key host to the database secret in the default values.yaml recently. That default values yaml doesn't use sealed secrets at all, it defines them in a raw format. But because that value hadn't been added to pathoplexus's values.yaml in any form, and because pathoplexus values.yaml redefines the database secret as a sealed secret, what ended up happening was that the sealed secrets controller attempted to extract the host key, merged in from the default values yaml, even though that had never been sealed. This was so confusing to debug that I want to avoid it ever happening again, so now we will store sealed secrets under encryptedData not data so that you cannot get merging of a sealed and unsealed secret.

https://different-key-sealed-secr.loculus.org/

@theosanderson theosanderson added the preview Triggers a deployment to argocd label Aug 17, 2024
@corneliusroemer corneliusroemer removed the preview Triggers a deployment to argocd label Aug 21, 2024
@theosanderson theosanderson merged commit f5fea09 into main Aug 22, 2024
11 checks passed
@theosanderson theosanderson deleted the different-key-sealed-secret-data branch August 22, 2024 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants