Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RequirementError during deploy job in prod #3973

Closed
achave11-ucsc opened this issue Mar 22, 2022 · 1 comment
Closed

RequirementError during deploy job in prod #3973

achave11-ucsc opened this issue Mar 22, 2022 · 1 comment
Assignees
@nadove-ucsc
Labels
code [subject] Production code infra [subject] Project infrastructure like CI/CD, build and deployment scripts no demo [process] Not to be demonstrated at the end of the sprint orange [process] Done by the Azul team workaround [type] An enhancement that works around a defect of an external dependency

Comments

@achave11-ucsc
Copy link
Member

achave11-ucsc commented Mar 22, 2022 •
edited
Loading

https://gitlab.azul.data.humancellatlas.org/ucsc/azul/-/jobs/18206
The post_deploy make target fails with the following exception…

Traceback (most recent call last):
  File "/builds/ucsc/azul/scripts/post_deploy.py", line 87, in <module>
    main()
  File "/builds/ucsc/azul/scripts/post_deploy.py", line 83, in main
    verify_source_access()
  File "/builds/ucsc/azul/scripts/post_deploy.py", line 74, in verify_source_access
    require(not diff,
  File "/builds/ucsc/azul/src/azul/__init__.py", line 1273, in require
    reject(not condition, *args, exception=exception)
  File "/builds/ucsc/azul/src/azul/__init__.py", line 1292, in reject
    raise exception(*args)
azul.RequirementError: ('The public service account can access snapshots that the indexer service account cannot', {'c3f66226-3f1f-4fe5-b654-0bb12d62467c', 'da0b4a1e-c742-49e4-8134-6d9057abc837', 'bbc476e1-5f7a-47fd-aaab-1a5bcd461b16', 'cc901775-ca48-4588-b608-a6720225c367', 'e9faa1c3-e38b-42ca-accd-926533c76c7a', 'deb0ddec-b8c0-40e2-a2c2-695b171628e5', 'b5cd9eed-a005-4584-baef-1f1bde61fcb1', 'be00eaf7-9dd1-442d-96d1-d370107d1db1', 'bbe3aa2e-42b0-44db-83c8-a4b9ec8c31ab', 'cd9a498b-cbf9-4de3-9435-9c61cfc4d160', 'b8f00be9-f7b3-463b-8030-13b9275e394e', 'c66d67ae-81bf-43b3-9633-580cb45f4f08', 'cda9de73-d0a5-430b-943d-19d7fab8a5c4', 'c67542bb-0a6e-4ee1-9d12-0cf67400c2c1', 'bd75ce6a-9a03-40cc-a6b3-f424393f35dd', 'f842dca7-81ce-4d1b-bcba-eb91dbee92f0', 'db44f677-294e-49cc-ad76-1a46fb7870ca', 'b9aa492e-056b-490d-956a-950d33a4b852', 'dfc88eb3-ac32-4bf6-804d-9c57368e524c', 'd7de66b3-aaec-41b7-83ba-24d37343f61f', 'b87e03d7-1086-4cd9-bfb6-f60a9c173666', 'faf1d08b-b176-4a94-a6eb-869f0af5038b', 'bbfc986c-1d0d-49df-b109-bb0cbd95a7bc', 'd31df398-b23f-48ea-a2f3-1ec5bc3537f1', 'e93ccccf-6aa0-40bc-83f2-b5f062e2ee9e', 'fdc5d07b-a861-4b27-b539-d0b555d6621d', 'dee17f6d-8c5c-4f0f-b692-1277be521c91', 'f7b42ded-e60f-45cf-bf1b-54ff2747582a', 'fef7ec23-d73a-4aa2-a64e-2cdbd0b47559', 'ec82832e-27d7-464b-9e88-9d1fe395f396', 'e29906d7-7d5f-4383-9e47-f3aa67a5db79', 'fa0ea5d3-a529-4731-a0de-c17f33885971', 'b1e0c970-1bce-4e57-b707-8cdf09db08c1', 'f49bbbcf-f081-496f-8a21-51a245bc91e0', 'd71bb717-1117-4e81-8400-1bcb2b9f30b1', 'c4d27025-74ef-4cb4-b3a7-6c0287c68bd4', 'b9563933-e3fb-4513-922b-99c15b39124d', 'deece79d-9f4a-4baf-9fd1-d1de0c6bbe86', 'ff2e5d3c-e8d2-4cea-953f-9a53d244b098', 'f9c9ee4f-7e16-411b-b127-781930dc04dd', 'd5e12be3-cb9b-4442-ae03-f353a47a380d', 'd0d1c365-b46d-4ec8-b174-6cae5cd147fa', 'ccab8a08-2825-4d4c-bdc5-d79b29a6d9de', 'd96b2c52-c943-4f17-99ff-049b03bb46ba', 'd6b9db1e-7334-4143-a35a-05f82940dbbd', 'b35caf6f-7196-4d29-89f9-f26576c05ab8', 'df59cab8-a53d-4b9d-afaa-2a41970bc448', 'd3388230-361d-420c-9483-4832c7beb64e', 'b4a798c1-2d39-49ae-998e-80adbd2de5c4', 'e8d3a212-6eef-4b40-af3b-abe9cadb0a07', 'e7fc5d85-70a1-4a7c-81cf-952d0b3ba1e8', 'ff9add46-fcce-47d7-9ed2-af4c6bf275f7', 'd49eeacd-131c-4f7c-97cb-578b3f9c53fe', 'e01ea88e-00d8-464e-a750-8769ea11d934', 'e7ee828e-2aea-4e30-a1b4-0f6fb0332de2', 'f0896197-e9ba-41e9-97da-3d4c09f406c6', 'dc4b88aa-6803-4618-a8fa-160568e09838', 'bce60b88-654e-44de-b6cd-9a5d5079833c', 'c9f3337c-53ae-4709-bb3a-723ec418eeda', 'e717e0f8-5a6f-45a9-8f82-c04f37191eea', 'e04f12c0-29ee-41ef-b8bb-40bbfea3c89e', 'fe93e46d-0784-4849-ab22-b0db620085ee', 'f380e38a-ad98-4c2c-ba4a-db09da82fb6a'})
make[1]: *** [Makefile:91: post_deploy] Error 1
make[1]: Leaving directory '/builds/ucsc/azul'
make: *** [Makefile:87: auto_deploy] Error 2
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

Note: deploy job succeeded after third try.
@achave11-ucsc achave11-ucsc added the orange [process] Done by the Azul team label Mar 22, 2022
@achave11-ucsc achave11-ucsc changed the title GitLab prod fails deploy job due to RequirementError GitLab prod fails deploy job due to transient RequirementError Mar 23, 2022
@hannes-ucsc hannes-ucsc changed the title GitLab prod fails deploy job due to transient RequirementError RequirementError during deploy job prod Mar 23, 2022
@hannes-ucsc
Copy link
Member

This is caused by HumanCellAtlas/dcp2#61.

We'll work around this in a permanent hotfix by explicitly specifying a sort of created_date and a direction of asc. Sorting by date minimizes the chances of snapshot additions throwing the paging off. We still have no defense against removal of snapshots. Hopefully that doesn't happen all too often. I'll create a ticket about more reliable paging in the face of concurrent changes to the underlying set being paged over.

@hannes-ucsc hannes-ucsc added workaround [type] An enhancement that works around a defect of an external dependency code [subject] Production code infra [subject] Project infrastructure like CI/CD, build and deployment scripts labels Mar 23, 2022
@hannes-ucsc hannes-ucsc changed the title RequirementError during deploy job prod RequirementError during deploy job in prod Mar 23, 2022
@nadove-ucsc nadove-ucsc mentioned this issue Mar 23, 2022
Merged
58 tasks
hannes-ucsc pushed a commit that referenced this issue Mar 24, 2022
@nadove-ucsc @hannes-ucsc
[H] Specify explicit sort parameters when listing snapshots (#3973)
b9e2b1e
@hannes-ucsc hannes-ucsc added the no demo [process] Not to be demonstrated at the end of the sprint label Mar 24, 2022
achave11-ucsc added a commit that referenced this issue Mar 24, 2022
@achave11-ucsc
[H] Fix: RequirementError during deploy job in prod (#3973, PR #3980)
dc22c96
@achave11-ucsc achave11-ucsc mentioned this issue Mar 24, 2022
Merged
35 tasks
achave11-ucsc added a commit that referenced this issue Mar 25, 2022
@achave11-ucsc
[H] Backport: afacb8b (#3939, PR #3951) dc22c96 (#3973, PR #3980, PR #…
1ef9737 
…3983)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nadove-ucsc nadove-ucsc

Labels
code [subject] Production code infra [subject] Project infrastructure like CI/CD, build and deployment scripts no demo [process] Not to be demonstrated at the end of the sprint orange [process] Done by the Azul team workaround [type] An enhancement that works around a defect of an external dependency
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hannes-ucsc @achave11-ucsc @theathorn @nadove-ucsc