DO NOT MERGE: Debug gene info endpoint returning null #1153

tihuan · 2024-12-18T22:35:45Z

⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️

DO NOT MERGE UNTIL ADDING STAGING BUCKET PERMISSION TO DEV ROLE
Revert staging role iam to dev role

⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️

Argus Stack Details

Name	thuang-test-rdev-tests
Short Name	certain-bull
Base URL	https://certain-bull.dev-sc.dev.czi.team

This reverts commit 6d162bf.

tihuan · 2024-12-19T16:17:02Z

.infra/rdev/values.yaml

@@ -22,4 +22,4 @@ stack:
        - name: DATA_LOCATOR_DOMAIN
          value: $(__ARGUS_STACK_INGRESS_HOST)
        - name: CXG_BUCKET_PATH
-          value: hosted-cellxgene-dev
+          value: hosted-cellxgene-staging


@hspitzley-czi I'm trying to test a fix by changing Explorer BE from hitting dev to staging, but I'm getting:

Unable to discover s3 region name from s3://hosted-cellxgene-staging (type=value_error)

In https://argo.prod.platform.si.czi.technology/applications/certain-bull?resource=&node=%2FPod%2Fargus-single-cell-explorer-rdev%2Fcertain-bull-stack-explorer-7654d9cc58-8nzrg%2F0&tab=logs

Is that because rdev role permission doesn't have access to staging bucket?

The bug I'm trying to address is that rdev gene info endpoint always returns null 🤔

CURL

curl 'https://certain-bull.dev-sc.dev.czi.team/cellxgene/s3_uri/s3%253A%252F%252Fhosted-cellxgene-dev%252Fsuper-cool-spatial.cxg/api/v0.3/geneinfo?geneID=&gene=AR' \ -H 'accept: application/json' \ -H 'accept-language: en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7,zh-CN;q=0.6' \ -H 'cookie: argus_sticky_session=1734566048.418.1662.83615|e3f9a91a7a0db7bca0ae9c8322a67838' \ -H 'dnt: 1' \ -H 'priority: u=1, i' \ -H 'referer: https://certain-bull.dev-sc.dev.czi.team/d/super-cool-spatial.cxg/' \ -H 'sec-ch-ua: "Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'sec-ch-ua-platform: "macOS"' \ -H 'sec-fetch-dest: empty' \ -H 'sec-fetch-mode: cors' \ -H 'sec-fetch-site: same-origin' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'

whereas on staging it returns actual info:

curl 'https://api.cellxgene.staging.single-cell.czi.technology/cellxgene/s3_uri/s3%253A%252F%252Fhosted-cellxgene-staging%252Fpbmc3k.cxg/api/v0.3/geneinfo?geneID=&gene=ACD' \ -H 'accept: application/json' \ -H 'accept-language: en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7,zh-CN;q=0.6' \ -H 'cookie: argus_sticky_session=1734565349.449.38.739565|df4dba4aefd2407066eb62a1488673d2' \ -H 'dnt: 1' \ -H 'origin: https://cellxgene.staging.single-cell.czi.technology' \ -H 'priority: u=1, i' \ -H 'referer: https://cellxgene.staging.single-cell.czi.technology/' \ -H 'sec-ch-ua: "Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'sec-ch-ua-platform: "macOS"' \ -H 'sec-fetch-dest: empty' \ -H 'sec-fetch-mode: cors' \ -H 'sec-fetch-site: same-site' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'

Hmm, that seems likely. I'm not sure what the application code is running to produce that error but looking at the role used by rdev I'm not seeing permission for the staging bucket
https://github.com/chanzuckerberg/single-cell-explorer/blob/main/.infra/common.yaml#L24

I see! I think the code that produces the error is this line in the Explorer BE code:

single-cell-explorer/server/common/config/config_model.py

Line 201 in 31fd26b

raise ValueError(f"Unable to discover s3 region name from {path}")

My hypothesis is that the Explorer BE is returning null for gene info query now, because behind the scene Explorer GE actually queries Data Portal's dev gene info endpoint for data, and I believe dev data portal endpoint is no longer working, thus us seeing the issue

https://cellxgene.dev.single-cell.czi.technology/ <-- gets 503 service not available now

So I'm thinking the solution here is for Explorer rdev to switch from hitting dev to staging and thus needing permission to staging bucket

Should I just change the common iam from dev to staging ?

arn:aws:iam::699936264352:role/data_portal_staging_explorer

Let me try that out! Thanks for pointing me to the code!

You might want to check if there are other differences in the permissions between the dev role and staging role. You could probably pretty easily add permission to the dev role for the staging bucket

Ah thanks for the callout! Okay let me test if using staging role actually works, if so I'll just add staging bucket permission to dev role 🙆‍♂️

Huh yeah github org wide search didn't return anything o_o

https://github.com/search?q=org:chanzuckerberg%20jheath-argus-demo-deleteme&type=code

Ooh it looks like the access key and secret key set with argus secrets belongs to that user. (pass -r to see the values)

» argus list secrets --app single-cell-explorer --env rdev hspitzley@MacBookPro Listing environment variables on environment rdev +-----------------------+----------+-------------+ | KEY | VALUE | SOURCE | +-----------------------+----------+-------------+ | AWS_ACCESS_KEY_ID | ******** | environment | | AWS_SECRET_ACCESS_KEY | ******** | environment | +-----------------------+----------+-------------+

There was some reason we needed it to assume a user instead of a role (I'm forgetting why right now) but this gives us a path forward. You could copy those values elsewhere so you can restore them later and then overwrite them with the corresponding values from staging and I think that will make it able to access the staging bucket

oh fantastic!! Thanks so much for helping me out with this 🤩 🙏 That's really good to know 💡

Okay will do and retest 🫡

Thanks again!

Okay can confirm that my rdev can now hit staging endpoint thank you!

However, somehow the staging endpoint is still returning null, so I'm adding more logs to troubleshoot what's going on there. Will update this thread 👌

argocd-czi-prod · 2025-01-03T00:00:06Z

Stale Stack Warning

The stack associated with this PR has not been updated since 2024-12-23 18:51:13 UTC (~10.2 days ago).

The threshold for deletion specified in the 'stale_rdev_cleanup_threshold_days' setting for the associated app (single-cell-explorer) is 14 days.

Unless a new commit is made to this branch before the threshold is reached, the stack will be deleted.

tihuan · 2025-01-06T15:50:47Z

Closing this for now to focus on higher priority stuff. The latest investigation shows that the rdev hitting the staging API still returns null for some reason 🤔

#1153 (comment)

codecov · 2025-01-06T15:58:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (f283373) to head (a822c59).
Report is 1 commits behind head on main.

Additional details and impacted files

@@     Coverage Diff      @@
##   main   #1153   +/-   ##
============================
============================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tihuan added the stack label Dec 18, 2024

tihuan force-pushed the thuang-test-rdev-tests branch 3 times, most recently from b6f146d to 31fd26b Compare December 18, 2024 23:49

tihuan changed the title ~~DO NOT MERGE: Revert "feat: agent (#1148)"~~ DO NOT MERGE: Debug gene info endpoint returning null Dec 18, 2024

tihuan commented Dec 19, 2024

View reviewed changes

tihuan force-pushed the thuang-test-rdev-tests branch 3 times, most recently from ad4cde2 to f70b093 Compare December 19, 2024 22:57

tihuan added 4 commits December 23, 2024 08:33

fix: Update gene_info endpoint to staging

7d7dfaa

change iam to staging

4d8b744

add logs

0ca2e83

print iam role

366e793

tihuan force-pushed the thuang-test-rdev-tests branch from d0cbc71 to 366e793 Compare December 23, 2024 16:33

more debug

15b9998

tihuan force-pushed the thuang-test-rdev-tests branch from 4719a41 to 15b9998 Compare December 23, 2024 16:58

more logs

d98718d

tihuan force-pushed the thuang-test-rdev-tests branch from 67889c1 to d98718d Compare December 23, 2024 17:28

more logs

8f1bcd8

tihuan force-pushed the thuang-test-rdev-tests branch 2 times, most recently from d203f43 to 92a1302 Compare December 23, 2024 17:55

use logger

d81ca4c

tihuan force-pushed the thuang-test-rdev-tests branch from 92a1302 to d81ca4c Compare December 23, 2024 17:56

more logs

18feafd

tihuan force-pushed the thuang-test-rdev-tests branch from 89e3b23 to 18feafd Compare December 23, 2024 18:47

chore: Updated [rdev] values.yaml image tags to sha-18feafd2

a822c59

tihuan closed this Jan 6, 2025

tihuan deleted the thuang-test-rdev-tests branch January 6, 2025 15:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DO NOT MERGE: Debug gene info endpoint returning null #1153

DO NOT MERGE: Debug gene info endpoint returning null #1153

tihuan commented Dec 18, 2024 •

edited by github-actions bot

Loading

tihuan Dec 19, 2024

hspitzley-czi Dec 19, 2024

tihuan Dec 19, 2024

hspitzley-czi Dec 19, 2024

tihuan Dec 19, 2024

tihuan Dec 20, 2024

hspitzley-czi Dec 23, 2024

hspitzley-czi Dec 23, 2024

tihuan Dec 23, 2024

tihuan Dec 23, 2024

argocd-czi-prod bot commented Jan 3, 2025

tihuan commented Jan 6, 2025

codecov bot commented Jan 6, 2025

DO NOT MERGE: Debug gene info endpoint returning null #1153

DO NOT MERGE: Debug gene info endpoint returning null #1153

Conversation

tihuan commented Dec 18, 2024 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

argocd-czi-prod bot commented Jan 3, 2025

tihuan commented Jan 6, 2025

codecov bot commented Jan 6, 2025

Codecov Report

tihuan commented Dec 18, 2024 •

edited by github-actions bot

Loading