Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(BDC-326): Adds gen3-user-data-library #2674

Merged
merged 1 commit into from
Dec 5, 2024
Merged

feat(BDC-326): Adds gen3-user-data-library #2674

merged 1 commit into from
Dec 5, 2024

Conversation

k-burt-uch
Copy link
Contributor

Link to JIRA ticket if there is one: BDC-326

New Features

  • Adds gen3-user-data-library

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

@k-burt-uch k-burt-uch requested a review from Avantol13 November 18, 2024 21:24
@k-burt-uch k-burt-uch force-pushed the feat/gen3-user-data-library branch from d99e7d3 to 45aea17 Compare November 18, 2024 21:27
k-burt-uch
k-burt-uch commented Nov 18, 2024
View reviewed changes
kube/services/gen3-user-data-library/gen3-user-data-library-deploy.yaml
memory: 1024Mi
limits:
cpu: 2
memory: 2048Mi
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what to do about the high resource requirements of gen3-user-data-library at the moment. At 1CPU, 1024Mi is needed. Any lower and we get OOMKilled.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is concerning... there's some memory misuse somewhere. I guess it's okay for now but we should circle back and try to figure out how to get that requirement down

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, I forgot the worker count was dynamic based on the number of cores in the CPU. It could be overhead from spinning out so many workers? I just did a quick memtest with the pytest suite and don't see any obvious functions of ours jumping out, but could be something that only shows up in a real scenario with real interactions

EliseCastle23
EliseCastle23 previously approved these changes Nov 18, 2024
View reviewed changes
@k-burt-uch k-burt-uch force-pushed the feat/gen3-user-data-library branch 3 times, most recently from 49e30e2 to 9010ea4 Compare November 22, 2024 15:28
@k-burt-uch k-burt-uch force-pushed the feat/gen3-user-data-library branch from 9010ea4 to aecdd7c Compare November 22, 2024 15:47
@github-actions GitHub Actions
Copy link

filepath $$\textcolor{#23d18b}{\tt{passed}}$$ $$\textcolor{#f14c4c}{\tt{failed}}$$ $$\textcolor{#ffa500}{\tt{skipped}}$$ SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_oauth2.py}}$$ $$\textcolor{#23d18b}{\tt{15}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{15}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_centralized\_auth.py}}$$ $$\textcolor{#23d18b}{\tt{16}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{16}}$$
$$\textcolor{#ffa500}{\tt{tests/test\_graph\_submit\_and\_query.py}}$$ $$\textcolor{#23d18b}{\tt{13}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#ffa500}{\tt{1}}$$ $$\textcolor{#ffa500}{\tt{14}}$$
$$\textcolor{#ffa500}{\tt{tests/test\_data\_upload.py}}$$ $$\textcolor{#23d18b}{\tt{8}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#ffa500}{\tt{1}}$$ $$\textcolor{#ffa500}{\tt{9}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_guppy\_service.py}}$$ $$\textcolor{#23d18b}{\tt{8}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{8}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_presigned\_url.py}}$$ $$\textcolor{#23d18b}{\tt{7}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{7}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_requestor.py}}$$ $$\textcolor{#23d18b}{\tt{5}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{5}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_user\_token.py}}$$ $$\textcolor{#23d18b}{\tt{5}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{5}}$$
$$\textcolor{#ffa500}{\tt{tests/test\_dbgap.py}}$$ $$\textcolor{#23d18b}{\tt{4}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#ffa500}{\tt{1}}$$ $$\textcolor{#ffa500}{\tt{5}}$$
$$\textcolor{#f14c4c}{\tt{tests/test\_audit\_service.py}}$$ $$\textcolor{#23d18b}{\tt{4}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{5}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_drs\_endpoint.py}}$$ $$\textcolor{#23d18b}{\tt{4}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{4}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_study\_viewer.py}}$$ $$\textcolor{#23d18b}{\tt{3}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{3}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_workspace.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_ras\_authn.py}}$$ $$\textcolor{#23d18b}{\tt{3}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{3}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_indexing\_page.py}}$$ $$\textcolor{#23d18b}{\tt{2}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{2}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_login\_page.py}}$$ $$\textcolor{#23d18b}{\tt{2}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{2}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_oidc\_client.py}}$$ $$\textcolor{#23d18b}{\tt{2}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{2}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_pfb\_export.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_manifest\_service.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_metadata\_ingestion.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{tests/test\_google\_data\_access.py}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#ffa500}{\tt{tests/test\_gen3ff\_landing\_page.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#ffa500}{\tt{2}}$$ $$\textcolor{#ffa500}{\tt{2}}$$
$$\textcolor{#ffa500}{\tt{tests/test\_register\_user.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#ffa500}{\tt{2}}$$ $$\textcolor{#ffa500}{\tt{2}}$$
$$\textcolor{#f14c4c}{\tt{tests/test\_discoverypage.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$
$$\textcolor{#f14c4c}{\tt{tests/test\_client\_credentials.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$
$$\textcolor{#f14c4c}{\tt{tests/test\_etl.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$
$$\textcolor{#f14c4c}{\tt{tests/test\_aggregate\_mds.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$
$$\textcolor{#f14c4c}{\tt{tests/test\_gen3client.py}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$ $$\textcolor{#666666}{\tt{0}}$$ $$\textcolor{#f14c4c}{\tt{1}}$$
$$\textcolor{#f14c4c}{\tt{TOTAL}}$$ $$\textcolor{#23d18b}{\tt{107}}$$ $$\textcolor{#f14c4c}{\tt{6}}$$ $$\textcolor{#ffa500}{\tt{7}}$$ $$\textcolor{#f14c4c}{\tt{120}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

@github-actions GitHub Actions
Copy link

Please find the ci env pod logs here

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 3, 2024

filepath $$\textcolor{#666666}{\tt{SUBTOTAL}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$ $$\textcolor{#666666}{\tt{0}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

@k-burt-uch
Copy link
Contributor Author

Merging despite test failures. https://cdis.slack.com/archives/C04LAGYRQ59/p1733412408969329?thread_ts=1733411988.972259&cid=C04LAGYRQ59

@k-burt-uch k-burt-uch merged commit f8c5065 into master Dec 5, 2024
4 of 6 checks passed
@k-burt-uch k-burt-uch deleted the feat/gen3-user-data-library branch December 5, 2024 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AidanHilt AidanHilt AidanHilt approved these changes

@Avantol13 Avantol13 Avantol13 approved these changes

@EliseCastle23 EliseCastle23 EliseCastle23 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@k-burt-uch @Avantol13 @AidanHilt @EliseCastle23