Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf: Course dashboard performance issues #800

Closed
bmtcril opened this issue May 10, 2024 · 5 comments
Closed

Perf: Course dashboard performance issues #800

bmtcril opened this issue May 10, 2024 · 5 comments
Labels
backport PR backports a change from main to a named release.

Comments

@bmtcril
Copy link
Contributor

bmtcril commented May 10, 2024

Tested with the 300M row dataset on ClickHouse Cloud I've found the following issues:

OOM Errors:

  • Problem Results
  • Problem Interactions
  • Watched Video Segments
  • Partial and full views per video

Slow Queries (> 3 seconds):

  • Problems attempted per section/subsection (43 rows, 5 seconds)
  • Video views per section/subsection (24 rows, 4 seconds)
  • Page views per section/subsection (37 rows, 3 seconds)
@bmtcril bmtcril added aspects v1 backport PR backports a change from main to a named release. labels May 10, 2024
@SoryRawyer
Copy link
Contributor

I have been trying to make changes to int_problem_results in dbt, but so far the best results have been when I could filter the subqueries individually. This suggests to me that we should move int_problem_results into Superset and rely on templating to make sure queries make proper use of indexes.

I will look into a similar approach for the watched video segments query.

@Ian2012
Copy link
Contributor

Ian2012 commented May 10, 2024

@SoryRawyer I think I can move this dataset to a MV with MergeEngine: https://github.com/openedx/aspects-dbt/blob/b16a7957de4506e22768cb4a610798113c9868a2/models/problems/int_problem_results.sql#L4

Would that also be a suitable solution? cc @bmtcril

@bmtcril
Copy link
Contributor Author

bmtcril commented May 13, 2024

Fixes for everything but problem interactions and problem results are here #808

@Ian2012
Copy link
Contributor

Ian2012 commented May 24, 2024

fixes for problem interactions - problem results are here: #825

@bmtcril
Copy link
Contributor Author

bmtcril commented May 28, 2024

I believe these issues are all fixed now!

@bmtcril bmtcril closed this as completed May 28, 2024
@github-project-automation github-project-automation bot moved this from Doing to Done in Data Working Group May 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport PR backports a change from main to a named release.
Projects
Development

No branches or pull requests

3 participants