-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: vastly reduces number of db queries #252
Open
abaisero
wants to merge
3
commits into
usgo:main
Choose a base branch
from
abaisero:fix/reduce_db_queries
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Commits on Apr 11, 2022
-
fix: vastly reduces number of db queries
Problem: The creation of some tables was causing a large number of queries to be run which itself scaled with the size of the database, i.e., larger databases (such as the one in production) caused more queries to be made the the db itself (a problem which componds on itself). This issue was not apparent in development because of the smaller size of the development db. First, we can verify the above problem by increasing the size of the development db. We can do this by temporarily editing `scripts/entrypoint.sh`, chanding line python make_fake_fixtures.py 1000 1000 1000 > /tmp/fake_agagd_data.json with python make_fake_fixtures.py 100 10000 100 > /tmp/fake_agagd_data.json and then deleting and recreating the docker images/containers/volumes. Before the change, the development database was creating data for 1000 players, 1000 games, and 1000 tournaments. Note that this meant that, on average, each player only played one game and each tournament only contained one game. This was helping mask the primary issue. After the change, the development database is creating 100 players, 10_000 games, and 100 tournaments, meaning that, on average, each player played 100 games, and each tournament contained 100 games. We note that, although this new development db is still significantly smaller than the one in production, the local development app is already having trouble responding to requests in involving player and tournament pages. With the new larger development database in place, we observe the following concerning facts: * a random player page requires ~1000 queries to be made. * a random tournament page requires ~100 queries to be made. Solution: This commit addresses the above issues by making 2 primary changes in how some tables are computed or rendered: 1. The first involves the way player names and ids were rendered in the game tables using the custom `agagd_core.tables.players.LinkFullMembersNameColumn`. This custom column had a `.render()` method which internally made a query to the db. Then, for the rendering of the whole table, the `.render()` method was being called for every entry in the table data, causing many queries to be made. To solve this issue, we removed `LinkFullMembersNameColumn` altogether, and constructed the appropriate queries to construct the "player-name (player-id)" label once for all entries in the table. 2. The second involves the way opponent and tournament data was collected to create the opponent and tournament tables in the player_profile view. This data was collected and manipulated in pure python, using explicit python loops over the Games model, and at least one db query per iteration. To solve this issue, we created appropriate queries which gathered and aggregated the required data using a couple of large queries (rather than many small ones). After the above changes: * the number of queries made in a random player page dropped from ~1000 to ~10. * the number of queries made in a random tournament page dropped from ~100 to ~5.
Configuration menu - View commit details
-
Copy full SHA for 9ba7ffe - Browse repository at this point
Copy the full SHA 9ba7ffeView commit details
Commits on Apr 12, 2022
-
refactor: simplifies rendering of "player_name (player_id)" in tables
Before this commit, the construction of "player_name (player_id)" was performed at the db query level by creating a new composite data field. After this commit, the construction of "player_name (player_id)" is performed post-query as a simpler rendering function based on given fields.
Configuration menu - View commit details
-
Copy full SHA for 4e4b6d6 - Browse repository at this point
Copy the full SHA 4e4b6d6View commit details
Commits on May 2, 2022
-
fix: tournaments table in player profile
the previous commit which reduces the number of db queries was constructing tournament tables in player profiles with wrong numbers of won/lost games. This commit fixes that issue.
Configuration menu - View commit details
-
Copy full SHA for 65649ca - Browse repository at this point
Copy the full SHA 65649caView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.