Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: vastly reduces number of db queries #252

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Commits on Apr 11, 2022

  1. fix: vastly reduces number of db queries

    Problem:  The creation of some tables was causing a large number of
    queries to be run which itself scaled with the size of the database,
    i.e., larger databases (such as the one in production) caused more
    queries to be made the the db itself (a problem which componds on
    itself).  This issue was not apparent in development because of the
    smaller size of the development db.
    
    First, we can verify the above problem by increasing the size of the
    development db.  We can do this by temporarily editing
    `scripts/entrypoint.sh`, chanding line
    
            python make_fake_fixtures.py 1000 1000 1000 > /tmp/fake_agagd_data.json
    
    with
    
            python make_fake_fixtures.py 100 10000 100 > /tmp/fake_agagd_data.json
    
    and then deleting and recreating the docker images/containers/volumes.
    
    Before the change, the development database was creating data for 1000
    players, 1000 games, and 1000 tournaments.  Note that this meant that,
    on average, each player only played one game and each tournament only
    contained one game.  This was helping mask the primary issue.  After the
    change, the development database is creating 100 players, 10_000 games,
    and 100 tournaments, meaning that, on average, each player played 100
    games, and each tournament contained 100 games.  We note that, although
    this new development db is still significantly smaller than the one in
    production, the local development app is already having trouble
    responding to requests in involving player and tournament pages.
    
    With the new larger development database in place, we observe the
    following concerning facts:
    
    * a random player page requires ~1000 queries to be made.
    * a random tournament page requires ~100 queries to be made.
    
    Solution:
    
    This commit addresses the above issues by making 2 primary changes in
    how some tables are computed or rendered:
    
    1.  The first involves the way player names and ids were rendered in the
        game tables using the custom
        `agagd_core.tables.players.LinkFullMembersNameColumn`.  This custom
        column had a `.render()` method which internally made a query to the
        db.  Then, for the rendering of the whole table, the `.render()`
        method was being called for every entry in the table data, causing
        many queries to be made.  To solve this issue, we removed
        `LinkFullMembersNameColumn` altogether, and constructed the
        appropriate queries to construct the "player-name (player-id)" label
        once for all entries in the table.
    
    2. The second involves the way opponent and tournament data was
       collected to create the opponent and tournament tables in the
       player_profile view.  This data was collected and manipulated in pure
       python, using explicit python loops over the Games model, and at
       least one db query per iteration.  To solve this issue, we created
       appropriate queries which gathered and aggregated the required data
       using a couple of large queries (rather than many small ones).
    
    After the above changes:
    
    * the number of queries made in a random player page dropped from ~1000
      to ~10.
    
    * the number of queries made in a random tournament page dropped from
      ~100 to ~5.
    abaisero committed Apr 11, 2022
    Configuration menu
    Copy the full SHA
    9ba7ffe View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2022

  1. refactor: simplifies rendering of "player_name (player_id)" in tables

    Before this commit, the construction of "player_name (player_id)" was
    performed at the db query level by creating a new composite data field.
    After this commit, the construction of "player_name (player_id)" is
    performed post-query as a simpler rendering function based on given
    fields.
    abaisero committed Apr 12, 2022
    Configuration menu
    Copy the full SHA
    4e4b6d6 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2022

  1. fix: tournaments table in player profile

    the previous commit which reduces the number of db queries was
    constructing tournament tables in player profiles with wrong numbers of
    won/lost games.  This commit fixes that issue.
    abaisero committed May 2, 2022
    Configuration menu
    Copy the full SHA
    65649ca View commit details
    Browse the repository at this point in the history