Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TRT-1338: WIP: Potential replacement for the remaining 35 minute matview: Test analysis by job #2074

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

TRT-1338: WIP: Potential replacement for the remaining 35 minute matview: Test analysis by job #2074

wants to merge 15 commits into from
+473 −57

Commits on Oct 30, 2024

  1. Define new test_analysis_by_job_for_dates table to replace matview

    @dgoodwin
    dgoodwin committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    045c0c9 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Fix new table date column

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    aeff3a4 View commit details
    Browse the repository at this point in the history

  2. Running the query and parsing results into go types

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    94ac7c3 View commit details
    Browse the repository at this point in the history

  3. Deal with bigquery vs postgresql Date

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    1371aed View commit details
    Browse the repository at this point in the history

  4. Create rest of row for postgres insertion

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    30a59de View commit details
    Browse the repository at this point in the history

  5. Batch insertion

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    ed4535b View commit details
    Browse the repository at this point in the history

  6. Implement calculation for date range

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    dc5928f View commit details
    Browse the repository at this point in the history

  7. Use the date ranges

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    e6ef289 View commit details
    Browse the repository at this point in the history

  8. Now functional, but slow

    @dgoodwin
    dgoodwin committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    43eb6aa View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Wrap batch insert in one overall transaction for a ~40% reduction

    @dgoodwin
    dgoodwin committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    3acca89 View commit details
    Browse the repository at this point in the history

  2. temp

    @dgoodwin
    dgoodwin committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    55116ff View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. wip

    @dgoodwin
    dgoodwin committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    aa15118 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Refactor to import one day at a time for better transactions and timings

    @dgoodwin
    dgoodwin committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    1e08fad View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Partition the test analysis by job by dates table automatically

    @dgoodwin
    dgoodwin committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    1727522 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. Add schema to create the partitioned table, but not update it ever

    @dgoodwin
    dgoodwin committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    12c6d5b View commit details
    Browse the repository at this point in the history