Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix/too many partitions #132

Merged
merged 8 commits into from
Sep 18, 2024

Conversation

fivetran-joemarkiewicz
Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz commented Aug 23, 2024

PR Overview

This PR will address the following Issue/Feature: Issue #130

This PR will result in the following new package version: v0.14.0

This will be a breaking change for BigQuery users since this will adjust the partition grain. Therefore, we will want to mark this as a breaking change for all users.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes (Full refresh required after upgrading)

  • Partitioned models have had the partition_by logic adjusted to include a granularity of a month. This change should only impact BigQuery warehouses and was applied to avoid the common too many partitions error users have experienced due to over-partitioning by day. Therefore, adjusting the partition to a monthly granularity will increase the partition windows and allow for more performant querying.
  • This change was applied to the following models:
    • int_netsuite2__tran_with_converted_amounts
    • netsuite2__balance_sheet
    • netsuite2__income_statement
    • netsuite2__transaction_details

Under the Hood

  • Consistency tests added for each Netsuite2 end model in order to be used during integration test validations.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • dbt run (if incremental models are present)

Before marking this PR as "ready for review" the following have been applied:

  • The appropriate issue has been linked, tagged, and properly assigned
  • All necessary documentation and version upgrades have been applied
  • docs were regenerated (unless this PR does not include any code or yml updates)
  • BuildKite integration tests are passing
  • Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

See below the passing validation tests for both full refresh and incremental runs.

image

image

@fivetran-joemarkiewicz fivetran-joemarkiewicz marked this pull request as ready for review September 5, 2024 21:51
Copy link
Contributor

@fivetran-catfritz fivetran-catfritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-joemarkiewicz Gave this a test run, and all looks good here. Lgtm!

@fivetran-avinash fivetran-avinash self-requested a review September 18, 2024 10:01
CHANGELOG.md Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Contributor

@fivetran-avinash fivetran-avinash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-joemarkiewicz One comment but otherwise looks good to me!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-joemarkiewicz Think you want to remove this.

@fivetran-joemarkiewicz fivetran-joemarkiewicz changed the base branch from main to release-branch/v0.14.0 September 18, 2024 20:25
@fivetran-avinash fivetran-avinash merged commit bb3e777 into release-branch/v0.14.0 Sep 18, 2024
7 of 9 checks passed
fivetran-avinash added a commit that referenced this pull request Sep 25, 2024
* Update customer and vendor joins

* customer vendor joins

* seeds, docs, test

* schema bump

* final fixes before PR review

* tweak schema in integration tests

* remove run results

* PR review updates

* update quickstart.yml

* update run_models.sh

* release-branch/v0.14.0

* Bugfix/too many partitions (#132)

* bugfix/too-many-partitions

* validation schema addition

* docs regen and final changes before review

* intermediate model conditional update

* Apply suggestions from code review

Co-authored-by: Avinash Kunnath <[email protected]>

* removing conditional logic

---------

Co-authored-by: Avinash Kunnath <[email protected]>

* regen docs

* release review

---------

Co-authored-by: Jared Monger <[email protected]>
Co-authored-by: Joe Markiewicz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants