Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pandas 2.0 Support #94

Merged
merged 41 commits into from
Sep 29, 2023
Merged

Pandas 2.0 Support #94

merged 41 commits into from
Sep 29, 2023

Conversation

IzerOnadimQC
Copy link
Collaborator

@IzerOnadimQC IzerOnadimQC commented Sep 8, 2023 •
edited
Loading

Adds support for Pandas 2.0

  • Support pandas 2
  • Test pyarrow 12 and 13
  • Prevent dask from casting all object dtypes to strings
  • Remove tests for pyarrow<=3 as they fail with pandas>=2

@IzerOnadimQC
Avoid pandas 2.1.0 due to timestamp bug
a43c8ed 
Pandas 2.1.0 DataFrame constructor bug causeing timestamps to have
inconsistent units (pandas-dev/pandas#55014).
@IzerOnadimQC
Coerce timestamps to nanoseconds when converting to pandas
f2ea716
@IzerOnadimQC
Prevent dask from converting objects to strings
fa15a47
@IzerOnadimQC
Avoid dask 2023.9.2 due to failing tests
c182ffb
@IzerOnadimQC
Cast metadata bytes to object to get around pandas bug
8b1e0af
@IzerOnadimQC
Generate arrow-compat reference data for 13.0.0
d35b3e0
@IzerOnadimQC
Change package version in docs to match environment.yml
5715758
@IzerOnadimQC
Only use coerce timestamps arg on pyarrow>=13
1dbe229
@IzerOnadimQC
Remove pyarrow<8 tests from ci.yml
062f283
@IzerOnadimQC
Generate arrow-compat reference data for 12.0.0
f909ffe
@IzerOnadimQC
Avoid pandas 2.1.0 in numfocus nightly ci test
f0f0662
@IzerOnadimQC
Avoid pandas 2.1.0.* in numfocus_nightly pip install
3c56634
@IzerOnadimQC
Add changelog entry and update setup.cfg
60079ee
@IzerOnadimQC
Shrink PR
2ccb2fe
@IzerOnadimQC
Add dask tests for lines marked as uncovered by codecov
d52f76c
@IzerOnadimQC
Check conda env before verbose import
fae951d
@IzerOnadimQC
Use micromamba instead of mamba
91ecebf
@IzerOnadimQC
Pin pandas<2.1.0 due to bug in 2.1.0 and 2.1.1
6248192
@IzerOnadimQC
Check if adding pyarrow 13 tests improves coverage
807650c
@IzerOnadimQC
Fix yaml error
d45d160
@IzerOnadimQC
Fix pyarrow install command and re-add removed tests
fdcecc3
@IzerOnadimQC
Experiment with tests for backwards compatibility
0414b78
@IzerOnadimQC
Allow install of specific pandas version
1c2352d
@IzerOnadimQC
Fix yaml error
84cdbdb
@IzerOnadimQC
Update changelog and re-add pyarrow 4.0.1 to ci.yml
c14edda
@IzerOnadimQC
Remove test for pyarrow==3.0.0 as incompatible with pandas 2
2c80c10
@IzerOnadimQC
asv no longer supports dev, use run instead
657704a
@IzerOnadimQC
Add environment arg to asv run
1f97207
@IzerOnadimQC
Use astype when seting Series type due to change in pandas behaviour
c0218c3
@IzerOnadimQC
Pin dask<2023.9.2
efb827f
@IzerOnadimQC IzerOnadimQC changed the title [WIP] Pandas 2.0 Support Pandas 2.0 Support Sep 26, 2023
@IzerOnadimQC IzerOnadimQC marked this pull request as ready for review September 26, 2023 16:36
xhochy
xhochy reviewed Sep 27, 2023
View reviewed changes
CHANGES.rst Show resolved Hide resolved
This was referenced Sep 27, 2023
Closed
Closed
jtilly
jtilly reviewed Sep 27, 2023
View reviewed changes
docs/environment-docs.yml Outdated Show resolved Hide resolved
plateau/core/common_metadata.py Outdated Show resolved Hide resolved
jtilly
jtilly reviewed Sep 28, 2023
View reviewed changes
setup.cfg Outdated Show resolved Hide resolved
jtilly
jtilly approved these changes Sep 28, 2023
View reviewed changes
Copy link
Collaborator

@jtilly jtilly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@IzerOnadimQC IzerOnadimQC merged commit d939135 into data-engineering-collective:main Sep 29, 2023
19 of 20 checks passed
@IzerOnadimQC IzerOnadimQC deleted the pandas-2-support branch October 30, 2023 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xhochy xhochy xhochy left review comments

@jtilly jtilly jtilly approved these changes

@DamianB-BitFlipper DamianB-BitFlipper Awaiting requested review from DamianB-BitFlipper

@DamianBarabonkovQC DamianBarabonkovQC Awaiting requested review from DamianBarabonkovQC

@fjetter fjetter Awaiting requested review from fjetter fjetter is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@IzerOnadimQC @xhochy @jtilly