adding all_rows_overlap to fugue

[fdosani]

Ref: #214
Support for all_rows_overlap with Fugue

[fdosani]

@goodwanghan FYI. Had some time, so adding in some more functionality to the Fugue side.

[goodwanghan]

Wow nice

[fdosani]

Wow nice

Well this sucks. Seems like the last release of fugue is breaking some of the tests :(

[goodwanghan]

Oh what? I will take a look

…

On Fri, Nov 10, 2023 at 09:08 Faisal ***@***.***> wrote: Wow nice Well this sucks. Seems like the last release of fugue is breaking some of the tests :( — Reply to this email directly, view it on GitHub <#244 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFA5Q7ZDLOIP3LORSNNNFYTYDZNQRAVCNFSM6AAAAAA7GLTQDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGA4TSOJXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[goodwanghan]

I can reproduce this issue, will take a look today

[goodwanghan]

So after some initial investigation, it is related with pandas. Fugue 0.8.7 has a big update that is it relies more on pandas extension d types. It is an interesting case why the current datacompy logic breaks with this change. I will investigate more.

[fdosani]

So after some initial investigation, it is related with pandas. Fugue 0.8.7 has a big update that is it relies more on pandas extension d types. It is an interesting case why the current datacompy logic breaks with this change. I will investigate more.

Ahh interesting. Thanks for helping out here and taking a peek, really appreciate it.

[goodwanghan]

no problem, actually think this is a very good case, i have found an example
You know pandas will use object type for string columns, but in new pandas, it can be StringDType, and it can be an arrow type (it is a mess I know). So in this case we want to treat them as the same type? In the current logic, we think they are different. That is why we saw those failed tests.

To generalize, datacompy needs to consider the comparison between original type and extension dtypes, do we think they are the same?

Examples are:

object vs StringDType
int vs Int
bool vs Bool

[fdosani]

no problem, actually think this is a very good case, i have found an example

You know pandas will use object type for string columns, but in new pandas, it can be StringDType, and it can be an arrow type (it is a mess I know). So in this case we want to treat them as the same type? In the current logic, we think they are different. That is why we saw those failed tests.

To generalize, datacompy needs to consider the comparison between original type and extension dtypes, do we think they are the same?

Examples are:

object vs StringDType

int vs Int

bool vs Bool

Yup those should be considered the same type IMHO. Off the top of my head pandas does store other things as objects too though. I'm pretty there is some conversion logic which happens in the compare class. Just travelling but I can take a look once I'm home.

[fdosani]

Waiting on #245 first before rebasing, reviewing, and merging

[jdawang]

We should probably list the added functionality for fugue in the docs. Right now only is_match and report are mentioned.

[fdosani]

We should probably list the added functionality for fugue in the docs. Right now only is_match and report are mentioned.

Good call! Will add this in a bit.

[CLAassistant]

All committers have signed the CLA.

[fdosani]

@jdawang updated now