TST: Add test for pd.read_csv date parsing not working with dtype_backend="pyarrow" and missing values

[KevsterAmp]

• closes BUG: pd.read_csv date parsing not working with dtype_backend="pyarrow" and missing values #59904 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[KevsterAmp]

I tried using

    assert pd.api.types.is_datetime64_any_dtype(df["date"])

but it seems that the code checks doesn't allow it. Not sure if the assertion is correct:

    assert (df["date"].dtype) == "datetime64[s]"

[mroeschke]

Can you build an expected DataFrame and use tm.assert_frame_equal?

[KevsterAmp]

Having a bit of struggle with the dtype casting, Tried two methods:

    # put dtype string[pyarrow] on the Series
    expected = pd.DataFrame(
        {
            "date": pd.Series(
                pd.to_datetime(["20/12/2025", pd.NaT, "31/12/2020"], dayfirst=True),
            ),
            "id": pd.Series(["a", "b", "c"], dtype="string[pyarrow]"),
        },
    )
    
    ###############
    
    # cast dtype using .astype()
    expected["id"] = expected["id"].astype("string[pyarrow]")

Returns error:

E       AssertionError: Attributes of DataFrame.iloc[:, 1] (column name="id") are different
E
E       Attribute "dtype" are different
E       [left]:  StringDtype(storage=pyarrow, na_value=<NA>)
E       [right]: string[pyarrow]

For a band-aid fix, I tried casting string[pyarrow] as well to the same column in the df variable.

@td.skip_if_no("pyarrow")
def test_pyarrow_read_csv_datetime_dtype():
    data = "date,id\n20/12/2025,a\n,b\n31/12/2020,c"
    df = pd.read_csv(
        StringIO(data), parse_dates=["date"], dayfirst=True, dtype_backend="pyarrow"
    )
    expected = pd.DataFrame(
        {
            "date": pd.Series(
                pd.to_datetime(["20/12/2025", pd.NaT, "31/12/2020"], dayfirst=True),
            ),
            "id": pd.Series(["a", "b", "c"], dtype="string[pyarrow]"),
        },
    )
    expected["id"] = expected["id"].astype("string[pyarrow]")
    df["id"] = df["id"].astype("string[pyarrow]")

    assert tm.assert_frame_equal(expected, df)
    assert (df["date"].dtype) == "datetime64[s]"

But for some reason, pytest returns:

>       assert tm.assert_frame_equal(expected, df)
E       AssertionError

Hard to check what's the error exaclty, since the error isn't verbose.

[mroeschke]

Based on the simplifying the bug report, I don't think we need the string column, only the "date" column.

[KevsterAmp]

@mroeschke - Tried this:

@td.skip_if_no("pyarrow")
def test_pyarrow_read_csv_datetime_dtype():
    # GH 59904
    data = '"date"\n"20/12/2025"\n""\n"31/12/2020"'
    result = pd.read_csv(
        StringIO(data), parse_dates=["date"], dayfirst=True, dtype_backend="pyarrow"
    )
    expected_dict = {
        "date": pd.Series(
            pd.to_datetime(["20/12/2025", pd.NaT, "31/12/2020"], dayfirst=True)
        )
    }
    expected = pd.DataFrame(expected_dict)

    assert (result["date"].dtype) == "datetime64[s]"
    assert tm.assert_frame_equal(expected, result)

Still returns assertion error

>       assert tm.assert_frame_equal(expected, result)
E       AssertionError

pandas/tests/io/test_common.py:696: AssertionError

[KevsterAmp]

Finally saw the problem lol
tm.assert_frame_equal should be run without assert. That's why it was showing AssertionError 😆

    assert tm.assert_frame_equal(expect, result) # returns AssertionError

    tm.assert_frame_equal(expect, result) # passes

Fixed it now and the test is passing

[mroeschke]

Suggested change

assert (result["date"].dtype) == "datetime64[s]"

This is done in assert_frame_equal

[mroeschke]

Suggested change

	expect_data = pd.Series(
	pd.to_datetime(["20/12/2025", pd.NaT, "31/12/2020"], dayfirst=True)
	)
	expect_data = pd.to_datetime(["20/12/2025", pd.NaT, "31/12/2020"], dayfirst=True)

[mroeschke]

Thanks @KevsterAmp