GH-39914: [pyarrow] Reorder to_pandas extension dtype mapping #44720
Conversation
Addresses pandas-dev/pandas#53011 `types_mapper` always had highest priority as it overrode what was set before. However, switching the logical ordering, it means that we don't need to call `_pandas_api.pandas_dtype(dtype)` when using the pyarrow backend. Resolving the issue of complex `dtype` with `list` or `struct`
|
❌ GitHub issue #53011 could not be retrieved. |
bretttully
changed the title
|
|
And because you added a |
| # Round trip df0 into df1 | ||
| with io.BytesIO() as stream: | ||
| df0.to_parquet(stream, schema=schema) | ||
| stream.seek(0) | ||
| df1 = pd.read_parquet(stream, dtype_backend="pyarrow") |
There was a problem hiding this comment.
You might not need the roundtrip to parquet, but a table = pa.table(df); result = table.to_pandas(types_mapper=pd.ArrowDtype) should be sufficient to test this?
I know this doesn't test exactly pd.read_parquet in its entirety, but it should test the relevant part on the pyarrow side, and an actual pd.read_parquet test can still be added to pandas
There was a problem hiding this comment.
The error only gets thrown once the pandas metadata is added to the table. That's why I have used a round-trip test. Is there another way to generate that metadata and set it on the table before calling to_pandas?
There was a problem hiding this comment.
The metadata gets added on the pyarrow side, so table = pa.table(df) will do that
There was a problem hiding this comment.
The pandas to_parquet method essentially just does a table = pa.Table.from_pandas(df) and then writes that to parquet (and pa.table(df) is a shorter less explicit version of that, but you can also use Table.from_pandas)
Yes, exactly. Priority remains the same, but functions are skipped if the field already has a type, meaning that the code causing the error is no longer called if types_mapper is provided. |
|
The |
Rationale for this change
This is a long standing pandas ticket with some fairly horrible workarounds, where complex arrow types do not serialise well to pandas as the pandas metadata string is not parseable. However,
types_mapperalways had highest priority as it overrode what was set before.What changes are included in this PR?
By switching the logical ordering, it means that we don't need to call
_pandas_api.pandas_dtype(dtype)when using the pyarrow backend, thus resolving the issue of complexdtypewithlistorstruct. It will likely still fail if the numpy backend is used, but at least this gives a working solution rather than an inability to load files at all.Are these changes tested?
Existing tests should stay unchanged and a new test for the complex type has been added
Are there any user-facing changes?
This PR contains a "Critical Fix".
This makes
pd.read_parquet(..., dtype_backend="pyarrow")work with complex data types where the metadata added by pyarrow duringpd.to_parquetis not serialisable and currently throwing an exception. This issue currently prevents the use of pyarrow as the default backend for pandas.