Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-39914: [pyarrow] Reorder to_pandas extension dtype mapping #44720

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+46 −20
Open

GH-39914: [pyarrow] Reorder to_pandas extension dtype mapping #44720

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1e8d4b7
Reorder to_pandas extension dtype mapping
bretttully Nov 13, 2024
cb87068
Added test
bretttully Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 20 additions & 20 deletions python/pyarrow/pandas_compat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -848,6 +848,25 @@ def _get_extension_dtypes(table, columns_metadata, types_mapper=None):
if _pandas_api.extension_dtype is None:
return ext_columns

# use the specified mapping of built-in arrow types to pandas dtypes
if types_mapper:
for field in table.schema:
typ = field.type
pandas_dtype = types_mapper(typ)
if pandas_dtype is not None:
ext_columns[field.name] = pandas_dtype

# infer from extension type in the schema
for field in table.schema:
typ = field.type
if field.name not in ext_columns and isinstance(typ, pa.BaseExtensionType):
try:
pandas_dtype = typ.to_pandas_dtype()
except NotImplementedError:
pass
else:
ext_columns[field.name] = pandas_dtype

# infer the extension columns from the pandas metadata
for col_meta in columns_metadata:
try:
Expand All @@ -856,33 +875,14 @@ def _get_extension_dtypes(table, columns_metadata, types_mapper=None):
name = col_meta['name']
dtype = col_meta['numpy_type']

if dtype not in _pandas_supported_numpy_types:
if name not in ext_columns and dtype not in _pandas_supported_numpy_types:
# pandas_dtype is expensive, so avoid doing this for types
# that are certainly numpy dtypes
pandas_dtype = _pandas_api.pandas_dtype(dtype)
if isinstance(pandas_dtype, _pandas_api.extension_dtype):
if hasattr(pandas_dtype, "__from_arrow__"):
ext_columns[name] = pandas_dtype

# infer from extension type in the schema
for field in table.schema:
typ = field.type
if isinstance(typ, pa.BaseExtensionType):
try:
pandas_dtype = typ.to_pandas_dtype()
except NotImplementedError:
pass
else:
ext_columns[field.name] = pandas_dtype

# use the specified mapping of built-in arrow types to pandas dtypes
if types_mapper:
for field in table.schema:
typ = field.type
pandas_dtype = types_mapper(typ)
if pandas_dtype is not None:
ext_columns[field.name] = pandas_dtype

return ext_columns


Expand Down
26 changes: 26 additions & 0 deletions python/pyarrow/tests/test_pandas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

import gc
import decimal
import io
import json
import multiprocessing as mp
import sys
Expand Down Expand Up @@ -4411,6 +4412,31 @@ def test_to_pandas_extension_dtypes_mapping():
assert isinstance(result['a'].dtype, pd.PeriodDtype)



def test_to_pandas_extension_dtypes_mapping_complex_type():
pa_type = pa.struct(
[
pa.field("bar", pa.bool_(), nullable=False),
pa.field("baz", pa.float32(), nullable=True),
],
)
pd_type = pd.ArrowDtype(pa_type)
schema = pa.schema([pa.field("foo", pa_type)])
df0 = pd.DataFrame(
[
{"foo": {"bar": True, "baz": np.float32(1)}},
{"foo": {"bar": True, "baz": None}},
],
).astype({"foo": pd_type})

# Round trip df0 into df1
with io.BytesIO() as stream:
df0.to_parquet(stream, schema=schema)
stream.seek(0)
df1 = pd.read_parquet(stream, dtype_backend="pyarrow")
Comment on lines +4432 to +4436
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might not need the roundtrip to parquet, but a table = pa.table(df); result = table.to_pandas(types_mapper=pd.ArrowDtype) should be sufficient to test this?

I know this doesn't test exactly pd.read_parquet in its entirety, but it should test the relevant part on the pyarrow side, and an actual pd.read_parquet test can still be added to pandas

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error only gets thrown once the pandas metadata is added to the table. That's why I have used a round-trip test. Is there another way to generate that metadata and set it on the table before calling to_pandas?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The metadata gets added on the pyarrow side, so table = pa.table(df) will do that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pandas to_parquet method essentially just does a table = pa.Table.from_pandas(df) and then writes that to parquet (and pa.table(df) is a shorter less explicit version of that, but you can also use Table.from_pandas)

pd.testing.assert_frame_equal(df0, df1)


def test_array_to_pandas():
if Version(pd.__version__) < Version("1.1"):
pytest.skip("ExtensionDtype to_pandas method missing")
Expand Down
Loading