@richard-berg thanks for the issue.

import numpy as np
import pandas as pd
import xarray as xr
index1 = np.array([1, 2, 3])
index2 = np.array([1, 2, 4])
srs = pd.Series(index=index1, data=1).convert_dtypes()
arr = srs.to_xarray()
arr + 5

works for me on main maybe because of #10278. But I'm not sure it's the same as your example since you have x there in the addition example.

As for the text coverage, I agree we should increase it. I will add the cases you raise here but since you appear to be aware of more, I would love more guidance. If you look through my PRs here, it's a bit of whack-a-mole because while I think I am using relatively sound practices as I go, xarray has a lot of edge cases in its API that I am not familiar with. If you're aware of some, it would be great to handle them.

I would be opposed to somehow going around special casing because now we actually do let through datetimes as well as interval arrays (which are both tested even if it is not immediately obvious). The reason I had to do #9042 was exactly because all of the special casing that existed before was so complex that unraveling it required a massive PR.

So with special casing, we would be looking at categoricals, interval types, and datetimes passed through and only numerics excluded, until another type comes along. So I'd somewhat rather be very clear here that everything passes through.

P.S I see:

TypeError: 'PandasExtensionArray' object is not callable

And we just made it so that PandasExtensionArray is no longer part of the public API. So hopefully this condition is not something that can be run into

Whoops, int float and string should be converted to the numpy types. Looks like we lost this somehow and aren't testing it :(

While I could see floats being a good idea, nullable integers do not exist in numpy:

import numpy as np


In [3]: np.array([np.nan, 1, 2], dtype="int32")

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[3], line 1
----> 1 np.array([np.nan, 1, 2], dtype="int32")

ValueError: cannot convert float NaN to integer

Thanks for checking, and for the quick fillna patch! FYI I do have a robust fix that handles all of the above in a general way, while preserving strong types (including nullable ints). Just may take a few days to get legal approval for (and work thru the mechanical details of) exfiltrating code off my network.

nullable integers do not exist in numpy:

I know but this is what we do with numpy masked arrays today.

Ideally we'd convert to https://github.com/mdhaber/marray in the near future.

Regardless, for now we'd like to enable any "extra" dtypes (categorical, intervals) while using the numpy dtypes as much as we can.

Ideally we'd convert to https://github.com/mdhaber/marray in the near future.

Interesting!

@richard-berg I would be interested why not just limit this behavior to from_dataframe i.e., users who wish to convert into an xarray object from pandas will lose the type but going the other way would be disabled. We rely on integer arrays and boolean arrays in https://github.com/scverse/anndata and projects like https://geopandas.org/en/stable/ rely on geometry extension types. So users constructing xarray objects should be allowed to continue to do that, no? And then they can exit xarray with their types in tact?

Regardless, for now we'd like to enable any "extra" dtypes (categorical, intervals) while using the numpy dtypes as much as we can.

+1 here given the above example of geometry which is not in pandas core but is widely used int he geospatial community.

I should say @richard-berg been thinking about this since you posted and I really appreciate your contribution and effort here btw! Looking forward to seeing you PR :)))) I love to "users" get involved with open source, wish it was easier given corporate situations sometimes. Thanks a million again!

@ilan-gold finally got past the firewall!

I see there's been lots of related activity in the last few weeks -- while I catch up on all the recent commits & PR discussions, mind glancing at #10380 to assess which changes are (a) still relevant (b) a reasonable direction to evolve the codebase? No sense slogging thru merge conflicts that'll just need to be backed out after feedback...

Looking now!

On first glance, I think this is a great PR. I think it's more opinionated and/or thorough than mine #10304, but seems to do roughly the same thing (I think the co. The find_result_type from pandas is awesome, didn't know about it.

It also has some other nice-ities I really like as well, so in general in favor of this PR (even though I'm not an official maintainer here, although I do appear to be pinged every time something happens with this feature haha). I've been splitting my PRs up to make it clear what changes are exactly needed to fix which issue and why, and that helps with reviewing.

For example, for this original issue around re-indexing, I think only a subset of the PR you opened is needed, but maybe I'm wrong. For example is https://github.com/pydata/xarray/pull/10380/files#diff-ca5c2de2fe6e9e25fbf22bd53e4976c15da74900dfb14deb7e6e87f5377230e3R7292-R7296 relevant to this issue specifically?

But the rest of it is great, at first glance the categorical stuff resolves #10247 which was opened by a contributor to our library: https://github.com/scverse/anndata and is of course super applicable across domains (categoricals are so much faster than strings for operations where they make sense). There should be a link in that PR to the code we have in our codebase for handling the different categoricals right now, so would be great to have that done as well inside xarray.

Thanks so much!

UPDATE: I see now in your description (bad habit of going right to the code) you say clearly that the PR goes beyond this issue. But still would be good to understand what is needed for this bugfix vs. others