Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: iloc with Series as indexer fails for __getitem__ but works with __setitem__ #60994

Open
3 tasks done
luxedo opened this issue Feb 23, 2025 · 7 comments · May be fixed by #61162
Open
3 tasks done

BUG: iloc with Series as indexer fails for __getitem__ but works with __setitem__ #60994

luxedo opened this issue Feb 23, 2025 · 7 comments · May be fixed by #61162
Assignees
@arthurlw
Labels
Bug Indexing Related to indexing on series/frames, not to indexes themselves

Comments

@luxedo
Copy link

luxedo commented Feb 23, 2025 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# __getitem__
>>> a = pd.Series([0, 1, 2])
>>> a.iloc[pd.Series([True, False, False])]
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[8], line 1
----> 1 a.iloc[pd.Series([True, False, False])]

File ~/.local/share/hatch/env/virtual/lontras/VBVTu9RT/lontras/lib/python3.11/site-packages/pandas/core/indexing.py:1191, in _LocationIndexer.__getitem__(self, key)
   1189 maybe_callable = com.apply_if_callable(key, self.obj)
   1190 maybe_callable = self._check_deprecated_callable_usage(key, maybe_callable)
-> 1191 return self._getitem_axis(maybe_callable, axis=axis)

File ~/.local/share/hatch/env/virtual/lontras/VBVTu9RT/lontras/lib/python3.11/site-packages/pandas/core/indexing.py:1738, in _iLocIndexer._getitem_axis(self, key, axis)
   1735     key = np.asarray(key)
   1737 if com.is_bool_indexer(key):
-> 1738     self._validate_key(key, axis)
   1739     return self._getbool_axis(key, axis=axis)
   1741 # a list of integers

File ~/.local/share/hatch/env/virtual/lontras/VBVTu9RT/lontras/lib/python3.11/site-packages/pandas/core/indexing.py:1578, in _iLocIndexer._validate_key(self, key, axis)
   1576 if hasattr(key, "index") and isinstance(key.index, Index):
   1577     if key.index.inferred_type == "integer":
-> 1578         raise NotImplementedError(
   1579             "iLocation based boolean "
   1580             "indexing on an integer type "
   1581             "is not available"
   1582         )
   1583     raise ValueError(
   1584         "iLocation based boolean indexing cannot use "
   1585         "an indexable as a mask"
   1586     )
   1587 return

NotImplementedError: iLocation based boolean indexing on an integer type is not available

# __setitem__
>>> a.iloc[pd.Series([True, False, False])] = 10; a
0    10
1     1
2     2
dtype: int64

Issue Description

Behavior of loc with a Series as argument shows inconsistent behavior for __getitem__ and __setitem__

Expected Behavior

Either both __getitem__ and __setitem__ should work or both should fail

Installed Versions

INSTALLED VERSIONS ------------------ commit : 0691c5c python : 3.11.10 python-bits : 64 OS : Linux OS-release : 6.12.10-arch1-1 Version : #1 SMP PREEMPT_DYNAMIC Sat, 18 Jan 2025 02:26:57 +0000 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 2.2.2
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.3.1
Cython : None
sphinx : 8.1.3
IPython : 8.31.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.5
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None
@luxedo luxedo added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Feb 23, 2025
@arthurlw
Copy link
Contributor

arthurlw commented Mar 3, 2025

take

@rhshadrach
Copy link
Member

rhshadrach commented Mar 5, 2025 •
edited
Loading

Thanks for the report. This was discussed in #3631, I think the conclusion from that was we'd be okay supporting Boolean masks in .iloc. I believe this is why it was closed by raising a NotImplemented (yet) error.

#3631 (comment)

It seems to me __setitem__ should also raise here.
It seems to me __getitem__ should also work here.

@rhshadrach rhshadrach added Indexing Related to indexing on series/frames, not to indexes themselves and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 5, 2025
@arthurlw arthurlw mentioned this issue Mar 9, 2025
Closed
5 tasks
@mroeschke
Copy link
Member

@rhshadrach if the conclusion is that boolean masks should be supported in iloc then the current iloc.__setitem__ behavior is OK and the resolution would be to implement iloc.__getitem__?

@arthurlw
Copy link
Contributor

arthurlw commented Mar 10, 2025 •
edited
Loading

I also think implementing iloc.__getitem__ would be the better solution for consistency. Changing iloc.__setitem__ to raise an error seems to have affected other parts of the codebase in my testing.

@rhshadrach
Copy link
Member

@mroeschke - indeed, I somehow negated the conclusion I was going for. Agreed __getitem__ should work here. I've edited my comment above.

@arthurlw arthurlw linked a pull request Mar 21, 2025 that will close this issue
Open
5 tasks
@rhshadrach
Copy link
Member

@jbrockmendel - any opposition to having .iloc work with Boolean-valued masks?

@jbrockmendel
Copy link
Member

if by "mask" you mean ndarray, then im totally on board. if you mean Series[bool] then im concerned about the case where key has a non-matching index.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arthurlw arthurlw

Labels
Bug Indexing Related to indexing on series/frames, not to indexes themselves
Projects
None yet
Milestone
No milestone
5 participants
@jbrockmendel @mroeschke @luxedo @rhshadrach @arthurlw