BUG: read_excel index-name detection for non-contiguous MultiIndex header

[aleks-drozy]

Investigation

While looking into a reported read_excel issue with header=[0, 2], I found the attached .xlsx reproduction file was internally corrupted: its own sharedStrings.xml is missing the string "h2", so cell B1 ends up pointing at the same shared-string index as A1 -- the file simply does not contain the data its screenshot implies. Re-creating a clean .xlsx with the same claimed values (header=[0, 2], non-contiguous, matching column layout) and reading it with read_excel produces output that matches read_csv on equivalent data exactly. So the originally reported symptom looks like a corrupted-attachment artifact, not a live bug.

The real, adjacent bug this PR fixes

While investigating, I found a genuine bug in the same code path: BaseExcelReader._parse_sheet in pandas/io/excel/_base.py.

For a non-contiguous header list (e.g. header=[0, 2]), the code that decides whether the row right after the header holds MultiIndex index names did:

if len(header) < len(data):
    potential_index_names = data[len(header)]

len(header) (2) is not the row index immediately following the last header row for a non-contiguous header -- max(header) + 1 (3) is. So this looked at data[2], which is the second header row itself, instead of data[3], the actual row after the header. That corrupts has_index_names detection, which in turn shifts the forward-fill offset used for a MultiIndex index_col by one row, and can leak the index-name text into the actual index values.

The fix mirrors the offset = 1 + max(header) computation a few lines below in the very same function, which already handles non-contiguous headers correctly for the forward-fill offset itself -- this brings the index-name-row lookup a few lines above it in line with that existing, correct logic.

Testing

Added test_read_excel_noncontiguous_header_index_names in pandas/tests/io/excel/test_readers.py, using a minimal .xlsx built with openpyxl in the test itself (not a corrupted attachment). It uses header=[0, 2] with a MultiIndex index_col, an intervening non-header row, and a genuine index-name row, and asserts the resulting index values/names are correct.

Confirmed the test fails before this fix (the corrupted has_index_names detection leaks the index-name text "ilvl1" into an actual index value that should stay blank/NaN) and passes after it. Ran the full pandas/tests/io/excel/ slice locally; no regressions (the only failures/errors present are pre-existing on main, unrelated to this change -- an openpyxl version mismatch in an unrelated writer test and a network-dependent test_read_from_http_url test).

Added a whatsnew entry under doc/source/whatsnew/v3.1.0.rst (I/O section).

[github-actions]

Thanks for the pull request, @aleks-drozy! It's linked to #66373, but that issue isn't assigned to you yet. To make sure two people don't unknowingly work on the same thing, we ask contributors to claim an issue first. Just comment /take on #66373 to claim it, and you're good to go. See the contributing guide for the full flow.

[kuraga]

@aleks-drozy , thanks very much about investigation!

1. Yes, BUG: read_excel (with tupled header): incorrect .columns #66373 was incorrect.
   You can safely remove comments on it from this PR, can't you?

2. Should we document this (correct) behavior?

3. (BTW, there was a simillar question at API: read_excel: .index.names vs .columns.names #66377 (comment).)

4. And interesting, does read_csv have the same behavior?

Thanks.

[aleks-drozy]

Thanks for the review, @kuraga.

1. Done -- the BUG: read_excel (with tupled header): incorrect .columns #66373-specific framing is removed from the description; it now just describes the investigation and the actual bug/fix without referencing that issue.

2. Happy to add a short docstring note if you think it's worth it. The rule worth documenting: when header is a list and index_col is given, the index-name row is the row immediately after the last header row (max(header) + 1); rows between non-contiguous header rows are skipped. I'd add a sentence to that effect under the header parameter in the read_excel docstring (mirroring the same rule already implicit in read_csv's). Let me know if that's the scope you want and I'll push it as a follow-up commit here.

3. Will take a look at the API: read_excel: .index.names vs .columns.names #66377 thread you linked -- looks like the same len-vs-position confusion family; the fix here brings read_excel in line with the position-based convention discussed there.

4. No, read_csv does not share this bug -- it was already correct on both engines. I ran the exact CSV analog of this PR's xlsx repro (same shape: header rows at 0 and 2, a skip row at 1, an index-name row at 3, data with a blank first-level index cell) with header=[0, 2], index_col=[0, 1]:

=== engine=c ===
            foo
              a  b
ilvl1 ilvl2
nan   p       1  2
y     q       3  4
index: [(nan, 'p'), ('y', 'q')] names: ['ilvl1', 'ilvl2']
engine=c: MATCHES expected (correct behavior)

=== engine=python ===
(identical)
engine=python: MATCHES expected (correct behavior)

Both engines produce clean index names/values with no leakage or shift. That's because read_csv locates the index-name row by physical position relative to the last header row (header[-1] + 1) rather than by count of header rows -- the python engine appends it via header = [*list(header), header[-1] + 1] (python_parser.py:587), the C tokenizer does the same in parsers.pyx:633, and it's popped off as the index names in BaseParser._extract_multi_indexer_columns (base_parser.py:241). read_excel instead indexed the materialized sheet with data[len(header)], which only equals max(header) + 1 when the header list is contiguous from 0 -- that's the bug this PR fixes, and it brings read_excel in line with the convention the CSV parsers already followed.