String arrays written with zarr python 3 zarr spec 2 cannot be read in zarr python 2

[msschwartz21]

Zarr version

v2.18.7, v3.0.8

Numcodecs version

v0.15.1

Python Version

3.12.10

Operating System

Mac

Installation

Pixi from conda

Description

Current behavior:

• Write an array of strings using zarr python 3 and zarr format 2
• The same array cannot be read using zarr python 2

Traceback (most recent call last):
  File "/Users/schwartzm10/Code/geff/sandbox/zarr_string.py", line 33, in <module>
    read(path)
  File "/Users/schwartzm10/Code/geff/sandbox/zarr_string.py", line 20, in read
    ids = z['strings'][:]
          ~~~~~~~~~~~~^^^
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 799, in __getitem__
    result = self.get_basic_selection(pure_selection, fields=fields)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 925, in get_basic_selection
    return self._get_basic_selection_nd(selection=selection, out=out, fields=fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 967, in _get_basic_selection_nd
    return self._get_selection(indexer=indexer, out=out, fields=fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 1342, in _get_selection
    self._chunk_getitems(
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 2187, in _chunk_getitems
    self._process_chunk(
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 2100, in _process_chunk
    chunk = self._decode_chunk(cdata)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/zarr/core.py", line 2370, in _decode_chunk
    chunk = chunk.view(self._dtype)
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/schwartzm10/Code/geff/.pixi/envs/zarr2/lib/python3.12/site-packages/numpy/_core/_internal.py", line 565, in _view_is_safe
    raise TypeError("Cannot change data-type for array of references.")
TypeError: Cannot change data-type for array of references.

Steps to reproduce

write.py

# /// script
# requires-python = ">=3.12,<3.13"
# dependencies = [
#   "zarr>3,<4",
# ]
# ///
import zarr
import numpy as np


if __name__ == "__main__":
    ids = np.array([f"1_{i}" for i in range(1000)])

    print(zarr.__version__)

    z = zarr.open(f'test.zarr', mode='a', zarr_format=2)
    z['strings'] = ids

read.py

# /// script
# requires-python = ">=3.12,<3.13"
# dependencies = [
#   "zarr>=2,<3",
# ]
# ///
import zarr

if __name__ == "__main__":
    z = zarr.open('test.zarr')
    ids = z['strings'][:]
    print(ids)

Additional output

No response

[d-v-b]

@msschwartz21 thanks for the report, this should have been fixed by #3100, can you run your scripts against the main branch to confirm?

[d-v-b]

for context, zarr-python 2 is complaining because it has decoded a chunk into a numpy object dtype array, and the reason it did that is because zarr-python 3 was erroneously inserting a vlen-utf8 codec into the filters for arrays with dtype U. Because the numpy U dtype is fixed-size, there is no reason to encode it with vlen-utf8, so removing that behavior brings back zarr-python 2 compatibility.

[msschwartz21]

@d-v-b you're right! Installing from main fixed the bug. I'll keep an eye out for the next release.