Support dimension_separator

[orenbenkiki]

Using "/" would allow using M directories w/ N files per directory instead of M*N files in a single directory, which matters for large matrices. This is allowed by the Zarr spec. Arguably "/" would have been a better default than ".", but at least allow overriding it.

[lazarusA]

Hi,

could you share more details of what is not working? We have tests for that case (I think):

Zarr.jl/test/v3_julia.jl

Line 27 in 56eda22

dimension_separator='/')

so, I'm not sure I follow fully.

[orenbenkiki]

My bad. I was using 0.9.6 - this is fixed in 0.10. That said... what I really want to use is the shard file (and not one file per chunk on disk). Which means using V3. It reorganized the pipeline (array→array → array→bytes → bytes→bytes codec slots to replace v2's filters → compressor), and Zarr.jl 0.10's v3 codec registry is sparse. Three concrete gaps that bite me:

1. No vlen-utf8 v3 codec. v2 had VLenUTF8Filter auto-applied for String arrays. v3's BytesCodec is the default array→bytes codec and refuses String input. → I generated a local VLenUTF8V3Codec (see below).
2. No standalone shuffle codec. v2 has ShuffleFilter (used by :gzip_shuffle). v3 has no shuffle codec, only the shuffle setting bundled inside BloscV3Codec. → :gzip_shuffle errors on my v3 backend.
3. No standalone bitshuffle codec. Same story — bitshuffle is only available inside blosc. → :zstd_bitshuffle errors (and was already rejected in v2 too, just for a different reason).

I assume this will all get fixed/implemented eventually... thanks for doing this project!

# ==============================================================================
# BEGIN: temporary `vlen-utf8` v3 codec.
#
# Zarr.jl 0.10's V3 codec registry has no `vlen-utf8` entry, and its v3 metadata
# constructor silently drops the `filters` kwarg, so writing a Zarr v3 array of
# `String` fails (the default `BytesCodec` cannot reinterpret `String`). The
# codec below mirrors the wire format of Zarr.jl's v2 `VLenUTF8Filter` (numcodecs
# `vlen-utf8`) so v3 stores written here round-trip through the same standard
# encoding. The matching `string_zcreate` helper bypasses Zarr.jl's high-level
# `zcreate` to construct `MetadataV3` with this codec in the pipeline.
#
# `__init__` checks at module load whether Zarr.jl now ships its own `vlen-utf8`
# v3 codec; if so, it logs a warning and skips the local registration. When that
# warning fires in a release of `Zarr.jl`, this entire block (codec, helper, and
# the `__init__` arm) should be deleted in favour of the upstream version.
# ==============================================================================

struct VLenUTF8V3Codec <: Zarr.Codecs.V3Codecs.V3Codec{:array, :bytes}
end

Zarr.Codecs.V3Codecs.name(::VLenUTF8V3Codec) = "vlen-utf8"
Zarr.Codecs.V3Codecs.is_fixed_size(::VLenUTF8V3Codec) = false

JSON.lower(::VLenUTF8V3Codec) =
    Dict{String, Any}("name" => "vlen-utf8", "configuration" => Dict{String, Any}())

function Zarr.Codecs.V3Codecs.codec_encode(::VLenUTF8V3Codec, data::AbstractArray)::Vector{UInt8}
    buffer = IOBuffer()
    write(buffer, UInt32(length(data)))
    for element in data
        utf8_bytes = transcode(String, String(element))
        write(buffer, UInt32(ncodeunits(utf8_bytes)))
        write(buffer, utf8_bytes)
    end
    return take!(buffer)
end

function Zarr.Codecs.V3Codecs.codec_decode(
    ::VLenUTF8V3Codec,
    encoded::Vector{UInt8},
    ::Type{T},
    shape::NTuple{N, Int};
    fill_value = nothing,
)::Array{String, N} where {T, N}
    buffer = IOBuffer(encoded)
    nitems = Int(read(buffer, UInt32))
    output = Array{String, N}(undef, shape)
    for index in 1:nitems
        len = Int(read(buffer, UInt32))
        output[index] = String(read(buffer, len))
    end
    close(buffer)
    return output
end

# Build a v3 `String` ZArray under `group` at `name` with the given shape, with the local
# `VLenUTF8V3Codec` as the array→bytes step and no compression.
function string_zcreate(group::ZGroup, name::AbstractString, shape::NTuple{N, Int})::ZArray{String, N} where {N}
    pipeline = Zarr.V3Pipeline((), VLenUTF8V3Codec(), ())
    metadata = Zarr.MetadataV3{String, N, typeof(pipeline)}(
        3,
        "array",
        shape,
        shape,
        "string",
        pipeline,
        "",
        Zarr.ChunkKeyEncoding('/', true),
    )
    storage = group.storage
    array_path = Zarr._concatpath(group.path, String(name))
    Zarr.isemptysub(storage, array_path) || error("non-empty Zarr path: $(array_path)")  # UNTESTED
    Zarr.writemetadata(Zarr.ZarrFormat(3), storage, array_path, metadata)
    Zarr.writeattrs(Zarr.ZarrFormat(3), storage, array_path, Dict{String, Any}())
    array = ZArray(metadata, storage, array_path, Dict{String, Any}(), true)
    group.arrays[String(name)] = array
    return array
end

function __init__()
    if haskey(Zarr.Codecs.V3Codecs.codec_parsers, "vlen-utf8")
        @warn(
            "Zarr.jl now ships a built-in `vlen-utf8` v3 codec. The local copy in " *
            "`DataAxesFormats.ZarrFormat` (the BEGIN/END `vlen-utf8` block in " *
            "`src/zarr_format.jl` plus this branch of `__init__`) should be removed in " *
            "favour of the upstream version."
        )
    else
        Zarr.Codecs.V3Codecs.register_codec("vlen-utf8", VLenUTF8V3Codec) do _config, _ctx
            return VLenUTF8V3Codec()
        end
    end
    Zarr.typemap3["string"] = String
    return nothing
end

# ==============================================================================
# END: temporary `vlen-utf8` v3 codec.
# ==============================================================================

Oh, I also needed:

 # Zarr.jl 0.10's v3 metadata parser computes `sizeof(T)` unconditionally at `metadata3.jl:232` for the
 # blosc-typesize default of the codec context, which throws for non-bits types. The value is irrelevant on
 # our path because every blosc instance we emit carries an explicit `typesize`. Returning `1` byte makes the
 # parser accept v3 stores whose `data_type` resolves to `String`. Type piracy on `Base.sizeof` is bounded —
 # `sizeof(::Type{String})` is otherwise undefined in Julia and is virtually never called outside this Zarr.jl
 # code path. Drops together with the rest of the block when Zarr.jl ships its own `vlen-utf8` codec.
 Base.sizeof(::Type{String}) = 1

[lazarusA]

Indeed, the full pipeline (+shards) was just recently implemented by @mkitti and merged for the latest release. And, yes now it comes the users feedback, of course doing PRs could help move things forward faster.

[mkitti]

For new codecs, please propose them as a zarr-extension at this repository:
https://github.com/zarr-developers/zarr-extensions

Also note that the Rust implementation has been wrapped:
https://github.com/earth-mover/Zarrs.jl

Extensions should be implementation as 3rd party packages eventually. I think we have implementated registration for codecs and chunk-key-extensions.

[mkitti]

Base.sizeof(::Type{String}) = 1

This is type piracy. We probably need to implement our own function for this purpose.

[orenbenkiki]

I'd love to but I'm drowning in my own codebase right now :-( Zarr is only one of my backends... As for the type piracy, agreed, it is bad. You could make the default size nothing (instead of the sizeof) and in the body to fill in the default sizeof only if no explicit size was given...

[mkitti]

I propose that ee break this out into several issues:

1. Document sharding
2. Document how to extend Zarr
3. Vlen-utf8 codec
4. ByteShuffle codec
5. BitShuffle codec
6. Blosc codec fixes for strings

[orenbenkiki]

Oh, this is an issue as well:

function __init__()
    if haskey(Zarr.Codecs.V3Codecs.codec_parsers, "vlen-utf8")
        @warn(
            "Zarr.jl now ships a built-in `vlen-utf8` v3 codec. The local copy in " *
            "`DataAxesFormats.ZarrFormat` (the BEGIN/END `vlen-utf8` block in " *
            "`src/zarr_format.jl` plus this branch of `__init__`) should be removed in " *
            "favour of the upstream version."
        )
    else
        Zarr.Codecs.V3Codecs.register_codec("vlen-utf8", VLenUTF8V3Codec) do _config, _ctx
            return VLenUTF8V3Codec()
        end
    end
    Zarr.typemap3["string"] = String
    # Inject a v3-correct `getattrs` for `ConsolidatedStore` at runtime: Zarr.jl 0.10's stock implementation reads the
    # v2 `.zattrs` key from the consolidated dict, which never appears in a v3 `.zmetadata`; v3 attributes live inside
    # the consolidated `zarr.json` under the `attributes` field. The injection runs through `@eval` because Julia
    # forbids method overwriting at top-level during precompilation. Drops together with the rest of the temporary
    # block when Zarr.jl ships a v3-correct implementation.
    @eval Zarr begin
        function getattrs(::ZarrFormat{3}, store::ConsolidatedStore, path)
            json_key = _unconcpath(store, path, "zarr.json")
            if !haskey(store.cons, json_key)
                return Dict{String, Any}()
            end
            return get(store.cons[json_key], "attributes", Dict{String, Any}())
        end
    end
    return nothing
end

[mkitti]

Are you referring to the consolidated metadata portion as the new issue? We've had some difficulty figuring out what the standard is supposed to be there.

See #261 and zarr-developers/zarr-specs#309 .

If there are public datasets that use consolidated metadata that you need to access, we would appreciate a reference to them.

[orenbenkiki]

Yes, we want consolidated metadata same like we had in v2, e.g. for serving HTTP. I now do this manually... and once consolidated data is implemented, getattr needs to be updated. Right now, both consolidate_metadata and getattrs(::ZarrFormat{3}, ::ConsolidatedStore, p) are v2-shaped — the writer walks .zattrs and the reader queries .zattrs. Together they form a consistent no-op on v3, so the bug doesn't manifest as data loss in stock Zarr.jl. Implementing v3 consolidation (writer side) without fixing the reader would expose the bug.

This isn't for an existing dataset(s), it is for a Zarr back-end for my specialized storage library.

[mkitti]

There are unfortunately unresolved ambiguities with consolidated metadata in Zarr 3. For example, an outstanding question is does the consolidated metadata occur as a top-level key as in #261 or under attributes as you mentioned above. To implement it I need one of the following:

1. A specification
2. Emperical datasets
3. A reference implementation to compare compatibility with (zarr-python or zarrs, perhaps?)

[orenbenkiki]

Wouldn't (3) resolve this? Isn't it defined by the Zarr spec? I'm not a Zarr expect by any means...

[lazarusA]

I think getattrs has a bug. Here,

Zarr.jl/src/Storage/consolidated.jl

Line 41 in 56eda22

return get(d.cons, _unconcpath(d, p, ".zattrs"), Dict{String,Any}())

shouldn't be .zattrs but some .json , as proposed above.

[orenbenkiki]

It seems the v3 spec explicitly (intentionally?) leaves consolidated out, unlike the v2 spec. So it isn't a "core spec" feature because (I assume good?) "reasons". Python seems to implement it anyway "for read heavy use cases", which is my case. They put it in a consolidated_metadata in the zarr.json of the root group, which is different from having a separate .zmetadata (which was the v2 approach?). zarrs does the same. So... I guess doing the same thing here would make sense? There are 2 other implementations to compare against for tests etc.

[mkitti]

Isn't it defined by the Zarr spec?

For attempts to do so, see the following.

• consolidated metadata storage for Zarr v3 zarr-developers/zarr-specs#136
• Added consolidated metadata to spec zarr-developers/zarr-specs#309

Thinking about it, someone should just codify the current practice as a Zarr Convention.