Can't amend a writable snapshot following a rearrange snapshot

[tomwhite]

What happened?

I'm using amend in order to avoid keeping history, and I want to be able to mix data changes (in writable sessions) and structural modifications (in rearrange sessions). If I do a commit in a writable session then an amend in a rearrange session the snapshop is modified (so there is only one). But if I try to another amend in a writable session, it fails.

What did you expect to happen?

I expected amend to always amend the previous snapshot.

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "icechunk>=2.0.4",
# ]
# ///

import icechunk as ic
import zarr

ic.print_debug_info()

def main() -> None:
    # create a repo with an array
    repo = ic.Repository.create(ic.in_memory_storage())
    session = repo.writable_session("main")
    root = zarr.group(session.store)
    root.create_group("group")
    root.create_array("group/a", shape=(10,), dtype="f4")
    session.commit("create")
    print("\n".join([str((a.id, a.message)) for a in repo.ancestry(branch="main")]))

    # rename the array
    session = repo.rearrange_session("main")
    session.move("/group/a", "/group/b")
    session.amend("move")
    print("\n".join([str((a.id, a.message)) for a in repo.ancestry(branch="main")]))

    # update the array
    session = repo.writable_session("main")
    root = zarr.open_group(session.store, mode="r+")
    group = root["group"]
    arr = group["b"]
    arr[:] = -1
    session.amend("update")
    print("\n".join([str((a.id, a.message)) for a in repo.ancestry(branch="main")]))

if __name__ == "__main__":
    main()

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in icechunk.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of icechunk and its dependencies.

Relevant log output

('SKK7R5EYET9DC0G2VEX0', 'create')
('1CECHNKREP0F1RSTCMT0', 'Repository initialized')
('V5D2PPQD7DAEXP4QXCRG', 'move')
('1CECHNKREP0F1RSTCMT0', 'Repository initialized')
Traceback (most recent call last):
  File "/Users/tom/workspace/vczstore/repro-ic.py", line 39, in <module>
    main()
  File "/Users/tom/workspace/vczstore/repro-ic.py", line 35, in main
    session.amend("update")
  File "/Users/tom/.cache/uv/environments-v2/repro-ic-ffba12c202125ccc/lib/python3.12/site-packages/icechunk/session.py", line 537, in amend
    return self._session.amend(message, metadata, allow_empty=allow_empty)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
icechunk.IcechunkError:   × session error: This session was created to rearrange the hierarchy, other write operations cannot be executed. Commit or abandon the sessions and create a regular writable session
  │ 
  │ context:
  │    0: icechunk::session::commit_inner
  │            with update max_concurrent_nodes=1 rewrite_manifests=false commit_method=Amend allow_empty=false
  │              at icechunk/src/session.rs:1598
  │

Anything else we need to know?

This came up in some code I'm writing to do a simple rechunk. Ideally I'd like to be able to do a rechunk operation in a single commit. See sgkit-dev/vczstore#98

Environment

Details

platform: macOS-14.7-arm64-arm-64bit python: 3.12.0 icechunk: 2.0.4 zarr: 3.2.1 numcodecs: 0.16.5

[mpiannucci]

Hi @tomwhite, This is expected behavior actually, see #1744

The issue is that when you rearrange, the manifests are rewritten. We don't allow amending after this because then the transaction log enters a state where its now lossy.

@paraseba can give more details about possible workarounds

[paraseba]

Yeah, currently, for simplicity of implementation, we never merge transactions that do move and other operations. There are too many edge cases. For this reason we disallow amending a normal session on top of a rearrange (we do allow amending rearranges on top of rearranges iirc). For now the only workaround is doing a normal commit.

[tomwhite]

Thanks - unfortunately in our application we have to be able to guarantee that we don't retain any history at all, but we need icechunk for its transactional control.

Stepping back, what we actually need is a way to change the chunk structure of an array in-place (e.g. merging chunks). Is it possible to do that within a single icechunk amend operation?

[rabernat]

Hi Tom! 👋

We're really excited to support your use case. I think I understand what you want to do: safely swap out an array with one chunk scheme for a rechunked version of the same data.

Your requirement - "guarantee that we don't retain any history at all" - is an interesting one. We have started evolving in this direction with the addition of amend, recognizing that there are users who care about transactions but not about version history.

I'm wondering if you can say a bit more about what's driving that requirement. Is it storage costs? Some sort of compliance thing? Is it acceptable to have some history temporarily which is later squashed? Your answers will help us think of an optimal solution.

[tomwhite]

Hi Ryan!

I'm wondering if you can say a bit more about what's driving that requirement.

The requirement is due to compliance, not storage costs - in the vczstore project we need to be able to say that if a participant requests that their data is removed from the study that no copies are kept. Having history temporarily which is then squashed would meet that requirement I believe.

The feature set and performance of icechunk has been very impressive. Thanks for everyone's work on it!

[rabernat]

Ok, great answer. In this case, here's the workflow we would suggest for in-place rechunking.

1. Create a temporary branch.
2. Do your structural changes via a rearrange session (on that branch). Rename the old array, create the new array. Commit.
3. Now write the rechunked data. Commit.
4. Delete the old array in another rearrange session. Commit.
5. Reset main to point to the temporary branch. Delete temporary branch.
6. Run expiration and garbage collection, erase all history.

Writing that all out, it looks like a lot! 😅 But it will work today and it will meet your requirements.

[ianhi]

File "/Users/tom/.cache/uv/environments-v2/repro-ic-ffba12c202125ccc/lib/python3.12/site-packages/icechunk/session.py", > line 537, in amend
return self._session.amend(message, metadata, allow_empty=allow_empty)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
icechunk.IcechunkError: × session error: This session was created to rearrange the hierarchy, other write operations cannot > be executed. Commit or abandon the sessions and create a regular writable session

We need a better error message here. This should be celar that you aren't allowed allowed to amends MUST amend a session of hte same type. the other types of ops can't be executed is confsuing.

Also one small note on the steps Ryan proposed:

Rename the old array, create the new array.

creating will have to be in a new writeable session. you cannot create new arrays within a writable session.

[ianhi]

The other potentially important gap we have here is that to remove some history you have to remove all history. I.e. I don't think you can GC just the last commit but not prior ones. The exception being that tagged snapshots will never be GCed. So @tomwhite do you want the history of the repo (for reproducibility etc) but basically removing a single snapshot from the history?

I wonder if we could also better support this use case with https://icechunk.io/en/stable/reference/#icechunk.Repository.expire_snapshots accepting an explicit snapshot id instead of just older_than