Add support for OME-Zarr data

[lubianat]

xref #49

Initial draft; NOTE: GH copilot support for syntax while I get familiarized with the details.

Test case is a real-world CC0 dataset (https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BIAD1961). Probably a smaller one would be better before pull.

For the nature of (OME)-Zarr, it includes many files.

I'll be working here as a draft PR for transparency, but comments/tips/changes are welcome at any point.

It is not yet comprehensively tested.

[lubianat]

Currently considering the correct way of modelling Zarr datasets in Croissant, maybe the https://github.com/MIT-LCP/croissant-baker/blob/main/src/croissant_baker/handlers/parquet_handler.py is a good inspiration

[lubianat]

for reference

https://huggingface.co/datasets/LuciexJune/ome-vip-dataset

https://huggingface.co/api/datasets/LuciexJune/ome-vip-dataset/croissant

• Add Zarr / OME-Zarr Dataset Support huggingface/datasets#8135
  and the related
• Add Zarr streaming support (POC) huggingface/datasets#7983

{
  "@type": "https://schema.org/Dataset",
  "http://mlcommons.org/croissant/recordSet": [],
  "http://purl.org/dc/terms/conformsTo": {
    "@language": "en",
    "@value": "http://mlcommons.org/croissant/1.1"
  },
  "https://schema.org/alternateName": [
    {
      "@language": "en",
      "@value": "LuciexJune/ome-vip-dataset"
    },
    {
      "@language": "en",
      "@value": "Zarr / OME-Zarr Data Repository"
    }
  ],
  "https://schema.org/creator": {
    "@type": "https://schema.org/Person",
    "https://schema.org/name": {
      "@language": "en",
      "@value": "Biswanath Saha"
    },
    "https://schema.org/url": {
      "@language": "en",
      "@value": "https://huggingface.co/LuciexJune"
    }
  },
  "https://schema.org/description": {
    "@language": "en",
    "@value": "This image data has been taken from the IDR OME-NGFF Samples- Catalog of IDR images formatted as OME-NGFF (https://idr.github.io/ome-ngff-samples/)\nSample id: ExpA_VIP_ASLM_on.zarr\n  This dataset is used to show the viability of Zarr image streaming and uploading using 🤗 datasets library corresponding to the PR: [Add Zarr / OME-Zarr Dataset Support\n]\n\n\t\n\t\t\n\t\n\t\n\t\tZarr / OME-Zarr Data Repository\n\t\n\nThis repository stores Zarr / OME-Zarr array data for large-scale scientific imaging workflows.… See the full description on the dataset page: https://huggingface.co/datasets/LuciexJune/ome-vip-dataset."
  },
  "https://schema.org/distribution": {
    "@id": "repo",
    "@type": "http://mlcommons.org/croissant/FileObject",
    "https://schema.org/contentUrl": {
      "@language": "en",
      "@value": "https://huggingface.co/datasets/LuciexJune/ome-vip-dataset/tree/refs%2Fconvert%2Fparquet"
    },
    "https://schema.org/description": {
      "@language": "en",
      "@value": "The Hugging Face git repository."
    },
    "https://schema.org/encodingFormat": {
      "@language": "en",
      "@value": "git+https"
    },
    "https://schema.org/name": {
      "@language": "en",
      "@value": "repo"
    },
    "https://schema.org/sha256": {
      "@language": "en",
      "@value": "https://github.com/mlcommons/croissant/issues/80"
    }
  },
  "https://schema.org/keywords": [
    {
      "@language": "en",
      "@value": "apache-2.0"
    },
    {
      "@language": "en",
      "@value": "Datasets"
    },
    {
      "@language": "en",
      "@value": "🇺🇸 Region: US"
    },
    {
      "@language": "en",
      "@value": "zarr"
    },
    {
      "@language": "en",
      "@value": "ome-zarr"
    },
    {
      "@language": "en",
      "@value": "microscopy"
    },
    {
      "@language": "en",
      "@value": "bioimaging"
    },
    {
      "@language": "en",
      "@value": "scientific-imaging"
    },
    {
      "@language": "en",
      "@value": "multidimensional-arrays"
    },
    {
      "@language": "en",
      "@value": "streaming"
    },
    {
      "@language": "en",
      "@value": "datasets"
    }
  ],
  "https://schema.org/license": {
    "@language": "en",
    "@value": "https://choosealicense.com/licenses/apache-2.0/"
  },
  "https://schema.org/name": {
    "@language": "en",
    "@value": "ome-vip-dataset"
  },
  "https://schema.org/url": {
    "@language": "en",
    "@value": "https://huggingface.co/datasets/LuciexJune/ome-vip-dataset"
  }
}
---