[Dataset hub] more than storage: streaming, editing, connectors #2052

lhoestq · 2025-11-13T18:39:08Z

streaming
editing (cc @cfahlgren1 )
connectors

I also touched the libraries page a bit cc @davanstrien

HuggingFaceDocBuilderDev · 2025-11-13T18:41:01Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lhoestq · 2025-11-14T10:18:44Z

docs/hub/datasets-streaming.md

+<div class="flex justify-center">
+<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage.png"/>
+<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-dark.png"/>
+</div>


maybe we should have a toggle or something to show the Streaming code in addition to the Download code ? wdyt @cfahlgren1 ?

julien-c

this is great 🔥 important stuff

julien-c · 2025-11-14T18:02:14Z

docs/hub/datasets-downloading.md

 ## Integrated libraries

-If a dataset on the Hub is tied to a [supported library](./datasets-libraries), loading the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/Samsung/samsum?library=datasets) shows how to do so with 🤗 Datasets below.
+If a dataset on the Hub is tied to a [supported library](./datasets-libraries), loading the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.


julien-c · 2025-11-14T18:06:30Z

docs/hub/datasets-streaming.md

+
+## Integrated libraries
+
+If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.


Suggested change

If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.

If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`knkarthick/samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.

julien-c · 2025-11-14T18:07:26Z

docs/hub/datasets-streaming.md

+Streaming is especially useful to read big files on Hugging Face progressively or only small portion.
+For example `tarfile` can iterate on the files of TAR archives, `zipfile` can read files from ZIP archives and `pyarrow` can access row groups of Parquet files.
+
+>![TIP]


Suggested change

>![TIP]

> ![TIP]

julien-c · 2025-11-14T18:09:25Z

docs/hub/datasets-streaming.md

+from huggingface_hub import HfFileSystem
+
+fs = HfFileSystem()
+
+repo_id = "allenai/c4"
+path_in_repo = "en/c4-train.00000-of-01024.json.gz"
+
+# Stream the file
+with fs.open(f"datasets/{repo_id}/{path_in_repo}", "r", compression="gzip") as f:
+    print(f.readline())  # read only the first line
+    # {"text":"Beginners BBQ Class Taking Place in Missoula!...}


(nit, DX) any way to make the snippet even more simpler/more compact? for instance do we need to instantiate a HfFileSystem or we could have syntactic sugar maybe?

julien-c · 2025-11-14T18:11:28Z

docs/hub/datasets-streaming.md

+
+Parquet is a great format for AI datasets. It offers good compression, a columnar structure for efficient processing and projections, and multi-level metadata for fast filtering, and is suitable for datasets of all sizes.
+
+Parquet files are divided in row groups that are often around 100MB each. This lets data loaders and data processing frameworks stream data progressively, iterating on row groups.


we do need a doc page about CDC Parquet btw!! cc @kszucs too

julien-c · 2025-11-14T18:12:58Z

docs/hub/datasets-streaming.md

+
+### Efficient random access
+
+Row groups are further divied into columns, and columns into pages. Pages are often around 1MB and are the smallest unit of data in Parquet, since this is where compression is applied. Accessing pages enables loading specific rows without having to load a full row group, and is possible if the Parquet file has a page index. However not every Parquet frameworks supports reading at the page level. PyArrow doesn't for example, but the `parquet` crate in Rust does:


add a good visual of Parquet file format, and btw we should have a /docs/hub/parquet page probably...

julien-c · 2025-11-14T18:15:00Z

docs/hub/datasets-libraries.md

+
+### Optimized Parquet files
+
+Parquet files on Hugging Face are optimized to improve storage efficiency, accelerate downloads and uploads, and enable efficient dataset streaming and editing:


add a visual of how those files or datasets are marked on the Hub 😁

hint hint @lhoestq =)

julien-c · 2025-11-14T18:18:23Z

docs/hub/datasets-editing.md

+
+## Using the `huggingface_hub` client library
+
+The rich features set in the `huggingface_hub` library allows you to manage repositories, including editing dataset files on the Hub. Visit [the client library's documentation](/docs/huggingface_hub/index) to learn more.


should embed at least an example of two (same in the "downloading" doc btw)

julien-c · 2025-11-14T18:19:47Z

docs/hub/datasets-editing.md

+Therefore the amount of data to reupload depends on the edits and the file structure.
+
+The Parquet format is columnar and compressed at the page level (pages are around ~1MB).
+We optimized Parquet for Xet with [Parquet Content Defined Chunking](https://huggingface.co/blog/parquet-cdc), which ensures unchanged data generally result in unchanged pages.


add a visual of how those files and/or datasets are marked on the Hub 😁

lhoestq added 4 commits November 13, 2025 19:37

index

637c099

wip

9a13dab

update libraries page

a607ad2

datasets streaming

97fe466

lhoestq commented Nov 14, 2025

View reviewed changes

datasets editing

fd043a2

julien-c reviewed Nov 14, 2025

View reviewed changes


		## Integrated libraries

		If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.


		Parquet is a great format for AI datasets. It offers good compression, a columnar structure for efficient processing and projections, and multi-level metadata for fast filtering, and is suitable for datasets of all sizes.

		Parquet files are divided in row groups that are often around 100MB each. This lets data loaders and data processing frameworks stream data progressively, iterating on row groups.


		### Efficient random access

		Row groups are further divied into columns, and columns into pages. Pages are often around 1MB and are the smallest unit of data in Parquet, since this is where compression is applied. Accessing pages enables loading specific rows without having to load a full row group, and is possible if the Parquet file has a page index. However not every Parquet frameworks supports reading at the page level. PyArrow doesn't for example, but the `parquet` crate in Rust does:


		### Optimized Parquet files

		Parquet files on Hugging Face are optimized to improve storage efficiency, accelerate downloads and uploads, and enable efficient dataset streaming and editing:


		## Using the `huggingface_hub` client library

		The rich features set in the `huggingface_hub` library allows you to manage repositories, including editing dataset files on the Hub. Visit [the client library's documentation](/docs/huggingface_hub/index) to learn more.

[Dataset hub] more than storage: streaming, editing, connectors #2052

Are you sure you want to change the base?

[Dataset hub] more than storage: streaming, editing, connectors #2052

Conversation

lhoestq commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Nov 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julien-c left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lhoestq commented Nov 13, 2025 •

edited

Loading