Skip to content

Seed data creation and data preprocessing improvements #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Seed data creation and data preprocessing improvements #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
203cf74
Revise comments in chunking notebook
alimaredia Apr 23, 2025
0058f67
feat: Add sdg seed dataset creation notebook
alimaredia Apr 23, 2025
cc69a06
rename doc preprocessing-to-sdg
alimaredia May 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions notebooks/doc-preprocessing-to-sdg/requirements.txt
View file
Open in desktop

This file was deleted.

29 changes: 24 additions & 5 deletions ...s/doc-preprocessing-to-sdg/chunking.ipynb → ...ructlab-knowledge-pipeline/chunking.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,20 @@
"id": "7b33678f-67d2-48a1-801f-302622e43e0f",
"metadata": {},
"source": [
"## Goal\n",
"## Chunking\n",
"The goal of chunking for InstructLab SDG is to provide the teacher model small and logical pieces of the source document to generate data off of.\n",
"\n",
"In this notebook we are doing chunking with Docling[https://docling-project.github.io/docling/examples/hybrid_chunking/#hybrid-chunking].\n",
"\n",
"First let's ensure docling is installed."
"The input to this notebook is a docling JSON file created after a docling conversion, or a directory of docling JSON files."
]
},
{
"cell_type": "markdown",
"id": "d9f268fd-35d2-4c7a-8cfa-47630de00837",
"metadata": {},
"source": [
"### Dependencies"
]
},
{
Expand Down Expand Up @@ -272,8 +280,7 @@
" c = dict(chunk=chunk, file=file.stem)\n",
" all_chunks.append(c)\n",
" except ConversionError as e:\n",
" print(f\"Skipping file {file}\")\n",
"# print(all_chunks)"
" print(f\"Skipping file {file}\")"
]
},
{
Expand All @@ -286,14 +293,26 @@
"To view the chunks, run through the following cell. As you can see the document is broken into small pieces with metadata about the chunk based on the document's format"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ff88cf5c-1315-4eca-afcd-25706eaf7d6b",
"metadata": {},
"outputs": [],
"source": [
"# print(all_chunks)"
]
},
{
"cell_type": "markdown",
"id": "84826055-a7f1-4334-a12b-bbc07a523199",
"metadata": {
"tags": []
},
"source": [
"## Save the chunks to a text file each"
"## Save the chunks to a text file for each chunk\n",
"\n",
"Each chunk is saved to an individual text file in the format: `{docling-json-file-name}-{chunk #}.txt`. Having chunking in this format is important as an input to create-sdg-seed-data notebook."
]
},
{
Expand Down
Loading