fix(html): extract <details>/<summary> content instead of discarding it - #4451
Open
r0h1tb wants to merge 1 commit into
Open
fix(html): extract <details>/<summary> content instead of discarding it#4451r0h1tb wants to merge 1 commit into
r0h1tb wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Bug fix in HTML parsing: previously dropped <details>/<summary> content is now extracted, with tests pinned; low-risk mapping change.
Re-trigger cubic
Both tags were mapped to `RemovedBlock`, so `partition_html()` dropped every disclosure widget along with everything inside it. Pages that build their FAQ or documentation out of accordions lost all of their questions and answers, and the caller got a shorter element list rather than an error. `<details>` is an ordinary block container, so it becomes `Flow` like `<div>`. `<summary>` is the widget's heading -- the spec calls it "a summary, caption, or legend for the rest of the contents" -- so it becomes `Heading` and emits a `Title`. That is what the reporter asked for, and it is what makes the fix useful downstream: `chunk_by_title()` opens a new section at each `Title`, so a question stays attached to its own answer instead of blending into its neighbours. `<summary>` was listed twice, under both phrasing and the removed-form group; there is now a single mapping. Resolves Unstructured-IO#3919. Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
r0h1tb
force-pushed
the
fix/html-content-elements
branch
from
August 21, 2026 21:43
7b06863 to
67191e5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
<details>and<summary>were both mapped toRemovedBlock, sopartition_html()discarded the whole subtree. On the FAQ page from the issue, every question and answer disappears and the caller gets a shorter element list rather than an error:Fix
<details>is an ordinary block container, so it becomesFlow, like<div>.<summary>becomesHeading. The spec calls it "a summary, caption, or legend for the rest of the contents", the issue asks for the questions as titles, and it is what makes the fix useful downstream —chunk_by_title()opens a new section at eachTitle, so a question stays attached to its own answer instead of blending into the next entry.<summary>was also listed twice, under both phrasing and the removed-form group; there is now one mapping.Scope
Only these two tags.
<dl>,<figure>and the form/nav tags stay inRemovedBlock:<figure>/<figcaption>removal is asserted byDescribeRemovedBlock.it_is_skipped_during_parsing, so it is a deliberate decision, not an oversight.<dl>/<dt>/<dd>also drop their content, andListBlock's docstring already says "maybe a<dl>element at some point". That looks worth doing, but reclassifying it changes the ancestor chain that_category_depthwalks (there is a parametrised case attest_parser.py:298relying on<dl><dd>nesting), so it belongs in its own PR rather than riding along here.Tests
Four tests in
test_unstructured/partition/html/test_partition.py. With the parser change reverted on this branch:Also checked by hand and behaving: unclosed
<details>,<summary>not the first child, two<summary>siblings, empty<summary>,<summary>outside any<details>, attributes on<details>, inline markup inside<summary>,<details>inside a table cell and inside a list item, 20-deep nesting, and a JSON round-trip.Suite before 146 failed / 2531 passed, after 146 failed / 2535 passed — the same 146 IDs, all pre-existing here (pandoc and the ML extras aren't installed locally). 14 modules that cannot be collected without those extras were excluded from both runs.
Note: this bumps to
0.26.4, as does #4449 — whichever lands second needs the version and changelog re-pointed.