Skip to content

fix(html): extract <details>/<summary> content instead of discarding it - #4451

Open
r0h1tb wants to merge 1 commit into
Unstructured-IO:mainfrom
r0h1tb:fix/html-content-elements
Open

fix(html): extract <details>/<summary> content instead of discarding it#4451
r0h1tb wants to merge 1 commit into
Unstructured-IO:mainfrom
r0h1tb:fix/html-content-elements

Conversation

@r0h1tb

@r0h1tb r0h1tb commented Aug 21, 2026

Copy link
Copy Markdown

Problem

<details> and <summary> were both mapped to RemovedBlock, so partition_html() discarded the whole subtree. On the FAQ page from the issue, every question and answer disappears and the caller gets a shorter element list rather than an error:

>>> partition_html(text="<h1>Support FAQ</h1>"
...   "<details><summary>Which Bluetooth profiles are supported?</summary>"
...   "<p>Your device supports A2DP and HFP profiles.</p></details>")
[<Title 'Support FAQ'>]          # both the question and the answer are gone

Fix

<details> is an ordinary block container, so it becomes Flow, like <div>.

<summary> becomes Heading. The spec calls it "a summary, caption, or legend for the rest of the contents", the issue asks for the questions as titles, and it is what makes the fix useful downstream — chunk_by_title() opens a new section at each Title, so a question stays attached to its own answer instead of blending into the next entry.

<summary> was also listed twice, under both phrasing and the removed-form group; there is now one mapping.

Scope

Only these two tags. <dl>, <figure> and the form/nav tags stay in RemovedBlock:

  • <figure>/<figcaption> removal is asserted by DescribeRemovedBlock.it_is_skipped_during_parsing, so it is a deliberate decision, not an oversight.
  • <dl>/<dt>/<dd> also drop their content, and ListBlock's docstring already says "maybe a <dl> element at some point". That looks worth doing, but reclassifying it changes the ancestor chain that _category_depth walks (there is a parametrised case at test_parser.py:298 relying on <dl><dd> nesting), so it belongs in its own PR rather than riding along here.

Tests

Four tests in test_unstructured/partition/html/test_partition.py. With the parser change reverted on this branch:

FAILED test_partition_html_extracts_details_and_summary_content        - assert [] == [(...5 elements)]
FAILED test_partition_html_makes_a_summary_start_its_own_chunk_section
FAILED test_partition_html_extracts_a_details_block_that_has_no_summary
FAILED test_partition_html_extracts_nested_details_blocks

Also checked by hand and behaving: unclosed <details>, <summary> not the first child, two <summary> siblings, empty <summary>, <summary> outside any <details>, attributes on <details>, inline markup inside <summary>, <details> inside a table cell and inside a list item, 20-deep nesting, and a JSON round-trip.

Suite before 146 failed / 2531 passed, after 146 failed / 2535 passed — the same 146 IDs, all pre-existing here (pandoc and the ML extras aren't installed locally). 14 modules that cannot be collected without those extras were excluded from both runs.

Note: this bumps to 0.26.4, as does #4449 — whichever lands second needs the version and changelog re-pointed.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Shadow auto-approve: would auto-approve. Bug fix in HTML parsing: previously dropped <details>/<summary> content is now extracted, with tests pinned; low-risk mapping change.

Re-trigger cubic

Both tags were mapped to `RemovedBlock`, so `partition_html()` dropped every
disclosure widget along with everything inside it. Pages that build their FAQ or
documentation out of accordions lost all of their questions and answers, and the
caller got a shorter element list rather than an error.

`<details>` is an ordinary block container, so it becomes `Flow` like `<div>`.
`<summary>` is the widget's heading -- the spec calls it "a summary, caption, or
legend for the rest of the contents" -- so it becomes `Heading` and emits a
`Title`. That is what the reporter asked for, and it is what makes the fix
useful downstream: `chunk_by_title()` opens a new section at each `Title`, so a
question stays attached to its own answer instead of blending into its
neighbours.

`<summary>` was listed twice, under both phrasing and the removed-form group;
there is now a single mapping.

Resolves Unstructured-IO#3919.

Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
@r0h1tb
r0h1tb force-pushed the fix/html-content-elements branch from 7b06863 to 67191e5 Compare August 21, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant