feat: MarkdownHeaderSplitter

[OGuggenbuehl]

Proposed Changes:

Implement MarkdownHeaderSplitter to split Documents written in .md based on their headers

How did you test it?

unit tests

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[CLAassistant]

All committers have signed the CLA.

[sjrl]

@OGuggenbuehl definitely looks like an interesting approach! I've left an initial set of comments, but to further review I'd appreciate if you could add a set of tests like the ones we have for the DocumentSplitter https://github.com/deepset-ai/haystack/blob/main/test/components/preprocessors/test_document_splitter.py

This will help me be able to review the actual algorithm for splitting since it's easier to understand with examples.

[vercel]

@OGuggenbuehl is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[sjrl]

Does this mean that if keep_headers is True we don't store them in the metadata?

[sjrl]

Let's reuse the docstring from DocumentSplitter

Suggested change

	:param skip_empty_documents: If True, skip documents with empty content. If False, process empty documents.
	Defaults to True.
	:param skip_empty_documents: Choose whether to skip documents with empty content. Default is True.
	Set to False when downstream components in the Pipeline (like LLMDocumentContentExtractor) can extract text
	from non-textual documents.

[sjrl]

I just realized we are missing a warm_up method for this component MarkdownHeaderSplitter which calls the warm_up of DocumentSplitter if secondary_split is provided.

This is needed in the case where the secondary_split of sentence is chosen.

So I'd add a warm up method like

    def warm_up(self):
        """
        Warm up the MarkdownHeaderSplitter
        """
        if self.secondary_splitter and not self._is_warmed_up:
            self.secondary_splitter.warm_up()
            self._is_warmed_up = True

then add self._is_warmed_up = False in the init method of MarkdownHeaderSplitter

[sjrl]

Could we avoid using strip() here? This would remove any white space characters from the text we probably want to keep.

[sjrl]

Same here let's remove the .strip()

[sjrl]

If we don't use .strip() anywhere then we should be able to do

Suggested change

	header_line = f"{header_prefix} {header_text}"
	header_line = f"{header_prefix}{header_text}"

I believe

Also if this is only used if self.keep_headers is True we could move it's construction to inside that if statement.

[sjrl]

Also here we shouldn't just add in our own newline characters. If we avoid using strip earlier we should be able to do

Suggested change

	if pending_headers:
	chunk_content += "\n".join(pending_headers) + "\n"
	chunk_content += f"{header_line}\n{content}"
	if pending_headers:
	chunk_content += "".join(pending_headers)
	chunk_content += f"{header_line}{content}"

[sjrl]

I think we could refactor this a bit to make the code overall simpler. E.g.

Suggested change

	processed_documents = []
	for doc in documents:
	# handle empty documents
	if not doc.content or not doc.content.strip():
	if self.skip_empty_documents:
	logger.warning("Document ID {doc_id} has an empty content. Skipping this document.", doc_id=doc.id)
	continue
	# keep empty documents
	processed_documents.append(doc)
	logger.warning(
	"Document ID {doc_id} has an empty content. Keeping this document as per configuration.",
	doc_id=doc.id,
	)
	continue
	processed_documents.append(doc)
	if not processed_documents:
	return {"documents": []}
	header_split_docs = self._split_documents_by_markdown_headers(processed_documents)
	processed_documents = []
	for doc in documents:
	# handle empty documents
	if not doc.content or not doc.content.strip():
	if self.skip_empty_documents:
	logger.warning("Document ID {doc_id} has an empty content. Skipping this document.", doc_id=doc.id)
	continue
	# keep empty documents
	processed_documents.append(doc)
	logger.warning(
	"Document ID {doc_id} has an empty content. Keeping this document as per configuration.",
	doc_id=doc.id,
	)
	continue
	else:
	header_split_docs = self._split_documents_by_markdown_headers([doc])
	final_docs = self._apply_secondary_splitting(header_split_docs) if self.secondary_split else header_split_docs
	processed_documents.extend(final_docs)

this way we know we will only run the splitting logic on non-empty documents. Which hopefully means we don't need to worry about handling documents with empty contents within the splitting logic (i.e. all checks for empty strings can be removed)

[sjrl]

Let's explicitly check the page number of each split

[sjrl]

Let's make these tests explicit. So check the actual content of each one.

[sjrl]

Sorry to be a broken record but let's make this an explicit check for each split. It's easier then for me to review to see if the output is what we expect.

[sjrl]

This should be updated once this comment https://github.com/deepset-ai/haystack/pull/9660/files#r2432221323 is addressed