community: Add custom sitemap URL parameter to GitbookLoader #30549
+235
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
dosubot
bot
added
size:L
eyurtsev
reviewed
Mar 31, 2025
libs/community/tests/unit_tests/document_loaders/test_gitbook.py
Outdated
Show resolved
Hide resolved
eyurtsev
reviewed
Apr 1, 2025
| assert paths == ["/page1", "/page2", "/page3"] | ||
|
|
||
|
|
||
| @patch("requests.get") |
There was a problem hiding this comment.
Thanks!
@andrasfe I don't know if you're familiar with the responses library, but it's generally a much cleaner way to patch requests.get / patch / post if you're comparing it to using the built in patch methods.
Going to merge as is (not a blocker). But if you have time to re-write w/ responses will be appreciated. It'll make the code much less brittle!
I've observed that the usage of @patch w/ strings specifying the patched code introduces more technical debt sometimes than it solves. patch object is slightly better since it's easier to catch namespacing changes, but for network requests responses is better in virtually all cases
eyurtsev
reviewed
Apr 1, 2025
libs/community/tests/integration_tests/document_loaders/test_gitbook.py
Outdated
Show resolved
Hide resolved
dosubot
bot
added
the
lgtm
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new
sitemap_urlparameter to theGitbookLoaderclass that allows users to specify a custom sitemap URL when loading content from a GitBook site. This is particularly useful for GitBook sites that use non-standard sitemap file names likesitemap-pages.xmlinstead of the defaultsitemap.xml.The standard
GitbookLoaderassumes that the sitemap is located at/sitemap.xml, but some GitBook instances (including GitBook's own documentation) use different paths for their sitemaps. This parameter makes the loader more flexible and helps users extract content from a wider range of GitBook sites.Issue
Fixes bug 30473 where the
GitbookLoaderwould fail to find pages on GitBook sites that use custom sitemap URLs.Dependencies
No new dependencies required.
I've added:
The changes are fully backward compatible, as the parameter is optional with a sensible default.