Skip to content

Conversation

@Pijukatel
Copy link
Collaborator

Description

  • Wrap Timeout errors raised by user-defined handler by UserHandlerTimeoutError (Otherwise it would be misunderstood as Request handler timed out after 60 s)
  • Ensure that get_one_line_error_summary_if_possible does not access a non-existing index

Issues

Testing

  • Unit tests

Checklist

  • CI passed

@github-actions github-actions bot added this to the 129th sprint - Tooling team milestone Dec 8, 2025
@github-actions github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Dec 8, 2025
@Pijukatel
Copy link
Collaborator Author

Pijukatel commented Dec 8, 2025

Reported example would now log something like this

...
[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to https://a.placeholder.com due to: Timeout raised by user defined handler. File "...crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1576, in default_handler,     await asyncio.wait_for(Future(), timeout=1)
[crawlee.crawlers._basic._basic_crawler] ERROR Request to https://a.placeholder.com failed and reached maximum retries
   File "...crawlee-python/src/crawlee/router.py", line 106, in __call__
    return await user_defined_handler(context)
  File "...crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1576, in default_handler
    await asyncio.wait_for(Future(), timeout=1)
asyncio.exceptions.TimeoutError
...

@Pijukatel Pijukatel requested a review from vdusek December 8, 2025 10:54
@Pijukatel Pijukatel marked this pull request as ready for review December 8, 2025 10:55
Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment, otherwise LGTM

import traceback

import crawlee.errors
import crawlee.router
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to import this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Only the first one is needed.

@Pijukatel Pijukatel merged commit b751208 into master Dec 8, 2025
28 checks passed
@Pijukatel Pijukatel deleted the fix-short-error-summary-for-user-defined-timeout branch December 8, 2025 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TimeoutError from asyncio.wait_for is not handled gracefully and crashes Crawlee

4 participants