Skip to content

Conversation

@yblanke
Copy link

@yblanke yblanke commented Dec 22, 2025

Summary

Add configurable, recoverable error handling for Kafka poll in async source, with docs and tests.

Changes

  • Core behavior: poll-fn now:
    • Still lets wakeups behave as before for normal shutdown.
    • On other exceptions, either:
      • Calls a user-provided :ketu.source/custom-catch-fn (if given), or
      • Uses a built-in handler that logs the problem, skips over the problematic messages, and keeps consuming new ones instead of treating it as a fatal error.
  • Offset skipping:
    • The number of messages to skip can be configured via :ketu.source/error-skip-offset-amount.
    • If not set, it is derived from the Kafka max.poll.records setting, with a safe default of 1.
  • Docs: README.md Consumer-source options now document:
    • :ketu.source/custom-catch-fn
    • :ketu.source/error-skip-offset-amount

@github-actions
Copy link

github-actions bot commented Dec 24, 2025

Unit Test Results

0 tests   0 ✔️  0s ⏱️
0 suites  0 💤
0 files    0

Results for commit 0068246.

♻️ This comment has been updated with latest results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants