Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): add redis lock during insertion of event in event table during initial attempt of outgoing webhook delivery #7579

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

cookieg13
Copy link
Contributor

@cookieg13 cookieg13 commented Mar 20, 2025

Type of Change

  • Bugfix
  • New feature
  • Enhancement
  • Refactoring
  • Dependency updates
  • Documentation
  • CI/CD

Description

The current implementation of insertion of an event in event table does not handle the case where multiple requests with the same idempotent_event_id might be processed concurrently, leading to multiple unique constraint violation errors in the database.

This was reported by a merchant who was facing high volumes of unique constraint violation errors.

NOTE: this unique constraint log error issue is only applicable for theinitial attempt of outgoing webhook delivery, not for retry attempts

Why does this happen?

  1. Multiple requests with the same idempotent_event_id might reach the database at the same time.
  2. Each request checks for the event, finds nothing (since it’s not committed yet), and proceeds to insert.
  3. The first request succeeds, while the others fail due to the unique constraint on idempotent_event_id.

Solution (Using Redis Lock)
Before querying the database, acquire a lock in Redis for the given idempotent_event_id.
Only one process/thread will hold the lock, preventing others from inserting at the same time.

If a lock is acquired:

  • Check if the event already exists in the database.
  • If it exists, return early
  • Otherwise, insert the event.
  • Once done, release the lock

Additional Changes

  • This PR modifies the API contract
  • This PR modifies the database schema
  • This PR modifies application configuration/environment variables

Motivation and Context

How did you test it?

Checklist

  • I formatted the code cargo +nightly fmt --all
  • I addressed lints thrown by cargo clippy
  • I reviewed the submitted code
  • I added unit tests for my changes where possible

@cookieg13 cookieg13 self-assigned this Mar 20, 2025
@cookieg13 cookieg13 requested a review from a team as a code owner March 20, 2025 07:28
Copy link

semanticdiff-com bot commented Mar 20, 2025

Review changes with  SemanticDiff

Changed Files
File Status
  crates/router/src/core/webhooks/outgoing.rs  1% smaller

@cookieg13 cookieg13 added the A-core Area: Core flows label Mar 20, 2025
…uring initial attempt of outgoing webhook delivery
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-core Area: Core flows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant