Skip to content

[#4802] Adopt a concurrently created config token instead of failing to start - #4813

Merged
smcvb merged 1 commit into
mainfrom
bug/4802/jdbc-config-token-concurrent-init
Aug 6, 2026
Merged

[#4802] Adopt a concurrently created config token instead of failing to start#4813
smcvb merged 1 commit into
mainfrom
bug/4802/jdbc-config-token-concurrent-init

Conversation

@schananas

@schananas schananas commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Fixes #4802

What changed

initializeConfigToken rethrew the insert's primary-key violation as UnableToRetrieveIdentifierException without re-reading the row the winning instance had just committed. Several instances starting at once against a fresh token table therefore left all but one unable to start. Measured at three failures in four.

The insert is now wrapped in a JDBC savepoint. On failure it rolls back to the savepoint, re-reads the config token, and adopts the winner's identifier, rethrowing with the original cause only when there is genuinely no row. The savepoint is what makes the re-read legal on databases such as PostgreSQL that abort the whole transaction on a constraint violation.

Tests

New: JdbcTokenStoreTest.ConcurrentIdentifierInitialization, a four-thread barrier race. Green on 5 consecutive runs; with JdbcTokenStore.java reverted it fails 3 of 3, 3 of 4 threads throwing UnableToRetrieveIdentifierException -- the three-in-four rate reported on the issue.

Full messaging module: green. No existing test needed changing.

The fix only works at READ COMMITTED

Measured across four HSQLDB configurations, same 4-thread race:

tx mode isolation succeeded failed
mvcc READ_COMMITTED 4 0
locks READ_COMMITTED 4 0
mvcc SERIALIZABLE 1 3
locks SERIALIZABLE 1 3

At SERIALIZABLE the cause is SQLTransactionRollbackException: transaction rollback: serialization failure -- the whole transaction is dead before the savepoint rollback runs, so the re-read never happens and 3 of 4 processors still fail to start. The same reasoning applies to PostgreSQL REPEATABLE READ and SERIALIZABLE, and to MySQL and MariaDB, whose default is REPEATABLE READ: a consistent read pins the snapshot at the first read, so a plain non-FOR UPDATE re-read cannot see the winner.

Not a regression, and not a correctness hazard -- a 40001 serialization failure is exactly what a caller retries as a whole transaction. But it is a real limit on where this helps.

For the reviewer

The savepoint is deliberately optional: savepointOrNull returns null when the connection auto-commits, where a failed statement poisons nothing, and when the driver rejects setSavepoint. The re-read is still attempted, so both degrade to today's behaviour rather than breaking. The savepoint is never explicitly released, because releaseSavepoint is unsupported on some drivers.

The original SQLException is preserved as the cause and surfaces only when the re-read finds no row, so a genuine schema or connectivity fault still fails loudly instead of being swallowed as a lost race.

Two gaps worth naming. The savepoint half of the diff has no test: HSQLDB does not abort the transaction on a PK violation, so the adopt path works even with setSavepoint throwing, leaving the property claimed for the savepoint unverified here. And the test asserts no landing evidence -- if the barrier ever serialised so all four threads found the row on the first read, it would pass without exercising the catch block.

Not covered here: JpaTokenStore.getConfig has the same race. After a failed em.flush() the persistence context is rollback-only, so recovering there needs a fresh transaction rather than a re-read. Called out in #4802 as deserving its own issue rather than widened into this one.

…to start

The config token row is created on first use, so instances starting together against a fresh
token table all raced to insert it. Only one could win, and the losers rethrew the primary-key
violation as UnableToRetrieveIdentifierException without ever re-reading the row the winner had
just committed, leaving all but one unable to start. Measured at three failures in four.

The insert is now wrapped in a JDBC savepoint. On failure it rolls back to the savepoint, re-reads
the config token and adopts the winner's identifier, rethrowing with the original cause only when
there is genuinely no row to adopt. The savepoint is what makes the re-read legal on databases
that abort the whole transaction on a constraint violation, and it is skipped when the connection
auto-commits, since a failed insert cannot then poison anything else.

Fixes #4802
@schananas
schananas requested a review from a team as a code owner July 29, 2026 11:47
@schananas
schananas requested review from MateuszNaKodach, laura-devriendt-lemon and zambrovski and removed request for a team July 29, 2026 11:47
@schananas schananas self-assigned this Jul 29, 2026

@smcvb smcvb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍

@smcvb
smcvb merged commit c9dfd5a into main Aug 6, 2026
7 checks passed
@smcvb
smcvb deleted the bug/4802/jdbc-config-token-concurrent-init branch August 6, 2026 14:00
@smcvb

smcvb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

/backport axon-5.3.x

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@smcvb smcvb added Type: Bug Use to signal issues that describe a bug within the system. Priority 1: Must Highest priority. A release cannot be made if this issue isn’t resolved. labels Aug 6, 2026
@smcvb smcvb modified the milestones: Release 5.3.1, Release 5.4.0 Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Priority 1: Must Highest priority. A release cannot be made if this issue isn’t resolved. Type: Bug Use to signal issues that describe a bug within the system.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Several instances starting at once against a fresh JDBC token store: all but one fail to start

2 participants