fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490) by Seb-MIGUEL · Pull Request #6493 · OpenAEV-Platform/openaev

Seb-MIGUEL · 2026-06-30T10:25:32Z

Problem

The custom dashboard shows 100% detected for some injects. Clicking opens a drawer with inject_expectation_status = SUCCESS from Elasticsearch. Clicking an inject navigates to the inject detail page which shows 100% not detected — inconsistent data between ES and DB.

Fixes #6490

Root Cause

findForIndexing uses a strict > comparison on updated_at:

WHERE inject_expectation_updated_at > :from
ORDER BY inject_expectation_updated_at ASC
LIMIT 500

When bulkComputeTechnicalExpectations expires hundreds of expectations in one saveAll() call, Hibernate's @UpdateTimestamp assigns the same millisecond to all of them. If there are >500 with the same timestamp, the cursor advances to T, then updated_at > T permanently skips the rest.

Fix: Compound keyset cursor

Add a secondary inject_expectation_id cursor so the query can resume within a timestamp:

WHERE (inject_expectation_updated_at > :from)
   OR (inject_expectation_updated_at = :from AND inject_expectation_id > :lastId)
ORDER BY inject_expectation_updated_at ASC, inject_expectation_id ASC

Changes

File	Change
`V5_35__Add_indexing_last_id.java`	Migration: add `indexing_last_id TEXT NULL` to `indexing_status`
`IndexingStatus.java`	Add `lastId` field for compound cursor
`InjectExpectationRepository.java`	Add `findForIndexingAfter(from, lastId, limit)` query
`Handler.java`	Add default `fetch(from, lastId, limit)` method (backward-compatible)
`InjectExpectationHandler.java`	Override compound-cursor fetch + extract shared `map()` helper
`OpenSearchService.java`	Track `lastId` in `bulkProcessing`, pass to handler
`ElasticService.java`	Same as OpenSearch

Cursor logic

Full batch returned (size == 500): save lastId = lastItem.id, keep same lastIndexing
Partial batch (size < 500): clear lastId = null, advance lastIndexing as before

Copilot

Pull request overview

Implements a compound keyset cursor for ES/OpenSearch indexing to prevent permanently skipping inject expectations when more than one full batch shares the same updated_at timestamp, keeping Elasticsearch/OpenSearch results consistent with the database.

Changes:

Adds indexing_last_id storage to persist a secondary cursor (ID) alongside lastIndexing.
Extends the Handler contract with a backward-compatible fetch(from, lastId, limit) default method and implements it for InjectExpectationHandler.
Updates Elastic/OpenSearch bulk processing to pass/persist lastId and to clear it when the batch is not full.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
openaev-model/src/main/java/io/openaev/service/OpenSearchService.java	Passes compound cursor to handlers and persists `lastId` in `IndexingStatus`.
openaev-model/src/main/java/io/openaev/service/ElasticService.java	Same compound-cursor tracking logic for the Elasticsearch client.
openaev-model/src/main/java/io/openaev/engine/model/injectexpectation/InjectExpectationHandler.java	Implements compound-cursor fetch and refactors mapping into a helper.
openaev-model/src/main/java/io/openaev/engine/Handler.java	Adds default compound-cursor `fetch(from, lastId, limit)` API.
openaev-model/src/main/java/io/openaev/database/repository/InjectExpectationRepository.java	Adds `findForIndexingAfter(from, lastId, limit)` with compound cursor ordering.
openaev-model/src/main/java/io/openaev/database/model/IndexingStatus.java	Adds persisted `lastId` field used by the compound cursor.
openaev-api/src/main/java/io/openaev/migration/V5_35__Add_indexing_last_id.java	Flyway migration adding `indexing_last_id` column.

Seb-MIGUEL · 2026-06-30T10:42:10Z

+        SELECT ie.inject_expectation_id FROM injects_expectations ie
+          JOIN injects i ON i.inject_id = ie.inject_id
+          WHERE i.inject_updated_at > :from


Fixed: the UNION branch for \i.inject_updated_at\ now uses the compound cursor: \WHERE i.inject_updated_at > :from OR (i.inject_updated_at = :from AND ie.inject_expectation_id > :lastId).

Seb-MIGUEL · 2026-06-30T10:42:16Z

+        SELECT ie.inject_expectation_id FROM injects_expectations ie
+          JOIN injects i ON i.inject_id = ie.inject_id
+          JOIN injectors_contracts ic ON ic.injector_contract_id = i.inject_injector_contract
+          WHERE ic.injector_contract_updated_at > :from


Fixed: same compound cursor applied to the injector-contract UNION branch: WHERE ic.injector_contract_updated_at > :from OR (ic.injector_contract_updated_at = :from AND ie.inject_expectation_id > :lastId).

Seb-MIGUEL · 2026-06-30T10:42:29Z

+  /**
+   * Variant of {@link #fetch(Instant, int)} using a compound keyset cursor (timestamp + last entity
+   * ID) to avoid missing items that share the same {@code updated_at} timestamp when the previous
+   * batch was full.
+   *
+   * <p>The default implementation ignores {@code lastId} and falls back to the basic cursor,
+   * providing backward-compatible behaviour for handlers that do not need compound pagination.
+   * Override in handlers where the same-millisecond duplicate issue is observable (e.g.
+   * InjectExpectationHandler).


Added integration test compound_cursor_returns_all_expectations_at_same_timestamp in IndexingRegressionTest.java (nested class InjectExpectationIndexing). Creates 3 expectations forced to the same updated_at, verifies fetch(FROM, 2) returns first 2, then fetch(FROM, lastId, 2) returns the 3rd proving no expectation is skipped.

…ith same timestamp When multiple InjectExpectation entities are saved in the same batch (e.g. during expiration of many expectations), Hibernate sets the same @UpdateTimestamp millisecond for all of them. The existing polling query uses a strict > comparison on updated_at, so when the indexing batch size (500) cuts through a set of items sharing the same timestamp, the remaining items are permanently skipped. Fix: introduce a compound keyset cursor (updated_at, inject_expectation_id). - Add indexing_last_id column to indexing_status (V5_35 migration) - Add findForIndexingAfter query with compound cursor in repository - Add fetch(from, lastId, limit) default method to Handler interface; InjectExpectationHandler overrides it with the compound-cursor query - OpenSearchService and ElasticService track lastId per indexing cycle: when a full batch is returned, lastId is persisted so the next cycle continues from the exact item boundary; otherwise lastId is cleared This guarantees all expectations are eventually indexed regardless of how many share the same updated_at timestamp, fixing the inconsistency between the dashboard detection percentage (ES) and the inject detail page (DB).

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Seb-MIGUEL · 2026-07-01T07:36:39Z

+      // Act — first batch (limit=2): returns 2 expectations ordered by (updated_at, id)
+      List<EsInjectExpectation> batch1 = injectExpectationHandler.fetch(FROM, 2);
+      assertThat(batch1).hasSize(2);
+
+      // Act — compound cursor continues from last item of batch1; must return the 3rd
+      String lastId = batch1.getLast().getBase_id();
+      List<EsInjectExpectation> batch2 = injectExpectationHandler.fetch(FROM, lastId, 2);


Fixed: the second batch now uses lastUpdatedAt = batch1.getLast().getBase_updated_at() as the from parameter, which mirrors how the production indexing job updates lastIndexing after a full batch. With from = sharedTimestamp and lastId = exp2.id, only exp3 satisfies (updated_at = sharedTimestamp AND id > exp2.id), correctly proving the compound cursor skips nothing.

Seb-MIGUEL · 2026-07-01T07:36:49Z

+    try (Statement statement = context.getConnection().createStatement()) {
+      statement.execute(
+          "ALTER TABLE indexing_status ADD COLUMN IF NOT EXISTS indexing_last_id TEXT NULL");
+    }


Fixed: the migration now also resets the indexing cursor for expectation-inject (sets indexing_last = epoch and indexing_last_id = NULL) so that the next indexing cycle re-processes all expectations from the beginning, picking up any rows that were previously skipped due to the same-timestamp overflow.

…eset indexing on migration

RomuDeuxfois · 2026-07-01T07:42:39Z

Hey,
If we have a limit, why not iterate on this limit to batch indexing until we batch everything ?

…match main branch

Seb-MIGUEL · 2026-07-01T09:42:22Z

Hi @RomuDeuxfois! The one-batch-per-invocation design is intentional for three reasons:

Bounded execution time: looping until done could run indefinitely on large datasets (e.g. millions of newly-updated expectations after a bulk update). A single bounded batch keeps each scheduler tick predictable and fast.
Resilience: the cursor (lastIndexing + lastId) is persisted to DB after each batch. If the process crashes or restarts mid-processing, the next scheduled invocation resumes exactly where it left off - no data loss, no full re-scan.
Non-blocking scheduler: the job interleaves all model types (expectations, injects, assets) on each tick. Looping on one model would starve the others until that model is fully caught up.

The compound cursor fix in this PR ensures correctness at batch boundaries: when more than one full batch shares the same updated_at, successive scheduled invocations now correctly page through them with (updated_at = :from AND id > :lastId) rather than skipping them.

RomuDeuxfois · 2026-07-01T11:42:34Z

+    LIMIT :limit
+    """,
+      nativeQuery = true)
+  List<RawInjectExpectationIndexing> findForIndexingAfter(


todo: This query is really closed from the previous one. Can we do some refacto here ?

Done - merged findForIndexing and findForIndexingAfter into a single query. The compound-cursor clause is now guarded by :lastId IS NOT NULL, so passing null gives the simple > :from behaviour (first batch) and passing a non-null ID gives the full compound cursor. Removed 70 lines of duplicated SQL. InjectExpectationHandler.fetch(from, limit) now delegates to findForIndexingAfter(from, null, limit) (commit 5e83a21).

…gle nullable-lastId method

…atus update (#6490)

Copilot AI review requested due to automatic review settings June 30, 2026 10:25

Copilot started reviewing on behalf of Seb-MIGUEL June 30, 2026 10:25 View session

Seb-MIGUEL mentioned this pull request Jun 30, 2026

[WIP] Fix inconsistency between dashboard detection percentage and inject detail page #6492

Closed

8 tasks

Seb-MIGUEL force-pushed the fix/es-expectation-indexing-compound-cursor-v2 branch from d8716d8 to 2d56f5f Compare June 30, 2026 10:30

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Seb-MIGUEL force-pushed the fix/es-expectation-indexing-compound-cursor-v2 branch 2 times, most recently from b116ffd to caf91fb Compare June 30, 2026 10:41

Seb-MIGUEL requested a review from Copilot June 30, 2026 12:46

Copilot started reviewing on behalf of Seb-MIGUEL June 30, 2026 12:46 View session

Seb-MIGUEL force-pushed the fix/es-expectation-indexing-compound-cursor-v2 branch from caf91fb to 0ec87d7 Compare June 30, 2026 12:50

Copilot AI reviewed Jun 30, 2026

View reviewed changes

[backend] fix(engine): address Copilot review - fix test cursor and r…

391144f

…eset indexing on migration

Seb-MIGUEL changed the title ~~[backend] fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490)~~ fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490) Jul 1, 2026

[backend] fix(migration): rename V5_34 signatures migration to V6 to …

e5f9f32

…match main branch

RomuDeuxfois reviewed Jul 1, 2026

View reviewed changes

Seb-MIGUEL added 2 commits July 1, 2026 14:17

[backend] refactor(engine): merge duplicate indexing queries into sin…

5e83a21

…gle nullable-lastId method

[docs] chore: add anti-pattern rule for duplicate query methods

2bd91c4

RomuDeuxfois approved these changes Jul 1, 2026

View reviewed changes

[backend] fix(migration): fix wrong column names in V5_35 indexing_st…

b37e477

…atus update (#6490)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490)#6493

fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490)#6493
Seb-MIGUEL wants to merge 6 commits into
mainfrom
fix/es-expectation-indexing-compound-cursor-v2

Seb-MIGUEL commented Jun 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Seb-MIGUEL Jun 30, 2026

Uh oh!

Seb-MIGUEL Jun 30, 2026

Uh oh!

Seb-MIGUEL Jun 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Seb-MIGUEL Jul 1, 2026

Uh oh!

Seb-MIGUEL Jul 1, 2026

Uh oh!

RomuDeuxfois commented Jul 1, 2026

Uh oh!

Seb-MIGUEL commented Jul 1, 2026

Uh oh!

RomuDeuxfois Jul 1, 2026

Uh oh!

Seb-MIGUEL Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Seb-MIGUEL commented Jun 30, 2026

Problem

Root Cause

Fix: Compound keyset cursor

Changes

Cursor logic

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Seb-MIGUEL Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Seb-MIGUEL Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Seb-MIGUEL Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Seb-MIGUEL Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Seb-MIGUEL Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

RomuDeuxfois commented Jul 1, 2026

Uh oh!

Seb-MIGUEL commented Jul 1, 2026

Uh oh!

RomuDeuxfois Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Seb-MIGUEL Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants