Prevent Stale UPDATE Packet Reprocessing

[Varaniya201]

1. Problem Statement (Mandatory)

As part of the Draft API reimplementation, the Registration Processor now reconstructs the draft for each packet during packet reprocessing. This introduces a regression scenario for UPDATE packets.
When multiple UPDATE packets exist for the same resident, reprocessing an older packet can overwrite the identity changes already applied by a newer successfully processed packet, causing the resident's identity to regress to an older state.

Update packet A (pkt_cr_dtimes = T1)
    ↓
Created and got stuck in one stage and waiting for reprocessor to reprocess
Identity = {name: "Alice"}

Update packet B (pkt_cr_dtimes = T2 > T1)
    ↓
PROCESSED
Identity = {name: "Alice Smith"}

Later...

UpdateA is reprocessed
    ↓
populateDraftWithIdentity(...)
    ↓
Identity becomes {name: "Alice"}   ❌

The resident identity no longer represents the latest successful update.
This behavior must be prevented while continuing to allow legitimate retries of NEW packets.

Purpose (Mandatory)

• Prevents resident identity regression.
• Ensures the latest successful UPDATE always remains the source of truth.
• Maintains data consistency after Draft API reimplementation.
• Preserves existing retry capability for NEW packet failures.
• Provides clear feedback to residents when an outdated packet is rejected.

2. Goals

1. Prevent Resident Identity Regression

   • Ensure that reprocessing an older UPDATE packet cannot overwrite identity information that has already been updated by a newer successfully processed UPDATE packet.

2. Preserve Data Consistency

   • Maintain the resident's identity as the latest valid state by enforcing chronological processing during UPDATE packet reprocessing.

3. Support Safe Reprocessing

   • Continue to support legitimate reprocessing of packets, particularly NEW packets, without introducing unnecessary validation that could block valid retry scenarios.

4. Improve Reprocessing Reliability

   • Introduce validation during UPDATE packet reprocessing to identify and reject stale packets before they modify the resident's identity.

5. Provide Clear Failure Handling

   • Mark stale UPDATE packets as failed with an appropriate reason and notify the resident so they can initiate a new update request if required.

6. Maintain Backward Compatibility

   • Ensure existing processing behavior remains unchanged for legacy packets and packet types that are not impacted by the Draft API reimplementation, such as NEW, LOST, ACTIVATED, and DEACTIVATED packets.

7. Protect the Integrity of the Draft API Flow

   • Ensure the Draft API reconstruction (populateDraftWithIdentity) is executed only when the packet being reprocessed represents the latest valid UPDATE for the resident, preventing stale data from being propagated to the draft and published identity.

3. Out of Scope

• NEW packet processing logic
• Initial packet processing flow
• LOST packet flow
• ACTIVATED packet flow
• DEACTIVATED packet flow
• Changes to Draft API implementation

4. Acceptance Criteria (Mandatory)

• AC1 – Allow valid UPDATE reprocessing
  Given an UPDATE packet is being reprocessed, When the created date of UPDATE packet id latest than the previous processed packet Then
  - the packet shall continue through the normal processing flow
  - Draft API shall be invoked
  - resident identity shall be updated
  - packet shall complete successfully.
• AC2 – Prevent stale UPDATE reprocessing
  Given an UPDATE packet is being reprocessed, When a latest UPDATE packet has already been successfully processed, Then
  - the packet shall not proceed further
  - Draft API shall not be invoked
  - resident identity shall remain unchanged
  - packet status shall be marked as FAILED
  - failure event shall be published
  - resident shall be notified about request have been failed and prompt them to re try on updating details.
• AC3 – NEW packet retry remains supported
  Given a NEW packet is being reprocessed, When the packet is retried, Then
  - no stale packet validation shall be executed
  - normal reprocessing shall continue
  - duplicate detection, if applicable, shall continue through existing ABIS logic.
• AC4 – Status publication
  Whenever a stale UPDATE packet is rejected:
  - packet status shall be updated to FAILED
  - failure reason shall indicate that a newer UPDATE has already been processed
  - notification shall be triggered for the resident.
• AC5 – Data integrity
  The system shall never allow an older UPDATE packet to overwrite identity information already committed by a newer successfully processed UPDATE packet.

Basic Flow

1. An UPDATE packet enters the reprocessing flow.

2. The Registration Processor identifies the resident using the packet's reference registration ID (UIN).

3. The processor verifies whether the packet being reprocessed is an UPDATE packet.

   • If No, continue with the existing reprocessing flow without any additional validation.
   • If Yes, proceed to the stale packet validation.

4. The processor retrieves the latest successfully processed UPDATE packet for the same resident.

5. The processor compares the packetCreateDateTime (pkt_cr_dtimes) of the current packet with the latest successfully processed UPDATE packet.

   • If no newer processed UPDATE packet exists, continue with the normal reprocessing flow.
   • If a newer processed UPDATE packet exists, the current packet is identified as stale.

6. For a stale UPDATE packet:

   • Stop further processing.
   • Do not invoke the Draft API
   • Mark the packet status as FAILED.
   • Record the failure reason indicating that a newer UPDATE has already been processed.
   • Publish the packet status update.
   • Notify the resident that the update could not be completed because a newer update has already been processed and advise them to submit a new update request.

7. For a valid UPDATE packet:

   • Continue with the existing reprocessing flow.
   • Invoke the Draft API to reconstruct the draft.
   • Process the packet through the remaining Registration Processor stages.
   • Update the resident identity and complete processing successfully.

8. The reprocessing flow ends with either:

   • SUCCESS, when the packet is eligible for reprocessing, or
   • FAILED, when the packet is identified as stale.

Flow Summary

Alternate Scenarios

Scenario	Expected Behavior
UPDATE packet is reprocessed and no newer PROCESSED UPDATE packet exists	The packet passes the stale packet validation, proceeds through the normal reprocessing flow, invokes the Draft API , updates the resident identity, and completes successfully.
UPDATE packet is reprocessed and a newer PROCESSED UPDATE packet exists	The packet is identified as stale. Processing is stopped, the Draft API is not invoked, the packet is marked as FAILED, the failure reason is recorded, the failure status is published, and the resident is notified to submit a new update request.
Update packet A is reprocessed while Update packet B is still IN_PROGRESS	Only PROCESSED UPDATE packets are considered during stale packet validation. Since Update packet B is still IN_PROGRESS, it is ignored and Update packet A proceeds with reprocessing. If Update packet B completes successfully later, its changes overwrite UpdateA, ensuring the latest resident state is preserved.
packetCreateDateTime is null (legacy packets)	The stale packet validation is skipped when packetCreateDateTime is null. The packet continues through the existing reprocessing flow to maintain backward compatibility with legacy packets created before this field was introduced.
Two UPDATE packets have the same packetCreateDateTime	The validation uses a strict greater-than (>) comparison. Since neither packet has a later timestamp, neither is considered stale. Both packets are eligible for processing, and the packet that completes last determines the final resident identity.
UPDATE packet has a null reference registration ID (UIN)	No matching processed UPDATE packets can be identified for comparison. The stale packet validation returns no results, and the packet continues through the normal reprocessing flow.
NEW packet is reprocessed after a previous FAILED attempt	The stale UPDATE validation is not executed. The NEW packet proceeds through the normal reprocessing flow. This supports legitimate retry scenarios such as failures at ABIS or downstream processing stages.
NEW packet is reprocessed after it was already PROCESSED successfully	The stale UPDATE validation is not executed. The packet is reprocessed as per the existing flow, and any duplicate enrollment continues to be handled by the existing ABIS deduplication mechanism.
LOST packet is reprocessed	LOST packets continue through their existing processing branch (lostAndUpdateUin). No stale packet validation is performed, and no behavior changes are introduced as part of this feature.
ACTIVATED packet is reprocessed	ACTIVATED packets continue through their existing processing branch. Since they do not reconstruct or publish draft data, they are not impacted by this feature.
DEACTIVATED packet is reprocessed	DEACTIVATED packets continue through their existing processing branch. Since they do not reconstruct or publish draft data, they are not impacted by this feature.
Multiple consecutive UPDATE packets of the same UIN are reprocessed in chronological order	Each packet is validated independently. Only the latest UPDATE packet that does not have a newer successfully processed UPDATE packet is allowed to proceed. Any older UPDATE packets are marked as FAILED to prevent resident identity regression.
Multiple consecutive UPDATE packets of the same UIN are reprocessed out of chronological order	Every reprocessed UPDATE packet is compared against the latest successfully processed UPDATE packet for the resident. Any packet older than the latest processed UPDATE is rejected, irrespective of the order in which reprocessing is initiated.

Non Functional Requirement

1. Performance

• The stale packet validation shall introduce minimal overhead to the reprocessing flow.
• The lookup for newer processed UPDATE packets shall be optimized using appropriate database indexes
• The validation shall not significantly increase the overall packet reprocessing time.

2. Scalability

• The solution shall support high-volume packet reprocessing without performance degradation.
• The validation shall scale efficiently as the number of packets per resident increases.

3. Reliability

• The validation shall consistently prevent stale UPDATE packets from modifying resident identity.
• Failures during validation shall not leave packets in an inconsistent processing state.

4. Data Integrity

• Resident identity shall always represent the latest successfully processed UPDATE.
• Under no circumstance shall an older UPDATE packet overwrite data from a newer successfully processed UPDATE packet.
• NEW packet retry behaviour shall remain unaffected.

5. Backward Compatibility

• Existing packet processing flows for NEW, LOST, and Correction packet types shall continue without modification.
• Legacy UPDATE packets without packetCreateDateTime shall continue to be processed using the existing behaviour.

6. Maintainability

• The stale packet validation logic shall be implemented as a reusable and modular component within the reprocessing flow.
• The implementation shall follow existing coding standards and design patterns to simplify future enhancements.

7. Testability

• The feature shall be fully covered by unit, integration, and regression tests.
• Test cases shall include positive, negative, boundary, legacy compatibility, and concurrency scenarios.

Related Issues / PRs (As Applicable)

Draft API Implementation : #2326