Improve BackbeatAPI pause/resume logic

[benzekrimaha]

Issue: BB-658

[bert-e]

Hello benzekrimaha,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue BB-658 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.0.7

• 9.1.0

Please check the Fix Version/s of BB-658, or the target
branch of this pull request.

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.61%. Comparing base (66c19b0) to head (0b81a26).

Files with missing lines	Patch %	Lines
lib/util/LocationStatusManager.js	85.71%	11 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/util/LocationStatusManager.js	87.05% <85.71%> (+4.67%)	⬆️

... and 3 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	80.36% <ø> (ø)
Core Library	80.59% <85.71%> (-0.44%)	⬇️
Ingestion	70.31% <ø> (ø)
Lifecycle	78.35% <ø> (ø)
Oplog Populator	85.06% <ø> (ø)
Replication	61.15% <ø> (-0.04%)	⬇️
Bucket Scanner	85.60% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/9.1    #2644      +/-   ##
===================================================
- Coverage            74.81%   74.61%   -0.20%     
===================================================
  Files                  201      200       -1     
  Lines                13579    13553      -26     
===================================================
- Hits                 10159    10113      -46     
- Misses                3410     3430      +20     
  Partials                10       10              

Flag	Coverage Δ
api:retry	9.24% <6.49%> (-0.25%)	⬇️
api:routes	9.05% <7.79%> (-0.25%)	⬇️
bucket-scanner	85.60% <ø> (ø)
ft_test:queuepopulator	9.07% <0.00%> (-1.27%)	⬇️
ingestion	12.58% <0.00%> (+0.02%)	⬆️
lib	7.79% <0.00%> (+0.01%)	⬆️
lifecycle	18.97% <0.00%> (+0.03%)	⬆️
notification	1.04% <0.00%> (+<0.01%)	⬆️
replication	18.79% <0.00%> (+0.03%)	⬆️
unit	50.79% <85.71%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bert-e]

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

[bert-e]

Incorrect fix version

The Fix Version/s in issue BB-658 contains:

• 9.1.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.0.7

• 9.1.0

Please check the Fix Version/s of BB-658, or the target
branch of this pull request.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[williamlardier]

I tried to review everything but due to some changes I discuss, some of my comments will likely end up irrelevant, I suggest reading the whole review and work on the "big part"s first, so you don't waste time addressing some of my comments I wrote with the current code

[francoisferrand]

Overall, still seems very complex to me, mixing different layers and trying to introduce functions which "abstract everything" and make it very complex to understand what happens.

• We need to find a way to keep the "abstraction" (mongo vs zookeeper) somewhat cleanly separated, and not intermingle it with state management ;
• The state is already managed in memory (that is the starting point of the bug), in LocationStatus, so this is where the state management would need to be updated ;
• Since we want to fix inconsistent state, it means we need intermediary states (like PAUSE_IN_PROGRESS) in the FSM, and transition both before initiating the backend operation and once it has succeeded (or failed)

---

Since we have multiple processes (or restart), I wonder however if this approach really works in every case : we may write the state to the backend, but then another process could overwrite this state in between, and we would still be inconsistent?
(i.e. maybe the safer -and simpler- way is to rely on the backend information and using more "transactional" operations instead of in-memory cache)

[francoisferrand]

also please rebase 🙏