Replace backbeat client with cloudserver client #2679

SylvainSenechal · 2025-10-06T14:55:48Z

WIP
Issue: BB-706

bert-e · 2025-10-06T14:55:52Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

codecov · 2025-10-06T15:58:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.84%. Comparing base (66c19b0) to head (454ceaf).
⚠️ Report is 3 commits behind head on development/9.1.

Additional details and impacted files

Files with missing lines	Coverage Δ
extensions/gc/tasks/GarbageCollectorTask.js	`86.72% <ø> (-0.46%)`	⬇️
...sions/lifecycle/tasks/LifecycleDeleteObjectTask.js	`93.28% <ø> (+0.76%)`	⬆️
extensions/lifecycle/tasks/LifecycleTaskV2.js	`88.88% <ø> (ø)`
extensions/replication/tasks/CopyLocationTask.js	`66.49% <ø> (-6.43%)`	⬇️
...xtensions/replication/tasks/MultipleBackendTask.js	`55.55% <ø> (-3.88%)`	⬇️
extensions/replication/tasks/ReplicateObject.js	`91.07% <ø> (-0.74%)`	⬇️
lib/BackbeatMetadataProxy.js	`62.16% <ø> (+12.16%)`	⬆️
lib/clients/ClientManager.js	`63.26% <ø> (-1.45%)`	⬇️
lib/clients/utils.js	`71.42% <ø> (-10.39%)`	⬇️
lib/queuePopulator/IngestionProducer.js	`76.06% <ø> (+6.12%)`	⬆️
... and 1 more

... and 5 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	`80.36% <ø> (ø)`
Core Library	`81.61% <ø> (+0.58%)`	⬆️
Ingestion	`71.07% <ø> (+0.75%)`	⬆️
Lifecycle	`78.29% <ø> (-0.06%)`	⬇️
Oplog Populator	`85.06% <ø> (ø)`
Replication	`59.33% <ø> (-1.86%)`	⬇️
Bucket Scanner	`85.76% <ø> (+0.15%)`	⬆️

@@                 Coverage Diff                 @@
##           development/9.1    #2679      +/-   ##
===================================================
+ Coverage            74.81%   74.84%   +0.02%     
===================================================
  Files                  201      200       -1     
  Lines                13579    13102     -477     
===================================================
- Hits                 10159     9806     -353     
+ Misses                3410     3286     -124     
  Partials                10       10

Flag	Coverage Δ
bucket-scanner	`85.76% <ø> (+0.15%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

SylvainSenechal · 2025-10-06T16:04:00Z

package.json

    "vaultclient": "scality/vaultclient#8.5.1",
-    "werelogs": "scality/werelogs#8.2.1"
+    "werelogs": "scality/werelogs#8.2.1",
+    "@scality/cloudserverclient": "file:../cloudserverclient"


SylvainSenechal · 2025-10-06T16:04:09Z

conf/config.json

-            "replicaSet": "rs0",
-            "readPreference": "primary",
-            "database": "metadata"
+            "replicaSetHosts": "localhost:27017",


bert-e · 2025-10-13T15:22:09Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

SylvainSenechal · 2025-10-14T09:52:05Z

extensions/gc/tasks/GarbageCollectorTask.js

-const { attachReqUids } = require('../../../lib/clients/utils');
+
 const BackbeatTask = require('../../../lib/tasks/BackbeatTask');
+const { BatchDeleteCommand } = require('../../../lib/clients/smithy/build/smithy/source/typescript-codegen');


Do not pay attention to these imports path. They will be updated once the new library is released

SylvainSenechal · 2025-10-14T10:01:12Z

extensions/gc/tasks/GarbageCollectorTask.js

                return done(err);
            }

            const backbeatClient = this.getBackbeatClient(accountId);


Side note : There are a lot of variables, function names and possibly file names that would benefit from a rename, from backbeatClient to cloudserverClient etc.
I don't wanna do it now because it will add a lot of noise to that PR, and it's not even that straightforward to do with this silly programming language

SylvainSenechal · 2025-10-14T13:27:28Z

extensions/replication/tasks/ReplicateObject.js

            Key: destEntry.getObjectKey(),
            CanonicalID: destEntry.getOwnerId(),
+            // TODO : missing content length
            ContentLength: partSize,


I removed content-length. Smithy doesn't allow us to have a content length variable that we defined ourselves to be used as the header, because it's calculating the header itself.
Only thing I'm worried about is that the new ContentLength from Smithy might not be equal to the previous one : I think the new ones uses the whole request, when our old content length was partObj.getObjSize()

SylvainSenechal · 2025-10-14T14:07:42Z

extensions/replication/tasks/ReplicateObject.js

                // eslint-disable-next-line no-param-reassign
                err.origin = 'target';
-                if (err.ObjNotFound || err.code === 'ObjNotFound') {
+                if (err.ObjNotFound || err.code === 'ObjNotFound' ||


This isn't super nice error checking 🫤

we cannot guaarantee the type of error there? is it always an arsenal error, or an sdk one? if sdk you should be able to use instanceof, no?
Side note: can't it also be a NoSuchVersion in this case, if the versionID is not empty?

SylvainSenechal · 2025-10-14T14:33:40Z

extensions/replication/tasks/MultipleBackendTask.js

    }

    _getAndPutObjectOnce(sourceEntry, log, done) {
        log.debug('getting object data', { entry: sourceEntry.getLogInfo() });


TODO: review this function again

SylvainSenechal · 2025-10-14T14:46:33Z

extensions/replication/tasks/MultipleBackendTask.js

+            .then(data => {
+                sourceEntry.setReplicationSiteDataStoreVersionId(this.site,
+                    data.versionId);
+                // TODO : review metadata metrics


@francoisferrand I think using JSON.stringify will work, I see that the function .getSerialized() in Arsenal also returns a string from JSON.stringify that is then used in this same _publishMetadataWriteMetrics, but I'm not sure that "command.input" is equivalent to the old "destReq.httpRequest.body" data we used to call it with

There are a few occurences where this _publishMetadataWriteMetrics is an issue for me.
Actually, we can add a middleware to extract the request body like this, and pass it to the metrics just as before.
Although, I have found that here it would likely be useless, because as you can see in the command above, there is Body in the command, so the length would always be 0.
I have only found one occurence where we the request has a body

command.middlewareStack.add( next => async args => { const request = args.request as any; console.log("content-length:", request?.headers?.['content-length']) console.log("len", Buffer.byteLength(request.body as any)) return next(args); }, { step: 'finalizeRequest' } );

One more thought : With the old SDK, we were using destReq.httpRequest.body, but if you look at the command, there is no proper body parameter, and with the new v3 client, I tried to get the body with the middleware and it is undefined...
Anyways, probably not worth it to overthink this problem, we just need to clearly state what is the metric we want to compute first (I believe that here it's the Tags, since the api is putTagging), so we probably just want to do something like
this._publishMetadataWriteMetrics(JSON.stringify(command.input.Tags), writeStartTime)

SylvainSenechal · 2025-10-14T15:01:14Z

lib/clients/ClientManager.js

     * @return {BackbeatClient} The S3 client instance to make requests with
     */
    getBackbeatClient(accountId) {
+        // TODO : review why credentials creation before client cache check


I wonder if we could check if the client is in the cache directly, the getCredentials function seems rather expensive. But I also wonder if there is some logic in getCredentials that makes it important to run it even when we later return a client from the cache

SylvainSenechal · 2025-10-14T15:05:44Z

lib/queuePopulator/IngestionProducer.js

-            if (err) {
+        return this._ringReader.send(command)
+            .then(data => {
+            // TODO : Review : data[0] doesn't make sense, considering the output shape.. : 


Need input from other ppl who know this stuff.
Considering the output of getRaftId is a string,
Either,

We indeed want to read the first character of that string

Or we expect it to be an array and want to read the first element, but in that case there is a problem with api definition

Either way, weird api design

The output looks like this, but can RaftID be multiple digit ?

Yes each raft session has an id, so if you have 10 then it'll be 2 digits

For instance in Scuba we type it like htis: https://github.com/scality/scuba/blob/d6c285163fb7fa2113aecc32f14623bda60e7508/lib/utils/logId.ts#L6

Mhh, so if raftId == '13', sounds like the old code getting the raftId from data[0] would be getting '1' instead of '13' 🤔

maybe @scality/hopocalypse team can help better understand this, i.e. how the API behaves and maybe why we used to take only the first character of the string?

This is not used in S3C, we don't have an IngestionPopulator.

Here is our list of entrypoints for backbeat:

https://github.com/scality/Federation/blob/development/9.5/roles/run-backbeat/templates/supervisord.conf.j2

https://github.com/scality/Federation/blob/development/9.5/roles/run-bucket-notifications/templates/supervisord.conf.j2

No bin/ingestion.js in there

SylvainSenechal · 2025-10-14T15:09:45Z

lib/queuePopulator/IngestionProducer.js

        ], done);
    }

+    // TODO : review streaming logic..


TODO : review

SylvainSenechal · 2025-10-16T10:05:46Z

extensions/replication/tasks/ReplicateObject.js

            Key: entry.getObjectKey(),
            VersionId: entry.getEncodedVersionId(),
            AccountId: accountId,
+            // TODO : content length not available


Probably gonna remove this :
Smithy won't let use create Requests with a field named Content-length that would be used as a header : It's a reserved name because the value will be auto-computed

SylvainSenechal · 2025-10-16T10:10:51Z

lib/queuePopulator/IngestionProducer.js

        ], done);
    }

    getRaftLog(raftId, begin, limit, targetLeader, done) {


This function is heavily modified, in theory it should be simpler, as the Api doesn't returns a stream anymore (example of returned response below)
But I'm worried about the impact, it means that the function calling getRaftLog() : _processReadRecords() will get a completely different type compared with before. In this function, we assign the variable batchState.logRes with the response of that getRaftLog(), but it also looks like batchState.logRes is assigned in other places in the code, and there is potentially broader impact (see function _processPrepareEntries below)

Example new Api Response :

I don't know how big this can be, but if big, this is probably not scalable not to use any stream... When you say the api doesn't return a stream anymore, do you mean because we stop using ListRecordStream but the class from the smithy generated lib? Did we change the way of getting the oplog?

I'm not a backbeat expert but we usually use bucketclient to consume the oplog, please see the latest version of the approach in scuba (the utilization service that uses the oplog to get the events from the metadata backend) https://github.com/scality/scuba/blob/d6c285163fb7fa2113aecc32f14623bda60e7508/lib/reader/LogConsumerV2.ts#L110

The api is paginated though, although I can't be sure the pagination is always used.. (but on the metadata doc, it says if there is no pagination, it will return max 10k elements)
I looked in metadata and it seems like the response from this api is indeed a stream, it's weird that in the old backbeat client, the response was not marked as a stream, I think I should change it and use a stream.

williamlardier

some random comments after seeing your ask for early reviews

williamlardier · 2025-10-16T13:10:05Z

extensions/lifecycle/tasks/LifecycleDeleteObjectTask.js

+                done(null);
+            })
+            .catch(err => {
+                if (err?.$metadata?.httpStatusCode === errors.MethodNotAllowed.code) {


shall we test for the err "instanceof" first?

williamlardier · 2025-10-16T13:12:56Z

extensions/replication/tasks/ReplicateObject.js

                // eslint-disable-next-line no-param-reassign
                err.origin = 'target';
-                if (err.ObjNotFound || err.code === 'ObjNotFound') {
+                if (err.ObjNotFound || err.code === 'ObjNotFound' ||


we cannot guaarantee the type of error there? is it always an arsenal error, or an sdk one? if sdk you should be able to use instanceof, no?
Side note: can't it also be a NoSuchVersion in this case, if the versionID is not empty?

williamlardier · 2025-10-16T13:15:09Z

lib/queuePopulator/IngestionProducer.js

-            if (err) {
+        return this._ringReader.send(command)
+            .then(data => {
+            // TODO : Review : data[0] doesn't make sense, considering the output shape.. : 


Yes each raft session has an id, so if you have 10 then it'll be 2 digits

For instance in Scuba we type it like htis: https://github.com/scality/scuba/blob/d6c285163fb7fa2113aecc32f14623bda60e7508/lib/utils/logId.ts#L6

williamlardier · 2025-10-16T13:21:16Z

lib/queuePopulator/IngestionProducer.js

        ], done);
    }

    getRaftLog(raftId, begin, limit, targetLeader, done) {


I don't know how big this can be, but if big, this is probably not scalable not to use any stream... When you say the api doesn't return a stream anymore, do you mean because we stop using ListRecordStream but the class from the smithy generated lib? Did we change the way of getting the oplog?

I'm not a backbeat expert but we usually use bucketclient to consume the oplog, please see the latest version of the approach in scuba (the utilization service that uses the oplog to get the events from the metadata backend) https://github.com/scality/scuba/blob/d6c285163fb7fa2113aecc32f14623bda60e7508/lib/reader/LogConsumerV2.ts#L110

williamlardier · 2025-10-16T13:22:30Z

lib/BackbeatMetadataProxy.js

-                if (err.ObjNotFound || err.code === 'ObjNotFound' || err.code === 'InvalidBucketState') {
+                // TODO : Review these errors..
+                if (err.ObjNotFound || 
+                    err.code === 'ObjNotFound' || err.name === 'ObjNotFound' ||


Maybe if the .name & . code are both required I think not, but as we are lacking typescript...), you could do a kind of "macro" function you may use to reduce the code duplication

SylvainSenechal · 2025-10-20T14:34:15Z

extensions/replication/tasks/ReplicateObject.js

            CanonicalID: destEntry.getOwnerId(),
-            ContentLength: partSize,
            ContentMD5: partObj.getPartETag(),
            Body: incomingMsg,


When Maha finished the sdk v3 migration, incomingMsg's type should change (it will be a StreamingBlobPayloadOutputTypes) and be compatible

Issue: BB-706

SylvainSenechal force-pushed the improvement/BB-706 branch from 6398439 to 269de9b Compare October 6, 2025 15:57

SylvainSenechal commented Oct 6, 2025

View reviewed changes

SylvainSenechal force-pushed the improvement/BB-706 branch 5 times, most recently from d06be27 to 11d7667 Compare October 13, 2025 14:47

scality deleted a comment from bert-e Oct 13, 2025

SylvainSenechal force-pushed the improvement/BB-706 branch 2 times, most recently from 130fc22 to 5c4933f Compare October 14, 2025 09:51

SylvainSenechal commented Oct 14, 2025

View reviewed changes

SylvainSenechal force-pushed the improvement/BB-706 branch from 5c4933f to 616c43b Compare October 14, 2025 09:58

SylvainSenechal commented Oct 14, 2025

View reviewed changes

SylvainSenechal force-pushed the improvement/BB-706 branch from 616c43b to dcd1111 Compare October 14, 2025 11:48

SylvainSenechal commented Oct 14, 2025

View reviewed changes

SylvainSenechal force-pushed the improvement/BB-706 branch 3 times, most recently from 404c946 to 897ee3e Compare October 14, 2025 14:06

SylvainSenechal commented Oct 14, 2025

View reviewed changes

lib/queuePopulator/IngestionProducer.js Outdated

], done);

}

// TODO : review streaming logic..

Copy link

Contributor Author

SylvainSenechal Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO : review

SylvainSenechal force-pushed the improvement/BB-706 branch from 897ee3e to 3f6b0a3 Compare October 14, 2025 15:13

SylvainSenechal commented Oct 16, 2025

View reviewed changes

williamlardier reviewed Oct 16, 2025

View reviewed changes

SylvainSenechal commented Oct 20, 2025

View reviewed changes

SylvainSenechal force-pushed the improvement/BB-706 branch 8 times, most recently from f17fa5f to adf2539 Compare October 23, 2025 12:45

SylvainSenechal added 8 commits October 23, 2025 15:03

replace backbeat client with cloudserver client

77c0433

Issue: BB-706

putdata and batch delete

fe66185

lifecycle and indexes apis

f374bf4

raft apis

cfaca2b

finished multi backend apis

f8166b6

use stream on raftlog function

661e6a8

fixup lint

6c3d3f1

fixup imports

615de56

SylvainSenechal force-pushed the improvement/BB-706 branch 2 times, most recently from 7304ff0 to 7fd666c Compare October 23, 2025 14:29

tests for new client

c2ec8ea

SylvainSenechal force-pushed the improvement/BB-706 branch 7 times, most recently from 454ceaf to c2ec8ea Compare October 24, 2025 13:55

Replace backbeat client with cloudserver client #2679

Are you sure you want to change the base?

Replace backbeat client with cloudserver client #2679

Uh oh!

Conversation

SylvainSenechal commented Oct 6, 2025

Uh oh!

bert-e commented Oct 6, 2025

Hello sylvainsenechal,

Uh oh!

codecov bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bert-e commented Oct 13, 2025

Waiting for approval

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

williamlardier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 6, 2025 •

edited

Loading

SylvainSenechal Oct 16, 2025 •

edited

Loading

SylvainSenechal Oct 17, 2025 •

edited

Loading