Skip to content

Replace backbeat client with cloudserver client #2679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: development/9.1
Choose a base branch
from
+1,666 −1,880
Draft

Replace backbeat client with cloudserver client #2679

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions extensions/gc/tasks/GarbageCollectorTask.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ const async = require('async');
const errors = require('arsenal').errors;
const { ObjectMD } = require('arsenal').models;

const { attachReqUids } = require('../../../lib/clients/utils');
const BackbeatTask = require('../../../lib/tasks/BackbeatTask');
const { BatchDeleteCommand } = require('@scality/cloudserverclient');
const { GarbageCollectorMetrics } = require('../GarbageCollectorMetrics');
/** @typedef { import('../GarbageCollector.js') } GarbageCollector */

Expand Down Expand Up @@ -122,16 +122,20 @@ class GarbageCollectorTask extends BackbeatTask {
}

const backbeatClient = this.getBackbeatClient(accountId);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note : There are a lot of variables, function names and possibly file names that would benefit from a rename, from backbeatClient to cloudserverClient etc.
I don't wanna do it now because it will add a lot of noise to that PR, and it's not even that straightforward to do with this silly programming language


if (!backbeatClient) {
log.error('failed to get backbeat client', { accountId });
return done(errors.InternalError
.customizeDescription('Unable to obtain client'));
}

const req = backbeatClient.batchDelete(params);
attachReqUids(req, log);
return req.send(done);
const command = new BatchDeleteCommand({
...params,
RequestUids: log.getSerializedUids(),
});

return backbeatClient.send(command)
.then(() => done())
.catch(err => done(err));
});
}

Expand Down
48 changes: 29 additions & 19 deletions extensions/lifecycle/tasks/LifecycleDeleteObjectTask.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ const ObjectMD = require('arsenal').models.ObjectMD;
const BackbeatTask = require('../../../lib/tasks/BackbeatTask');
const { attachReqUids } = require('../../../lib/clients/utils');
const { LifecycleMetrics } = require('../LifecycleMetrics');
const {
DeleteObjectFromExpirationCommand
} = require('@scality/cloudserverclient');
/** @typedef { import('../objectProcessor/LifecycleObjectProcessor.js') } LifecycleObjectProcessor */

class ObjectLockedError extends Error {}
Expand Down Expand Up @@ -140,26 +143,33 @@ class LifecycleDeleteObjectTask extends BackbeatTask {
LifecycleMetrics.onLifecycleStarted(log,
'expiration',
location, startTime - transitionTime);
const req = client.deleteObjectFromExpiration(reqParams);
attachReqUids(req, log);
req.send(err => {
if (err?.statusCode === errors.MethodNotAllowed.code) {
log.warn('deleteObjectFromExpiration API not supported, falling back to deleteObject',
logDetails);
// fallback to s3 deleteObject when using a cloudserver that
// doesn't support deleteObjectFromExpiration
const s3Client = this.getS3Client(accountId);
if (!s3Client) {
log.error('failed to get s3 client', logDetails);
return done(errors.InternalError
.customizeDescription('Unable to obtain client'));
}
const s3Req = s3Client.deleteObject(reqParams);
attachReqUids(s3Req, log);
return s3Req.send(done);
}
return done(err);
const command = new DeleteObjectFromExpirationCommand({
...reqParams,
RequestUids: log.getSerializedUids(),
});

client.send(command)
.then(() => {
done(null);
})
.catch(err => {
if (err?.$metadata?.httpStatusCode === errors.MethodNotAllowed.code) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we test for the err "instanceof" first?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer to stay with the codes here, with the smithy client we have guarantees to have error codes in err?.$metadata?.httpStatusCode, but not sure we have proper error types

log.warn('deleteObjectFromExpiration API not supported, falling back to deleteObject',
logDetails);
// fallback to s3 deleteObject when using a cloudserver that
// doesn't support deleteObjectFromExpiration
const s3Client = this.getS3Client(accountId);
if (!s3Client) {
log.error('failed to get s3 client', logDetails);
return done(errors.InternalError
.customizeDescription('Unable to obtain client'));
}
const s3Req = s3Client.deleteObject(reqParams);
attachReqUids(s3Req, log);
return s3Req.send(done);
}
return done(err);
});
}

_executeDelete(entry, startTime, log, done) {
Expand Down
4 changes: 2 additions & 2 deletions extensions/lifecycle/tasks/LifecycleTaskV2.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ class LifecycleTaskV2 extends LifecycleTask {
// Undefined params can happen when lifecycle configuration rule is disabled for example.
if (!params) {
log.debug('no appropriate listing found', {
method: 'LifecycleTaskV2._getObjectList',
method: 'LifecycleTaskV2._getObjectVersions',
bucketLCRules,
currentDate,
bucketData,
Expand Down Expand Up @@ -213,7 +213,7 @@ class LifecycleTaskV2 extends LifecycleTask {
if (!err) {
log.debug(
'sent kafka entry for bucket consumption', {
method: 'LifecycleTaskV2._getObjectList',
method: 'LifecycleTaskV2._getObjectVersions',
});
}
});
Expand Down
Loading
Loading