bucket notifications - facilitate notif conf to override connection #8932
Conversation
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
2 times, most recently
from
391fe76 to
b32378a
Compare
src/util/notifications_util.js
Outdated
| const filename_parts = connect_filename_with_overrides.split('?'); | ||
| const connect_filename_no_overrides = filename_parts[0]; | ||
| const overrides_str = filename_parts[1]; |
There was a problem hiding this comment.
I think this ad hoc arn structure is not a good idea, we should try to map it to an arn format.
For kafka I expect the kafka arn format like in aws where ClusterName is our connection name, ClusterUuid should match the kafka cluster uuid , and TopicName would be optional:
arn:aws:kafka:::cluster/${ClusterName}/${ClusterUuid}
arn:aws:kafka:::topic/${ClusterName}/${ClusterUuid}/${TopicName}
For http webhook there is no matching arn in aws, however there is api-gateway or lambda. I would go with something in the spirit of api-gateway execute-api arn where ApiId is our connection name, Stage is less relevant as we do not manage the webhook lifecycle but I would still keep it as part of the arn structure, and finally method and path are very relevant:
arn:aws:execute-api:::${ApiId}/${Stage}/${Method}/${ApiSpecificResourcePath}
There was a problem hiding this comment.
I've made the topic for kafka to optionally be "kafka:::topic".
Still haven't found any mention of cluster UUID in the client.
Note that using '/' separator in containerized is somewhat confusing because I also need secret name in addition to connection name (currently, at least).
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
b32378a to
5d5ee14
Compare
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
5d5ee14 to
f1e6bd3
Compare
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
4 times, most recently
from
56759b8 to
3871f68
Compare
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
3871f68 to
6447e8e
Compare
| @@ -46,8 +46,8 @@ const FROM_FILE = 'from_file'; | |||
| const ANONYMOUS = 'anonymous'; | |||
|
|
|||
| const VALID_OPTIONS_ACCOUNT = { | |||
| 'add': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', FROM_FILE, ...CLI_MUTUAL_OPTIONS]), | |||
| 'update': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', 'new_name', 'regenerate', ...CLI_MUTUAL_OPTIONS]), | |||
| 'add': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', 'default_connection', FROM_FILE, ...CLI_MUTUAL_OPTIONS]), | |||
There was a problem hiding this comment.
why is this part of the account configuration. shouldn't it be per bucket?
There was a problem hiding this comment.
Apparently in Marc's project the user and connection creation would be done by one admin user (with manage_nsfs cli), while buckets and their notif conf would be done by other (not-admin) users (with s3 rest api). So the admin would be able to do a once-per-system users and connection creation (including user's default connection), and then the other users wouldn't need to know about connections at all.
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
6447e8e to
9d199ad
Compare
|
|
||
| //adapt to db shcema | ||
| if (topic_configuration) { | ||
| for (const conf of topic_configuration) { | ||
| conf.id = conf.Id; | ||
| conf.event = conf.Event; | ||
| conf.topic = conf.Topic; | ||
| //handle Kafka's topic synax, if present | ||
| if (conf.Topic && conf.Topic.length > 0 && conf.Topic[0].startsWith('kafka:::topic/')) { |
There was a problem hiding this comment.
wouldn't it be also good to have a validation on the cli level. that you wouldn't be able to configure default_connection of this format?
Describe the Problem
We want to allow user to describe an external Kafka server in a connection file, but allow to specify to specify Kafka topic per notification conf (instead of in the connection).
Explain the Changes
TopicConfigurations' TopicArn field can be of this syntax "kafka:::topic//", where:
-if connection is not specified, the default account's connection is used.
-Kafka topic in TopicArn takes precedence over Kafka topic in connection.
-backward compatible - you can still put just the connection in TopicArn, like now
Issues: Fixed #xxx / Gap #xxx
Testing Instructions:
Set "default_connection" field in NC user.
Create a connection file for kafka (with or without topic field).
{
"notification_protocol": "kafka",
"name": "k",
"topic": "mytopic",
"kafka_options_object": "{"metadata.broker.list":"localhost:9092"}"
}
Configure a notification with override object:
{
"TopicConfigurations": [
{
"Id": "first",
"TopicArn": "kafka:::topic//override",
"Events": []
}
]
}
Create an event (ie upload object to bucket with notification).
The notification will be sent with user's default connection with topic "override".