bucket notifications - facilitate notif conf to override connection #8932
Conversation
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
2 times, most recently
from
391fe76 to
b32378a
Compare
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
b32378a to
5d5ee14
Compare
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
5d5ee14 to
f1e6bd3
Compare
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
4 times, most recently
from
56759b8 to
3871f68
Compare
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
3871f68 to
6447e8e
Compare
| @@ -46,8 +46,8 @@ const FROM_FILE = 'from_file'; | |||
| const ANONYMOUS = 'anonymous'; | |||
|
|
|||
| const VALID_OPTIONS_ACCOUNT = { | |||
| 'add': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', FROM_FILE, ...CLI_MUTUAL_OPTIONS]), | |||
| 'update': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', 'new_name', 'regenerate', ...CLI_MUTUAL_OPTIONS]), | |||
| 'add': new Set(['name', 'uid', 'gid', 'supplemental_groups', 'new_buckets_path', 'user', 'access_key', 'secret_key', 'fs_backend', 'allow_bucket_creation', 'force_md5_etag', 'iam_operate_on_root_account', 'default_connection', FROM_FILE, ...CLI_MUTUAL_OPTIONS]), | |||
There was a problem hiding this comment.
why is this part of the account configuration. shouldn't it be per bucket?
There was a problem hiding this comment.
Apparently in Marc's project the user and connection creation would be done by one admin user (with manage_nsfs cli), while buckets and their notif conf would be done by other (not-admin) users (with s3 rest api). So the admin would be able to do a once-per-system users and connection creation (including user's default connection), and then the other users wouldn't need to know about connections at all.
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
6447e8e to
9d199ad
Compare
|
|
||
| //adapt to db shcema | ||
| if (topic_configuration) { | ||
| for (const conf of topic_configuration) { | ||
| conf.id = conf.Id; | ||
| conf.event = conf.Event; | ||
| conf.topic = conf.Topic; | ||
| //handle Kafka's topic synax, if present | ||
| if (conf.Topic && conf.Topic.length > 0 && conf.Topic[0].startsWith('kafka:::topic/')) { |
There was a problem hiding this comment.
wouldn't it be also good to have a validation on the cli level. that you wouldn't be able to configure default_connection of this format?
There was a problem hiding this comment.
I'm not sure I understand.
Account's default_connection is just a string. It doesn't have a "special" syntax (as opposed to conf.Topic's optional "kafka:::topic" syntax).
There was a problem hiding this comment.
ok, if it doesn't have a "special" syntax, its fine
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
2 times, most recently
from
7961043 to
399e208
Compare
|
Guy's comments were resolved, we can merge |
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
Signed-off-by: Amit Prinz Setter <[email protected]>
alphaprinz
force-pushed
the
notif_kafka_connect2
branch
from
399e208 to
3c9806d
Compare
Describe the Problem
We want to allow user to describe an external Kafka server in a connection file, but allow to specify to specify Kafka topic per notification conf (instead of in the connection).
Explain the Changes
TopicConfigurations' TopicArn field can be of this syntax "kafka:::topic//", where:
-if connection is not specified, the default account's connection is used.
-Kafka topic in TopicArn takes precedence over Kafka topic in connection.
-backward compatible - you can still put just the connection in TopicArn, like now
Issues: Fixed #xxx / Gap #xxx
Testing Instructions:
Set "default_connection" field in NC user.
Create a connection file for kafka (with or without topic field).
{
"notification_protocol": "kafka",
"name": "k",
"topic": "mytopic",
"kafka_options_object": "{"metadata.broker.list":"localhost:9092"}"
}
Configure a notification with override object:
{
"TopicConfigurations": [
{
"Id": "first",
"TopicArn": "kafka:::topic//override",
"Events": []
}
]
}
Create an event (ie upload object to bucket with notification).
The notification will be sent with user's default connection with topic "override".