Rabbitmq vhost and user support #320
Conversation
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/43a7be44149e4a80a82fc6b816e9f29b ✔️ openstack-meta-content-provider-master SUCCESS in 2h 24m 19s |
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/fcb50a37da904b558dc9d03f6c6b6eb0 ✔️ openstack-meta-content-provider-master SUCCESS in 2h 08m 29s |
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ee8b8635fe8241a7aae1b57c091777e0 ✔️ openstack-meta-content-provider-master SUCCESS in 1h 31m 19s |
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
e216e40 to
8133c4d
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/5deaf237b4c84f9394a94fed49bfe09c ✔️ openstack-meta-content-provider-master SUCCESS in 2h 53m 25s |
amoralej
left a comment
amoralej
left a comment
There was a problem hiding this comment.
I'd like to also modify the kuttl tests to cover the new parameters. You can use the existing kuttl tests, i.e. the watcher-notification or some other, watcher-topology, etc... to test non-default values for the new spec params.
I also left some inline comments.
| description: Name of the cluster | ||
| minLength: 1 | ||
| type: string | ||
| user: |
There was a problem hiding this comment.
im not sure if this makes seense either.
the reaosin i say that is when i filed https://issues.redhat.com/browse/OSPRH-92 orgianly it was also covering rothation of the rabbit mq password
it was scoped down to only the db for GA but
the intent was to intoduce a messaging busss account CR following the mariadb CR patthern to allow password rotation while having 2 active password/users
to do that we would need to generate a new user/password password pair so that the old pass word can remain active after the contolplane has rotaited to allow time of for the edpm deployment.
inother words we cannot have the user be part of this struct unless rabbit supprot having 2 active password for the same user.
There was a problem hiding this comment.
Thanks for the feedback. The way credentials rotation can be achieved is by simply switching the user in the cr, so you could start from "user-old" and switch to "user-new". Both credentials will be valid until a human admin decides to manually remove the unused one. We did not implement auto cleanup as it would break edpm nodes, plus we want to allow rolling back to old credentials if required.
There was a problem hiding this comment.
that seams messy vs the dedicated maria db account where the db is account is removed when the account CR is removed and we use a finalsize to prevent tis deletion until all usage of it is remvoed form all the crs that refence it.
are there plans to model the rabbit acocunts as CRDs in a similar way?
watcher does nto have any edpm compoents so its less relenvet for this PR but it a factor for nova
ideally we shoudl not have human operators direcly interacting with rabbit.
so my suggestion is to replace user with an account name which fence a rabbit account opbejct like the database account CRD.
the rotation woudl basically happen the way you suggest you create a new account obejct which will be reconsiled by the infra operator and update the value in the service template which will propagate to the service operator to reconisle.
during that reconsiliation they will remove there finaliser form the old account CR and add it to the new one.
once teh human has completed the edpm deploy ment they will delete the old account cr.
the infra operator would hten remove that user and password form rabit.
if they want to revert at any point before they do that delete they just need to revert the refence rabbit account object.
There was a problem hiding this comment.
That's how it has been implemented already. Human operators only have to specify the username and a rabbitmquser crd is created for them, similarly to the db account. Finalizers are also moved to the new account and removed from the old one for rotation purposes. See openstack-k8s-operators/infra-operator@f4eab21 .
There was a problem hiding this comment.
the db account have to be created seperalty and then the name of the account passed in
so this is not workign the saem. unless this is the name of the rabbitmquer cr not the username.
if we have a rabbitmquser CRD then we shoudl be takign the name of the rabbitmquser CR here not the user name ot have this work the same as the db password.
can you make that change?
There was a problem hiding this comment.
for what its worht i kind of see the current propsoal to autocare the user (and password) just bsed on teh user name as a security problem that would prevent use merging this.
bsiclly it provde a very easy way to typo the name have a user created and then not notice this again since it wont show up in the controlplan CR
you would have to expclitly go looking for the RabbitmqUser crs to keep track fo them and manually corralate them.
There was a problem hiding this comment.
I have implemented the automatic cleanup of orphaned users, either created by typos then corrected or as result of credentials rotation, so hopefully this should take care of the security aspect (transporturl controller in infra-operator will take care of the cleanup).
As for what happens if there is no username specified - the service will use the default rabbitmq user as it does today, so from a end user perspective this is an opt-in to using new dedicated credentials.
There was a problem hiding this comment.
ok. i woudl still prefer to have the user refer to a RabbitmqUser cr name rather then the username in rabit itself but if you have impleneted the cleanup the i can consider the security element (leaking user accounts) to be adressed and the functioal blocker to be reoslved.
There was a problem hiding this comment.
ill resolve this an remove the requested chagne state ocne i review where/how you did that got a pointer?
There was a problem hiding this comment.
ah its here openstack-k8s-operators/infra-operator#523
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
8133c4d to
f5abf3f
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/18dac81b57c3435e8ee3c89c8255337d ✔️ openstack-meta-content-provider-master SUCCESS in 4h 47m 04s |
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
4 times, most recently
from
db92d54 to
ff28815
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/ae2a6a20346043d7b23e2e2dc5675058 ✔️ openstack-meta-content-provider-master SUCCESS in 2h 52m 59s |
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
2 times, most recently
from
f749ebf to
e2c7b37
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/dab5bd6c4e9c47219f0d96d09cbf2caf ✔️ openstack-meta-content-provider-master SUCCESS in 3h 36m 47s |
| description: Name of the cluster | ||
| minLength: 1 | ||
| type: string | ||
| user: |
There was a problem hiding this comment.
for what its worht i kind of see the current propsoal to autocare the user (and password) just bsed on teh user name as a security problem that would prevent use merging this.
bsiclly it provde a very easy way to typo the name have a user created and then not notice this again since it wont show up in the controlplan CR
you would have to expclitly go looking for the RabbitmqUser crs to keep track fo them and manually corralate them.
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
e2c7b37 to
a5d1879
Compare
|
Master job is failing with: Note that the openstackcontrolplane is creating in that job includes old-style notifications declaration using notificationsBusInstance https://logserver.rdoproject.org/e73/rdoproject.org/e73efdd0de6440b2b1aac1ec3c485810/controller/ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-5a235be74bde76726d2fe232519162c11cacc7ebe133103b4cd177993898d231/namespaces/openstack/crs/openstackcontrolplanes.core.openstack.org/controlplane.yaml Configuration for that job is in: watcher-operator/ci/scenarios/edpm.yml Lines 60 to 85 in add353f |
|
This change depends on a change that failed to merge. Change openstack-k8s-operators/openstack-operator#1779 is needed. |
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
e5cad3a to
ec9703c
Compare
|
This change depends on a change that failed to merge. Change openstack-k8s-operators/openstack-operator#1779 is needed. |
|
check-rdo |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/a3a7c90e1fdc44a9ac0d4ef27203dc35 ✔️ openstack-meta-content-provider-master SUCCESS in 2h 35m 51s |
Add new messagingBus and notificationsBus interfaces to hold cluster,
user and vhost names for optional usage.
The controller adds these values to the TransportURL create request when present.
Additionally, we migrate RabbitMQ cluster name to RabbitMq config struct
using DefaultRabbitMqConfig from infra-operator to automatically
populate the new Cluster field from legacy RabbitMqClusterName.
Example usage:
spec:
messagingBus:
cluster: rpc-rabbitmq
user: rpc-user
vhost: rpc-vhost
notificationsBus:
cluster: notifications-rabbitmq
user: notifications-user
vhost: notifications-vhost
Jira: https://issues.redhat.com/browse/OSPRH-23882
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
ec9703c to
8b30c5e
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/470494c2f6ae44f8a4c235e85c208a03 ❌ openstack-meta-content-provider-master FAILURE in 34m 51s |
|
check-rdo |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c24687696303477da1e8550bfb025939 ✔️ openstack-meta-content-provider-master SUCCESS in 2h 50m 25s |
|
check-rdo |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/e90ffd4b0d2444b79af831f53b6d4e51 ✔️ openstack-meta-content-provider-master SUCCESS in 2h 42m 41s |
|
This change depends on a change that failed to merge. Change openstack-k8s-operators/openstack-operator#1779 is needed. |
amoralej
left a comment
amoralej
left a comment
There was a problem hiding this comment.
Just some minor questions, other than that this patch LGTM.
I've also tested this manually with watcher CRs in isolation and worked as expected. The openstack-operator patch in depends-on is still on progress at this point.
| } | ||
|
|
||
| // validateDeprecatedFieldsCreate validates deprecated fields during CREATE operations | ||
| func (spec *WatcherSpecCore) validateDeprecatedFieldsCreate(basePath *field.Path) ([]string, field.ErrorList) { |
There was a problem hiding this comment.
validateDeprecatedFieldsCreate is never returning errors, only warnings. The interface may be simplified to return only []string at this point. Said this, we can leave it for potential future cases.
ci/scenarios/edpm.yml
Outdated
| - patch: |- | ||
| apiVersion: core.openstack.org/v1beta1 | ||
| kind: OpenStackControlPlane | ||
| metadata: | ||
| name: controlplane | ||
| spec: | ||
| keystone: | ||
| template: | ||
| rabbitMqClusterName: "" |
There was a problem hiding this comment.
I guess this is unrelated to this patch? (i guess related to openstack-operator change)
| notificationsRabbitMqConfig := *instance.Spec.NotificationsBus | ||
|
|
||
| // Append cluster name to the TransportURL name to make it unique when using different clusters | ||
| notificationTransportURLName := fmt.Sprintf("%s-watcher-notification-%s", instance.Name, notificationsRabbitMqConfig.Cluster) |
There was a problem hiding this comment.
This is effectively changing tha name of the transporturl during operator update. I guess openstack-operator will automatically remove the old one?
There was a problem hiding this comment.
Thanks, I added the transporturl delete logic modeled after nova's. Tested switching notifications on/off/changing cluster and it seems to be working as expected.
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
06571b5 to
7447609
Compare
|
This change depends on a change that failed to merge. Change openstack-k8s-operators/openstack-operator#1779 is needed. |
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
7447609 to
041ac0b
Compare
|
This change depends on a change that failed to merge. Change openstack-k8s-operators/openstack-operator#1779 is needed. |
|
recheck |
|
check-rdo |
api/v1beta1/watcher_webhook.go
Outdated
| // Default NotificationsBus.Cluster if NotificationsBus is present but Cluster is empty | ||
| if spec.NotificationsBus != nil && spec.NotificationsBus.Cluster == "" { | ||
| spec.NotificationsBus.Cluster = "rabbitmq" | ||
| } |
There was a problem hiding this comment.
This means the user set up user and/or vhost but not cluster iiuc. For me that should fail validation. Setting the same rabbitmq for messagingBus and notificationBus should be something that could only be done by explicitely setting it, not by defaulting rules, IMO.
Also, i checked that adding any wrong content to NotificationsBus, i.e. {}, ends up in this setting rabbitmq for notifications.
api/v1beta1/watcher_webhook.go
Outdated
| spec.MessagingBus.Cluster = "rabbitmq" | ||
| } | ||
|
|
||
| // Default NotificationsBus.Cluster if NotificationsBus is present but Cluster is empty |
There was a problem hiding this comment.
I liked the previous way of overriding spec.NotificationsBus.Cluster from NotificationsBusInstance if this is set and NotificationsBus.Cluster was empty. I think it'd avoid racy situations between openstack-operators and watcher-operator in update conditions. Said this, i guess it will run fine in most cases and will end up in reconciling properly (unless there is any other problem in the update related to the openstackcontrolplane content, i.e.)
There was a problem hiding this comment.
As reminded by Luca, the webhook only runs in the openstack-operator for the openstackcontrolplane, not in the watcher-operator, so actually, adding that logic to the webhook will not help to fix the race i mentioned before.
There was a problem hiding this comment.
I think this is a valid concern if each service operator runs in isolation (like in CI). Since we embed the webhooks in the openstack-operator I feel like we are somehow guaranteed that the defaulting/migration logic will run in the expected order, and that the watcher-operator will receive a valid configuration.
|
This change depends on a change that failed to merge. Change openstack-k8s-operators/openstack-operator#1779 is needed. |
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
041ac0b to
83cb18f
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/707dc3a9310d47399eeca91b1f3871fd ❌ openstack-meta-content-provider-master FAILURE in 46m 03s |
lmiccini
force-pushed
the
rabbitmq_vhosts
branch
from
83cb18f to
6da4849
Compare
|
This change depends on a change that failed to merge. Change openstack-k8s-operators/openstack-operator#1779 is needed. |
|
check-rdo |
4 participants
Add new messagingBus and notificationsBus interfaces to hold cluster, user and vhost names for optional usage.
The controller adds these values to the TransportURL create request when present.
Additionally, we migrate RabbitMQ cluster name to RabbitMq config struct using DefaultRabbitMqConfig from infra-operator to automatically populate the new Cluster field from legacy RabbitMqClusterName.
Example usage:
Jira: https://issues.redhat.com/browse/OSPRH-23882
Depends-on: openstack-k8s-operators/openstack-operator#1779