Deprecate opendistro_security_roles and add opensearch_security_roles #5113

shikharj05 · 2025-02-18T09:39:34Z

Description

Deprecate opendistro_security_roles and add opensearch_security_roles for internal-user APIs.

Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Maintenance
Why these changes are required?
Deprecate legacy language from the security plugin repo in 3.0.0 for removal in 4.0.0 #5092
What is the old behavior before changes and new behavior after changes?

curl -XPUT -k -u admin:<password> "https://localhost:9200/_plugins/_security/api/internalusers/testuser" -H 'Content-Type: application/json' -d'
{
  "password": "<test-password>",
  "opensearch_security_roles": ["manage_snapshots"],
  "backend_roles": ["role 1", "role 2"],
  "attributes": {
    "attribute1": "value1",
    "attribute2": "value2"
  }
}
'

curl -k -u admin:<password> 'https://localhost:9200/_plugins/_security/api/user/testuser?pretty'                                                
{
  "testuser" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "role 1",
      "role 2"
    ],
    "attributes" : {
      "attribute1" : "value1",
      "attribute2" : "value2"
    },
    "opendistro_security_roles" : [
      "manage_snapshots"
    ],
    "opensearch_security_roles" : [
      "manage_snapshots"
    ],
    "static" : false
  }
}



curl -k -u admin:<password> 'https://localhost:9200/_plugins/_security/api/internalusers/testuser?pretty' -X PATCH -d '[{"op": "replace", "path": "/opensearch_security_roles", "value": ["notebooks_full_access","all_access"]}]' -H 'content-type: application/json'

{
  "status" : "OK",
  "message" : "'testuser' updated."
}


curl -k -u admin:<password> 'https://localhost:9200/_plugins/_security/api/user/testuser?pretty'
{
  "testuser" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "role 1",
      "role 2"
    ],
    "attributes" : {
      "attribute1" : "value1",
      "attribute2" : "value2"
    },
    "opendistro_security_roles" : [
      "notebooks_full_access",
      "all_access"
    ],
    "opensearch_security_roles" : [
      "notebooks_full_access",
      "all_access"
    ],
    "static" : false
  }
}

curl -k -u admin:<password> 'https://localhost:9200/_plugins/_security/api/internalusers/testuser?pretty' -X PATCH -d '[{"op": "replace", "path": "/opendistro_security_roles", "value": ["notebooks_full_access"]}]' -H 'content-type: application/json'     

{
  "status" : "OK",
  "message" : "'testuser' updated."
}


curl -k -u admin:<password> 'https://localhost:9200/_opendistro/_security/api/user/testuser?pretty'                                             
{
  "testuser" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "role 1",
      "role 2"
    ],
    "attributes" : {
      "attribute1" : "value1",
      "attribute2" : "value2"
    },
    "opendistro_security_roles" : [
      "manage_snapshots"
    ],
    "opensearch_security_roles" : [
      "manage_snapshots"
    ],
    "static" : false
  }
}


curl -k -u 'testuser:<test-password>' -k  'https://localhost:9200?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [cluster:monitor/main] and User [name=testuser, backend_roles=[role 1, role 2], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [cluster:monitor/main] and User [name=testuser, backend_roles=[role 1, role 2], requestedTenant=null]"
  },
  "status" : 403
}


 curl -k -u 'testuser:<test-password>' 'https://localhost:9200/_plugins/_security/api/internalusers/testuser?pretty' -X PATCH -d '[{"op": "replace", "path": "/opendistro_security_roles", "value": ["notebooks_full_access"]}]' -H 'content-type: application/json'

{
  "status" : "FORBIDDEN",
  "message" : "No permission to access REST API: User testuser with Security roles [manage_snapshots, own_index] does not have any role privileged for admin access. No client TLS certificate found in request"
}

Issues Resolved

#5098

Testing

Manual testing results above.

Logs

[2025-02-18T15:03:11,188][WARN ][org.opensearch.security.dlic.rest.api.InternalUsersApiAction] The field 'opendistro_security_roles' is deprecated and will be removed in a future release. Please use 'opensearch_security_roles' instead.

Check List

New functionality includes testing
[TODO] New functionality has been documented
[NA] New Roles/Permissions have a corresponding security dashboards plugin PR
[NA] API changes companion pull request created
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… for InternalUsersApiAction Signed-off-by: shikharj05 <[email protected]>

nibix · 2025-02-18T10:14:08Z

src/main/java/org/opensearch/security/securityconf/impl/v7/InternalUserV7.java

@@ -51,6 +51,7 @@ public class InternalUserV7 implements Hideable, Hashed, StaticDefinable {
    private Map<String, String> attributes = Collections.emptyMap();
    private String description;
    private List<String> opendistro_security_roles = Collections.emptyList();
+    private List<String> opensearch_security_roles = Collections.emptyList();


One high-level remark:

IMHO, config property names should be as self-descriptive as possible. However, neither opendistro_security_roles nor opensearch_security_roles are really self-descriptive.

The function of this property is as follows: Roles which are specified via role property first go through role-mapping so that they are mapped onto the effective roles of the user at runtime. Roles specified via opendistro_security_roles do not go through role mapping, but get effective immediately.

Can we find a property name that is (kind of) self-descriptive?

what do you think about opensearch_security_inline_roles/opensearch_security_static_roles/opensearch_security_direct_roles or any other name :) ?

I do not think the opensearch_security prefix is necessary, as the other attribute is also just called roles.

So, that would be inline_roles or static_roles or direct_roles. I would not recommend static_roles, as this could be confused with the static role configuration in https://github.com/opensearch-project/security/blob/main/src/main/resources/static_config/static_roles.yml .

I could also imagine directly_effective_roles, that would be most telling.

@cwperks @DarshitChanpura @derek-ho wdyt?

Personally, I think we should consider removing this attribute from the InternalUser object and only do roles mappings with RolesMappings.

What is the advantage of having multiple ways to map roles to usernames?

What is the advantage of having multiple ways to map roles to usernames?

I would consider these roles as default permissions that a user has when you create such an user. Also, it's simpler to setup for small number of users.

Using role mappings is better for more centralized and flexible management, better for managing similar users, etc.

I think it should be okay to keep this feature for it's simplicity (single API setup/configuration) - also, maybe it will prove helpful with service/system accounts feature in the plugin - i.e. using inline_roles or such directly_effective_roles for accounts.

What is the advantage of having multiple ways to map roles to usernames?

I would agree with @shikharj05 that one advantage is simplicity. On simple clusters without any external auth systems, this can improve the first user experience - under one constraint: It must be clear to the user which option has which purpose. If it is not clear to the user, this possibility actually makes first user experience worse, as this will cause confusion.

To make things even more complicated: There's also another config option plugins.security.roles_mapping_resolution which can be used to use the roles property of the internal users database without roles mapping.

My gist from this is: IMHO, on medium term, the property should be kept with a telling name and a crystal clear documentation. On long term, one might need to re-think the whole role mapping concept to address its larger issues.

I have renamed the property to direct_security_roles - let me know your thoughts @nibix @cwperks .

@cwperks should we create an issue to track the long-term re-evaluation of role mappings.

@cwperks should we create an issue to track the long-term re-evaluation of role mappings.

Sounds like we are going to keep supporting inline roles based on this conversation. I think we can add an issue for migration assistance here. In past versions, there was a Migrate API to help migrating the security config so should we start implementing one to account for these upcoming changes as well?

Signed-off-by: shikharj05 <[email protected]>

codecov · 2025-02-19T15:31:22Z

Codecov Report

Attention: Patch coverage is 73.52941% with 9 lines in your changes missing coverage. Please review.

Project coverage is 71.68%. Comparing base (01dd98c) to head (5b85ba6).

Files with missing lines	Patch %	Lines
...ch/security/securityconf/DynamicConfigFactory.java	30.76%	5 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5113      +/-   ##
==========================================
+ Coverage   71.67%   71.68%   +0.01%     
==========================================
  Files         337      337              
  Lines       22783    22811      +28     
  Branches     3604     3610       +6     
==========================================
+ Hits        16329    16352      +23     
- Misses       4653     4655       +2     
- Partials     1801     1804       +3

Files with missing lines	Coverage Δ
...security/dlic/rest/api/InternalUsersApiAction.java	`94.02% <100.00%> (+0.48%)`	⬆️
.../security/securityconf/impl/v7/InternalUserV7.java	`67.18% <100.00%> (+4.68%)`	⬆️
...ch/security/securityconf/DynamicConfigFactory.java	`60.49% <30.76%> (-2.01%)`	⬇️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: shikharj05 <[email protected]>

src/main/java/org/opensearch/security/dlic/rest/api/InternalUsersApiAction.java

src/integrationTest/java/org/opensearch/security/api/InternalUsersRestApiIntegrationTest.java

nibix · 2025-02-20T10:06:51Z

src/main/java/org/opensearch/security/dlic/rest/api/InternalUsersApiAction.java

+        // Add OpenDistro roles if present
+        if (openDistroRoles != null) {
+            mergedRoles.addAll(openDistroRoles);
+            log.warn(


Indirectly related: What's the migration path towards having only the new attribute?

This will only log warnings when the REST API is used. No warnings will be logged for usage of the securityadmin tool or just the existing data in the config index.

Will there be an additional change in the future to address existing data?

What's the migration path towards having only the new attribute?

Looking at other migrations/deprecation of terms, I think we can announce deprecation/removal in docs.
e.g. - https://opensearch.org/docs/latest/breaking-changes/#deprecate-non-inclusive-terms

Will there be an additional change in the future to address existing data?

Yes, I was thinking Security plugin should log a warning during initialization if configs have opendistro_security_role present. will this suffice?

src/main/java/org/opensearch/security/securityconf/DynamicConfigFactory.java

nibix · 2025-02-20T10:09:17Z

src/main/java/org/opensearch/security/securityconf/impl/v7/InternalUserV7.java

+        return direct_security_roles;
+    }
+
+    public void setDirect_security_roles(List<String> direct_security_roles) {


How does the serialized JSON produced from this bean look like? I reckon, it will contain both attributes, even if one is unused, correct?

correct, I have not removed/modified the old attribute to keep it backwards compatible. Do you think it will lead to confusion?

I am generally wondering how clients consuming that API should handle this.

This actually already applies to the security Dashboards plugin, which uses this API.

Ideally, an API could just switch the the new attribute and use that. However, in this case, all API clients would also need to have a kind of detection mechanism to see which attribute is the one that is actually in use.

This is especially critical for the PATCH API, which such clients cannot use anymore without making a GET in advance to find out what attribute would be in use.

Generally, that seems to be not a good design of an API.

I am generally wondering how clients consuming that API should handle this. This actually already applies to the security Dashboards plugin, which uses this API.

Since both the fields are maintained and updated on changes, clients using existing field will not break. As the existing field is being deprecated, guidance to use new field will be documented. As both the fields are available, I don't think a detection mechanism is required. I agree it would have been cleaner to switch to the new attribute, however I am not sure if we can do it without breaking existing clients.

See example below-

curl -k -u admin:<password> 'https://localhost:9200/_plugins/_security/api/internalusers/testuser?pretty' -X PATCH -d '[{"op": "replace", "path": "/opendistro_security_roles", "value": ["notebooks_full_access"]}]' -H 'content-type: application/json' { "status" : "OK", "message" : "'testuser' updated." } curl -k -u admin:<password> 'https://localhost:9200/_opendistro/_security/api/user/testuser?pretty' { "testuser" : { "hash" : "", "reserved" : false, "hidden" : false, "backend_roles" : [ "role 1", "role 2" ], "attributes" : { "attribute1" : "value1", "attribute2" : "value2" }, "opendistro_security_roles" : [ "manage_snapshots" ], "opensearch_security_roles" : [ "manage_snapshots" ], "static" : false } }

Signed-off-by: shikharj05 <[email protected]>

nibix · 2025-03-07T07:05:09Z

src/main/java/org/opensearch/security/securityconf/impl/v7/InternalUserV7.java

    public void setOpendistro_security_roles(List<String> opendistro_security_roles) {
+        log.warn("Deprecated configuration opendistro_security_roles set. Kindly use direct_security_roles instead.");
        this.opendistro_security_roles = opendistro_security_roles;
+        this.direct_security_roles = opendistro_security_roles;
+    }


Sorry for the delay, I was a bit busy with other stuff. I have thought about this log message again.

This method will be called by Jackson when it de-serializes the internal user JSON. It will be called for each user object.

As in the security config index, all user records are stored as one big JSON in a single document, Jackson needs to de-serialize all user objects, even the security plugin needs only one (for example for a REST request). This will have the effect that whenever the user config is being loaded from the index, you'll get this log message repeated for each user stored in that index. On clusters with many users, these will be quite a lot repeated log messages at once.

This would happen in these cases:

On each node: On cluster startup, when the index is initially read

On each node: Whenever the configuration is updated

On a single node: Whenever internalusers REST API is used (be it GET or PUT or PATCH), the index is freshly read initially, so these logs would also appear for all users, even if the REST API only addresses a single user. If this is a PUT or PATCH request, you'd see the logs repeated twice, once for reading the existing users, once for updating the changed user information.

I think such repeated log messages won't be helpful to the user. I would guess that we need a different solution to notify the user about the deprecated attribute.

No worries. Thanks for the detailed explanation. My understanding was that the log lines would be added only during initial read and during updates (limited to a specific user); And that once the configuration is in memory, it won't be fetched from the index unless a flush was called.

I will remove the log line to avoid excessive logging.

I would guess that we need a different solution to notify the user about the deprecated attribute.

Do you think documentation should be enough? something like- https://opensearch.org/docs/latest/breaking-changes/#deprecate-non-inclusive-terms

Do you think documentation should be enough?

This is actually a question which I cannot answer just on my own. This really depends on the deprecation philosophy/strategy of the whole project. I am not sure if there is actually something like this formally defined.

In my own humble opinion, a good solution would be like following (disclaimer: I really do not want to give a direction here, I am just sharing my private opinion):

There are two main alternative options:

Change name of the attribute only along with a major update of the config REST APIs. The config REST APIs have many more shortcomings, like very limited validation capabilities, no bulk update capabilities, unusual configuration of privileges, etc. This could be seen as a reason to rebuild the config REST APIs in a major effort. In that effort, this attribute could be also renamed. Up until this point, the "obsolete" name of this attribute is such a minor disturbance that it can be tolerated.

Support both attribute names, but have already at the beginning a solution that can be kept for the longer future. That means that any retrieval of the configuration will only show one attribute name, either the opendistro one or the new one. Which one is shown is determined by the last update: If the client used the new attribute name in the update, the new one is also used when retrieving the document. If the client used the old attribute name in the update, the old one is also used when retrieving the document. This state can be kept for the foreseeable future. Because there will be never a time when an admin needs to get active and convert the config, there is also no need for a deprecation message.

Which one is shown is determined by the last update: If the client used the new attribute name in the update, the new one is also used when retrieving the document. If the client used the old attribute name in the update, the old one is also used when retrieving the document.

nit: This might get problematic with number of clients? For e.g. client A using old attribute v/s client B using the new attribute.

I am more aligned with option 1 - i.e. track with a major update if required. Alternatively, we could introduce new APIs to deprecate old functionality. i.e. new APIs return new attribute names, existing ones use old one. (would have been cleaner if this was done when APIs moved to _plugins/_security from _opendistro API paths)

This might get problematic with number of clients? For e.g. client A using old attribute v/s client B using the new attribute.

Yes, good point.

Deprecate opendistro_security_roles and add opensearch_security_roles…

8c6601b

… for InternalUsersApiAction Signed-off-by: shikharj05 <[email protected]>

shikharj05 requested review from cwperks, DarshitChanpura, derek-ho, nibix, peternied, RyanL1997, reta and willyborankin as code owners February 18, 2025 09:39

shikharj05 marked this pull request as draft February 18, 2025 09:39

nibix reviewed Feb 18, 2025

View reviewed changes

shikharj05 added 6 commits February 18, 2025 17:19

Fix rest integ tests

54c72fa

Signed-off-by: shikharj05 <[email protected]>

Get combined list of roles while validating the roles

ab26317

Signed-off-by: shikharj05 <[email protected]>

spotless apply

4b27f5c

Signed-off-by: shikharj05 <[email protected]>

Merge branch 'opensearch-project:main' into opendistro-api-req-resp

afe19d8

Rename opensearch_security_roles to direct_security_roles

cb3e1cd

Signed-off-by: shikharj05 <[email protected]>

spotless apply

879f575

Signed-off-by: shikharj05 <[email protected]>

Adding tests for getRoles method

23b5916

Signed-off-by: shikharj05 <[email protected]>

shikharj05 marked this pull request as ready for review February 19, 2025 17:12

shikharj05 added 2 commits February 19, 2025 22:43

Remove test log configuration

47d755a

Signed-off-by: shikharj05 <[email protected]>

Merge branch 'main' into opendistro-api-req-resp

ba574b6

cwperks reviewed Feb 19, 2025

View reviewed changes

src/main/java/org/opensearch/security/dlic/rest/api/InternalUsersApiAction.java Outdated Show resolved Hide resolved

nibix reviewed Feb 20, 2025

View reviewed changes

src/integrationTest/java/org/opensearch/security/api/InternalUsersRestApiIntegrationTest.java Outdated Show resolved Hide resolved

nibix reviewed Feb 20, 2025

View reviewed changes

src/main/java/org/opensearch/security/securityconf/DynamicConfigFactory.java Outdated Show resolved Hide resolved

nibix reviewed Feb 20, 2025

View reviewed changes

Merge branch 'opensearch-project:main' into opendistro-api-req-resp

20768b0

cwperks previously approved these changes Feb 27, 2025

View reviewed changes

Update integ tests for both roles

736dc78

Signed-off-by: shikharj05 <[email protected]>

shikharj05 dismissed cwperks’s stale review via 736dc78 March 3, 2025 12:50

shikharj05 added 2 commits March 3, 2025 18:34

Adding log line for depreacted opendistro_security_roles

7e0fcc9

Signed-off-by: shikharj05 <[email protected]>

Add more logs for deprecation notice

5f10ad6

Signed-off-by: shikharj05 <[email protected]>

shikharj05 requested review from nibix and cwperks March 5, 2025 18:32

shikharj05 added 2 commits March 6, 2025 00:03

Merge branch 'main' into opendistro-api-req-resp

9e28622

Merge branch 'main' into opendistro-api-req-resp

0674469

nibix reviewed Mar 7, 2025

View reviewed changes

Merge branch 'main' into opendistro-api-req-resp

5b85ba6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deprecate opendistro_security_roles and add opensearch_security_roles #5113

Deprecate opendistro_security_roles and add opensearch_security_roles #5113

shikharj05 commented Feb 18, 2025 •

edited

Loading

nibix Feb 18, 2025

shikharj05 Feb 18, 2025

nibix Feb 18, 2025 •

edited

Loading

cwperks Feb 18, 2025

shikharj05 Feb 18, 2025

nibix Feb 19, 2025

shikharj05 Feb 19, 2025

cwperks Feb 19, 2025

codecov bot commented Feb 19, 2025 •

edited

Loading

nibix Feb 20, 2025

shikharj05 Feb 24, 2025

nibix Feb 20, 2025

shikharj05 Feb 24, 2025

nibix Mar 1, 2025

shikharj05 Mar 3, 2025

nibix Mar 7, 2025 •

edited

Loading

shikharj05 Mar 7, 2025

nibix Mar 10, 2025 •

edited

Loading

shikharj05 Mar 13, 2025

nibix Mar 17, 2025

Deprecate opendistro_security_roles and add opensearch_security_roles #5113

Are you sure you want to change the base?

Deprecate opendistro_security_roles and add opensearch_security_roles #5113

Conversation

shikharj05 commented Feb 18, 2025 • edited Loading

Description

Issues Resolved

Testing

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nibix Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 19, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nibix Mar 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nibix Mar 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shikharj05 commented Feb 18, 2025 •

edited

Loading

nibix Feb 18, 2025 •

edited

Loading

codecov bot commented Feb 19, 2025 •

edited

Loading

nibix Mar 7, 2025 •

edited

Loading

nibix Mar 10, 2025 •

edited

Loading