[Attack Discovery][Scheduling] Attack Discovery scheduling rule management (#12003)

[e40pud]

Summary

Main ticket (Internal link)

To allow users to schedule Attack Discovery generations, we will use either Alerting Framework. These changes add functionality to manage new alerts type - Attack Discovery Schedule.

Introduced endpoints

• Create AD scheduling rule route: POST /internal/elastic_assistant/attack_discovery/schedules
• Read/Get AD scheduling rule by id route: GET /internal/elastic_assistant/attack_discovery/schedules/{id}
• Update AD scheduling rule by id route: PUT /internal/elastic_assistant/attack_discovery/schedules/{id}
• Delete AD scheduling rule by id route: DELETE /internal/elastic_assistant/attack_discovery/schedules/{id}
• Enable AD scheduling rule by id route: POST /internal/elastic_assistant/attack_discovery/schedules/{id}/_enable
• Disable AD scheduling rule by id route: POST /internal/elastic_assistant/attack_discovery/schedules/{id}/_disable
• Find all existing AD scheduling rules route: GET /internal/elastic_assistant/attack_discovery/schedules/_find

NOTES

The feature is hidden behind the feature flag:

xpack.securitySolution.enableExperimental: ['assistantAttackDiscoverySchedulingEnabled']

cURL examples

Create AD scheduling rule route

curl --location 'http://localhost:5601/internal/elastic_assistant/attack_discovery/schedules' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'Content-Type: application/json' \
--data '{
    "name": "Test Schedule",
    "schedule": {
        "interval": "10m"
    },
    "params": {
        "alertsIndexPattern": ".alerts-security.alerts-default",
        "apiConfig": {
            "connectorId": "gpt-4o",
            "actionTypeId": ".gen-ai"
        },
        "end": "now",
        "size": 100,
        "start": "now-24h"
    }
}'

Read/Get AD scheduling rule by id route

curl --location 'http://localhost:5601/internal/elastic_assistant/attack_discovery/schedules/{id}' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: security-solution'

Update AD scheduling rule by id route

curl --location --request PUT 'http://localhost:5601/internal/elastic_assistant/attack_discovery/schedules/{id}' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'Content-Type: application/json' \
--data '{
    "name": "Test Schedule - Updated",
    "schedule": {
        "interval": "123m"
    },
    "params": {
        "alertsIndexPattern": ".alerts-security.alerts-default",
        "apiConfig": {
            "connectorId": "gpt-4o",
            "actionTypeId": ".gen-ai"
        },
        "end": "now",
        "size": 35,
        "start": "now-24h"
    },
    "actions": []
}'

Delete AD scheduling rule by id route

curl --location --request DELETE 'http://localhost:5601/internal/elastic_assistant/attack_discovery/schedules/{id}' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: security-solution'

Enable AD scheduling rule by id route

curl --location --request POST 'http://localhost:5601/internal/elastic_assistant/attack_discovery/schedules/{id}/_enable' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: security-solution'

Disable AD scheduling rule by id route

curl --location --request POST 'http://localhost:5601/internal/elastic_assistant/attack_discovery/schedules/{id}/_disable' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: security-solution'

Find all existing AD scheduling rules route

curl --location 'http://localhost:5601/internal/elastic_assistant/attack_discovery/schedules/_find' \
--header 'kbn-xsrf: true' \
--header 'elastic-api-version: 1' \
--header 'x-elastic-internal-origin: security-solution'

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[e40pud]

Same privileges as we have for the detection alerts https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/packages/features/src/security/v2_features/kibana_features.ts#L95

[e40pud]

Updated because we added a new attack discovery alerts aad index that maps the ecs fields. Similarly to this https://github.com/elastic/kibana/pull/194322/files#r1781620194 and this https://github.com/elastic/kibana/pull/184541/files#r1628183656.

[nreese]

kibana-presentation integration test assertion update changes LGTM
code review only

[andrew-goldstein]

Thanks @e40pud for the new scheduling APIs!
✅ Desk tested locally
LGTM 🚀

[pmuellr]

Usually these files are generated from files with ES mappings, I believe mainly in the peer directory src/field_maps. Not seeing where it might have been generated for.

Which makes me wonder if we've defined mappings for these fields in the alerts documents. Perhaps this is done differently for security rules, or this is part of a feature branch where some of that work was already done.

In any case, we should make sure we have the ES mappings in place for these field, if we want them indexed.

[e40pud]

The mappings are defined in this file https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/elastic_assistant/server/lib/attack_discovery/schedules/fields/field_map.ts.

I followed other solutions including security while adding these mappings and this new alert type. Similarly to o11y, ml, security, stack etc. the mappings are defined inside those solutions and the generated schemas then added via FTR test script inside the src/platform/packages/shared/kbn-alerts-as-data-utils/src/schemas/generated/.

Here is example of similarly defined mappings https://github.com/elastic/kibana/blob/main/x-pack/solutions/observability/plugins/apm/server/routes/alerts/register_apm_rule_types.ts

[pmuellr]

I'm curious why a rule is used here instead of a task manager task. Task manager tasks are generally more open-ended, but there's also a lot of built-in stuff with rules like metrics, APM, logging, etc. Not a problem, just curious.

@mikecote Code looks good to me, but I noticed you added it to our board last week. The rule executor is empty at the moment, so presumably that'll come in the future :-). Was wondering if there was anything specific to look for here ...

[mikecote]

@mikecote Code looks good to me, but I noticed you added it to our board last week. The rule executor is empty at the moment, so presumably that'll come in the future :-). Was wondering if there was anything specific to look for here ...

@pmuellr I think the PR scope might have changed since I added it. But I basically added it for a codeowners review. We should see a follow up PR soon for the rule type @e40pud ?

[e40pud]

@pmuellr I think the PR scope might have changed since I added it. But I basically added it for a codeowners review. We should see a follow up PR soon for the rule type @e40pud ?

@mikecote correct. There will be a follow up PR and I will add you both to it to review. With this PR I setup rules management APIs and introduced a new rule type.

@pmuellr we decided to move with the alerting framework rules because of:

• After many discussions everyone aligned on a vision of attack discoveries as alerts/signals. At some point they will be presented on the same page with existing detection alerts/signals in security solution.
• Time limits that we have. This feature needs to be mostly done by the end of April and fully available in 8.19/9.1 versions.
• We need to have notifications/actions support (including conditional actions) and we did not want to reimplement all this due to time limits. Same applies to other existing tools handled already by the framework.

[semd]

LGTM

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1962ca4

Failed CI Steps

• FTR Configs #49

Test Failures

• [job] [logs] FTR Configs #49 / discover/group1 discover accessibility top nav menu buttons should return focus to the alerts button when dismissing the alerts popover

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
automaticImport	728	731	+3
securitySolution	7192	7195	+3
total			+6

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/alerts-as-data-utils	33	34	+1
@kbn/elastic-assistant-common	446	505	+59
elasticAssistant	44	46	+2
total			+62

Unknown metric groups

API count

id	before	after	diff
@kbn/alerts-as-data-utils	33	34	+1
@kbn/elastic-assistant-common	519	592	+73
elasticAssistant	60	62	+2
total			+76

History

• 💚 Build #291292 succeeded 9ddd16a
• 💔 Build #291218 failed 9312d21
• 💔 Build #291178 failed 36f2c44
• 💔 Build #291119 failed 61ab656
• 💔 Build #291097 failed d4285c4
• 💔 Build #291063 failed 71eb297

cc @e40pud

[pmuellr]

@e40pud thanks for the description of task vs rule.

I was curious to see the wrapper over alerting client in
.../elastic_assistant/server/lib/attack_discovery/schedules/data_client/index.ts.

The wrappers aren't doing much now, but in the future I imagine they will be doing some interesting things.

Folks will be able to create a rule of this type outside of this wrapper, for example using the alerting HTTP APIs. As well as update, enable/disable, etc. I don't think we have a way of "hiding" rule types to prevent this sort of thing. Is that going to be ok?

We added a capability to connectors to allow code to be run "around" some APIs - we only added what was needed at the time. And have talked about doing a similar sort of thing with rules. I can't find an issue to track that (so no plan to implement now), but am curious if something like that could essentially replace your existing wrapper.

[pmuellr]

ResponseOps changes (minimal) LGTM

[afharo]

kibana.jsonc changes LGTM

[e40pud]

@e40pud thanks for the description of task vs rule.

I was curious to see the wrapper over alerting client in .../elastic_assistant/server/lib/attack_discovery/schedules/data_client/index.ts.

The wrappers aren't doing much now, but in the future I imagine they will be doing some interesting things.

Folks will be able to create a rule of this type outside of this wrapper, for example using the alerting HTTP APIs. As well as update, enable/disable, etc. I don't think we have a way of "hiding" rule types to prevent this sort of thing. Is that going to be ok?

We added a capability to connectors to allow code to be run "around" some APIs - we only added what was needed at the time. And have talked about doing a similar sort of thing with rules. I can't find an issue to track that (so no plan to implement now), but am curious if something like that could essentially replace your existing wrapper.

This wrapper is mostly to add some common utils in those wrappers and to align API differences:

• passed parameters
• conversions alerting framework <=> security solution. We will strip out some of the fields returned by alerting framework and return those as attack discovery schedule type
• potentially it will be one point of logging/telemetry reporting (if ever needed)

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/14376549179

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 216656

Questions ?

Please refer to the Backport tool documentation

[e40pud]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation