Skip to content

First pass spec for taking thread dumps#403

Open
breedx-splk wants to merge 8 commits into
signalfx:mainfrom
breedx-splk:add_thread_dumpery
Open

First pass spec for taking thread dumps#403
breedx-splk wants to merge 8 commits into
signalfx:mainfrom
breedx-splk:add_thread_dumpery

Conversation

@breedx-splk

Copy link
Copy Markdown
Contributor

We are being requested to make this quick hack to cause thread dumps to be taken and reported via the profiling signal. I have a number of notable concerns with this, but am trying to make the most out of an unfortunately regettable thing.

Comment thread specification/opamp_datamodel.md Outdated
Comment on lines +358 to +360
opamp/development:
endpoint: http://some.opamp-host.com:3420/v1/opamp
experimental_control: true

@ysolomchenko ysolomchenko Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
opamp/development:
endpoint: http://some.opamp-host.com:3420/v1/opamp
experimental_control: true
opamp/development:
endpoint: http://some.opamp-host.com:3420/v1/opamp
features:
remote_control:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, this declarative config stuff hates explicit feature toggles and prefers empty blocks to do the same thing. Ugh. Regardless, I think it very much needs the word experimental in there -- so that it sends a clear message to the user.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be experimental_remote_control: or remote_control/development: ?

Comment on lines +342 to +367
When using environment-variable based agent configuration, the following
configuration option MAY be provided:

| Name | Default | Description |
|--------------------------------------------|---------|---------------------------------------------------------------|
| `SPLUNK_OPAMP_EXPERIMENTAL_REMOTE_CONTROL` | false | Set to `true` to accept experimental remote-control commands. |

The equivalent Java system property is
`splunk.opamp.experimental_remote_control`.

When using declarative configuration, `experimental_control` enables the
feature:

```yaml
distribution:
splunk:
opamp/development:
endpoint: http://some.opamp-host.com:3420/v1/opamp
experimental_control: true
```

OpAMP must be enabled in the agent for this feature to work.

This feature is independent of the call graph profiler (also known as the
snapshot profiler) and does not require `SPLUNK_SNAPSHOT_PROFILER_ENABLED` to
be set.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be placed in the configuration file?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe, I guess yeah, and then links in both directions or something.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opamp data model file already contains both wire format, behavioral stuff, and config stuff. I think having all the "opamp stuff" in one place is nice. Maybe we should just rename the file from opamp_datamodel.md to opamp.md or something?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have an OpAMP configuration section:

https://github.com/signalfx/gdi-specification/blob/main/specification/configuration.md#opamp

So I think it would be better to move the configuration-related part there.

@MrAlias MrAlias left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for carrying the compatibility bridge through. I think we need a few contract details resolved before we merge it: the present wording can bypass the opt-in and allow duplicate or unexpected command execution. Once those are nailed down, it should give the Java implementation—and future agents—a dependable target.

Comment thread specification/opamp_datamodel.md Outdated
`AgentConfigFile` whose body is UTF-8 encoded. Its `content_type` SHOULD be
`text/plain; charset=utf-8`.

When `COMMAND_HACKS` is present, the agent MUST interpret its body as a command

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make handling COMMAND_HACKS conditional on the experimental-control option, and say agents advertise the command capability only when that option is enabled? As written, this unconditional MUST applies to an agent with ordinary remote configuration enabled but remote control still at its default of false; receiving this filename would execute a command anyway.

Comment thread specification/opamp_datamodel.md Outdated

The first thread dump is collected immediately. When `count` is greater than
one, the agent waits `interval_millis` after completing one thread dump before
starting the next. The agent MUST use the command's `job_id` for every thread

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we define duplicate-delivery semantics here? OpAMP can repeat remote-configuration content after reconnects or status reports, and job_id currently appears only in the output labels. A bounded in-memory record of accepted or completed job IDs, together with defined accepted, rejected, and completed outcomes, would keep a resend from collecting the same sequence again.

Comment thread specification/opamp_datamodel.md
Comment thread specification/opamp_datamodel.md Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants