adding latency config for data prepper #11196
Conversation
Signed-off-by: Anton Rubin <[email protected]>
AntonEliatra
requested review from
AMoo-Miki,
dlvenable,
epugh,
kolchfa-aws,
natebower and
sumobrian
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
Signed-off-by: Anton Rubin <[email protected]>
|
@dlvenable Could you review this PR or assign someone to review? Thanks! |
Signed-off-by: Anton Rubin <[email protected]>
| For each individual secret configuration, the latest secret value is polled on a regular interval to support refreshing secrets in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, such as connection and authentication to the backend service. | ||
| For each secret configuration, Data Prepper polls the latest value on a regular interval to support rotating secrets in AWS Secrets Manager. Refreshed values are utilized by plugins that can refresh their connections/authentication (for example, sinks). For multiple secret configurations, jitter within `60s` is applied across all configurations during the initial polling. | ||
|
|
||
| ## Pipeline latency tuning guide |
There was a problem hiding this comment.
I'm thinking this should go into another section and perhaps be split up.
I view Data Prepper users broadly as fitting into two personas: 1) Pipeline authors; and 2) Data Prepper managers. The first persona is updating pipeline configurations. The second deals with the data-prepper-config.yaml. In many situations, they are the same persona, but not always.
The "Managing Data Prepper" section is intended to cover Data Prepper operations and the configurations that are not related specifically to pipelines. I think this section should only cover improvements to the data-prepper-config.yaml.
Information for pipeline authors should be part of the "Pipelines" section.
3 participants
Description
adding latency config for data prepper
Version
all
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.