Skip to content

allow s3 output plugin to get keys from secrets #1688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 26, 2025
Merged

allow s3 output plugin to get keys from secrets #1688

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ae1b000
allow s3 output plugin to get keys from secrets
v-davegillies-upscale Sep 24, 2025
80944bf
update based on review feedback
v-davegillies-upscale Sep 25, 2025
80a0c80
Add generated manifests and docs
v-davegillies-upscale Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions apis/fluentd/v1alpha1/plugins/output/s3.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
package output

import "github.com/fluent/fluent-operator/v3/apis/fluentd/v1alpha1/plugins"

// S3 defines the parameters for out_s3 output plugin
type S3 struct {
// The AWS access key id.
AwsKeyId *string `json:"awsKeyId,omitempty"`
// The AWS secret key.
AwsSecKey *string `json:"awsSecKey,omitempty"`
// The AWS access key id from Secrets.
AwsKeyIdFromSecret *plugins.Secret `json:"awsKeyIdFromSecret,omitempty"`
// The AWS secret key from Secrets.
AwsSecKeyFromSecret *plugins.Secret `json:"awsSecKeyFromSecret,omitempty"`
// The Amazon S3 bucket name.
S3Bucket *string `json:"s3Bucket,omitempty"`
// The Amazon S3 region name
Expand Down
16 changes: 14 additions & 2 deletions apis/fluentd/v1alpha1/plugins/output/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -677,10 +677,22 @@ func (o *Output) kafka2Plugin(parent *params.PluginStore, loader plugins.SecretL
}

func (o *Output) s3Plugin(parent *params.PluginStore, loader plugins.SecretLoader) *params.PluginStore {
if o.S3.AwsKeyId != nil {
if o.S3.AwsKeyIdFromSecret != nil {
value, err := loader.LoadSecret(*o.S3.AwsKeyIdFromSecret)
if err != nil {
return nil
}
parent.InsertPairs("aws_key_id", value)
} else if o.S3.AwsKeyId != nil {
parent.InsertPairs("aws_key_id", fmt.Sprint(*o.S3.AwsKeyId))
}
if o.S3.AwsSecKey != nil {
if o.S3.AwsSecKeyFromSecret != nil {
value, err := loader.LoadSecret(*o.S3.AwsSecKeyFromSecret)
if err != nil {
return nil
}
parent.InsertPairs("aws_sec_key", value)
} else if o.S3.AwsSecKey != nil {
parent.InsertPairs("aws_sec_key", fmt.Sprint(*o.S3.AwsSecKey))
}
if o.S3.S3Bucket != nil {
Expand Down
72 changes: 72 additions & 0 deletions charts/fluent-operator/charts/fluentd-crds/crds/fluentd.fluent.io_clusteroutputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2358,9 +2358,81 @@ spec:
awsKeyId:
description: The AWS access key id.
type: string
awsKeyIdFromSecret:
description: The AWS access key id from Secrets.
properties:
valueFrom:
description: ValueSource defines how to find a value's
key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's
namespace
properties:
key:
description: The key of the secret to select
from. Must be a valid secret key.
type: string
name:
default: ""
description: |-
Name of the referent.
This field is effectively required, but due to backwards compatibility is
allowed to be empty. Instances of this type with an empty value here are
almost certainly wrong.
TODO: Add other useful fields. apiVersion, kind, uid?
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Drop `kubebuilder:default` when controller-gen doesn't need it https://github.com/kubernetes-sigs/kubebuilder/issues/3896.
type: string
optional:
description: Specify whether the Secret or its
key must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
awsSecKey:
description: The AWS secret key.
type: string
awsSecKeyFromSecret:
description: The AWS secret key from Secrets.
properties:
valueFrom:
description: ValueSource defines how to find a value's
key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's
namespace
properties:
key:
description: The key of the secret to select
from. Must be a valid secret key.
type: string
name:
default: ""
description: |-
Name of the referent.
This field is effectively required, but due to backwards compatibility is
allowed to be empty. Instances of this type with an empty value here are
almost certainly wrong.
TODO: Add other useful fields. apiVersion, kind, uid?
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Drop `kubebuilder:default` when controller-gen doesn't need it https://github.com/kubernetes-sigs/kubebuilder/issues/3896.
type: string
optional:
description: Specify whether the Secret or its
key must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
forcePathStyle:
description: This prevents AWS SDK from breaking endpoint
URL
Expand Down
72 changes: 72 additions & 0 deletions charts/fluent-operator/charts/fluentd-crds/crds/fluentd.fluent.io_outputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2358,9 +2358,81 @@ spec:
awsKeyId:
description: The AWS access key id.
type: string
awsKeyIdFromSecret:
description: The AWS access key id from Secrets.
properties:
valueFrom:
description: ValueSource defines how to find a value's
key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's
namespace
properties:
key:
description: The key of the secret to select
from. Must be a valid secret key.
type: string
name:
default: ""
description: |-
Name of the referent.
This field is effectively required, but due to backwards compatibility is
allowed to be empty. Instances of this type with an empty value here are
almost certainly wrong.
TODO: Add other useful fields. apiVersion, kind, uid?
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Drop `kubebuilder:default` when controller-gen doesn't need it https://github.com/kubernetes-sigs/kubebuilder/issues/3896.
type: string
optional:
description: Specify whether the Secret or its
key must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
awsSecKey:
description: The AWS secret key.
type: string
awsSecKeyFromSecret:
description: The AWS secret key from Secrets.
properties:
valueFrom:
description: ValueSource defines how to find a value's
key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's
namespace
properties:
key:
description: The key of the secret to select
from. Must be a valid secret key.
type: string
name:
default: ""
description: |-
Name of the referent.
This field is effectively required, but due to backwards compatibility is
allowed to be empty. Instances of this type with an empty value here are
almost certainly wrong.
TODO: Add other useful fields. apiVersion, kind, uid?
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Drop `kubebuilder:default` when controller-gen doesn't need it https://github.com/kubernetes-sigs/kubebuilder/issues/3896.
type: string
optional:
description: Specify whether the Secret or its
key must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
forcePathStyle:
description: This prevents AWS SDK from breaking endpoint
URL
Expand Down
72 changes: 72 additions & 0 deletions config/crd/bases/fluentd.fluent.io_clusteroutputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2358,9 +2358,81 @@ spec:
awsKeyId:
description: The AWS access key id.
type: string
awsKeyIdFromSecret:
description: The AWS access key id from Secrets.
properties:
valueFrom:
description: ValueSource defines how to find a value's
key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's
namespace
properties:
key:
description: The key of the secret to select
from. Must be a valid secret key.
type: string
name:
default: ""
description: |-
Name of the referent.
This field is effectively required, but due to backwards compatibility is
allowed to be empty. Instances of this type with an empty value here are
almost certainly wrong.
TODO: Add other useful fields. apiVersion, kind, uid?
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Drop `kubebuilder:default` when controller-gen doesn't need it https://github.com/kubernetes-sigs/kubebuilder/issues/3896.
type: string
optional:
description: Specify whether the Secret or its
key must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
awsSecKey:
description: The AWS secret key.
type: string
awsSecKeyFromSecret:
description: The AWS secret key from Secrets.
properties:
valueFrom:
description: ValueSource defines how to find a value's
key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's
namespace
properties:
key:
description: The key of the secret to select
from. Must be a valid secret key.
type: string
name:
default: ""
description: |-
Name of the referent.
This field is effectively required, but due to backwards compatibility is
allowed to be empty. Instances of this type with an empty value here are
almost certainly wrong.
TODO: Add other useful fields. apiVersion, kind, uid?
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Drop `kubebuilder:default` when controller-gen doesn't need it https://github.com/kubernetes-sigs/kubebuilder/issues/3896.
type: string
optional:
description: Specify whether the Secret or its
key must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
forcePathStyle:
description: This prevents AWS SDK from breaking endpoint
URL
Expand Down
Loading
Loading