could not unmarshal secret containing access credentials: invalid character 'd' looking for beginning of value

[kekjoshi]

Hi Team,

While using below KeptnMetricsProvider and KeptnMetric when I am trying to use a DQL query I am getting error as
Err Msg: could not unmarshal secret containing access credentials: invalid character 'd' looking for beginning of value

kind: KeptnMetricsProvider
metadata:
  name: dynatrace-provider
  namespace:test 
  labels:
    application: "dynatrace" 
spec:
  type: dql
  targetServer: ""
  secretKeyRef:
    name: dt-api-token
    key: DT_TOKEN

---
apiVersion: v1
kind: Secret
metadata:
  name: dt-api-token
  namespace:test 
data:
  DT_TOKEN: "" #encoded value
type: Opaque

apiVersion: metrics.keptn.sh/v1alpha2
kind: KeptnMetric
metadata:
  name: keptn-metric
  namespace:test 
  labels:
    application: "dynatrace"
    
spec:
  provider: 
    name: dynatrace-provider 
  query: "timeseries  by: { k8s.pod.name, k8s.namespace.name, k8s.cluster.name },container_count=avg(dt.kubernetes.containers)"

Ant the same is running fine if I am using builtin query and
spec:
type:dynatrace

Could you please help in debugging the root cause.

[BDunneInTheSun]

Quick update: I work with @kekjoshi
Wanted to provide an update since we made some progress, and to save anyone else trouble of looking into the same.

We found that this issue was due to KeptnMetricsProvider type: dql requires a secret in a specific format (different than type:dynatrace). We could not find this documented in the Docs or examples, ie https://keptn.sh/stable/docs/reference/crd-reference/metricsprovider/
But it seems clear from the provider implementation and tests that the format for the secret string value should be: {"token": "dt0s08.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "authUrl":"https://my-auth-url.test"}

So we have generated an oauth token and updated the secret in this format.

Our new, related issue:
Error : Err Msg: secret contains invalid credentials: secret does not start with required prefix dt0s08: the Dynatrace token has an invalid format

So looking in the code, it seems that only tokens that begin with that prefix are accepted. DT docs on token prefixes. User generated oAuth tokens have a different prefix, and so, are rejected.

We're checking with DT Support to see if we can get a token that will be accepted here. Curious to see if this is a hard requirement, or an artifact of development...

[bacherfl]

Hi @BDunneInTheSun - you are right - back then that prefix was the only possible one, but in the meantime the list of allowed prefixes has been extended. The restriction in https://github.com/keptn/lifecycle-toolkit/blob/main/metrics-operator/controllers/common/providers/dynatrace/common.go#L43 can therefore be adapted to also consider the other prefixes listed in the docs

[mowies]

Yes, this seems to be a bug. Care to contribute a fix for it @BDunneInTheSun @kekjoshi ?
Should be easy to just validate against the dt0s prefix instead of dt0s08.