Add ConfigProvider to make use of environment variables

[Antoine7773]

Hello,

Is your feature request related to a problem? Please describe.
I would like to run Jikkou in a GitHub action to connect to a SASL_PLAINTEXT cluster.
Currently it seems that Jikkou does not support the load of env vars in jikkouconfig file

Describe the solution you'd like
I would like to be able to define in the jikkouconfig file something like this :

"my_cluster_A" : {
"configFile" : null,
"configProps" : {
"kafka.client.bootstrap.servers" : "my_host_A:9092",
"kafka.client.security.protocol" : "SASL_PLAINTEXT",
"kafka.client.sasl.mechanism" : "SCRAM-SHA-256",
"kafka.client.sasl.jaas.config" : "org.apache.kafka.common.security.scram.ScramLoginModule required username="jikkou-user" password="${env:JIKKOU_PASSWORD}";",
"schemaRegistry.url" : "schema_host_A:8081"
}

Then I would set env variable JIKKOU_PASSWORD from GitHub secret that would be used by Jikkou

Describe alternatives you've considered
Thought about using sed to replace the variable by the actual secret but it means writing the password in clear so not the best option.

Additional context
Kafka implemented this feature according to this ticket https://issues.apache.org/jira/browse/KAFKA-14376
Would it be possible to have the same kind of feature in Jikkou ?

[fhussonnois]

Hi @Antoine7773, Jikkou uses HOCON for configuration which support environment subsitutiuon. So doing just ${JIKKOU_PASSWORD} with JIKKOU_PASSWORD being your env var should work.

[Antoine7773]

Hello @fhussonnois,

I tired but I have auth failure. Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256

Here is the jikkouconfig file
{
"currentContext": "my_cluster_A",
"my_cluster_A": {
"configFile": null,
"configProps": {
"kafka.client.bootstrap.servers": "my_host_A:9092",
"kafka.client.security.protocol": "SASL_PLAINTEXT",
"kafka.client.sasl.mechanism": "SCRAM-SHA-256",
"kafka.client.sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username="jikkou-user" password="${JIKKOU_PASSWORD}";",
"schemaRegistry.url": "schema_host_A:8081"
}
}
}

Docker command ran :
docker run -it -e JIKKOU_PASSWORD=jikkou-user-secret --net host --mount type=bind,source="$(pwd)"/jikkouconfig,target=/etc/jikkou/config streamthoughts/jikkou:latest get kafkatopics --logger-level=TRACE

Checked the env var in the container and it is correct
docker exec -it 91a766fad135 sh printenv JIKKOU_PASSWORD=jikkou-user-secret
When putting the password in clear text like this in the jikkouconfig

"kafka.client.sasl.jaas.config" : "org.apache.kafka.common.security.scram.ScramLoginModule required username="jikkou-user" password="jikkou-user-secret";",

The command works well.

Do you know if I am missing something ?

[obiremiv]

Actually i have the same issue with mTLS it ends up picking up the PLAINTEXT as the security. also the security was stated in the config_props but in logger-level=DEBUG

15:48:06.781 [main] INFO  o.a.k.c.admin.AdminClientConfig - AdminClientConfig values: 
        auto.include.jmx.reporter = true
        bootstrap.controllers = []
        bootstrap.servers = [cnflntbrkrsb1s1.alsac.stjude.org:9092]
        client.dns.lookup = use_all_dns_ips
        client.id = 
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        enable.metrics.push = true
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.max.ms = 1000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        **security.protocol = PLAINTEXT**
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS

@fhussonnois hi since it needs the DN that is being stated in the super-users and whats stated in the cert i am using the cert used by the broker itself

[Shawcs]

Hi guys ! now working with @Antoine7773 and I found a workaround for this:

in the config file do that:

"my_cluster_A" : {
  "configFile" : "/etc/jikkou/additional_conf.conf",
  "configProps" : {
    "kafka.client.bootstrap.servers" : "my_host_A:9092",
    "kafka.client.security.protocol" : "SASL_PLAINTEXT",
    "kafka.client.sasl.mechanism" : "SCRAM-SHA-256",
    "schemaRegistry.url" : "schema_host_A:8081",
    [...]
  }
}

create a file named additional_conf.conf in the right path mentionned in the configFile. And in here you can put

jikkou.kafka.client.sasl.jaas.config:"org.apache.kafka.common.security.scram.ScramLoginModule required username="${JIKKOU_USER}" password="${JIKKOU_PASSWORD}";"

So @fhussonnois I think the env injection work but not every where