Command line option to avoid loading the local telegraf.conf and telegraf.d/**

[poulhs]

Use Case

When using the telegraf-controller (or any other http-configuration end-point) the presence of /etc/telegraf/telegraf.conf (and telegraf.d/) may be in conflict with what is fetched from the http end-point.
e.g.:
W! Overlapping settings in multiple agent tables are not supported: may cause undefined behavior

Being able to avoid loading the default configuration would be appreciated, as managing configuration both central and local may be obfuscating, cause much more debugging and may produce unexpected behaviors.
(The telegraf.conf MUST be present for telegraf to start).
As the telegraf.service includes the
-config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
the preferred way may be to to include (in $TELEGRAF_OPTS) some flag indicating that the previous configuration flags should be ignored...

There will be cases where one has full control and wants both remote and local configuration.

Expected behavior

It should be possible to add a command line flag (or an environment variable) so that telegraf ignores the local configuration.

Actual behavior

when telegraf starts and fetches a remote configuration that contains the [agent] section it produces a warning:
W! Overlapping settings in multiple agent tables are not supported: may cause undefined behavior

Additional info

[agent]
  interval = "10s"
  round_interval = true

and all the other things in [agent] that one may want to override centally

[Daryes]

Given that the /etc/defaults/telegraf file has to be updated anyway, cannot you also override the service definition with a custom file like : /etc/systemd/system/telegraf.service.d/override.conf

[Service]
ExecStart=/usr/bin/telegraf  ...my custom command line and alternate config file...

Any file under a directory named as <service name>.d will be loaded by systemd in addition, its content having precedence over the common parameters from the original service unit definition.
Redhat has a good documentation about it, which is valid for any distribution using systemd.

[poulhs]

Sure it is possible to modify the /usr/lib/telegraf/scripts/telegraf.service, but that is not a configuration file:

# rpm --verify --verbose  telegraf
.........  c /etc/logrotate.d/telegraf
.........  c /etc/telegraf/telegraf.conf
.........  c /etc/telegraf/telegraf.d/.ignore
.........    /usr/bin/telegraf
.........    /usr/lib/telegraf/scripts/init.sh
S.5....T.    /usr/lib/telegraf/scripts/telegraf.service
.....UG..    /var/log/telegraf

and it will be overwritten by dnf reinstall

Not all our customers will be happy if we modify official RPM's like that.

[Daryes]

Please point me to where I wrote to modify the original service unit, which I never did, as there is no need to.
Creating an override is done through an additional file without altering the original service unit. Both will merged automatically by systemd when read, then executing the resulting combination.

Please read the fine link I provided before.

[poulhs]

Thanks @Daryes - I had complete forgotten about the /etc/systemd/system/ directory - thanks for reminding me!

[poulhs]

I was a bit too hasty...:

Creating: /etc/systemd/system/telegraf.service.d/execstart.conf with an ExecStart is not valid:
systemd[1]: telegraf.service: Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.

So I will have to replace the complete /etc/systemd/system/multi-user.target.wants/telegraf.service with /etc/systemd/system/telegraf.service that I will have to maintain. Hope there will not be too many changes to the telegraf systemd file...

[Daryes]

@poulhs
Try this instead for your override.conf

[service]
ExecStart=
ExecStart=/usr/bin/telegraf  #... and other parameters

This should work.

Some unit parameters are allowed to exists multiple times and merged together, which is the case for execStart, due to other service types.
To force systemd to replace such parameters instead of merging them, they need to be declared a first time with a blank value.

[srebhan]

@poulhs and @Daryes your discussion is a tiny bit off-topic. ;-)

Anyway, what we can (and should) do from the Telegraf side is to remove the default values from the service file. Those locations will be loaded anyways if there is no configuration is given. However, this would be a breaking change and we usually try to avoid those. Let me discuss with the team on what we should do here...