Error occurred when using 1.5.0 version, enabled es.all | two es instances running on the server

[Edvision]

Error occurred when using 1.5.0 version, but it's running fine when I use 1.1.0 version. I have two es instances running on the server, I think this is the root cause of the error.

Command:
./elasticsearch_exporter --es.uri https://USERNAME:PASSWORD@es-host1:9200 --es.all --es.ssl-skip-verify

Error info:

An error has occurred while serving metrics:

347 error(s) occurred:

• collected metric "elasticsearch_nodes_roles" { label:<name:"cluster" value:"escluster" > label:<name:"host" value:"es-host1" label:<name:"name" value:"es-host1" > label:<name:"role" value:"data" > gauge:<value:1 > } was collected before with the same name and label values
• collected metric "elasticsearch_nodes_roles" { label:<name:"cluster" value:"escluster" > label:<name:"host" value:"es-host1" label:<name:"name" value:"es-host1" > label:<name:"role" value:"client" > gauge:<value:1 > } was collected before with the same name and label values
• collected metric "elasticsearch_nodes_roles" { label:<name:"cluster" value:"escluster" > label:<name:"host" value:"es-host1" label:<name:"name" value:"es-host1" > label:<name:"role" value:"ingest" > gauge:<value:1 > } was collected before with the same name and label values
• collected metric "elasticsearch_nodes_roles" { label:<name:"cluster" value:"escluster" > label:<name:"host" value:"es-host2" label:<name:"name" value:"es-host2" > label:<name:"role" value:"data" > gauge:<value:1 > } was collected before with the same name and label values
• collected metric "elasticsearch_nodes_roles" { label:<name:"cluster" value:"escluster" > label:<name:"host" value:"es-host2" label:<name:"name" value:"es-host2" > label:<name:"role" value:"client" > gauge:<value:1 > } was collected before with the same name and label values
• collected metric "elasticsearch_nodes_roles" { label:<name:"cluster" value:"escluster" > label:<name:"host" value:"es-host2" label:<name:"name" value:"es-host2" > label:<name:"role" value:"ingest" > gauge:<value:1 > } was collected before with the same name and label values
  ......

[Edvision]

error version:
es version: 7.8.1/7.16.6
es exporter version: 1.5.0

ex exporter 1.1.0 is not compatible for es 7.16.6 so I upgrade to 1.5.0

[sysadmind]

I believe that this is a bug based on adding the host name as a label. If there are 2 instances running on the same host, this will cause a conflict. This is likely an unpopular configuration, however the exporter should still work in this scenario.

There are 2 options that come to mind to fix this. 1 is to add the port to the host label. That would make each of these instances unique and fix the problem. The other is to drop the host label entirely. This is something we have discussed before but when collecting data from multiple nodes, there still needs to be some way to distinguish the metrics. Both of these would be a breaking change to metrics however.

I think this requires more research and discussion to solve.

[Edvision]

I believe that this is a bug based on adding the host name as a label. If there are 2 instances running on the same host, this will cause a conflict. This is likely an unpopular configuration, however the exporter should still work in this scenario.

There are 2 options that come to mind to fix this. 1 is to add the port to the host label. That would make each of these instances unique and fix the problem. The other is to drop the host label entirely. This is something we have discussed before but when collecting data from multiple nodes, there still needs to be some way to distinguish the metrics. Both of these would be a breaking change to metrics however.

I think this requires more research and discussion to solve.

Thanks @sysadmind for your analysis and provide the solution.