ES template creation does not respect custom index names

[iserko]

I'm using an ES server with multiple Logstash indexers and for each one I'm using it's own ES index prefix. Here's an example config for one:

output {
    elasticsearch {
        hosts => ["esserver:9200"]
        index => "example-logstash-%{+YYYY.MM.dd}"
        template_name => "example-logstash"
    }
}

When I check the index template created for this I see:

% curl -XGET http://esserver:9200/_template/ | json_pp
{
   "example-logstash" : {
      "template" : "logstash-*",
      "order" : 0,
      "aliases" : {},
      "settings" : {
         "index" : {
            "refresh_interval" : "5s"
         }
      },
      "mappings" : {
      ...
}

The problem is that the template setting is set to logstash-* which will never match the indexes example-logstash-* upon index creation. See https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html for more examples.

[iserko]

ok I see that it's hardcoded in https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

I'll create my own template file and pass it using the template setting.

But still ... it's an annoying thing.

[andrewvc]

This is an interesting issue @iserko. I suppose we should have a separate template_pattern setting for that.

[untergeek]

@andrewvc We tried that a while back. It was somewhat problematic then. Perhaps we could revisit it again.

[andrewvc]

@untergeek what were the problems?

[untergeek]

It's kind of vague to me now. @talevy was involved then too. Perhaps he remembers why we reverted it.

[talevy]

template_pattern is a dangerous config option to enable because it would make it difficult to reason about which template was used for a specific index, and could be prone to newly applied mappings that would yield errors. I think it is OK to be explicit here with maintained template files

[iserko]

I think the solution with maintaining your own template file is perfectly fine, however it needs to be documented that if you change the template_name, you should provide the file as well.

Currently that's not the case. See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template_name

[djschny]

+1 to being able to override the template index pattern name, as opposed to having to manually manage the template

[kovyrin]

Along with documenting the issue and the expected approach for fixing it, I'd argue logstash should at least provide a very visible warning for this in the logs or, better, blow up for cases when an index name is changed, but no custom template is provided.

[jsvd]

there's an issue already about improving documentation about the template usage in this plugin in #416

what's missing then is a user experience improvement that warns users that, if they have manage_template => true, a check could be made between the index name and the index pattern.