Skip to content

Commit b81f418

Browse files
csoulioscsoulios
authored
[TSDB] Metric fields in the field caps API (elastic#88695)
To assist the user in configuring the visualizations correctly while leveraging TSDB functionality, information about TSDB configuration should be exposed via the field caps API per field. Especially for metrics fields, it must be clear which fields are metrics and if they belong to only time-series indexes or mixed time-series and non-time-series indexes. To further distinguish metric fields when they belong to any of the following indices: - Standard (non-time-series) indexes - Time series indexes - Downsampled time series indexes This PR modifies the field caps API so that the mapping parameters time_series_dimension and time_series_dimension are presented only when they are set on fields of time-series indexes. Those parameters are completely ignored when they are set on standard (non-time-series) indexes. This PR revisits some of the conventions adopted by elastic#78790
1 parent 188f887 commit b81f418

File tree

11 files changed

+474
-79
lines changed

11 files changed

+474
-79
lines changed

‎docs/changelog/88695.yaml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pr: 88695
2+
summary: "[TSDB] Metric fields in the field caps API"
3+
area: TSDB
4+
type: enhancement
5+
issues: []

‎docs/reference/search/field-caps.asciidoc

+6-4
Original file line numberDiff line numberDiff line change
@@ -137,11 +137,13 @@ field types are all described as the `keyword` type family.
137137

138138
`time_series_dimension`::
139139
preview:[]
140-
Whether this field is used as a time series dimension.
140+
Whether this field is used as a time series dimension on all indices.
141+
For non-time-series indices this field is not present.
141142

142143
`time_series_metric`::
143144
preview:[]
144-
Contains metric type if this fields is used as a time series metrics, absent if the field is not used as metric.
145+
Contains the metric type if the field is used as a time series metric on all indices, absent if the field is
146+
not used as a metric. For non-time-series indices this field is not included.
145147

146148
`indices`::
147149
The list of indices where this field has the same type family, or null if all indices
@@ -157,12 +159,12 @@ field types are all described as the `keyword` type family.
157159

158160
`non_dimension_indices`::
159161
experimental:[]
160-
If this list is present in response then some indices have the field marked as a dimension and other indices, the
162+
If this list is present in the response, some indices have the field marked as a dimension and other indices, the
161163
ones in this list, do not.
162164

163165
`metric_conflicts_indices`::
164166
experimental:[]
165-
The list of indices where this field is present if these indices don't have the same `time_series_metric` value for
167+
The list of indices where this field is present, if these indices don't have the same `time_series_metric` value for
166168
this field.
167169

168170
`meta`::

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/field_caps/40_time_series.yml

+10
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ setup:
1212
index:
1313
number_of_replicas: 0
1414
number_of_shards: 2
15+
mode: time_series
16+
routing_path: [ metricset, k8s.pod.uid ]
17+
time_series:
18+
start_time: 2021-04-28T00:00:00Z
19+
end_time: 2021-04-29T00:00:00Z
1520
mappings:
1621
properties:
1722
"@timestamp":
@@ -59,6 +64,11 @@ setup:
5964
index:
6065
number_of_replicas: 0
6166
number_of_shards: 2
67+
mode: time_series
68+
routing_path: [ k8s.pod.uid ]
69+
time_series:
70+
start_time: 2021-04-28T00:00:00Z
71+
end_time: 2021-04-29T00:00:00Z
6272
mappings:
6373
properties:
6474
"@timestamp":

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/tsdb/05_dimension_and_metric_in_non_tsdb_index.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -183,8 +183,8 @@ can't shadow metrics:
183183
# Test that _tsid field is not added if an index is not a time-series index
184184
no _tsid in standard indices:
185185
- skip:
186-
version: " - 8.0.99"
187-
reason: _tsid support introduced in 8.1.0
186+
version: " - 8.4.99"
187+
reason: time series params only on time series indices introduced in 8.5.0
188188

189189
- do:
190190
indices.create:
@@ -209,11 +209,11 @@ no _tsid in standard indices:
209209

210210
- match: {fields.metricset.keyword.searchable: true}
211211
- match: {fields.metricset.keyword.aggregatable: true}
212-
- match: {fields.metricset.keyword.time_series_dimension: true}
213212
- is_false: fields.metricset.keyword.indices
214213
- is_false: fields.metricset.keyword.non_searchable_indices
215214
- is_false: fields.metricset.keyword.non_aggregatable_indices
216215
- is_false: fields._tsid # _tsid metadata field must not exist in non-time-series indices
216+
- is_false: fields.metricset.keyword.time_series_dimension # time_series_dimension param is ignored in non-time-series indices
217217

218218
---
219219
no nested dimensions:

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/tsdb/110_field_caps.yml

+231
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
setup:
2+
- skip:
3+
version: " - 8.4.99"
4+
reason: metric params only on time series indexes introduced in 8.5.0
5+
6+
- do:
7+
indices.create:
8+
index: test_time_series
9+
body:
10+
settings:
11+
index:
12+
number_of_replicas: 0
13+
number_of_shards: 2
14+
mode: time_series
15+
routing_path: [ metricset, k8s.pod.uid ]
16+
time_series:
17+
start_time: 2021-04-28T00:00:00Z
18+
end_time: 2021-04-29T00:00:00Z
19+
mappings:
20+
properties:
21+
"@timestamp":
22+
type: date
23+
metricset:
24+
type: keyword
25+
time_series_dimension: true
26+
k8s:
27+
properties:
28+
pod:
29+
properties:
30+
availability_zone:
31+
type: short
32+
time_series_dimension: true
33+
uid:
34+
type: keyword
35+
time_series_dimension: true
36+
name:
37+
type: keyword
38+
ip:
39+
type: ip
40+
time_series_dimension: true
41+
network:
42+
properties:
43+
tx:
44+
type: long
45+
time_series_metric: counter
46+
rx:
47+
type: long
48+
time_series_metric: gauge
49+
50+
- do:
51+
indices.create:
52+
index: test_non_time_series
53+
body:
54+
settings:
55+
index:
56+
number_of_replicas: 0
57+
number_of_shards: 2
58+
mappings:
59+
properties:
60+
"@timestamp":
61+
type: date
62+
metricset:
63+
type: keyword
64+
time_series_dimension: true
65+
k8s:
66+
properties:
67+
pod:
68+
properties:
69+
availability_zone:
70+
type: short
71+
time_series_dimension: true
72+
uid:
73+
type: keyword
74+
time_series_dimension: true
75+
name:
76+
type: keyword
77+
ip:
78+
type: ip
79+
time_series_dimension: true
80+
network:
81+
properties:
82+
tx:
83+
type: long
84+
time_series_metric: counter
85+
rx:
86+
type: long
87+
time_series_metric: gauge
88+
89+
---
90+
field caps on time_series indices:
91+
- skip:
92+
version: " - 8.3.99"
93+
reason: metric params only on time series indexes introduced in 8.4.0
94+
95+
- do:
96+
field_caps:
97+
index: test_time_series
98+
fields: [ k8s.pod.uid, k8s.pod.network.rx, k8s.pod.network.tx, k8s.pod.ip, metricset, _tsid ]
99+
100+
- match: { fields.k8s\.pod\.uid.keyword.type: keyword }
101+
- match: { fields.k8s\.pod\.uid.keyword.searchable: true }
102+
- match: { fields.k8s\.pod\.uid.keyword.aggregatable: true }
103+
- match: { fields.k8s\.pod\.uid.keyword.time_series_dimension: true }
104+
- is_false: fields.k8s\.pod\.uid.keyword.indices
105+
- is_false: fields.k8s\.pod\.uid.keyword.non_searchable_indices
106+
- is_false: fields.k8s\.pod\.uid.keyword.non_aggregatable_indices
107+
108+
- match: { fields.k8s\.pod\.network\.rx.long.type: long }
109+
- match: { fields.k8s\.pod\.network\.rx.long.searchable: true }
110+
- match: { fields.k8s\.pod\.network\.rx.long.aggregatable: true }
111+
- match: { fields.k8s\.pod\.network\.rx.long.time_series_metric: gauge }
112+
- is_false: fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices
113+
- is_false: fields.k8s\.pod\.network\.rx.long.indices
114+
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
115+
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
116+
117+
118+
- match: { fields.k8s\.pod\.network\.tx.long.type: long }
119+
- match: { fields.k8s\.pod\.network\.tx.long.searchable: true }
120+
- match: { fields.k8s\.pod\.network\.tx.long.aggregatable: true }
121+
- match: { fields.k8s\.pod\.network\.tx.long.time_series_metric: counter }
122+
- is_false: fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices
123+
- is_false: fields.k8s\.pod\.network\.tx.long.indices
124+
- is_false: fields.k8s\.pod\.network\.tx.long.non_searchable_indices
125+
- is_false: fields.k8s\.pod\.network\.tx.long.non_aggregatable_indices
126+
127+
- match: { fields.k8s\.pod\.ip.ip.type: ip }
128+
- match: { fields.k8s\.pod\.ip.ip.searchable: true }
129+
- match: { fields.k8s\.pod\.ip.ip.aggregatable: true }
130+
- is_false: fields.k8s\.pod\.ip.ip.indices
131+
- is_false: fields.k8s\.pod\.ip.ip.non_searchable_indices
132+
- is_false: fields.k8s\.pod\.ip.ip.non_aggregatable_indices
133+
134+
- match: { fields.metricset.keyword.type: keyword }
135+
- match: { fields.metricset.keyword.searchable: true }
136+
- match: { fields.metricset.keyword.aggregatable: true }
137+
- match: { fields.metricset.keyword.time_series_dimension: true }
138+
- is_false: fields.metricset.keyword.non_dimension_indices
139+
- is_false: fields.metricset.keyword.indices
140+
- is_false: fields.metricset.keyword.non_searchable_indices
141+
- is_false: fields.metricset.keyword.non_aggregatable_indices
142+
143+
- match: { fields._tsid._tsid.metadata_field: true }
144+
- match: { fields._tsid._tsid.searchable: false }
145+
- match: { fields._tsid._tsid.aggregatable: true }
146+
- is_false: fields._tsid._tsid.indices
147+
- is_false: fields._tsid._tsid.non_searchable_indices
148+
- is_false: fields._tsid._tsid.non_aggregatable_indices
149+
150+
151+
---
152+
field caps on standard indices:
153+
- skip:
154+
version: " - 8.3.99"
155+
reason: metric params only on time series indexes introduced in 8.4.0
156+
157+
- do:
158+
field_caps:
159+
index: test_non_time_series
160+
fields: [ _tsid, metricset, k8s.pod.network.rx, k8s.pod.network.tx, k8s.pod.network.rx ]
161+
162+
- match: { fields.metricset.keyword.type: keyword }
163+
- match: { fields.metricset.keyword.searchable: true }
164+
- match: { fields.metricset.keyword.aggregatable: true }
165+
- is_false: fields.metricset.keyword.time_series_dimension
166+
- is_false: fields.metricset.keyword.non_dimension_indices
167+
- is_false: fields.metricset.keyword.indices
168+
- is_false: fields.metricset.keyword.non_searchable_indices
169+
- is_false: fields.metricset.keyword.non_aggregatable_indices
170+
171+
- is_false: fields._tsid # _tsid metadata field must not exist in non-time-series indices
172+
173+
- match: { fields.k8s\.pod\.network\.rx.long.type: long }
174+
- match: { fields.k8s\.pod\.network\.rx.long.searchable: true }
175+
- match: { fields.k8s\.pod\.network\.rx.long.aggregatable: true }
176+
- is_false: fields.k8s\.pod\.network\.rx.long.time_series_metric
177+
- is_false: fields.k8s\.pod\.network\.rx.long.metric_conflicts_indices
178+
- is_false: fields.k8s\.pod\.network\.rx.long.indices
179+
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
180+
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
181+
- is_false: fields.k8s\.pod\.network\.rx.gauge
182+
183+
- match: { fields.k8s\.pod\.network\.tx.long.type: long }
184+
- match: { fields.k8s\.pod\.network\.tx.long.searchable: true }
185+
- match: { fields.k8s\.pod\.network\.tx.long.aggregatable: true }
186+
- is_false: fields.k8s\.pod\.network\.tx.long.time_series_metric
187+
- is_false: fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices
188+
- is_false: fields.k8s\.pod\.network\.tx.long.indices
189+
- is_false: fields.k8s\.pod\.network\.tx.long.non_searchable_indices
190+
- is_false: fields.k8s\.pod\.network\.tx.long.non_aggregatable_indices
191+
- is_false: fields.k8s\.pod\.network\.tx.counter
192+
193+
194+
---
195+
field caps on mixed indices:
196+
- skip:
197+
version: " - 8.3.99"
198+
reason: metric params only on time series indexes introduced in 8.4.0
199+
200+
- do:
201+
field_caps:
202+
index: test_*
203+
fields: [ metricset, k8s.pod.availability_zone, k8s.pod.network.tx, k8s.pod.network.rx ]
204+
205+
- match: { fields.metricset.keyword.type: keyword }
206+
- match: { fields.metricset.keyword.searchable: true }
207+
- match: { fields.metricset.keyword.aggregatable: true }
208+
- is_false: fields.metricset.keyword.time_series_dimension
209+
- match: { fields.metricset.keyword.non_dimension_indices: [ "test_non_time_series" ] }
210+
- is_false: fields.metricset.keyword.indices
211+
- is_false: fields.metricset.keyword.non_searchable_indices
212+
- is_false: fields.metricset.keyword.non_aggregatable_indices
213+
214+
- match: { fields.k8s\.pod\.network\.rx.long.type: long }
215+
- match: { fields.k8s\.pod\.network\.rx.long.searchable: true }
216+
- match: { fields.k8s\.pod\.network\.rx.long.aggregatable: true }
217+
- match: { fields.k8s\.pod\.network\.rx.long.metric_conflicts_indices: [ "test_non_time_series", "test_time_series" ] }
218+
- is_false: fields.k8s\.pod\.network\.rx.long.time_series_metric
219+
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
220+
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
221+
- is_false: fields.k8s\.pod\.network\.rx.long.indices
222+
223+
- match: { fields.k8s\.pod\.network\.tx.long.type: long }
224+
- match: { fields.k8s\.pod\.network\.tx.long.searchable: true }
225+
- match: { fields.k8s\.pod\.network\.tx.long.aggregatable: true }
226+
- match: { fields.k8s\.pod\.network\.tx.long.metric_conflicts_indices: [ "test_non_time_series", "test_time_series" ] }
227+
- is_false: fields.k8s\.pod\.network\.tx.long.time_series_metric
228+
- is_false: fields.k8s\.pod\.network\.tx.long.non_searchable_indices
229+
- is_false: fields.k8s\.pod\.network\.tx.long.non_aggregatable_indices
230+
- is_false: fields.k8s\.pod\.network\.tx.long.indices
231+

‎rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/tsdb/40_search.yml

-39
Original file line numberDiff line numberDiff line change
@@ -304,45 +304,6 @@ aggregate a tag:
304304
term:
305305
_tsid: wont't work
306306

307-
---
308-
field capabilities:
309-
- skip:
310-
version: " - 8.1.99"
311-
reason: tsdb indexing changed in 8.2.0
312-
313-
- do:
314-
field_caps:
315-
index: test
316-
fields: [k8s.pod.uid, k8s.pod.network.rx, k8s.pod.ip, metricset, _tsid]
317-
318-
- match: {fields.k8s\.pod\.uid.keyword.searchable: true}
319-
- match: {fields.k8s\.pod\.uid.keyword.aggregatable: true}
320-
- match: {fields.k8s\.pod\.uid.keyword.time_series_dimension: true}
321-
- is_false: fields.k8s\.pod\.uid.keyword.indices
322-
- is_false: fields.k8s\.pod\.uid.keyword.non_searchable_indices
323-
- is_false: fields.k8s\.pod\.uid.keyword.non_aggregatable_indices
324-
- match: {fields.k8s\.pod\.network\.rx.long.searchable: true}
325-
- match: {fields.k8s\.pod\.network\.rx.long.aggregatable: true}
326-
- is_false: fields.k8s\.pod\.network\.rx.long.indices
327-
- is_false: fields.k8s\.pod\.network\.rx.long.non_searchable_indices
328-
- is_false: fields.k8s\.pod\.network\.rx.long.non_aggregatable_indices
329-
- match: {fields.k8s\.pod\.ip.ip.searchable: true}
330-
- match: {fields.k8s\.pod\.ip.ip.aggregatable: true}
331-
- is_false: fields.k8s\.pod\.ip.ip.indices
332-
- is_false: fields.k8s\.pod\.ip.ip.non_searchable_indices
333-
- is_false: fields.k8s\.pod\.ip.ip.non_aggregatable_indices
334-
- match: {fields.metricset.keyword.searchable: true}
335-
- match: {fields.metricset.keyword.aggregatable: true}
336-
- match: {fields.metricset.keyword.time_series_dimension: true}
337-
- is_false: fields.metricset.keyword.indices
338-
- is_false: fields.metricset.keyword.non_searchable_indices
339-
- is_false: fields.metricset.keyword.non_aggregatable_indices
340-
- match: {fields._tsid._tsid.metadata_field: true}
341-
- match: {fields._tsid._tsid.searchable: false}
342-
- match: {fields._tsid._tsid.aggregatable: true}
343-
- is_false: fields._tsid._tsid.indices
344-
- is_false: fields._tsid._tsid.non_searchable_indices
345-
- is_false: fields._tsid._tsid.non_aggregatable_indices
346307

347308
---
348309
sort by tsid:

‎server/src/internalClusterTest/java/org/elasticsearch/search/fieldcaps/FieldCapabilitiesIT.java

+10-1
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,9 @@
2424
import org.elasticsearch.common.breaker.CircuitBreakingException;
2525
import org.elasticsearch.common.io.stream.StreamInput;
2626
import org.elasticsearch.common.settings.Settings;
27+
import org.elasticsearch.index.IndexMode;
2728
import org.elasticsearch.index.IndexService;
29+
import org.elasticsearch.index.IndexSettings;
2830
import org.elasticsearch.index.mapper.DocumentParserContext;
2931
import org.elasticsearch.index.mapper.KeywordFieldMapper;
3032
import org.elasticsearch.index.mapper.MetadataFieldMapper;
@@ -119,7 +121,14 @@ public void setUp() throws Exception {
119121
.endObject()
120122
.endObject()
121123
.endObject();
122-
assertAcked(prepareCreate("old_index").setMapping(oldIndexMapping));
124+
125+
Settings settings = Settings.builder()
126+
.put(IndexSettings.MODE.getKey(), IndexMode.TIME_SERIES)
127+
.putList(IndexMetadata.INDEX_ROUTING_PATH.getKey(), List.of("some_dimension"))
128+
.put(IndexSettings.TIME_SERIES_START_TIME.getKey(), "2006-01-08T23:40:53.384Z")
129+
.put(IndexSettings.TIME_SERIES_END_TIME.getKey(), "2106-01-08T23:40:53.384Z")
130+
.build();
131+
assertAcked(prepareCreate("old_index").setSettings(settings).setMapping(oldIndexMapping));
123132

124133
XContentBuilder newIndexMapping = XContentFactory.jsonBuilder()
125134
.startObject()

0 commit comments

Comments
 (0)