Skip to content

[DOCS] Clarify supported rollup fields #70045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 9, 2021
Merged

[DOCS] Clarify supported rollup fields #70045

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
07e0f23
[DOCS] Clarify supported rollup fields
jrodewig Mar 5, 2021
983eb07
Note metric field
jrodewig Mar 9, 2021
4f8d5ca
Reword
jrodewig Mar 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 34 additions & 20 deletions docs/reference/rollup/apis/rollup-api.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,10 @@ POST /my-index-000001/_rollup/rollup-my-index-000001

* You can only roll up an index that contains:

** A <<date,`date`>> or <<date_nanos,`date_nanos`>> timestamp field
** One or more <<number,numeric>> fields
** A <<date,`date`>> or <<date_nanos,`date_nanos`>> timestamp field.
** At least one metric to roll up. This field must be a <<number,numeric>>,
`date`, or `date_nanos` field other than the timestamp field. This field
cannot be an `unsigned_long`.

* If the {es} {security-features} are enabled, you must have the `manage`
<<privileges-list-indices,index privilege>> for the index you roll up.
Expand Down Expand Up @@ -85,7 +87,7 @@ for the same stream.
// tag::rollup-config[]
`groups`::
(Required, object)
Aggregates and stores fields in the rollup.
Aggregates and groups documents in the rollup.
+
.Properties of `groups`
[%collapsible%open]
Expand All @@ -104,6 +106,9 @@ Groups documents based on a provided time interval.
<<date,`date`>> or <<date_nanos,`date_nanos`>> field containing a timestamp. If
you're rolling up a backing index or using the {ecs-ref}[Elastic Common Schema
(ECS)], we recommend using `@timestamp`.
+
WARNING: Do not use this field in `metrics`. If you do, the rollup attempt will
fail.

`calendar_interval` or `fixed_interval`::
(Required, <<time-units,time units>>)
Expand All @@ -125,18 +130,18 @@ Defaults to `+00:00` (UTC).
`histogram`::
(Optional, <<search-aggregations-bucket-histogram-aggregation,`histogram`
aggregation>> object)
Groups and stores <<number,numeric>> field values based on a provided interval.
Groups documents based on a numeric interval.
+
.Properties of `histogram`
[%collapsible%open]
======
`fields`::
(Required*, string or array of strings)
<<number,Numeric>> fields to group. If you specify a `histogram` object, this
property is required.
<<number,Numeric>> fields to group. `unsigned_long` fields are not supported. If
you specify a `histogram` object, this property is required.
+
WARNING: Do not use the same fields in `histogram` and `metrics`. If you specify
the same field in both `histogram` and `metrics`, the rollup attempt will fail.
WARNING: Do not use these fields in `metrics`. If you do, the rollup attempt
will fail.

`interval`::
(Required*, integer)
Expand All @@ -147,36 +152,44 @@ object, this property is required.
`terms`::
(Optional, <<search-aggregations-bucket-terms-aggregation,`terms`
aggregation>> object)
Groups and stores unique values for <<keyword,keyword family>> and
<<number,numeric>> fields.
Groups documents based on unique field values.
+
.Properties of `terms`
[%collapsible%open]
======
`fields`::
(Required*, string or array of strings)
<<keyword,Keyword family>> and <<number,numeric>> fields to store. If you
specify a `terms` object, this property is required.
+
--
(Required*, string or array of strings)
Fields to store unique values for. Supports the following field types:

* <<keyword,Keyword family>> types
* <<number,Numeric>> types, excluding `unsigned_long`
* <<text,`text`>> fields with <<fielddata-mapping-param,`fielddata`>> enabled

If you specify a `terms` object, this property is required.

TIP: Avoid storing high-cardinality fields. High-cardinality fields can greatly
increase the size of the resulting rollup index.
--
======
=====

`metrics`::
(Required, object or array of objects)
Collects and stores metrics for <<number,numeric>> fields. You must specify at
least one `metrics` object.
Collects and stores metrics for fields. You must specify at least one `metrics`
object.
+
.Properties of `metrics` objects
[%collapsible%open]
=====
`field`::
(Required, string)
<<number,Numeric>> field to collect metrics for.
<<number,Numeric>>, <<date,`date`>>, or <<date_nanos,`date_nanos`>> field to
collect metrics for. `unsigned_long` fields are not supported.
Comment on lines +188 to +189
Copy link
Contributor Author

@jrodewig jrodewig Mar 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seems to be a minor bug with date_nanos fields. Here's reproduction steps:

// Map a date_nanos field.
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "my-date-nanos-field": {
        "type": "date_nanos" 
      }
    }
  }
}

// Index a document with the date_nanos field.
POST my-index-000001/_doc
{ 
  "@timestamp": "2099-11-15T13:12:00",
  "my-date-nanos-field": "2015-01-01T12:10:30.123456789Z"
}

// Attempt to roll up the index with the date_nanos field as a metric.
POST my-index-000001/_rollup/rollup-my-index-000001
{
  "groups": {
    "date_histogram": {
      "field": "@timestamp",
      "calendar_interval": "1d"
    }
  },
  "metrics": [
    {
      "field": "my-date-nanos-field",
      "metrics": [
        "min",
        "max"
      ]
    }
  ]
}

The rollup attempt returns an error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "trying to convert a known date time formatter to a nanosecond one, wrong field used?"
      }
    ],
    "type" : "exception",
    "reason" : "Unable to rollup index [my-index-000001]",
    "caused_by" : {
      "type" : "exception",
      "reason" : "[my-index-000001/9DveCFw8RR67S2tul3Qt_w][[my-index-000001][0]] org.elasticsearch.action.support.broadcast.BroadcastShardOperationFailedException:",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "trying to convert a known date time formatter to a nanosecond one, wrong field used?"
      }
    }
  },
  "status" : 500
}

This seems to be the case even if an epoch_millis format is used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^^ @talevy

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will take a look. thank you!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened #70161

+
WARNING: Do not use the same fields in `histogram` and `metrics`. If you specify
the same field in both `histogram` and `metrics`, the rollup attempt will fail.
WARNING: Do not use fields specified in `histogram` or `date_histogram`. If you
do, the rollup attempt will fail.

`metrics`::
(Required, string or array of strings)
Expand All @@ -186,8 +199,9 @@ Metrics to collect. Each value corresponds to a
<<search-aggregations-metrics-max-aggregation,`max`>>,
<<search-aggregations-metrics-sum-aggregation,`sum`>>,
<<search-aggregations-metrics-avg-aggregation,`avg`>>, and
<<search-aggregations-metrics-valuecount-aggregation,`value_count`>>. You must
specify at least one value.
<<search-aggregations-metrics-valuecount-aggregation,`value_count`>>. <<date,`date`>> and <<date_nanos,`date_nanos`>>
fields only support the `max`, `min`, and `value_count` metrics. You must specify
at least one value.
+
NOTE: The rollup index stores these metrics in an
<<aggregate-metric-double,`aggregate_metric_double`>> field. The `avg` metric
Expand Down