You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _data-prepper/pipelines/pipelines.md
+65-13Lines changed: 65 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -65,31 +65,83 @@ If a pipeline component fails to process and send an event, then the source rece
65
65
66
66
Pipelines also support conditional routing, which enables the routing of events to different sinks based on specific conditions. To add conditional routing, specify a list of named routes using the `route` component and assign specific routes to sinks using the `routes` property. Any sink with the `routes` property only accepts events matching at least one of the routing conditions.
67
67
68
-
In the following example pipeline, `application-logs` is a named route with a condition set to `/log_type == "application"`. The route uses [Data Prepper expressions](https://github.com/opensearch-project/data-prepper/tree/main/examples) to define the condition. Data Prepper routes events satisfying this condition to the first OpenSearch sink. By default, Data Prepper routes all events to sinks without a defined route, as shown in the third OpenSearch sink of the given pipeline:
68
+
In the following pipeline, routes are defined at the pipeline level under `route`. The route uses [Data Prepper expressions](https://github.com/opensearch-project/data-prepper/tree/main/examples) to define the condition. Two named routes are declared:
69
+
70
+
- `errors: /level == "ERROR"`
71
+
72
+
- `slow_requests: /latency_ms != null and /latency_ms >= 1000`
73
+
74
+
Each OpenSearch sink can opt in to one or more routes using the `routes:` setting. Events that satisfy a route's condition are delivered to the sinks that reference that route. For example, the first sink receives events matching `errors`, and the second sink receives events matching `slow_requests`.
75
+
76
+
By default, any sink without a `routes:` list receives all events, regardless of whether they matched other routes. In the following example, the third sink has no `routes:` setting, so it receives all events, including those already routed to the first two sinks:
69
77
70
78
```yml
71
-
conditional-routing-sample-pipeline:
79
+
routes-demo-pipeline:
72
80
source:
73
81
http:
74
-
processor:
82
+
path: /logs
83
+
ssl: false
84
+
75
85
route:
76
-
- application-logs: '/log_type == "application"'
77
-
- http-logs: '/log_type == "apache"'
86
+
- errors: '/level == "ERROR"'
87
+
- slow_requests: '/latency_ms != null and /latency_ms >= 1000'
88
+
78
89
sink:
90
+
# 1) Only events matching the "errors" route
79
91
- opensearch:
80
-
hosts: [ "https://opensearch:9200" ]
81
-
index: application_logs
82
-
routes: [application-logs]
92
+
hosts: ["https://opensearch:9200"]
93
+
insecure: true
94
+
username: admin
95
+
password: admin_pass
96
+
index_type: custom
97
+
index: routed-errors-%{yyyy.MM.dd}
98
+
routes: [errors]
99
+
100
+
# 2) Only events matching the "slow_requests" route
83
101
- opensearch:
84
-
hosts: [ "https://opensearch:9200" ]
85
-
index: http_logs
86
-
routes: [http-logs]
102
+
hosts: ["https://opensearch:9200"]
103
+
insecure: true
104
+
username: admin
105
+
password: admin_pass
106
+
index_type: custom
107
+
index: routed-slow-%{yyyy.MM.dd}
108
+
routes: [slow_requests]
109
+
110
+
# 3) All events
87
111
- opensearch:
88
-
hosts: [ "https://opensearch:9200" ]
89
-
index: all_logs
112
+
hosts: ["https://opensearch:9200"]
113
+
insecure: true
114
+
username: admin
115
+
password: admin_pass
116
+
index_type: custom
117
+
index: routed-other-%{yyyy.MM.dd}
90
118
```
91
119
{% include copy.html %}
92
120
121
+
You can test this pipeline using the following command:
0 commit comments