Skip to content

Commit f10cd68

Browse files
meehanmanmeehanman
committed
docs: out_opentelemetry: Add 'encoding' option for OTLP/HTTP JSON export
Signed-off-by: Dean Meehan <d3an.meehan@hotmail.com>
1 parent d3d0a7a commit f10cd68

1 file changed

Lines changed: 77 additions & 46 deletions

File tree

‎pipeline/outputs/opentelemetry.md‎

Lines changed: 77 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@
44
**Supported event types:** `logs` `metrics` `traces` `profiles`
55
{% endhint %}
66

7-
The OpenTelemetry plugin lets you take logs, metrics, traces, and profiles from Fluent Bit and submit them to an OpenTelemetry HTTP endpoint.
7+
The OpenTelemetry plugin lets you take logs, metrics, traces, and profiles from Fluent Bit and submit them to an OpenTelemetry receiver over OTLP/HTTP or OTLP/gRPC.
88

9-
Only HTTP endpoints are supported.
9+
OTLP/HTTP supports both HTTP/1.1 and HTTP/2. OTLP/gRPC requires HTTP/2 and is enabled with the `grpc` option.
1010

1111

1212
| Key | Description | Default |
@@ -22,42 +22,43 @@ Only HTTP endpoints are supported.
2222
| `aws_sts_endpoint` | Custom endpoint for the AWS STS API, used with the `aws_role_arn` option. | _none_ |
2323
| `batch_size` | Set the maximum number of log records to be flushed at a time. | `1000` |
2424
| `compress` | Set payload compression mechanism. Options available are `gzip` and `zstd`. | _none_ |
25-
| `grpc` | Enable, disable or auto-detect gRPC usage. Accepted values: `on`, `off`, `auto`. | `off` |
26-
| `grpc_logs_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` |
27-
| `grpc_metrics_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` |
28-
| `grpc_profiles_uri` | Specify an optional gRPC URI for profiles OTel endpoint. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` |
29-
| `grpc_traces_uri` | Specify an optional gRPC URI for the target OTel endpoint. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` |
25+
| `encoding` | Set the encoding used to send logs, metrics, and traces over OTLP/HTTP. Accepted values: `protobuf` (default, `Content-Type: application/x-protobuf`) or `json` (`Content-Type: application/json`). Profiles always use `protobuf`. | `protobuf` |
26+
| `grpc` | Enable gRPC transport (OTLP/gRPC). When enabled, HTTP/2 is automatically activated. Accepted values: `on`, `off`. | `off` |
27+
| `grpc_logs_uri` | gRPC endpoint URI for log exports. | `/opentelemetry.proto.collector.logs.v1.LogsService/Export` |
28+
| `grpc_metrics_uri` | gRPC endpoint URI for metric exports. | `/opentelemetry.proto.collector.metrics.v1.MetricsService/Export` |
29+
| `grpc_profiles_uri` | gRPC endpoint URI for profile exports. | `/opentelemetry.proto.collector.profiles.v1experimental.ProfilesService/Export` |
30+
| `grpc_traces_uri` | gRPC endpoint URI for trace exports. | `/opentelemetry.proto.collector.trace.v1.TraceService/Export` |
3031
| `header` | Add a HTTP header key/value pair. Multiple headers can be set. | _none_ |
3132
| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` |
32-
| `http2` | Enable, disable or force HTTP/2 usage. Accepted values : `on`, `off`, or `force`. | `off` |
33+
| `http2` | Enable, disable or force HTTP/2 usage. Accepted values: `on`, `off`, or `force`. | `off` |
3334
| `http_passwd` | Set HTTP auth password. | _none_ |
3435
| `http_user` | Set HTTP auth user. | _none_ |
3536
| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` |
3637
| `log_response_payload` | Specify if the response payload should be logged or not. | `true` |
3738
| `log_suppress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` disables suppression. | `0` |
38-
| `logs_attributes_metadata_key` | Specify an `Attributes` key. | `$Attributes` |
39-
| `logs_body_key` | Specify an optional HTTP URI for the target OTel endpoint. | _none_ |
40-
| `logs_body_key_attributes` | If set and it matched a pattern, it includes the remaining fields in the record as attributes. | `false`|
41-
| `logs_instrumentation_scope_metadata_key` | Specify an `InstrumentationScope` key. | `InstrumentationScope` |
39+
| `logs_attributes_metadata_key` | Metadata key used to read log record attributes. | `$Attributes` |
40+
| `logs_body_key` | Record accessor pattern(s) used to populate the OTLP log body. The first matching key wins. May be specified multiple times. Defaults to `$log` then `$message`. | _none_ |
41+
| `logs_body_key_attributes` | When `true`, record fields not matched by `logs_body_key` are added to the log record's attributes instead of being dropped. | `false`|
42+
| `logs_instrumentation_scope_metadata_key` | Metadata key used to read the instrumentation scope. | `InstrumentationScope` |
4243
| `logs_max_resources` | Set the maximum number of OTLP log resources per export request (`0` disables the limit). | `0` |
4344
| `logs_max_scopes` | Set the maximum number of OTLP log scopes per resource (`0` disables the limit). | `0` |
44-
| `logs_metadata_key` | Set the key to look up in the metadata. | `otlp` |
45-
| `logs_observed_timestamp_metadata_key` | Specify an `ObservedTimestamp` key. | `$ObservedTimestamp` |
46-
| `logs_resource_metadata_key` | Specify a `Resource` key. | `Resource` |
47-
| `logs_severity_number_message_key` | Specify a `SeverityNumber` key. | `$severityNumber` |
48-
| `logs_severity_number_metadata_key` | Specify a `SeverityNumber` key. | `$SeverityNumber` |
49-
| `logs_severity_text_message_key` | Specify a `SeverityText` key. | `$SeverityText` |
50-
| `logs_severity_text_metadata_key` | Specify a `SeverityText` key. | `$SeverityText` |
51-
| `logs_span_id_message_key` | Specify a `SpanId` key. | `$SpanId` |
52-
| `logs_span_id_metadata_key` | Specify a `SpanId` key. | `$SpanId` |
53-
| `logs_timestamp_metadata_key` | Specify a `Timestamp` key. | `$Timestamp` |
54-
| `logs_trace_flags_metadata_key` | Specify a `TraceFlags` key. | `$TraceFlags` |
55-
| `logs_trace_id_message_key` | Specify a `TraceId` key. | `$TraceId` |
56-
| `logs_trace_id_metadata_key` | Specify a `TraceId` key. | `$TraceId` |
57-
| `logs_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/logs` |
45+
| `logs_metadata_key` | Top-level metadata key used to look up OTLP metadata added by `in_opentelemetry`. | `otlp` |
46+
| `logs_observed_timestamp_metadata_key` | Metadata key used to read the observed timestamp. | `$ObservedTimestamp` |
47+
| `logs_resource_metadata_key` | Metadata key used to read the OTLP resource. | `Resource` |
48+
| `logs_severity_number_message_key` | Record body key used to read the severity number. | `$SeverityNumber` |
49+
| `logs_severity_number_metadata_key` | Metadata key used to read the severity number. | `$SeverityNumber` |
50+
| `logs_severity_text_message_key` | Record body key used to read the severity text. | `$SeverityText` |
51+
| `logs_severity_text_metadata_key` | Metadata key used to read the severity text. | `$SeverityText` |
52+
| `logs_span_id_message_key` | Record body key used to read the span ID. | `$SpanId` |
53+
| `logs_span_id_metadata_key` | Metadata key used to read the span ID. | `$SpanId` |
54+
| `logs_timestamp_metadata_key` | Metadata key used to read the log timestamp. | `$Timestamp` |
55+
| `logs_trace_flags_metadata_key` | Metadata key used to read the trace flags. | `$TraceFlags` |
56+
| `logs_trace_id_message_key` | Record body key used to read the trace ID. | `$TraceId` |
57+
| `logs_trace_id_metadata_key` | Metadata key used to read the trace ID. | `$TraceId` |
58+
| `logs_uri` | HTTP URI for the logs OTLP endpoint. | `/v1/logs` |
5859
| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ |
5960
| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to wildcards. | _none_ |
60-
| `metrics_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/metrics` |
61+
| `metrics_uri` | HTTP URI for the metrics OTLP endpoint. | `/v1/metrics` |
6162
| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` |
6263
| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` |
6364
| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ |
@@ -89,7 +90,7 @@ Only HTTP endpoints are supported.
8990
| `oauth2.timeout` | Timeout for `OAuth 2.0` token requests. | `0s` |
9091
| `oauth2.token_url` | `OAuth 2.0` token endpoint URL. | _none_ |
9192
| `port` | TCP port of the target HTTP server. | `80` |
92-
| `profiles_uri` | Specify an optional HTTP URI for the profiles OTel endpoint. | `/v1development/profiles` |
93+
| `profiles_uri` | HTTP URI for the profiles OTLP endpoint. | `/v1development/profiles` |
9394
| `proxy` | Specify an HTTP Proxy. The expected format of this value is `http://host:port`.| _none_ |
9495
| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` |
9596
| `tls` | Enable or disable TLS/SSL support. | `off` |
@@ -107,22 +108,58 @@ Only HTTP endpoints are supported.
107108
| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ |
108109
| `tls.windows.certstore_name` | Sets the `certstore` name on an output (Windows). | _none_ |
109110
| `tls.windows.use_enterprise_store` | Sets whether using enterprise `certstore` or not on an output (Windows). | _none_ |
110-
| `traces_uri` | Specify an optional HTTP URI for the target OTel endpoint. | `/v1/traces` |
111+
| `traces_uri` | HTTP URI for the traces OTLP endpoint. | `/v1/traces` |
111112
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
112113

113114
## Get started
114115

115-
The OpenTelemetry plugin works with logs and only the metrics collected from one of the metric input plugins. In the following example, log records generated by the dummy plugin and the host metrics collected by the node exporter metrics plugin are exported by the OpenTelemetry output plugin.
116+
The OpenTelemetry plugin supports all four OTLP signal types: logs, metrics, traces, and profiles. In the following example, log records generated by the dummy plugin and host metrics from the node exporter metrics plugin are exported together over OTLP/HTTP.
117+
118+
### Encoding
119+
120+
By default the plugin sends all signals as binary protobuf (`Content-Type: application/x-protobuf`). Set `encoding json` to send logs, metrics, and traces as OTLP JSON (`Content-Type: application/json`) instead. This is useful when sending to receivers that only accept JSON, such as HTTP webhook endpoints.
121+
122+
{% hint style="info" %}
123+
The `encoding` option applies to **OTLP/HTTP** exports only. When `grpc` is `on`, logs, metrics, and traces are sent over OTLP/gRPC using protobuf framing.
124+
125+
Profiles are always sent as protobuf regardless of the `encoding` setting — no OTLP JSON encoder exists for the profiles signal yet.
126+
{% endhint %}
127+
128+
{% tabs %}
129+
{% tab title="fluent-bit.yaml" %}
130+
131+
```yaml
132+
pipeline:
133+
outputs:
134+
- name: opentelemetry
135+
match: "*"
136+
host: localhost
137+
port: 4318
138+
encoding: json
139+
```
140+
141+
{% endtab %}
142+
{% tab title="fluent-bit.conf" %}
143+
144+
```text
145+
[OUTPUT]
146+
Name opentelemetry
147+
Match *
148+
Host localhost
149+
Port 4318
150+
Encoding json
151+
```
152+
153+
{% endtab %}
154+
{% endtabs %}
155+
156+
### Full example
116157

117158
{% tabs %}
118159
{% tab title="fluent-bit.yaml" %}
119160

120161
```yaml
121-
# Dummy Logs and traces with Node Exporter Metrics export using OpenTelemetry output plugin
122-
# -------------------------------------------
123-
# The following example collects host metrics on Linux and dummy logs and traces and delivers
124-
# them through the OpenTelemetry plugin to a local collector :
125-
#
162+
# Collect host metrics and dummy logs, export via OpenTelemetry output
126163
service:
127164
flush: 1
128165
log_level: info
@@ -156,7 +193,6 @@ pipeline:
156193
logs_trace_id_message_key: trace_id
157194
logs_severity_text_message_key: loglevel
158195
logs_severity_number_message_key: lognum
159-
# add user-defined labels
160196
add_label:
161197
- app fluent-bit
162198
- color blue
@@ -166,11 +202,7 @@ pipeline:
166202
{% tab title="fluent-bit.conf" %}
167203
168204
```text
169-
# Dummy Logs and traces with Node Exporter Metrics export using OpenTelemetry output plugin
170-
# -------------------------------------------
171-
# The following example collects host metrics on Linux and dummy logs and traces and delivers
172-
# them through the OpenTelemetry plugin to a local collector :
173-
#
205+
# Collect host metrics and dummy logs, export via OpenTelemetry output
174206
[SERVICE]
175207
Flush 1
176208
Log_Level info
@@ -200,12 +232,11 @@ pipeline:
200232
Log_response_payload True
201233
Tls On
202234
Tls.verify Off
203-
Logs_Body_Key $message
204-
Logs_Span_Id_Message_Key span_id
205-
Logs_Trace_Id_Message_Key trace_id
206-
Logs_Severity_Text_Message_Key loglevel
235+
Logs_Body_Key $message
236+
Logs_Span_Id_Message_Key span_id
237+
Logs_Trace_Id_Message_Key trace_id
238+
Logs_Severity_Text_Message_Key loglevel
207239
Logs_Severity_Number_Message_Key lognum
208-
# add user-defined labels
209240
Add_Label app fluent-bit
210241
Add_Label color blue
211242
```

0 commit comments

Comments
 (0)