consolidate/clarify parsers info (#2275)

* consolidate/clarify parsers info

Signed-off-by: Alexa Kreizinger <[email protected]>

* Apply suggestions from code review

Signed-off-by: Alexa Kreizinger <[email protected]>

* Apply suggestions from code review

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Alexa Kreizinger <[email protected]>

---------

Signed-off-by: Alexa Kreizinger <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Author: alexakreizinger

SUMMARY.md

@@ -125,12 +125,12 @@
   * [Windows exporter metrics](pipeline/inputs/windows-exporter-metrics.md)
   * [Windows System Statistics (winstat)](pipeline/inputs/windows-system-statistics.md)
 * [Parsers](pipeline/parsers.md)
-  * [Configuring parsers](pipeline/parsers/configuring-parser.md)
-  * [Decoders](pipeline/parsers/decoders.md)
-  * [JSON](pipeline/parsers/json.md)
-  * [Logfmt](pipeline/parsers/logfmt.md)
-  * [LTSV](pipeline/parsers/ltsv.md)
-  * [Regular expression](pipeline/parsers/regular-expression.md)
+  * [Configuring custom parsers](pipeline/parsers/configuring-parser.md)
+  * [JSON format](pipeline/parsers/json.md)
+  * [Logfmt format](pipeline/parsers/logfmt.md)
+  * [LTSV format](pipeline/parsers/ltsv.md)
+  * [Regular expression format](pipeline/parsers/regular-expression.md)
+  * [Decoder settings](pipeline/parsers/decoders.md)
 * [Processors](pipeline/processors.md)
   * [Content modifier](pipeline/processors/content-modifier.md)
   * [Labels](pipeline/processors/labels.md)

administration/configuring-fluent-bit/yaml.md

@@ -8,14 +8,14 @@ don't support, like processors.
 
 YAML configuration files support the following top-level sections:
 
-- `env`: Configures [environment variables](./yaml/environment-variables-section).
-- `includes`: Specifies additional YAML configuration files to [include as part of a parent file](./yaml/includes-section).
-- `service`: Configures global properties of the Fluent Bit [service](./yaml/service-section).
-- `pipeline`: Configures active [`inputs`, `filters`, and `outputs`](./yaml/pipeline-section).
-- `parsers`: Defines [custom parsers](./yaml/parsers-section).
-- `multiline_parsers`: Defines [custom multiline parsers](./yaml/multiline-parsers-section).
-- `plugins`: Defines paths for [custom plugins](./yaml/plugins-section).
-- `upstream_servers`: Defines [nodes](./yaml/upstream-servers-section) for output plugins.
+- `env`: Configures [environment variables](../administration/configuring-fluent-bit/yaml/environment-variables-section.md).
+- `includes`: Specifies additional YAML configuration files to [include as part of a parent file](../administration/configuring-fluent-bit/yaml/includes-section.md).
+- `service`: Configures global properties of the Fluent Bit [service](../administration/configuring-fluent-bit/yaml/service-section.md).
+- `pipeline`: Configures active [`inputs`, `filters`, and `outputs`](../administration/configuring-fluent-bit/yaml/pipeline-section.md).
+- `parsers`: Defines [custom parsers](../administration/configuring-fluent-bit/yaml/parsers-section.md).
+- `multiline_parsers`: Defines [custom multiline parsers](../administration/configuring-fluent-bit/yaml/multiline-parsers-section.md).
+- `plugins`: Defines paths for [custom plugins](../administration/configuring-fluent-bit/yaml/plugins-section.md).
+- `upstream_servers`: Defines [nodes](../administration/configuring-fluent-bit/yaml/upstream-servers-section.md) for output plugins.
 
 {% hint style="info" %}
 YAML configuration is used in the smoke tests for containers. An always-correct up-to-date example is here: <https://github.com/fluent/fluent-bit/blob/master/packaging/testing/smoke/container/fluent-bit.yaml>.

administration/configuring-fluent-bit/yaml/parsers-section.md

@@ -1,23 +1,86 @@
 # Parsers
 
-Parsers enable Fluent Bit components to transform unstructured data into a structured internal representation. You can define YAML parsers either directly in the main configuration file or in separate external files for better organization.
+You can define customer [parsers](../pipeline/parsers.md) in the `parsers` section of YAML configuration files.
 
-This page provides a general overview of how to declare parsers.
+{% hint style="info" %}
 
-The main section name is `parsers`, and it lets you define a list of parser configurations. The following example demonstrates how to set up two basic parsers:
+To define custom multiline parsers, use [the `multiline_parsers` section](../administration/configuring-fluent-bit/yaml/multiline-parsers-section.md) of YAML configuration files.
+
+{% endhint %}
+
+## Syntax
+
+To define customers parsers in the `parsers` section of a YAML configuration file, use the following syntax.
+
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
 
 ```yaml
 parsers:
-  - name: json
+  - name: custom_parser1
     format: json
+    time_key: time
+    time_format: '%Y-%m-%dT%H:%M:%S.%L'
+    time_keep: on
+
+  - name: custom_parser2
+    format: regex
+    regex: '^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$'
+    time_key: time
+    time_format: '%Y-%m-%dT%H:%M:%S.%L'
+    time_keep: on
+    types: pid:integer
+```
+
+{% endtab %}
+{% endtabs %}
+
+For information about supported configuration options for custom parsers, see [configuring parsers](../pipeline/parsers/configuring-parser.md).
+
+## Standalone parsers files
+
+In addition to defining parsers in the `parsers` section of YAML configuration files, you can store parser definitions in standalone files. These standalone files require the same syntax as parsers defined in a standard YAML configuration file.
+
+To add a standalone parsers file to Fluent Bit, use the `parsers_file` parameter in the `service` section of your YAML configuration file.
 
-  - name: docker
+### Add a standalone parsers file to Fluent Bit
+
+To add a standalone parsers file to Fluent Bit, follow these steps.
+
+1. Define custom parsers in a standalone YAML file. For example, `my-parsers.yaml` defines two custom parsers:
+
+{% tabs %}
+{% tab title="my-parsers.yaml" %}
+
+```yaml
+parsers:
+  - name: custom_parser1
     format: json
     time_key: time
-    time_format: "%Y-%m-%dT%H:%M:%S.%L"
-    time_keep: true
+    time_format: '%Y-%m-%dT%H:%M:%S.%L'
+    time_keep: on
+
+  - name: custom_parser2
+    format: regex
+    regex: '^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$'
+    time_key: time
+    time_format: '%Y-%m-%dT%H:%M:%S.%L'
+    time_keep: on
+    types: pid:integer
 ```
 
-You can define multiple parsers sections, either within the main configuration file or distributed across included files.
+{% endtab %}
+{% endtabs %}
+
+1. Update the `parsers_file` parameter in the `service` section of your YAML configuration file:
+
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
+
+```yaml
+service:
+  parsers_file: my-parsers.yaml
+```
 
-For more detailed information on parser options and advanced configurations, refer to the [Configuring Parsers](../../../pipeline/parsers/configuring-parser.md) documentation.
+{% endtab %}
+{% endtabs %}

administration/configuring-fluent-bit/yaml/pipeline-section.md

@@ -65,7 +65,7 @@ pipeline:
 
 In the cases where each value in a list requires two values they must be separated by a space, such as in the `record` property for the `record_modifier` filter.
 
-### Input
+### Inputs
 
 An `input` section defines a source (related to an input plugin). Each section has a base configuration. Each input plugin can add it own configuration keys:
 
@@ -88,7 +88,7 @@ pipeline:
           tag: my_cpu
 ```
 
-### Filter
+### Filters
 
 A `filter` section defines a filter (related to a filter plugin). Each section has a base configuration and each filter plugin can add its own configuration keys:
 
@@ -113,7 +113,7 @@ pipeline:
           regex: log aa
 ```
 
-### Output
+### Outputs
 
 The `outputs` section specifies a destination that certain records should follow after a `Tag` match. Fluent Bit can route up to 256 `OUTPUT` plugins. The configuration supports the following keys:
 

administration/configuring-fluent-bit/yaml/service-section.md

@@ -2,7 +2,6 @@
 
 The `service` section defines global properties of the service. The available configuration keys are:
 
-
 | Key | Description | Default Value |
 | --- | ----------- | ------------- |
 | `flush`  | Sets the flush time in `seconds.nanoseconds`. The engine loop uses a flush timeout to define when to flush the records ingested by input plugins through the defined output plugins. | `1` |
@@ -11,9 +10,9 @@ The `service` section defines global properties of the service. The available co
 | `dns.mode` | Sets the primary transport layer protocol used by the asynchronous DNS resolver. Can be overridden on a per-plugin basis. | `UDP` |
 | `log_file` | Absolute path for an optional log file. By default, all logs are redirected to the standard error interface (`stderr`). | _none_ |
 | `log_level` | Sets the logging verbosity level. Possible values: `off`, `error`, `warn`, `info`, `debug`, and `trace`. Values are cumulative. For example, if `debug` is set, it will include `error`, `warning`, `info`, and `debug`. The `trace` mode is only available if Fluent Bit was built with the `WITH_TRACE` option enabled. | `info` |
-| `parsers_file` | Path for a parsers configuration file. Multiple `parsers_file` entries can be defined within the section. Parsers can be declared directly in the [`parsers` section](./parsers-section.md) of YAML configuration files. | _none_ |
-| `plugins_file` | Path for a `plugins` configuration file. This file specifies the paths to custom plugins (.so files) that Fluent Bit can load at runtime. Plugins can be declared directly in the [`plugins` section](./plugins-section.md) of YAML configuration files. | _none_ |
-| `streams_file` | Path for the [stream processor](../../../stream-processing/overview.md) configuration file. This file defines the rules and operations for stream processing in Fluent Bit. Stream processor configurations can also be defined directly in the `streams` section of YAML configuration files. | _none_ |
+| `parsers_file` | Path for [standalone parsers configuration files](../administration/configuring-fluent-bit/yaml/parsers-section.md#standalone-parsers-files). You can include one or more files. | _none_ |
+| `plugins_file` | Path for a `plugins` configuration file. This file specifies the paths to custom plugins (.so files) that Fluent Bit can load at runtime. Plugins can be declared directly in the [`plugins` section](../administration/configuring-fluent-bit/yaml/plugins-section.md) of YAML configuration files. | _none_ |
+| `streams_file` | Path for the [stream processor](../stream-processing/overview.md) configuration file. This file defines the rules and operations for stream processing in Fluent Bit. Stream processor configurations can also be defined directly in the `streams` section of YAML configuration files. | _none_ |
 | `http_server` | Enables the built-in HTTP server. | `off` |
 | `http_listen` | Sets the listening interface for the HTTP Server when it's enabled. | `0.0.0.0` |
 | `http_port` | Sets the TCP port for the HTTP server. | `2020` |
@@ -28,7 +27,7 @@ The `service` section defines global properties of the service. The available co
 
 ## Storage configuration
 
-The following storage-related keys can be set in the `service` section:
+The following storage-related keys can be set as children to the `storage` key:
 
 | Key | Description | Default Value |
 | --- | ----------- | ------------- |

pipeline/parsers.md

@@ -1,8 +1,6 @@
 # Parsers
 
-Dealing with raw strings or unstructured messages is difficult. Having a structure makes data more usable. Set a structure for the incoming data by using input plugins as data is collected.
-
-Parsers are fully configurable and are independently and optionally handled by each input plugin.
+You can use parsers to transform unstructured log entries into structured log entries.
 
 ```mermaid
 graph LR
@@ -18,15 +16,15 @@ graph LR
     style B stroke:darkred,stroke-width:2px;
 ```
 
-The parser converts unstructured data to structured data. As an example, consider the following Apache (HTTP Server) log entry:
+For example, a parser can turn an unstructured log entry like this:
 
 ```text
 192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395
 ```
 
-This log line is a raw string without format. Structuring the log makes it easier to process the data later. If the [regular expression parser](./parsers/regular-expression.md) is used, the log entry could be converted to:
+...into a structured JSON object like this:
 
-```javascript
+```json
 {
   "host":    "192.168.2.20",
   "user":    "-",
@@ -38,3 +36,44 @@ This log line is a raw string without format. Structuring the log makes it easie
   "agent":   ""
  }
 ```
+
+## How parsers work
+
+Parsers modify the data ingested by input plugins. This modification happens before Fluent Bit applies any [filters](../pipeline/filters.md) or [processors](..pipeline/processors.md) to that data.
+
+Each input plugin can have one active parser. Multiple plugins within the same Fluent Bit configuration file can use the same parser or use different parsers from each other.
+
+### Default parsers and custom parsers
+
+Fluent Bit includes a variety of [default parsers](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) for parsing common data formats, like Apache and Docker logs. You can also [define custom parsers](../configuring-fluent-bit/yaml/parsers-section.md).
+
+## Add a parser to an input plugin
+
+To add a parser to an input plugin, follow these steps.
+
+1. Either identify the name of the [default parser](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) you want to use, or [define a custom parser](../configuring-fluent-bit/yaml/parsers-section.md) with your desired [configuration settings](../pipeline/parsers/configuring-parser.md).
+
+1. Add a `parsers` key to the plugin's settings in the [`inputs`](../administration/configuring-fluent-bit/yaml/pipeline-section.md#inputs) section of your YAML configuration file.
+
+For example, the following configuration file adds the default [`apache` parser](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf#L2) to one input plugin and a custom parser named `custom_parser1` to another input plugin:
+
+
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
+
+```yaml
+pipeline:
+  inputs:
+    - name: tail
+      path: /input/input.log
+      refresh_interval: 1
+      parser: apache
+
+    - name: http
+      listen: 0.0.0.0
+      port: 8888
+      parser: custom_parser1
+```
+
+{% endtab %}
+{% endtabs %}