You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Fluent Bit](https://fluentbit.io) collects, parses, filters, and ships logs to a
6
-
central place. A critical piece of this workflow is the ability to do _buffering_: a
6
+
central place. A critical piece of this workflow is the ability to do _buffering_: a
7
7
mechanism to place processed data into a temporary location until is ready to be
8
8
shipped.
9
9
10
10
By default when Fluent Bit processes data, it uses Memory as a primary and temporary
11
-
place to store the records. There are scenarios where it's would be ideal
11
+
place to store the records. There are scenarios where it would be ideal
12
12
to have a persistent buffering mechanism based in the filesystem to provide
13
13
aggregation and data safety capabilities.
14
14
@@ -78,7 +78,7 @@ Look for messages in the Fluent Bit log output like:
78
78
79
79
Using `mem_buf_limit` is good for certain scenarios and environments. It
80
80
helps to control the memory usage of the service. However, if a file rotates while
81
-
the plugin paused, data can be lost since it won't be able to
81
+
the plugin is paused, data can be lost since it won't be able to
82
82
register new records. This can happen with any input source plugin. The goal of
83
83
`mem_buf_limit` is memory control and survival of the service.
84
84
@@ -121,7 +121,7 @@ Enabling filesystem buffering changes the behavior of the engine. Upon chunk
121
121
creation, the engine stores the content in memory and also maps a copy on disk
122
122
through [mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html). The newly
123
123
created chunk is active in memory, backed up on disk, and called to be
124
-
`up` which means the chunk content is up in memory.
124
+
`up`, which means the chunk content is up in memory.
125
125
126
126
Fluent Bit controls the number of chunks that are `up` in memory by using the
127
127
filesystem buffering mechanism to deal with high memory usage and
@@ -134,7 +134,7 @@ and are still receiving records. Any other remaining chunk is in a `down`
134
134
state, which means that it's only in the filesystem and won't be `up` in memory
135
135
unless it's ready to be delivered. Chunks are never much larger than 2 MB,
136
136
so with the default `storage.max_chunks_up` value of 128, each input is limited to
137
-
roughly 256MB of memory.
137
+
roughly 256 MB of memory.
138
138
139
139
If the input plugin has enabled `storage.type` as `filesystem`, when reaching the
140
140
`storage.max_chunks_up` threshold, instead of the plugin being paused, all new data
@@ -171,7 +171,7 @@ It's common to find cases where multiple destinations with different response ti
171
171
exist for a chunk, or one of the destinations is generating backpressure.
172
172
173
173
To limit the amount of filesystem chunks logically queueing, Fluent Bit v1.6 and
174
-
greater, includes the `storage.total_limit_size` configuration property for output
174
+
later includes the `storage.total_limit_size` configuration property for output
175
175
This property limits the total size in bytes of chunks that can exist in the
176
176
filesystem for a certain logical output destination. If one of the destinations
177
177
reaches the configured `storage.total_limit_size`, the oldest chunk from its queue
@@ -197,12 +197,12 @@ The Service section refers to the section defined in the main
197
197
| Key | Description | Default |
198
198
| :--- | :--- | :--- |
199
199
|`storage.path`| Set an optional location in the file system to store streams and chunks of data. If this parameter isn't set, Input plugins can only use in-memory buffering. |_none_|
200
-
|`storage.sync`| Configure the synchronization mode used to store the data in the file system. Possible values: `normal`, `full` . Using `full` increases the reliability of the filesystem buffer and ensures that data is guaranteed to be synced to the filesystem even if Fluent Bit crashes. On Linux, `full` corresponds with the `MAP_SYNC` option for [memory mapped files](https://man7.org/linux/man-pages/man2/mmap.2.html). |`normal`|
201
-
|`storage.checksum`| Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. |`Off`|
200
+
|`storage.sync`| Configure the synchronization mode used to store the data in the file system. Using `full` increases the reliability of the filesystem buffer and ensures that data is guaranteed to be synced to the filesystem even if Fluent Bit crashes. On Linux, `full` corresponds with the `MAP_SYNC` option for [memory mapped files](https://man7.org/linux/man-pages/man2/mmap.2.html). Accepted values: `normal`, `full`. |`normal`|
201
+
|`storage.checksum`| Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. Accepted values: `Off`, `On`. |`Off`|
202
202
|`storage.max_chunks_up`| If the input plugin has enabled `filesystem` storage type, this property sets the maximum number of chunks that can be `up` in memory. Use this setting to control memory usage when you enable `storage.type filesystem`. |`128`|
203
203
|`storage.backlog.mem_limit`| If `storage.path` is set, Fluent Bit looks for data chunks that weren't delivered and are still in the storage layer. These are called _backlog_ data. _Backlog chunks_ are filesystem chunks that were left over from a previous Fluent Bit run; chunks that couldn't be sent before exit that Fluent Bit will pick up when restarted. Fluent Bit will check the `storage.backlog.mem_limit` value against the current memory usage from all `up` chunks for the input. If the `up` chunks currently consume less memory than the limit, it will bring the _backlog_ chunks up into memory so they can be sent by outputs. |`5M`|
204
204
|`storage.metrics`| If `http_server` option is enabled in the main `[SERVICE]` section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the [Monitoring](monitoring.md) section. |`off`|
205
-
|`storage.delete_irrecoverable_chunks`| When enabled, [irrecoverable chunks](./buffering-and-storage.md#irrecoverable-chunks) will be deleted during runtime, and any other irrecoverable chunk located in the configured storage path directory will be deleted when Fluent-Bit starts. |`Off`|
205
+
|`storage.delete_irrecoverable_chunks`| When enabled, [irrecoverable chunks](./buffering-and-storage.md#irrecoverable-chunks) will be deleted during runtime, and any other irrecoverable chunk located in the configured storage path directory will be deleted when Fluent-Bit starts. Accepted values: 'Off`, 'On`. |`Off`|
206
206
207
207
A Service section will look like this:
208
208
@@ -222,12 +222,12 @@ running a checksum and up to a maximum of 5 MB of memory when processing ba
222
222
223
223
### Input Section Configuration
224
224
225
-
Optionally, any Input plugin can configure their storage preference, the following
225
+
Optionally, any Input plugin can configure their storage preference. The following
226
226
table describes the options available:
227
227
228
228
| Key | Description | Default |
229
229
| :--- | :--- | :--- |
230
-
|`storage.type`| Specifies the buffering mechanism to use. Options: `memory` or`filesystem`. |`memory`|
230
+
|`storage.type`| Specifies the buffering mechanism to use. Accepted values: `memory`,`filesystem`. |`memory`|
231
231
|`storage.pause_on_chunks_overlimit`| Specifies if the input plugin should pause (stop ingesting new data) when the `storage.max_chunks_up` value is reached. |`off`|
232
232
233
233
The following example configures a service offering filesystem buffering
@@ -261,7 +261,7 @@ options available:
261
261
262
262
| Key | Description | Default |
263
263
| :--- | :--- | :--- |
264
-
|`storage.total_limit_size`| Limit the maximum disk space size in bytes for buffering chunks in the filesystem for the current output logical destination. |_none_|
264
+
|`storage.total_limit_size`| Limit the maximum disk space size in bytes for buffering chunks in the filesystem for the current output logical destination. |_none_|
265
265
266
266
The following example creates records with CPU usage samples in the filesystem which
267
267
are delivered to Google Stackdriver service while limiting the logical queue
0 commit comments