diff --git a/pipeline/outputs/azure_blob.md b/pipeline/outputs/azure_blob.md index 36796e7e2..632dada0d 100644 --- a/pipeline/outputs/azure_blob.md +++ b/pipeline/outputs/azure_blob.md @@ -16,34 +16,63 @@ Ensure you have an Azure Storage account. [Azure Blob Storage Tutorial (video)]( Fluent Bit exposes the following configuration properties. -| Key | Description | Default | -| :--------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------- | -| `account_name` | Azure Storage account name. | _none_ | -| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` | -| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ | -| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ | -| `container_name` | Name of the container that will contain the blobs. | _none_ | -| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` | -| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` | -| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ | -| `compress` | Sets payload compression in network transfer. Supported value: `gzip` | _none_ | -| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | _none_ | -| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `off` | -| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ | -| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` | -| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | -| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` | -| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` | -| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and ago haven't reached `upload_file_size` limit yet. | `30m` | -| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` | -| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` | -| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` | -| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` | -| `blob_uri_length` | Set the length of generated blob URI before ingesting to Azure Kusto. | `64` | -| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` | -| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` | -| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` | -| `io_timeout` | HTTP IO timeout. | `60s` | +| Key | Description | Default | +| :--------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------- | +| `account_name` | Azure Storage account name. | _none_ | +| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` | +| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ | +| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ | +| `container_name` | Name of the container that will contain the blobs. | _none_ | +| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` | +| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` | +| `path` | Optional. Stores blobs under this prefix and now supports [path templating](#path-templating) so you can inject time, tag fragments, or random suffixes. Example: `path=/logs/%Y/%m/$TAG/$UUID` results in `/logs/2025/12/my.tag/abcd1234/myblob`. Trailing slashes are trimmed automatically. | _none_ | +| `compress` | Sets payload compression in network transfer. Supported value: `gzip` | _none_ | +| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | _none_ | +| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `off` | +| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ | +| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` | +| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | +| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` | +| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` | +| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and ago haven't reached `upload_file_size` limit yet. | `30m` | +| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` | +| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` | +| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` | +| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` | +| `blob_uri_length` | Set the length of generated blob URI before ingesting to Azure Kusto. | `64` | +| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` | +| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` | +| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` | +| `io_timeout` | HTTP IO timeout. | `60s` | + +### Path templating + +When `path` is set, Fluent Bit resolves the value as a template (similar to the Amazon S3 output) before each upload. The resolved prefix is persisted alongside buffered files, so retries and restarts keep writing to the same Azure path, and leading/trailing slashes are removed to avoid duplicate separators. If every token expands to an empty string, the blob is stored at the container root. + +| Token / syntax | Example output | Notes | +| :--------------------------------------------------------------- | :----------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `strftime` directives such as `%Y`, `%m`, `%d`, `%H`, `%M`, `%S` | `2025/12/16/05/42/00` | Uses the event timestamp when available; falls back to the current Fluent Bit time. | +| `%3N` | `123` | Millisecond portion of the event timestamp, zero-padded to three digits. | +| `%9N`, `%L` | `123456789` | Nanosecond portion of the event timestamp, zero-padded to nine digits. `%L` is an alias for `%9N`. | +| `$UUID` | `a1b2c3d4` | Eight-character random suffix generated once per blob so that all references (including block list commits) share the same value. | +| `$TAG` | `var.log.containers.app` | Expands to the full tag that matched the output. | +| `$TAG[n]` | `app` | Expands to the nth component (zero-based) of the tag split on dots. This uses Fluent Bit [record accessor syntax](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md); see its [templating limitations](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md#limitations-of-record_accessor-templating). | + +#### Example + +```yaml +pipeline: + outputs: + - name: azure_blob + match: '*' + account_name: YOUR_ACCOUNT_NAME + shared_key: YOUR_SHARED_KEY + container_name: logs + path: "/$TAG[0]/$TAG[4]/%Y/%m/%d/%H/%3N/$UUID" + buffering_enabled: true +``` + +If a chunk arrives with the tag `kube.var.log.containers.app-default`, the sample configuration creates blobs under `kube/app-default/2025/12/16/05/042/abcd1234/...`. Because the resolved prefix is stored with every buffered file, Fluent Bit keeps using the same directory structure even when disk buffering retries or daemon restarts occur. ## Get started