esf-terraform

You can find details on ESF in Elastic Serverless Forwarder for AWS.

This repository contains all necessary resources to deploy ESF.

Requirements

• curl
• terraform

How to use

1. Define secrets and variables in *.auto.tfvars files. See variables.tf for the list of variables declared or read section Inputs. Example:

# variables.auto.tfvars

lambda-name            = "my-esf-lambda"
release-version        = "lambda-v1.19.0" # See https://github.com/elastic/elastic-serverless-forwarder/tags
# config-file-bucket     = "my-esf-bucket" # Uncomment if s3 bucket pre-exists
aws_region             = "eu-central-1"
# config-file-local-path = "./config.yaml" # Uncomment if local config path is used
inputs = [
  {
    type = "cloudwatch-logs"
    id   = "<some_arn>"
    outputs = [
      {
        type = "elasticsearch"
        args = {
          elasticsearch_url  = "https://url.com"
          api_key            = "<some_api_key>"
          es_datastream_name = "logs-esf.cloudwatch-default"
        }
      }
    ]
  }
]

Please read section Inputs configuration for more details on how to configure the inputs. 2. Execute terraform init 3. Execute terraform apply

Inputs configuration

Note: Read Create and upload config.yaml to S3 bucket if you need more details on how the inputs should be configured.

Note: Read Fields to know which values are expected for each field input.

Warning: If you use s3-sqs input type, you also need to configure s3-buckets variable.

When applying these configuration files, a config.yaml file will always be uploaded to an S3 bucket. This S3 bucket will be the one specified in config-file-bucket, or, if the value is left empty, a new S3 bucket will be created.

Following this, we will create the content for the config.yaml file. This file will be built based on:

• Variable inputs. This variable is not required.
• Local configuration file found in config-file-local-path. This variable is also not required.

If both variables are provided, both will be considered. Otherwise, just the one that was given. If none are provided, the config.yaml file will be:

"inputs": []

It does not make sense to leave both empty.

You can see the following examples on the resulting config.yaml file.

Configure just the inputs variable

Configure the inputs variable as:

inputs = [
  {
    type = "cloudwatch-logs"
    id   = "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*"
    outputs = [
      {
        type = "elasticsearch"
        args = {
          elasticsearch_url  = "<url>"
          api_key            = "<api key>"
          es_datastream_name = "logs-esf.cloudwatch-default"
        }
      }
    ]
  }
]

Do not configure the config-file-bucket variable, which will be left as (empty) since that is the default.

The config.yaml placed inside the bucket will be:

"inputs":
  - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*"
    "outputs":
      - "args":
          "api_key": "<api key>"
          "elasticsearch_url": "<url>"
          "es_datastream_name": "logs-esf.cloudwatch-default"
        "type": "elasticsearch"
    "type": "cloudwatch-logs"

Configure just the config-file-local-path variable

Do not configure the inputs variable, which will be left as [] since that is the default.

Configure config-file-local-path variable:

config-file-local-path = "./config.yaml"

And the local config.yaml file looks like:

"inputs":
  - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-local-file:*"
    "outputs":
      - "args":
          "api_key": "<api key>"
          "elasticsearch_url": "<url>"
          "es_datastream_name": "logs-esf.cloudwatch-default"
        "type": "elasticsearch"
    "type": "cloudwatch-logs"

Configure both variables

Configure both inputs and config-file-local-path like in the previous examples.

The config.yaml placed inside the bucket will be:

"inputs":
- "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*"
  "outputs":
  - "args":
      "api_key": "<api key>"
      "elasticsearch_url": "<url>"
      "es_datastream_name": "logs-esf.cloudwatch-default"
    "type": "elasticsearch"
  "type": "cloudwatch-logs"
- "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-local-file:*"
  "outputs":
  - "args":
      "api_key": "<api key>"
      "elasticsearch_url": "<url>"
      "es_datastream_name": "logs-esf.cloudwatch-default"
    "type": "elasticsearch"
  "type": "cloudwatch-logs"

Requirements

Name	Version
terraform	>= 1.5.6
aws	~> 5.32.0
external	~> 2.3.1
local	~> 2.4.0

Providers

Name	Version
aws	5.32.1
terraform	n/a

Modules

Name	Source	Version
esf-lambda-function	terraform-aws-modules/lambda/aws	6.0.0

Resources

Name	Type
aws_cloudwatch_log_subscription_filter.esf-cloudwatch-log-subscription-filter	resource
aws_lambda_event_source_mapping.esf-event-source-mapping-continuing-queue	resource
aws_lambda_event_source_mapping.esf-event-source-mapping-kinesis-data-stream	resource
aws_lambda_event_source_mapping.esf-event-source-mapping-s3-sqs	resource
aws_lambda_event_source_mapping.esf-event-source-mapping-sqs	resource
aws_lambda_permission.esf-cloudwatch-logs-invoke-function-permission	resource
aws_s3_bucket.esf-config-bucket	resource
aws_s3_object.config-file	resource
aws_s3_object.dependencies-file	resource
aws_sqs_queue.esf-continuing-queue	resource
aws_sqs_queue.esf-continuing-queue-dlq	resource
aws_sqs_queue.esf-replay-queue	resource
aws_sqs_queue.esf-replay-queue-dlq	resource
aws_sqs_queue_redrive_allow_policy.esf-continuing-queue-dlq-redrive-allow-policy	resource
aws_sqs_queue_redrive_allow_policy.esf-replay-queue-dlq-redrive-allow-policy	resource
terraform_data.curl-dependencies-zip	resource

Inputs

Name	Description	Type	Default	Required
aws_region	AWS Region	string	n/a	yes
config-file-bucket	The name of the S3 bucket to place the config.yaml file and the dependencies zip. If the variable is left empty, a new bucket will be created. Otherwise, the bucket needs to preexist.	string	""	no
config-file-local-path	Local path to the configuration file. Define this variable only if you want to specify the local configuration file. If none given, make sure to set inputs variable. You can find instructions on how to set the configuration file in https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#sample-s3-config-file.	string	""	no
continuing-queue	Custom BatchSize and MaximumBatchingWindowInSeconds for the ESF SQS Continuing queue	object({ batch_size = optional(number, 10) batching_window_in_second = optional(number, 0) })	{ "batch_size": 10, "batching_window_in_second": 0 }	no
inputs	List of inputs to ESF. If none given, make sure to set config-file-local-path variable. You can find instructions on the variables in https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#s3-config-file-fields.	list(object({ type = string id = string outputs = list(object({ type = string args = object({ elasticsearch_url = optional(string) logstash_url = optional(string) cloud_id = optional(string) api_key = optional(string) username = optional(string) password = optional(string) es_datastream_name = string batch_max_actions = optional(number) batch_max_bytes = optional(number) ssl_assert_fingerprint = optional(string) compression_level = optional(string) }) })) }))	[]	no
kms-keys	List of KMS Keys ARNs to be used for decrypting AWS SSM Secrets, Kinesis Data Streams, SQS queue, or S3 buckets	list(string)	[]	no
lambda-name	ESF Lambda function name	string	n/a	yes
lambda-timeout	The amount of time your Lambda Function has to run in seconds.	number	3	no
log_level	Log level for ESF	string	"INFO"	no
release-version	ESF release version. You can find the possible values in https://github.com/elastic/elastic-serverless-forwarder/tags.	string	n/a	yes
s3-buckets	List of S3 bucket ARNs that are sources for the S3 SQS Event Notifications	list(string)	[]	no
ssm-secrets	List of SSM Secrets ARNs used in the config.yml	list(string)	[]	no
vpc	VPC to attach ESF to, identified by the list of its security group IDs and subnet IDs	object({ security-groups = list(string) subnets = list(string) })	{ "security-groups": [], "subnets": [] }	no

Outputs

Name	Description
config-bucket-name	Name of the bucket with the config.yaml and zip dependencies file.
esf-continuing-queue	Name of the ESF continuing queue.
esf-continuing-queue-dlq	Name of the Dead Letter Queue for the ESF continuing queue.
esf-replay-queue	Name of the ESF replay queue.
esf-replay-queue-dlq	Name of the Dead Letter Queue for the ESF replay queue.