Skip to content

Commit 1e4db62

Browse files
PettitWesleyPettitWesley
committed
Initial set of fluent bit examples
1 parent 6f0f3f6 commit 1e4db62

29 files changed

+637
-1
lines changed

‎README.md

+78-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,84 @@
1-
## Amazon Ecs Firelens Examples
1+
## Amazon ECS FireLens Examples
22

33
Sample logging architectures for FireLens on Amazon ECS and AWS Fargate.
44

5+
### Contributing
6+
7+
We want examples of as many use cases in this repository as possible! Submit a Pull Request if you would like to add something.
8+
9+
### Fluent Bit Examples
10+
11+
* [Send to CloudWatch Logs](examples/fluent-bit/cloudwatchlogs)
12+
* [Send to Kinesis Data Firehose](examples/fluent-bit/kinesis-firehose)
13+
* [Enable Debug Logging](examples/fluent-bit/enable-debug-logging)
14+
* [Forward to a Fluentd or Fluent Bit Log Aggregator](examples/fluent-bit/forward-to-aggregator)
15+
* [Parse Serialized JSON](examples/fluent-bit/parse-json)
16+
* [Parse common log formats](examples/fluent-bit/parse-common-log-formats)
17+
* [Send to multiple destinations](examples/fluent-bit/send-to-multiple-destinations)
18+
* [Add custom metadata to logs](examples/fluent-bit/add-keys)
19+
20+
### Fluentd Examples
21+
22+
TODO
23+
24+
### Setup for the examples
25+
26+
Before you use FireLens, familiarize yourself with [Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted_EC2.html) and with the [FireLens documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html).
27+
28+
In order to use these examples, you will need the following IAM resources:
29+
* A [Task IAM Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) with permissions to send logs to your log destination. Each of the examples in this repository that needs additional permissions has a sample policy.
30+
* A [Task Execution Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html). This role is used by the ECS Agent to make calls on your behalf. If you enable logging for your FireLens container with the [`awslogs` Docker Driver](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html), you will need permissions for CloudWatch. You also need to give it S3 permissions if you are pulling an external Fluent Bit or Fluentd configuration file from S3. See the the [FireLens documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html) for more.
31+
32+
Here is an example inline policy with S3 access for FireLens:
33+
34+
```
35+
{
36+
"Version": "2012-10-17",
37+
"Statement": [
38+
{
39+
"Effect": "Allow",
40+
"Action": [
41+
"s3:GetObject"
42+
],
43+
"Resource": [
44+
"arn:aws:s3:::examplebucket/folder_name/config_file_name"
45+
]
46+
},
47+
{
48+
"Effect": "Allow",
49+
"Action": [
50+
"s3:GetBucketLocation"
51+
],
52+
"Resource": [
53+
"arn:aws:s3:::examplebucket"
54+
]
55+
}
56+
]
57+
}
58+
```
59+
60+
### Using the Examples
61+
62+
You must update each Task Definition to reflect your own needs. Replace the IAM roles with your own roles. Update the log configuration with the values that you desire. And replace the app image with your own application image.
63+
64+
Additionally, several of these examples use a custom Fluent Bit/Fluentd configuration file in S3. You must upload it to your own bucket, and change the S3 ARN in the example Task Definition.
65+
66+
If you are using ECS on Fargate, then pulling a config file from S3 is not currently supported. Instead, you must create a custom Docker image with the config file.
67+
68+
Dockerfile to add a custom configs:
69+
```
70+
FROM amazon/aws-for-fluent-bit:latest
71+
ADD extra.conf /extra.conf
72+
```
73+
74+
Then update the `firelensConfiguration` `options` in the Task Definition to the following:
75+
```
76+
"options": {
77+
"config-file-type": "file",
78+
"config-file-value": "/extra.conf"
79+
}
80+
```
81+
582
## License Summary
683

784
This sample code is made available under the MIT-0 license. See the LICENSE file.

‎examples/fluent-bit/add-keys/README.md

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
### FireLens Example: Adding Keys to the Log Events
2+
3+
With the custom configuration file in this example, you can add a key to each log message. This is similar to how FireLens adds ECS Metadata to your logs.
4+
5+
In this example, we add a field called `app-version`- this will allow us to correlate log messages with the version of our app that generated them. The App Version is set via an environment variable, which can be referenced in the Fluent Bit configuration file.
6+
7+
Assuming ECS Log Metadata is enabled, the final log events in Firehose will look something like the following:
8+
```
9+
{
10+
"source": "stdout",
11+
"app-version": "v1.1.14",
12+
"log": "172.17.0.1 - - [03/Oct/2019:00:06:20 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.54.0\" \"-\"",
13+
"container_id": "e54cccfac2b87417f71877907f67879068420042828067ae0867e60a63529d35",
14+
"container_name": "/ecs-demo-6-container2-a4eafbb3d4c7f1e16e00"
15+
"ecs_cluster": "mycluster",
16+
"ecs_task_arn": "arn:aws:ecs:us-east-2:01234567891011:task/mycluster/3de392df-6bfa-470b-97ed-aa6f482cd7a6",
17+
"ecs_task_definition": "demo:7",
18+
"ec2_instance_id": "i-06bc83dbc2ac2fdf8"
19+
}
20+
```
21+
22+
Keys can be added and removed via the record_modifier filter- for more information see the [Fluent Bit documentation](https://fluentbit.io/documentation/0.12/filter/record_modifier.html).

‎examples/fluent-bit/add-keys/extra.conf

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
[FILTER]
2+
Name record_modifier
3+
Match *
4+
Record app-version ${APP_VERSION}

‎examples/fluent-bit/add-keys/permissions.json

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [{
4+
"Effect": "Allow",
5+
"Action": [
6+
"firehose:PutRecordBatch",
7+
],
8+
"Resource": "*"
9+
}]
10+
}

‎examples/fluent-bit/add-keys/task-definition.json

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
{
2+
"family": "firelens-example-firehose",
3+
"taskRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_iam_role",
4+
"executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_execution_role",
5+
"containerDefinitions": [
6+
{
7+
"essential": true,
8+
"image": "amazon/aws-for-fluent-bit:latest",
9+
"name": "log_router",
10+
"firelensConfiguration": {
11+
"type": "fluentbit",
12+
"options": {
13+
"config-file-type": "s3",
14+
"config-file-value": "arn:aws:s3:::yourbucket/yourdirectory/extra.conf"
15+
}
16+
},
17+
"logConfiguration": {
18+
"logDriver": "awslogs",
19+
"options": {
20+
"awslogs-group": "firelens-container",
21+
"awslogs-region": "us-west-2",
22+
"awslogs-create-group": "true",
23+
"awslogs-stream-prefix": "firelens"
24+
}
25+
},
26+
"environment": [
27+
{
28+
"name": "APP_VERSION",
29+
"value": "v1.1.14"
30+
}
31+
],
32+
"memoryReservation": 50
33+
},
34+
{
35+
"essential": true,
36+
"image": "httpd",
37+
"name": "app",
38+
"logConfiguration": {
39+
"logDriver":"awsfirelens",
40+
"options": {
41+
"Name": "firehose",
42+
"region": "us-west-2",
43+
"delivery_stream": "my-stream"
44+
}
45+
},
46+
"memoryReservation": 100
47+
}
48+
]
49+
}

‎examples/fluent-bit/cloudwatchlogs/README.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
### FireLens Example: Logging to CloudWatch Logs with Fluent Bit
2+
3+
For documentation on Fluent Bit & CloudWatch, see: [amazon-cloudwatch-logs-for-fluent-bit](https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit)

‎examples/fluent-bit/cloudwatchlogs/permissions.json

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [{
4+
"Effect": "Allow",
5+
"Action": [
6+
"logs:CreateLogStream",
7+
"logs:CreateLogGroup",
8+
"logs:DescribeLogStreams",
9+
"logs:PutLogEvents"
10+
],
11+
"Resource": "*"
12+
}]
13+
}

‎examples/fluent-bit/cloudwatchlogs/task-definition.json

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"family": "firelens-example-cloudwatch",
3+
"taskRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_iam_role",
4+
"executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_execution_role",
5+
"containerDefinitions": [
6+
{
7+
"essential": true,
8+
"image": "906394416424.dkr.ecr.us-east-1.amazonaws.com/aws-for-fluent-bit:latest",
9+
"name": "log_router",
10+
"firelensConfiguration": {
11+
"type": "fluentbit"
12+
},
13+
"logConfiguration": {
14+
"logDriver": "awslogs",
15+
"options": {
16+
"awslogs-group": "firelens-container",
17+
"awslogs-region": "us-west-2",
18+
"awslogs-create-group": "true",
19+
"awslogs-stream-prefix": "firelens"
20+
}
21+
},
22+
"memoryReservation": 50
23+
},
24+
{
25+
"essential": true,
26+
"image": "nginx",
27+
"name": "app",
28+
"logConfiguration": {
29+
"logDriver":"awsfirelens",
30+
"options": {
31+
"Name": "cloudwatch",
32+
"region": "us-west-2",
33+
"log_group_name": "firelens-fluent-bit",
34+
"auto_create_group": "true",
35+
"log_stream_prefix": "from-fluent-bit"
36+
}
37+
},
38+
"memoryReservation": 100
39+
}
40+
]
41+
}

‎examples/fluent-bit/enable-debug-logging/README.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
### FireLens Example: Enabling Debug Logging for Fluent Bit
2+
3+
Log level can be set in the [Service](https://docs.fluentbit.io/manual/service) section of the Fluent Bit configuration file. This section is not used by FireLens; you can set it yourself using an external configuration file.
4+
5+
To enable debug logging in the AWS Fluent Bit plugins; set the environment variable `FLB_LOG_LEVEL`. It can be set to `debug`, `info`, and `error`.

‎examples/fluent-bit/enable-debug-logging/extra.conf

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[SERVICE]
2+
Log_Level debug

‎examples/fluent-bit/enable-debug-logging/permissions.json

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [{
4+
"Effect": "Allow",
5+
"Action": [
6+
"logs:CreateLogStream",
7+
"logs:CreateLogGroup",
8+
"logs:DescribeLogStreams",
9+
"logs:PutLogEvents"
10+
],
11+
"Resource": "*"
12+
}]
13+
}

‎examples/fluent-bit/enable-debug-logging/task-definition.json

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
{
2+
"family": "firelens-example-debug",
3+
"taskRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_iam_role",
4+
"executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_execution_role",
5+
"containerDefinitions": [
6+
{
7+
"essential": true,
8+
"image": "906394416424.dkr.ecr.us-east-1.amazonaws.com/aws-for-fluent-bit:latest",
9+
"name": "log_router",
10+
"environment": [
11+
{ "name": "FLB_LOG_LEVEL", "value": "debug" }
12+
],
13+
"firelensConfiguration": {
14+
"type": "fluentbit",
15+
"options": {
16+
"config-file-type": "s3",
17+
"config-file-value": "arn:aws:s3:::yourbucket/yourdirectory/extra.conf"
18+
}
19+
},
20+
"logConfiguration": {
21+
"logDriver": "awslogs",
22+
"options": {
23+
"awslogs-group": "firelens-container",
24+
"awslogs-region": "us-west-2",
25+
"awslogs-create-group": "true",
26+
"awslogs-stream-prefix": "firelens"
27+
}
28+
},
29+
"memoryReservation": 50
30+
},
31+
{
32+
"essential": true,
33+
"image": "nginx",
34+
"name": "app",
35+
"logConfiguration": {
36+
"logDriver":"awsfirelens",
37+
"options": {
38+
"Name": "cloudwatch",
39+
"region": "us-west-2",
40+
"log_group_name": "firelens-fluent-bit",
41+
"auto_create_group": "true",
42+
"log_stream_prefix": "from-fluent-bit"
43+
}
44+
},
45+
"memoryReservation": 100
46+
}
47+
]
48+
}

‎examples/fluent-bit/forward-to-aggregator/README.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
### FireLens Example: Forward to an external Fluentd or Fluent Bit Log Aggregator
2+
3+
You can use FireLens to forward logs to an external Fluentd or Fluent Bit host. For more information on log aggregation see [Building a scalable log aggregator with AWS Fargate, Fluentd, and Amazon Kinesis Data Firehose](https://aws.amazon.com/blogs/compute/building-a-scalable-log-solution-aggregator-with-aws-fargate-fluentd-and-amazon-kinesis-data-firehose/).

‎examples/fluent-bit/forward-to-aggregator/task-definition.json

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{
2+
"family": "firelens-example-forward",
3+
"executionRoleArn": "arn:aws:iam::XXXXXXXXXXXX:role/ecs_task_execution_role",
4+
"containerDefinitions": [
5+
{
6+
"essential": true,
7+
"image": "amazon/aws-for-fluent-bit:latest",
8+
"name": "log_router",
9+
"firelensConfiguration": {
10+
"type": "fluentbit"
11+
},
12+
"logConfiguration": {
13+
"logDriver": "awslogs",
14+
"options": {
15+
"awslogs-group": "firelens-container",
16+
"awslogs-region": "us-west-2",
17+
"awslogs-create-group": "true",
18+
"awslogs-stream-prefix": "firelens"
19+
}
20+
},
21+
"memoryReservation": 50
22+
},
23+
{
24+
"essential": true,
25+
"image": "httpd",
26+
"name": "app",
27+
"logConfiguration": {
28+
"logDriver":"awsfirelens",
29+
"options": {
30+
"Name": "forward",
31+
"Host": "fluentdhost",
32+
"Port": "24224"
33+
}
34+
},
35+
"memoryReservation": 100
36+
}
37+
]
38+
}

‎examples/fluent-bit/kinesis-firehose/README.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
### FireLens Example: Logging to Kinesis Data Firehose with Fluent Bit
2+
3+
For documentation on Fluent Bit & Firehose, see: [amazon-kinesis-firehose-for-fluent-bit](https://github.com/aws/amazon-kinesis-firehose-for-fluent-bit)

‎examples/fluent-bit/kinesis-firehose/permissions.json

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [{
4+
"Effect": "Allow",
5+
"Action": [
6+
"firehose:PutRecordBatch",
7+
],
8+
"Resource": "*"
9+
}]
10+
}

0 commit comments

Comments
 (0)