rollback: rollback to 2.32.0

Signed-off-by: Matthew Fala <[email protected]>

Author: matthewfala

AWS_FOR_FLUENT_BIT_PREVIOUS_STABLE_VERSION

@@ -1 +1 @@
-2.32.0.20240122
+2.31.12.20231011

AWS_FOR_FLUENT_BIT_STABLE_VERSION

@@ -1 +1 @@
-2.32.0.20240304
+2.32.0.20240122

AWS_FOR_FLUENT_BIT_VERSION

@@ -1 +1 @@
-2.32.1
+2.32.0.20240304

CHANGELOG.md

@@ -1,16 +1,5 @@
 # Changelog
 
-### 2.32.1
-This release includes:
-* Fluent Bit [1.9.10](https://github.com/fluent/fluent-bit/tree/v1.9.10)
-* Amazon CloudWatch Logs for Fluent Bit 1.9.4
-* Amazon Kinesis Streams for Fluent Bit 1.10.2
-* Amazon Kinesis Firehose for Fluent Bit 1.7.2
-
-Compared to `2.32.0` this release adds:
-* Enhancement - Add `aws_fluent_bit_init_ignore_firelens_config` option to ECS init tag. As shown in the [example](https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/init-ignore-firelens) and [documentation](https://github.com/aws/aws-for-fluent-bit/tree/mainline/use_cases/init-process-for-fluent-bit#how-to-ignore-the-generated-firelens-configuration), this option tells init to ignore the built-in generated FireLens configuration which allows users to fully override it with their own configuration.
-* Enhancement - Upgrade to latest stable Go 1.20.1. Each future AWS for Fluent Bit release will be built with the latest stable Go release. Go is used to build the init process for ECS use cases and the [Go plugins](https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins).
-
 ### 2.32.0.20240304 Linux re-build
 
 *This release has the same Fluent Bit contents as 2.32.0, and is simply a linux-only re-build for recent patches in dependencies installed in the image. There are no windows images for this release.* 
@@ -39,8 +28,8 @@ This release includes:
 * Amazon Kinesis Firehose for Fluent Bit 1.7.2
 
 Compared to `2.31.12` this release adds:
-* Enhancement - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on `prometheus_remote_write`. Refer to [amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/amazon-managed-service-for-prometheus/README.md) for information on how to export metrics to AMP on ECS [aws-for-fluent-bit:256](https://github.com/aws/aws-for-fluent-bit/issues/256)
-* Enhancement - Support multiline parsers with the init tag. Multiline parsers can be specified same as a standard parser file [aws-for-fluent-bit:537](https://github.com/aws/aws-for-fluent-bit/issues/537)
+* Feature - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on `prometheus_remote_write`. Refer to [amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/amazon-managed-service-for-prometheus/README.md) for information on how to export metrics to AMP on ECS [aws-for-fluent-bit:256](https://github.com/aws/aws-for-fluent-bit/issues/256)
+* Feature - Support multiline parsers with the init tag. Multiline parsers can be specified same as a standard parser file [aws-for-fluent-bit:537](https://github.com/aws/aws-for-fluent-bit/issues/537)
 * Enhancement - Customers can route logs to CloudWatch Logs at higher throughputs by increasing number of output workers as `cloudwatch_logs` output plugin removed sequence tokens from API requests [aws-for-fluent-bit:526](https://github.com/aws/aws-for-fluent-bit/issues/526)
 * Fix - Fix multiline input behavior when multiple streams are parsed (stderr, stdout) together. Multiline logs are no longer terminated when streams are switched between [fluent-bit:7469](https://github.com/fluent/fluent-bit/pull/7469)
 * Fix - Fix networking edgecase causing data loss and OOM related issues via net timeout event injection resolution [fluent-bit:7728](https://github.com/fluent/fluent-bit/pull/7728/files)

README.md

@@ -54,8 +54,6 @@ The process for pushing out new builds with CVE patches in the base image or ins
 
 For Windows, every month after the [B release date/"patch tuesday"](https://learn.microsoft.com/en-us/windows/deployment/update/release-cycle#monthly-security-update-release), we re-build and update all Windows images currently found in the [windows.versions](windows.versions) file in this repo with the newest base images from Microsoft. The Fluent Bit and go plugin binaries are copied into the newly released base windows image. Thus, the windows image tags are not immutable images; only the Fluent Bit and Go plugin binaries are immutable over time.
 
-At any point in time, [windows.versions](windows.versions) file will contain at least 5 versions, including latest and latest stable. AWS for Fluent Bit Windows are guaranteed to be patched for 4 months after their release date. Therefore, the [windows.versions](windows.versions) file always contains all versions released in the last 4 months, and may contain more if the latest stable release is older than 4 months. 
-
 For Linux, each image tag is immutable. When there is a report of high or critical CVEs reported in the base amazon linux image or installed linux packages, we will work to push out a new image [per our patching policy](#compliance-and-patching). However, we will not increment the semantic version number to simply re-build to pull in new linux dependencies. Instead, we will add a 4th version number signifying the date the image was built.
 
 For example, a series of releases in time might look like:
@@ -161,7 +159,7 @@ Prior to being designated as the latest stable, a version must pass the followin
 
 * It has been out for at least 2 weeks or is a CVE patch with no Fluent Bit changes. Stable designation is based on the Fluent Bit code in the image. A version released for CVE patches can be made stable if the underlying if the underlying Fluent Bit code is already designated as stable.
 * No bugs have been reported in Fluent Bit which we expect will have high impact for AWS customers. This means bugs in the components that are most frequently used by AWS customers, such as the AWS outputs or the tail input.
-* The version has passed our long running stability tests for at least 2 weeks or is a CVE patch with no Fluent Bit changes that has passed our long running stability tests for at least 1 day. The version would have already passed our simple integration and load tests when it was first released as the latest image. 
+* The version has passed our long running stability tests for at least 2 weeks. The version would have already passed our simple integration and load tests when it was first released as the latest image. 
 
 #### CVE scans and latest stable
 

init/fluent_bit_init_process.go

@@ -3,6 +3,7 @@ package main
 import (
 	"encoding/json"
 	"io"
+	"io/ioutil"
 	"net/http"
 	"os"
 	"path/filepath"
@@ -18,20 +19,12 @@ import (
 	"github.com/sirupsen/logrus"
 )
 
-// env vars for user configuration
-// (?i) makes the match case insensitive
-const (
-	initS3ConfigFilePattern    = "(?i)aws_fluent_bit_init_s3"
-	initLocalConfigFilePattern = "(?i)aws_fluent_bit_init_file"
-	initIgnoreFireLensConfig   = "(?i)aws_fluent_bit_init_ignore_firelens"
-)
-
 // static paths
 const (
-	s3FileDirectoryPath             = "/init/fluent-bit-init-s3-files/"
-	initConfigFilePath              = "/init/fluent-bit-init.conf"
-	firelensGeneratedConfigFilePath = "/fluent-bit/etc/fluent-bit.conf"
-	invokeFilePath                  = "/init/invoke_fluent_bit.sh"
+	s3FileDirectoryPath    = "/init/fluent-bit-init-s3-files/"
+	mainConfigFile         = "/init/fluent-bit-init.conf"
+	originalMainConfigFile = "/fluent-bit/etc/fluent-bit.conf"
+	invokeFile             = "/init/invoke_fluent_bit.sh"
 )
 
 var (
@@ -84,7 +77,7 @@ func getECSTaskMetadata(httpClient HTTPClient) ECSTaskMetadata {
 		logrus.Fatalf("[FluentBit Init Process] Failed to get ECS Metadata via HTTP Get: %s\n", err)
 	}
 
-	response, err := io.ReadAll(res.Body)
+	response, err := ioutil.ReadAll(res.Body)
 	if err != nil {
 		logrus.Fatalf("[FluentBit Init Process] Failed to read ECS Metadata from HTTP response: %s\n", err)
 	}
@@ -107,7 +100,7 @@ func getECSTaskMetadata(httpClient HTTPClient) ECSTaskMetadata {
 	metadata.ECS_TASK_DEFINITION = metadata.ECS_FAMILY + ":" + metadata.ECS_REVISION
 
 	// per ECS task metadata docs, Cluster can be an ARN or the name
-	if strings.Contains(metadata.ECS_CLUSTER, "/") {
+	if (strings.Contains(metadata.ECS_CLUSTER, "/")) {
 		clusterARN, err := arn.Parse(metadata.ECS_CLUSTER)
 		if err != nil {
 			logrus.Fatalf("[FluentBit Init Process] Failed to parse ECS Cluster ARN: %s %s\n", metadata.ECS_CLUSTER, err)
@@ -164,9 +157,6 @@ func getAllConfigFiles() {
 	// get all env vars in the container
 	envs := os.Environ()
 
-	s3Regex := regexp.MustCompile(initS3ConfigFilePattern)
-	fileRegex := regexp.MustCompile(initLocalConfigFilePattern)
-
 	// find all env vars match specified prefix
 	for _, env := range envs {
 		var envKey string
@@ -179,59 +169,27 @@ func getAllConfigFiles() {
 		envKey = string(env_kv[0])
 		envValue = string(env_kv[1])
 
-		matchedS3 := s3Regex.MatchString(envKey)
-		matchedFile := fileRegex.MatchString(envKey)
+		s3_regex, _ := regexp.Compile("aws_fluent_bit_init_[sS]3")
+		file_regex, _ := regexp.Compile("aws_fluent_bit_init_[fF]ile")
+
+		matched_s3 := s3_regex.MatchString(envKey)
+		matched_file := file_regex.MatchString(envKey)
 
 		// if this env var's value is an arn, download the config file first, then process it
-		if matchedS3 {
+		if matched_s3 {
 			s3FilePath := getS3ConfigFile(envValue)
 			s3FileName := strings.SplitN(s3FilePath, "/", -1)
 			processConfigFile(s3FileDirectoryPath + s3FileName[len(s3FileName)-1])
 		}
-		// if this env var's value is a local config fil, process is directly
-		if matchedFile {
+		// if this env var's value is a path of our built-in config file, process is derectly
+		if matched_file {
 			processConfigFile(envValue)
 		}
 	}
 }
 
-func processFireLensConfigFile() {
-	includeFireLensConfig := true
-	envs := os.Environ()
-
-	ignoreRegex := regexp.MustCompile(initIgnoreFireLensConfig)
-
-	// docs say to use aws_fluent_bit_init_ignore_firelens
-	// this supports case insensitive prefix matching, in case someone
-	// tries to capitalize FireLens, or uses aws_fluent_bit_init_ignore_firelens_config
-	for _, env := range envs {
-		var envKey string
-		var envValue string
-		env_kv := strings.SplitN(env, "=", 2)
-		if len(env_kv) != 2 {
-			logrus.Fatalf("[FluentBit Init Process] Unrecognizable environment variables: %s\n", env)
-		}
-
-		envKey = string(env_kv[0])
-		envValue = string(env_kv[1])
-
-		matchedIgnore := ignoreRegex.MatchString(envKey)
-
-		if matchedIgnore {
-			if strings.EqualFold(envValue, "true") || strings.EqualFold(envValue, "on") {
-				includeFireLensConfig = false
-			}
-		}
-	}
-
-	if includeFireLensConfig {
-		// add @INCLUDE in main config file to include original main config file
-		writeInclude(firelensGeneratedConfigFilePath, initConfigFilePath)
-	}
-}
-
 func processConfigFile(path string) {
-	contentBytes, err := os.ReadFile(path)
+	contentBytes, err := ioutil.ReadFile(path)
 	if err != nil {
 		logrus.Errorln(err)
 		logrus.Fatalf("[FluentBit Init Process] Cannot open file: %s\n", path)
@@ -244,7 +202,7 @@ func processConfigFile(path string) {
 		updateCommand(path)
 	} else {
 		// this is not a parser config file. @INCLUDE
-		writeInclude(path, initConfigFilePath)
+		writeInclude(path, mainConfigFile)
 	}
 }
 
@@ -349,15 +307,15 @@ func downloadS3ConfigFile(s3Downloader S3Downloader, s3FilePath, bucketName, s3F
 }
 
 // use @INCLUDE to add config files to the main config file
-func writeInclude(configFilePath string, initConfigFilePath string) {
-	initConfigFile := openFile(initConfigFilePath)
-	defer initConfigFile.Close()
+func writeInclude(configFilePath, mainConfigFilePath string) {
+	mainConfigFile := openFile(mainConfigFilePath)
+	defer mainConfigFile.Close()
 
 	writeContent := "@INCLUDE " + configFilePath + "\n"
-	_, err := initConfigFile.WriteString(writeContent)
+	_, err := mainConfigFile.WriteString(writeContent)
 	if err != nil {
 		logrus.Errorln(err)
-		logrus.Fatalf("[FluentBit Init Process] Cannot write %s in main config file: %s\n", writeContent[:len(writeContent)-2], initConfigFilePath)
+		logrus.Fatalf("[FluentBit Init Process] Cannot write %s in main config file: %s\n", writeContent[:len(writeContent)-2], mainConfigFilePath)
 	}
 }
 
@@ -414,24 +372,23 @@ func main() {
 	// create the invoke_fluent_bit.sh
 	// which will declare ECS Task Metadata as environment variables
 	// and finally invoke Fluent Bit
-	createFile(invokeFilePath, true)
+	createFile(invokeFile, true)
 
 	// get ECS Task Metadata and set the region for S3 client
 	httpClient := &http.Client{}
 	metadata := getECSTaskMetadata(httpClient)
 
 	// set ECS Task Metada as env vars in the invoke_fluent_bit.sh
-	setECSTaskMetadata(metadata, invokeFilePath)
+	setECSTaskMetadata(metadata, invokeFile)
 
 	// create main config file which will be used invoke Fluent Bit
-	createFile(initConfigFilePath, true)
+	createFile(mainConfigFile, true)
 
-	// create Fluent Bit command to use "-c" to specify new main config file
-	createCommand(&baseCommand, initConfigFilePath)
+	// add @INCLUDE in main config file to include original main config file
+	writeInclude(originalMainConfigFile, mainConfigFile)
 
-	// include the FireLens generated config
-	// unless the user has set aws_fluent_bit_init_ignore_firelens
-	processFireLensConfigFile()
+	// create Fluent Bit command to use "-c" to specify new main config file
+	createCommand(&baseCommand, mainConfigFile)
 
 	// get our built in config files or files from s3
 	// process built-in config files directly
@@ -442,5 +399,5 @@ func main() {
 	// this function will be called at the end
 	// any error appear above will cause exit this process,
 	// will not write Fluent Bit command in the finvoke_fluent_bit.sh so Fluent Bit will not be invoked
-	modifyInvokeFile(invokeFilePath)
+	modifyInvokeFile(invokeFile)
 }

linux.version

@@ -1,6 +1,6 @@
 {
   "linux": {
-    "version": "2.32.1",
+    "version": "2.32.0.20240304",
     "latest": "true",
     "build": "1",
     "fluent-bit": "1.9.10",