Native S3 access log delivery to CloudWatch

[hannes-ucsc]

(Posted by Claude Code)

Every S3 server access log in the system reaches CloudWatch by way of a Lambda function we wrote ourselves. Buckets are configured with aws_s3_bucket_logging to deposit access logs into the shared logs bucket, an aws_s3_bucket_notification on that bucket triggers the forward_s3_logs function in the indexer app, and S3AccessLogForwardingService reads each object, parses it and writes the records to the function's own log group.

The buckets currently covered are trail, aws_config and shared in the shared component, storage and the provisioned mirror and mamirror buckets in the main component, and the Data Browser site buckets in the browser component.

Since this design was adopted, AWS has added delivery of S3 server access logs to CloudWatch Logs as a first-class feature, by way of vended log delivery. That is the same transformation the forwarder performs by hand, so we are maintaining a reimplementation of a managed service capability.

The cost of the reimplementation is concentrated in three places.

The parsing is ours to keep correct. S3AccessLogForwardingService carries a Latin-1 decode to avoid decoding failures on undocumented encoding, a double unquote of the key field, a reconstruction of the time field from two CSV columns because AWS does not quote it, and a hard-coded field list that has to track the documented log record format as AWS extends it. None of this is wrong, but all of it is ours to maintain, and it fails in the direction of dropped or malformed audit records.

The function is not narrowly privileged. manage_iam_role is false for the indexer app, so forward_s3_logs runs under the same role as every other indexer function, which carries OpenSearch, SQS, DynamoDB, mirror bucket read/write and secretsmanager:GetSecretValue on secret:*. A managed delivery path needs no execution role of ours at all.

Extending the design across an account boundary requires opening one. #8039 needs access logging for mirror buckets that live in accounts we do not control, and because S3 server access logs must be delivered to a bucket in the same account as their source, those logs land in a foreign bucket. #8219 therefore adds a resource-based policy on the forward_s3_logs alias that permits a bucket in another account to invoke it. The security design review on that PR records the consequences: the permission pins a reclaimable bucket name rather than an account, the resulting audit trail for our mirror buckets is authored and retained in an account we do not control, and the foreign side of the pipeline cannot be managed by our Terraform, so its absence would be silent.

Note that only the S3 half of the forwarder is at stake. forward_alb_logs handles a log type the S3 feature does not cover, so the function and its plumbing would survive in some form regardless.

[hannes-ucsc]

(Posted by Claude Code)

Proposed solution, per the convention that the description states the problem.

Evaluate replacing PutBucketLogging plus forward_s3_logs with CloudWatch Logs vended log delivery for the S3_SERVER_ACCESS_LOGS log type. Per-bucket, that is a delivery source, a delivery destination and a delivery, all of which have Terraform resources, in place of a bucket logging configuration and a slice of the S3 notification on the logs bucket. Records arrive as structured JSON, so S3AccessLogForwardingService and its parsing workarounds go away, as does the forwarder's dependence on the shared indexer role.

Two things should be settled before any of this is designed in earnest.

Whether cross-account delivery into our account is actually supported. This is the question that decides whether #8039 benefits at all, and the documentation does not answer it consistently. The comparison table in Logging requests with server access logging records cross-account aggregation as supported, annotated "Amazon S3 and Firehose destinations support cross-account delivery". But Logs sent to Amazon S3 states that "the service that delivers logs records the owner of the bucket to ensure that the logs are delivered only to a bucket belonging to this account", which cuts the other way for that destination. And the cross-account section of sal-cw-enabling.html offers only two mechanisms, both weaker than direct delivery: CloudWatch cross-account observability, which queries logs in place without copying them, and CloudWatch Logs centralization, which consolidates into a central log group using subscription filters.

The distinction matters for the compliance posture. Under the observability form the records still live in the partner's account and only the querying is ours, so the trust concern raised in the #8219 review survives. Under the centralization form we hold a copy. Either form removes the need for a foreign bucket to hold an invoke permission on a function in our account, which is the narrower win and is available regardless.

What it costs. The S3 bucket destination is free apart from storage, whereas CloudWatch Logs delivery bills at vended logs ingestion rates. Access logs for the storage and browser buckets are not low-volume, so this needs an estimate rather than an assumption, and the answer may differ per bucket.

Worth noting that the two paths are not exclusive — a bucket can deliver to both — so this can be trialled on one low-volume bucket in dev without disturbing the existing pipeline. trail or aws_config would be the natural candidates.

Neither prerequisite is an obstacle on our side: the destination log group must be in the same Region as the source bucket, and both mirror buckets and all four main deployments are in us-east-1.

If the cross-account question resolves against us, this is still worth doing for the in-account buckets on the maintenance argument alone, but it stops being relevant to #8039 and the mitigations recommended in the #8219 review remain the right ones for that path.