Frequent AccessDenied requests against manifest bucket from foreign VPC

[dsotirho-ucsc]

[
    {
        "eventTime": "2026-06-12T03:58:24Z",
        "@message": {
            "eventVersion": "1.11",
            "userIdentity": {
                "type": "AWSAccount",
                "principalId": "",
                "accountId": "anonymous"
            },
            "eventTime": "2026-06-12T03:58:24Z",
            "eventSource": "[s3.amazonaws.com](http://s3.amazonaws.com/)",
            "eventName": "GetObject",
            "awsRegion": "us-east-1",
            "sourceIPAddress": "172.20.33.14",
            "userAgent": "[Mozilla/5.0 (compatible)]",
            "errorCode": "AccessDenied",
            "errorMessage": "Access Denied",
            "requestParameters": {
                "bucketName": "edu-ucsc-gi-platform-hca-prod-storage-prod.us-east-1",
                "Host": "[s3.amazonaws.com](http://s3.amazonaws.com/)",
                "key": "manifests/3498fcc1-21e"
            },
            "responseElements": null,
            "additionalEventData": {
                "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
                "bytesTransferredIn": 0,
                "x-amz-id-2": "mgL3Jev35eGxTc6/ANhhNGX1aVUesXs39Iggg3oaOOEEeUlpZscrrlVqsNTlpV8SQeE9DU7GUQUdKE32EuR3nCdh+Hx3iiuP",
                "bytesTransferredOut": 263
            },
            "requestID": "8QE1NZKS5HHJZ3H9",
            "eventID": "6ac7ef24-00d7-30c3-98a1-7d38af075515",
            "readOnly": true,
            "resources": [
                {
                    "accountId": "542754589326",
                    "type": "AWS::S3::Bucket",
                    "ARN": "arn:aws:s3:::edu-ucsc-gi-platform-hca-prod-storage-prod.us-east-1"
                },
                {
                    "type": "AWS::S3::Object",
                    "ARN": "arn:aws:s3:::edu-ucsc-gi-platform-hca-prod-storage-prod.us-east-1/manifests/3498fcc1-21e"
                }
            ],
            "eventType": "AwsApiCall",
            "managementEvent": false,
            "recipientAccountId": "542754589326",
            "sharedEventID": "bc6d75fe-260b-4d1c-bdcd-70884ce9ddb8",
            "vpcEndpointId": "vpce-6690410f",
            "vpcEndpointAccountId": "328158692677",
            "eventCategory": "Data",
            "tlsDetails": {
                "tlsVersion": "TLSv1.2",
                "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
                "clientProvidedHostHeader": "[s3.amazonaws.com](http://s3.amazonaws.com/)"
            }
        }
    }
]

[dsotirho-ucsc]

Waiting to discuss once Hannes returns.

[hannes-ucsc]

If we add an exception to the log metric filer for this request, we may as well ignore all denied S3 read requests.

What we really need to do is automate the investigation of any access denied requests.

Last time I looked, there was no way to include the offending log entry in the alarm notification. Amazon simply did not not allow for that.

A good compromise would be writing a script selects each main deployment after switching to the respective branch, performs a logs insights query programmatically in that deployment and produces a report with all access denied errors in the last 24 hours. Claude should be able to cobble that together pretty quickly. The report should be an HTML page and the script should open the page in the browser (Claude knows how to start an HTTP server, serve the report, and open that page in the locally installed browser).

If we have any access denied alarms notifications we will review the report during stand-up. Unless there are any unusual events we'll just quickly dismiss all unauthorized alarms.

[nadove-ucsc]

Nine more occurrences this weekend of requests to truncated manifest keys. The requests were not made via a VPC however.

[nadove-ucsc]

@hannes-ucsc: "I think we cannot afford to continually triage these events, at least not at the rate they are currently occurring at. We already have an exception in place for AccessDenied caused by internal AWS services, so we should just add an extra exception for these events as well. The question is just how broad the exception should be."

[hannes-ucsc]

Assignee to mine the trail logs of all deployments to enumerate the buckets to which access was ever denied.

[nadove-ucsc]

HCA prod

CloudWatch Log Analytics Results
Region: us-east-1
Query:

SOURCE "azul-trail-prod" START=-93312000s END=0s |
fields eventTime, errorCode, eventSource, eventName,
        coalesce(userIdentity.invokedBy, sourceIPAddress) as source, 
        # requestParameters.key as requestKey,
        requestParameters.bucketName as requestBucket
| filter (userIdentity.invokedBy not in ['config.amazonaws.com', 'resource-explorer-2.amazonaws.com']) and (errorCode == 'AccessDenied') and (eventSource == 's3.amazonaws.com')
| stats count(*) as n by requestBucket
| sort n desc

requestBucket	n
edu-ucsc-gi-platform-hca-prod-schema.us-east-1	5405
edu-ucsc-gi-platform-hca-prod-storage-prod.us-east-1	151
edu-ucsc-gi-platform-hca-prod-logs.us-east-1	40
edu-ucsc-gi-platform-hca-prod-shared.us-east-1	10
edu-ucsc-gi-platform-hca-dev-logs.us-east-1	5
s3-inventory-reporter-shared-us-east-1-prod	2
scge	1
scg	1

AnVIL prod

CloudWatch Log Analytics Results
Region: us-east-1
Query:

SOURCE "azul-trail-anvilprod" START=-93312000s END=0s |
fields eventTime, errorCode, eventSource, eventName,
        coalesce(userIdentity.invokedBy, sourceIPAddress) as source, 
        # requestParameters.key as requestKey,
        requestParameters.bucketName as requestBucket
| filter (userIdentity.invokedBy not in ['config.amazonaws.com', 'resource-explorer-2.amazonaws.com']) and (errorCode == 'AccessDenied') and (eventSource == 's3.amazonaws.com')
| stats count(*) as n by requestBucket
| sort n desc

requestBucket	n
anvilproject	394
edu-ucsc-gi-platform-anvil-prod-storage-anvilprod.us-east-1	177
edu-ucsc-gi-platform-anvil-prod-logs.us-east-1	26
edu-ucsc-gi-platform-anvil-prod-shared.us-east-1	15
edu-ucsc-gi-platform-anvil-prod-mamirror-hammerbox.us-east-1	14
manifests	2
edu-ucsc-gi-platform-anvil-prod-awsconfig.us-east-1	2
edu-ucsc-gi-platform-anvil-prod-browser.us-east-1	2
edu-ucsc-gi-platform-anvil-prod-mirror-anvilprod.us-east-1	1
edu-ucsc-gi-platform-anvil-prod-mamirror-anvilprod.us-east-1	1
s3-inventory-reporter-shared-us-east-1-prod	1

HCA dev

CloudWatch Log Analytics Results
Region: us-east-1
Query:

SOURCE "azul-trail-dev" START=-93312000s END=0s |
fields eventTime, errorCode, eventSource, eventName,
        coalesce(userIdentity.invokedBy, sourceIPAddress) as source, 
        # requestParameters.key as requestKey,
        requestParameters.bucketName as requestBucket
| filter (userIdentity.invokedBy not in ['config.amazonaws.com', 'resource-explorer-2.amazonaws.com']) and (errorCode == 'AccessDenied') and (eventSource == 's3.amazonaws.com')
| stats count(*) as n by requestBucket
| sort n desc

requestBucket	n
edu-ucsc-gi-platform-hca-dev-logs.us-east-1	67
edu-ucsc-gi-platform-hca-dev-mamirror-nadove.us-east-1	26
edu-ucsc-gi-platform-hca-dev-mirror-nadove.us-east-1	17
edu-ucsc-gi-platform-hca-dev-shared.us-east-1	12
edu-ucsc-gi-platform-hca-dev-storage-dev.us-east-1	9
edu-ucsc-gi-platform-hca-prod-logs.us-east-1	8
edu-ucsc-gi-platform-hca-dev-mirror-dev.us-east-1	8
edu-ucsc-gi-platform-hca-dev-storage-nadove.us-east-1	7
edu-ucsc-gi-platform-hca-dev-storage-daniel.us-east-1	7
edu-ucsc-gi-platform-hca-dev-mirror-daniel.us-east-1	5
edu-ucsc-gi-platform-hca-dev-mirror-abrahamsc.us-east-1	5
edu-ucsc-gi-platform-hca-dev-mirror-hannes.us-east-1	5
edu-ucsc-gi-platform-hca-dev-browser.us-east-1	4
edu-ucsc-gi-platform-anvil-dev-mirror-anvildev.us-east-1	4
edu-ucsc-gi-platform-hca-dev-mamirror-dev.us-east-1	4
edu-ucsc-gi-platform-hca-dev-mirror-nadove2.us-east-1	4
edu-ucsc-gi-platform-hca-dev-storage-abrahamsc.us-east-1	2
edu-ucsc-gi-platform-hca-dev-awsconfig.us-east-1	2
edu-ucsc-gi-platform-hca-dev-storage-sandbox.us-east-1	2
edu-ucsc-gi-platform-hca-dev-storage-hannes.us-east-1	1

AnVIL dev

CloudWatch Log Analytics Results
Region: us-east-1
Query:

SOURCE "azul-trail-anvildev" START=-93312000s END=0s |
fields eventTime, errorCode, eventSource, eventName,
        coalesce(userIdentity.invokedBy, sourceIPAddress) as source, 
        # requestParameters.key as requestKey,
        requestParameters.bucketName as requestBucket
| filter (userIdentity.invokedBy not in ['config.amazonaws.com', 'resource-explorer-2.amazonaws.com']) and (errorCode == 'AccessDenied') and (eventSource == 's3.amazonaws.com')
| stats count(*) as n by requestBucket
| sort n desc

requestBucket	n
edu-ucsc-gi-platform-anvil-dev-logs.us-east-1	39
edu-ucsc-gi-platform-anvil-dev-storage-anvildev.us-east-1	15
edu-ucsc-gi-platform-anvil-dev-mirror-anvildev.us-east-1	6
edu-ucsc-gi-platform-anvil-dev-storage-achave11.us-east-1	2
edu-ucsc-gi-platform-anvil-dev-storage-daniel2.us-east-1	1

[hannes-ucsc]

Assignee to implement an exception in the corresponding CloudWatch Metric alarm for events with an eventSource of "s3.amazonaws.com", a userIdentity.accountId of "anonymous" and requestParameters.bucketName matching aws.qualified_bucket_name('*')