CloudFormation returned status code: Forbidden

[mdalvi]

When does cloud formation returns Forbidden status? This happened with me during create. I had a code where a the Lambda was supposed to upload file to S3 bucket and while that went successful the template got stuck!

Here is what the log looks like

[DEBUG]	2020-07-21T15:02:51.816Z	8f3cc89a-9521-4272-82cc-587d7e8a0d4b	Executing task PutObjectTask(transfer_id=0, {'bucket': 'my_bucket_name', 'key': 'faq_document.xlsx', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7fca11a7e220>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x7fca10db13a0>, 'bucket': 'my_bucket_name', 'key': 'faq_document.xlsx', 'extra_args': {}}
[DEBUG]	2020-07-21T15:03:23.593Z	8f3cc89a-9521-4272-82cc-587d7e8a0d4b	Releasing acquire 0/None
[INFO] 2020-07-21T15:03:23.594Z 8f3cc89a-9521-4272-82cc-587d7e8a0d4b File `faq_document.xlsx` uploaded successfully
[DEBUG]	2020-07-21T15:03:23.597Z	8f3cc89a-9521-4272-82cc-587d7e8a0d4b	enabling send_response
[DEBUG]	2020-07-21T15:03:23.616Z	8f3cc89a-9521-4272-82cc-587d7e8a0d4b	_send_response: True
[DEBUG] 2020-07-21T15:03:23.617Z 8f3cc89a-9521-4272-82cc-587d7e8a0d4b CFN response URL: https://cloudformation-custom-resource-response-my_region.s3.amazonaws.com ..........
[DEBUG]	2020-07-21T15:03:23.617Z	8f3cc89a-9521-4272-82cc-587d7e8a0d4b	
{
    "Status": "SUCCESS",
    "PhysicalResourceId": "S3PostAutoID000",
    "StackId": "arn:aws:cloudformation:my_region:my_account:stack/stack-cb3/f0da78d0-cb60-11ea-bbd9-0e3259ee6b3b",
    "RequestId": "0bed6b99-a998-49c0-933d-81386cad65be",
    "LogicalResourceId": "CustomS3Postrequisites",
    "Reason": "",
    "Data": {}
}
[INFO] 2020-07-21T15:03:23.843Z 8f3cc89a-9521-4272-82cc-587d7e8a0d4b CloudFormation returned status code: Forbidden
END RequestId: 8f3cc89a-9521-4272-82cc-587d7e8a0d4b

Its strange though never happened before, although one question, Am I suppose to randomize my PhysicalResourceId?, because right now its plain text

@helper.create
def create(event, _):
    try:
        bucket_name = event['ResourceProperties']['BucketName']
        s3 = boto3.resource('s3')
        s3.meta.client.upload_file('faq_document.xlsx', bucket_name, 'faq_document.xlsx')
        logger.info("File `faq_document.xlsx` uploaded successfully")
    except Exception as _:
        logger.critical(traceback.format_exc())

    return "S3PostAutoID000"

[mdalvi]

It might be because my Lambda was within VPC, but the SecuirtyGroup Ingress and Egress was 0.0.0.0/0 so would that matter?

[leohoare]

If you're getting forbidden, I don't think It'd be the Security Group:

    while True:
        try:
            connection = HTTPSConnection(host)
            connection.request(method="PUT", url=url, body=json_response_body, headers=headers)
            response = connection.getresponse()
            logger.info("CloudFormation returned status code: {}".format(response.reason))
            break
        except Exception as e:
            logger.error("Unexpected failure sending response to CloudFormation {}".format(e), exc_info=True)
            time.sleep(5)

Seems like you have the ability to hit url but getting unauthorized when hitting it.

Did you get this solved?

[alex-harvey-z3q]

@mdalvi Did you ever figure out what causes this?

[bisoldi]

My issue was that I am deploying a Custom Resource into a VPC with an S3 VPC Endpoint. And while creating the Endpoint, I followed instructions here (https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html#bucket-policies-s3) and added a Deny for any S3 buckets that are not owned by my account Id.

This then caused the custom resource to be blocked at the Endpoint.

Unfortunately, I don't see any sane way to block s3 access outside of either my account OR the AWS account that owns the particular bucket that cloudformation uses. I haven't asked AWS if they'd hand over the account number, but I doubt they would. They do recommend unblocking the specific buckets, but they don't provide a way to do that at the endpoint policy (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-vpce-bucketnames.html), so the below is what I'm trying:

PolicyDocument:
  Version: '2012-10-17'
  Statement:
    - Effect: Allow
      Principal: '*'
      Action: '*'
      Resource: '*'
      Condition:
        StringEquals:
          s3:ResourceAccount: !Ref 'AWS::AccountId'
    - Effect: Allow
      Principal: '*'
      Action: '*'
      Resource:
        - !Join
          - ''
          - - 'arn:aws:s3:::'
            - 'cloudformation-custom-resource-response-'
            - !Select [ 0, !Split [ "-" , !Ref 'AWS::Region' ] ]
            - !Select [ 1, !Split [ "-" , !Ref 'AWS::Region' ] ]
            - !Select [ 2, !Split [ "-" , !Ref 'AWS::Region' ] ]
        - !Join
          - ''
          - - 'arn:aws:s3:::'
            - 'cloudformation-custom-resource-response-'
            - !Select [ 0, !Split [ "-" , !Ref 'AWS::Region' ] ]
            - !Select [ 1, !Split [ "-" , !Ref 'AWS::Region' ] ]
            - !Select [ 2, !Split [ "-" , !Ref 'AWS::Region' ] ]
            - '/*'
        - !Sub 'arn:aws:s3:::cloudformation-waitcondition-${AWS::Region}'
        - !Sub 'arn:aws:s3:::cloudformation-waitcondition-${AWS::Region}/*'

[fabiatz]

I had a similar issue with my Lambda backed custom resource (was in a VPC) and AWS support helped me to realise it's the changed S3 VPC endpoint policy that stopped my custom resource to report back on to the pre-signed S3 URL (see the ResponseURL field in the incoming event object). As soon as I relaxed that policy it started working again. Now I need to define that policy properly...

[bisoldi]

I had a similar issue with my Lambda backed custom resource (was in a VPC) and AWS support helped me to realise it's the changed S3 VPC endpoint policy that stopped my custom resource to report back on to the pre-signed S3 URL (see the ResponseURL field in the incoming event object). As soon as I relaxed that policy it started working again. Now I need to define that policy properly...

If you figure out how to block access to external buckets but allow access to the Cloudformation buckets, please do share.

[leohoare]

If you figure out how to block access to external buckets but allow access to the Cloudformation buckets, please do share.

This could be achieved with a specific IAM policy in the lambda. IAM with an explicit allow will block anything else.
https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html

Such as:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::bucket-name"]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": ["arn:aws:s3:::bucket-name/*"]
        }
    ]
}