Skip to content

Commit 5f60bf4

Browse files
author
toshke
committed
initial commit
0 parents  commit 5f60bf4

14 files changed

+860
-0
lines changed

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
The MIT License (MIT)
2+
3+
Copyright (c) 2015 base2Services
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# CloudFormation custom resource catalogue
2+
3+
Collection of Cloud Formation custom resources written in python 3.6, result
4+
of months of continuous efforts to automate infrastructure management trough
5+
AWS CloudFormation. You may find some of these CloudFormation resources obsolete,
6+
as AWS team fills in the gaps. There is also some more complex ones, or developed
7+
to suite specific needs, such as copying s3 objects between buckets
8+
9+
## Custom resources
10+
11+
### Creating CloudFormation stack in specific region
12+
13+
It is easy to create sub-stacks in CloudFormation as long as they are in same region.
14+
In some cases, there is need to create stack in region different than region where
15+
parent stack is being create, or for example, to create same stack in multiple regions.
16+
Such (sub)stack lifecycle can be controlled via custom resource having it's code in
17+
`src/regional-cfn-stack` folder
18+
19+
handler: `src/regional-cfn-stack/handler.lambda_handler`
20+
runtime: `python3.6`
21+
22+
Required parameters:
23+
- `Region` - AWS Region to create stack in
24+
- `StackName` - Name of the stack to be created
25+
- `TemplateUrl` - S3 Url of stack template
26+
- `Capabilities` - Comma seperated list of capabilities. Set to empty value if no IAM capabilities required.
27+
- `EnabledRegions` - Comma separated list of regions that stack is allowed to be created in.
28+
Useful when passing this list is template parameters.
29+
30+
31+
Optional parameters:
32+
- `StackParam_Key` - Will pass value of this param down to stack's `Key` parameter
33+
- `OnFailure` - Behaviour on stack creation failure. Accepted values are `DO_NOTHING`,`ROLLBACK` and `DELETE`
34+
35+
### Copy or unpack objects between S3 buckets
36+
37+
This custom resource allows copying from source to destination s3 buckets. For source, if you provide prefix
38+
(without trailing slash), all objects under that prefix will be copied. Alternatively, if you provide s3 object
39+
with `*.zip` extensions, this object will be unpacked before it's files are unpacked to target bucket / prefix.
40+
Please note that this lambda function design does not include recursive calls if lambda is timing out, thus it does not
41+
permit mass file unpacking, but is rather designed for deployment of smaller files, such as client side web applications.
42+
43+
handler: `src/s3-copy/handler.lambda_handler`
44+
runtime: `python3.6`
45+
46+
Required parameters:
47+
48+
- `Source` - Source object/prefix/zip-file in `s3://bucket-name/path/to/prefix/or/object.zip` format
49+
- `Destination` - Destination bucket and prefix in `s3://bucket-name/destination-prefix` format
50+
- `CannedAcl` - Canned ACL for created objects in destination
51+
No optional parameters.

‎regional-cfn-stack/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# package marker

‎regional-cfn-stack/cr_response.py

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
import logging
2+
from urllib.request import urlopen, Request, HTTPError, URLError
3+
import json
4+
5+
logger = logging.getLogger()
6+
logger.setLevel(logging.INFO)
7+
8+
9+
class CustomResourceResponse:
10+
def __init__(self, request_payload):
11+
self.payload = request_payload
12+
self.response = {
13+
"StackId": request_payload["StackId"],
14+
"RequestId": request_payload["RequestId"],
15+
"LogicalResourceId": request_payload["LogicalResourceId"],
16+
"Status": 'SUCCESS',
17+
}
18+
19+
def respond(self):
20+
event = self.payload
21+
response = self.response
22+
####
23+
#### copied from https://github.com/ryansb/cfn-wrapper-python/blob/master/cfn_resource.py
24+
####
25+
26+
if event.get("PhysicalResourceId", False):
27+
response["PhysicalResourceId"] = event["PhysicalResourceId"]
28+
29+
logger.debug("Received %s request with event: %s" % (event['RequestType'], json.dumps(event)))
30+
31+
serialized = json.dumps(response)
32+
logger.info(f"Responding to {event['RequestType']} request with: {serialized}")
33+
34+
req_data = serialized.encode('utf-8')
35+
36+
req = Request(
37+
event['ResponseURL'],
38+
data=req_data,
39+
headers={'Content-Length': len(req_data),'Content-Type': ''}
40+
)
41+
req.get_method = lambda: 'PUT'
42+
43+
try:
44+
urlopen(req)
45+
logger.debug("Request to CFN API succeeded, nothing to do here")
46+
except HTTPError as e:
47+
logger.error("Callback to CFN API failed with status %d" % e.code)
48+
logger.error("Response: %s" % e.reason)
49+
except URLError as e:
50+
logger.error("Failed to reach the server - %s" % e.reason)

‎regional-cfn-stack/handler.py

+246
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
import boto3
2+
import json
3+
import os
4+
import sys
5+
6+
sys.path.append(f"{os.environ['LAMBDA_TASK_ROOT']}/lib")
7+
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
8+
9+
import cr_response
10+
import stack_manage
11+
import lambda_invoker
12+
import traceback
13+
14+
create_stack_success_states = ['CREATE_COMPLETE']
15+
update_stack_success_states = ['CREATE_COMPLETE', 'UPDATE_COMPLETE']
16+
delete_stack_success_states = ['DELETE_COMPLETE']
17+
18+
create_stack_failure_states = ['CREATE_FAILED',
19+
'DELETE_FAILED',
20+
'UPDATE_FAILED',
21+
'ROLLBACK_FAILED',
22+
'DELETE_COMPLETE',
23+
'ROLLBACK_COMPLETE']
24+
update_stack_failure_states = ['CREATE_FAILED', 'DELETE_FAILED', 'UPDATE_FAILED', 'ROLLBACK_COMPLETE','UPDATE_ROLLBACK_COMPLETE']
25+
delete_stack_failure_states = ['DELETE_FAILED']
26+
27+
28+
def respond_disabled_region(region, payload):
29+
cfn_response = cr_response.CustomResourceResponse(payload)
30+
payload['PhysicalResourceId'] = f"Disabled{region.replace('-','')}{payload['ResourceProperties']['StackName']}"
31+
cfn_response.response['Status'] = 'SUCCESS'
32+
cfn_response.respond()
33+
return
34+
35+
36+
def create_update_stack(cmd, payload):
37+
if 'Capabilities' not in payload['ResourceProperties']:
38+
payload['ResourceProperties']['Capabilities'] = 'CAPABILITY_IAM'
39+
40+
# compile stack parameters
41+
stack_params = {}
42+
for key, value in payload['ResourceProperties'].items():
43+
if key.startswith('StackParam_'):
44+
param_key = key.replace('StackParam_', '')
45+
param_value = value
46+
stack_params[param_key] = param_value
47+
48+
# instantiate and use management handler
49+
manage = stack_manage.StackManagement()
50+
51+
on_failure = 'DELETE'
52+
if 'OnFailure' in payload['ResourceProperties']:
53+
on_failure = payload['ResourceProperties']['OnFailure']
54+
55+
stack_id = ''
56+
if cmd == 'create':
57+
stack_id = manage.create(
58+
payload['ResourceProperties']['Region'],
59+
payload['ResourceProperties']['StackName'],
60+
payload['ResourceProperties']['TemplateUrl'],
61+
stack_params,
62+
payload['ResourceProperties']['Capabilities'].split(','),
63+
on_failure
64+
)
65+
elif cmd == 'update':
66+
stack_id = payload['PhysicalResourceId']
67+
result = manage.update(
68+
payload['ResourceProperties']['Region'],
69+
stack_id,
70+
payload['ResourceProperties']['TemplateUrl'],
71+
stack_params,
72+
payload['ResourceProperties']['Capabilities'].split(','),
73+
)
74+
# no updates to be performed
75+
if result is None:
76+
cfn_response = cr_response.CustomResourceResponse(payload)
77+
cfn_response.respond()
78+
return None
79+
else:
80+
raise 'Cmd must be create or update'
81+
82+
return stack_id
83+
84+
85+
def delete_stack(payload):
86+
manage = stack_manage.StackManagement()
87+
region = payload['ResourceProperties']['Region']
88+
stack_id = payload['PhysicalResourceId']
89+
manage.delete(region, payload['ResourceProperties']['StackName'])
90+
return stack_id
91+
92+
93+
def wait_stack_states(success_states, failure_states, lambda_payload, lambda_context):
94+
"""
95+
Wait for stack states, either be it success or failure. If none of the states
96+
appear and lambda is running out of time, it will be re-invoked with lambda_payload
97+
parameters
98+
:param lambda_context:
99+
:param stack_id:
100+
:param success_states:
101+
:param failure_states:
102+
:param lambda_payload:
103+
:return:
104+
"""
105+
manage = stack_manage.StackManagement()
106+
result = manage.wait_stack_status(
107+
lambda_payload['ResourceProperties']['Region'],
108+
lambda_payload['PhysicalResourceId'],
109+
success_states,
110+
failure_states,
111+
lambda_context
112+
)
113+
114+
# in this case we need to restart lambda execution
115+
if result is None:
116+
invoke = lambda_invoker.LambdaInvoker()
117+
invoke.invoke(lambda_payload)
118+
else:
119+
# one of the states is reached, and reply should be sent back to cloud formation
120+
cfn_response = cr_response.CustomResourceResponse(lambda_payload)
121+
cfn_response.response['PhysicalResourceId'] = lambda_payload['PhysicalResourceId']
122+
cfn_response.response['Status'] = result.status
123+
cfn_response.response['Reason'] = result.reason
124+
cfn_response.response['StackId'] = lambda_payload['StackId']
125+
cfn_response.respond()
126+
127+
128+
def lambda_handler(payload, context):
129+
# if lambda invoked to wait for stack status
130+
print(f"Received event:{json.dumps(payload)}")
131+
132+
# handle disable region situation
133+
if 'EnabledRegions' in payload['ResourceProperties']:
134+
region_list = payload['ResourceProperties']['EnabledRegions'].split(',')
135+
current_region = payload['ResourceProperties']['Region']
136+
print(f"EnabledRegions: {region_list}. Current region={current_region}")
137+
138+
if current_region not in region_list:
139+
# if this is create request just skip
140+
if payload['RequestType'] == 'Create' or payload['RequestType'] == 'Update':
141+
print(f"{current_region} not enabled, skipping")
142+
# report disabled
143+
# in case of region disable (update), physical record changes, so cleanup delete request is
144+
# sent subsequently via Cf, which will delete the stack
145+
respond_disabled_region(current_region, payload)
146+
return
147+
148+
149+
150+
# lambda was invoked by itself, we just have to wait for stack operation to be completed
151+
if ('WaitComplete' in payload) and (payload['WaitComplete']):
152+
print("Waiting for stack status...")
153+
if payload['RequestType'] == 'Create':
154+
wait_stack_states(
155+
create_stack_success_states,
156+
create_stack_failure_states,
157+
payload,
158+
context
159+
)
160+
161+
elif payload['RequestType'] == 'Update':
162+
wait_stack_states(
163+
update_stack_success_states,
164+
update_stack_failure_states,
165+
payload,
166+
context
167+
)
168+
169+
elif payload['RequestType'] == 'Delete':
170+
wait_stack_states(
171+
delete_stack_success_states,
172+
delete_stack_failure_states,
173+
payload,
174+
context
175+
)
176+
177+
# lambda was invoked directly by cf
178+
else:
179+
# depending on request type different handler is called
180+
print("Executing stack CRUD...")
181+
stack_id = None
182+
if 'PhysicalResourceId' in payload:
183+
stack_id = payload['PhysicalResourceId']
184+
try:
185+
manage = stack_manage.StackManagement()
186+
stack_name = payload['ResourceProperties']['StackName']
187+
region = payload['ResourceProperties']['Region']
188+
stack_exists = manage.stack_exists(region, stack_name)
189+
190+
if payload['RequestType'] == 'Create':
191+
# stack exists, create request => update
192+
# stack not exists, create request => create
193+
if stack_exists:
194+
print(f"Create request came for {stack_name}, but it already exists in {region}, updating...")
195+
payload['RequestType'] = 'Update'
196+
lambda_handler(payload, context)
197+
return
198+
else:
199+
stack_id = create_update_stack('create', payload)
200+
201+
elif payload['RequestType'] == 'Update':
202+
# stack exists, update request => update
203+
# stack not exists, update request => create
204+
if stack_exists:
205+
stack_id = create_update_stack('update', payload)
206+
if stack_id is None:
207+
# no updates to be performed
208+
return
209+
else:
210+
print(f"Update request came for {stack_name}, but it does not exist in {region}, creating...")
211+
payload['RequestType'] = 'Create'
212+
lambda_handler(payload, context)
213+
return
214+
215+
elif payload['RequestType'] == 'Delete':
216+
# stack exists, delete request => delete
217+
# stack not exists, delete request => report ok
218+
# for delete we are interested in actual stack id
219+
stack_exists = manage.stack_exists(region, stack_id)
220+
if stack_exists:
221+
delete_stack(payload)
222+
else:
223+
# reply with success
224+
print(f"Delete request came for {stack_name}, but it is nowhere to be found...")
225+
cfn_response = cr_response.CustomResourceResponse(payload)
226+
cfn_response.response['Reason'] = 'CloudFormation stack has not been found, may be removed manually'
227+
cfn_response.respond()
228+
return
229+
230+
# if haven't moved to other operation, set payloads stack id to created/updated stack and wait
231+
# for appropriate stack status
232+
payload['PhysicalResourceId'] = stack_id
233+
payload['WaitComplete'] = True
234+
invoker = lambda_invoker.LambdaInvoker()
235+
invoker.invoke(payload)
236+
237+
except Exception as e:
238+
print(f"Exception:{e}\n{str(e)}")
239+
print(traceback.format_exc())
240+
cfn_response = cr_response.CustomResourceResponse(payload)
241+
if 'PhysicalResourceId' in payload:
242+
cfn_response.response['PhysicalResourceId'] = payload['PhysicalResourceId']
243+
cfn_response.response['Status'] = 'FAILED'
244+
cfn_response.response['Reason'] = str(e)
245+
cfn_response.respond()
246+
raise e

‎regional-cfn-stack/lambda_invoker.py

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import boto3
2+
import os
3+
import json
4+
5+
6+
class LambdaInvoker:
7+
def __init__(self):
8+
print(f"Initialize lambda invoker")
9+
10+
def invoke(self, payload):
11+
bytes_payload = bytearray()
12+
bytes_payload.extend(map(ord, json.dumps(payload)))
13+
function_name = os.environ['AWS_LAMBDA_FUNCTION_NAME']
14+
function_payload = bytes_payload
15+
client = boto3.client('lambda')
16+
client.invoke(
17+
FunctionName=function_name,
18+
InvocationType='Event',
19+
Payload=function_payload
20+
)

0 commit comments

Comments
 (0)