|
| 1 | +# testing-helpers Specification |
| 2 | + |
| 3 | +## Purpose |
| 4 | + |
| 5 | +Provide a public testing surface (`cfn_handler.testing`) that lets users |
| 6 | +unit-test custom-resource handlers in-process — without HTTP, without |
| 7 | +`boto3`, without `moto` — and without reaching into the |
| 8 | +`cfn_handler._internal` namespace whose contract is explicitly unstable. |
| 9 | + |
| 10 | +The capability is built around `CustomResource.replay(event, context)`, |
| 11 | +which executes the full dispatch pipeline (handler resolution, handler |
| 12 | +invocation, polling deferral) and returns a structured `Replay` value |
| 13 | +capturing the response payload that *would* have been sent. Polling is |
| 14 | +stubbed so a deferred replay mutates the event with the same marker |
| 15 | +keys real polling adds, allowing a follow-up `replay()` to drive the |
| 16 | +poll handler without provisioning EventBridge rules. |
| 17 | + |
| 18 | +Adjacent surfaces — event/context factories, assertion helpers, and |
| 19 | +auto-discovered pytest fixtures — round out the kit so users can write |
| 20 | +TDD-shaped handler tests with minimal boilerplate. |
| 21 | + |
| 22 | +This capability is intentionally additive to the `lifecycle-handler` |
| 23 | +and `polling` capabilities: production behaviour is unchanged. The |
| 24 | +seams that make replay possible are private implementation details |
| 25 | +documented in the change archive (`openspec/changes/archive/`). |
| 26 | + |
| 27 | +## Requirements |
| 28 | + |
| 29 | +### Requirement: Public testing module is importable |
| 30 | + |
| 31 | +The library SHALL expose a `cfn_handler.testing` module importable in any Python environment where `cfn_handler` itself imports cleanly, without requiring `pytest`, `boto3`, or any other optional dependency. |
| 32 | + |
| 33 | +#### Scenario: Module imports without pytest installed |
| 34 | +- **WHEN** a user runs `import cfn_handler.testing` in an environment |
| 35 | + where pytest is not installed |
| 36 | +- **THEN** the import succeeds and the public names (`Replay`, |
| 37 | + `make_event`, `assert_success`, `assert_failed`, `assert_deferred`) |
| 38 | + are available |
| 39 | + |
| 40 | +#### Scenario: Module imports without boto3 installed |
| 41 | +- **WHEN** a user runs `import cfn_handler.testing` in an environment |
| 42 | + where boto3 is not installed |
| 43 | +- **THEN** the import succeeds and `Replay` / `make_event` / assertion |
| 44 | + helpers are available |
| 45 | + |
| 46 | +### Requirement: In-process replay of the dispatch flow |
| 47 | + |
| 48 | +`CustomResource` SHALL expose a `replay(event, context=None)` method that executes the full dispatch pipeline in-process and returns a `Replay` object capturing the outcome, without issuing HTTP requests, importing `boto3`, or mutating the registered handler functions. |
| 49 | + |
| 50 | +#### Scenario: Successful create handler is replayed |
| 51 | +- **WHEN** a `CustomResource` has a CREATE handler registered that |
| 52 | + returns `{"Endpoint": "https://x"}`, and `replay(create_event)` is |
| 53 | + invoked |
| 54 | +- **THEN** the returned `Replay` has `status="SUCCESS"`, |
| 55 | + `data={"Endpoint": "https://x"}`, and `payload` is the rendered |
| 56 | + CFN response payload that would have been PUT to the response URL |
| 57 | + |
| 58 | +#### Scenario: Handler raises during replay |
| 59 | +- **WHEN** a CREATE handler raises `RuntimeError("boom")` during |
| 60 | + replay |
| 61 | +- **THEN** the returned `Replay` has `status="FAILED"` and `reason` |
| 62 | + contains `"boom"` |
| 63 | + |
| 64 | +#### Scenario: Replay does not perform HTTP I/O |
| 65 | +- **WHEN** `replay()` is invoked with a valid event whose `ResponseURL` |
| 66 | + is `https://example.invalid/cfn-response` |
| 67 | +- **THEN** no HTTP request is made to any URL during the call |
| 68 | + |
| 69 | +#### Scenario: Replay does not import boto3 |
| 70 | +- **WHEN** `replay()` is invoked in an environment without boto3 |
| 71 | + installed AND no poll handler is registered |
| 72 | +- **THEN** the call completes successfully without raising |
| 73 | + `PollingDependencyError` or `ImportError` |
| 74 | + |
| 75 | +### Requirement: Replay produces a structured result |
| 76 | + |
| 77 | +The `Replay` type SHALL be a frozen, immutable dataclass with the fields `status` (literal `"SUCCESS" | "FAILED" | "DEFERRED"`), `physical_resource_id` (`str | None`), `data` (`dict[str, Any]`), `reason` (`str`), `no_echo` (`bool`), `payload` (`dict[str, Any]`), and `request_type` (literal `"Create" | "Update" | "Delete"`). |
| 78 | + |
| 79 | +#### Scenario: Replay result is immutable |
| 80 | +- **WHEN** a user attempts to mutate `replay.status = "FAILED"` after |
| 81 | + a SUCCESS replay |
| 82 | +- **THEN** `dataclasses.FrozenInstanceError` is raised |
| 83 | + |
| 84 | +#### Scenario: Replay payload matches what would be sent |
| 85 | +- **WHEN** `replay()` returns a `Replay` with `status="SUCCESS"` and |
| 86 | + `data={"Endpoint": "x"}` |
| 87 | +- **THEN** `replay.payload["Status"] == "SUCCESS"`, |
| 88 | + `replay.payload["Data"] == {"Endpoint": "x"}`, and the payload |
| 89 | + conforms to the CFN custom-resource response schema |
| 90 | + |
| 91 | +### Requirement: Replay supports the polling-deferral case |
| 92 | + |
| 93 | +`replay()` SHALL handle the polling-deferral path without invoking any AWS API or importing `boto3`: when a matching poll handler is registered, it MUST return a `Replay` with `status="DEFERRED"` and an empty `payload` dict, and MUST mutate the input event to add the polling marker keys (`CfnHandlerPoll`, `CfnHandlerRule`, `CfnHandlerPermission`) so a subsequent `replay()` call resumes into the poll handler path. |
| 94 | + |
| 95 | +#### Scenario: Create with poller defers |
| 96 | +- **WHEN** a `CustomResource` has both `@create` and `@poll_create` |
| 97 | + handlers registered, and `replay(create_event)` is invoked |
| 98 | +- **THEN** the returned `Replay` has `status="DEFERRED"`, no AWS API |
| 99 | + call is made, and the input event has been mutated to include |
| 100 | + `event["CfnHandlerPoll"] is True` |
| 101 | + |
| 102 | +#### Scenario: Poll re-invocation completes the flow |
| 103 | +- **WHEN** a deferred event is replayed a second time, and the |
| 104 | + registered poll handler returns response data |
| 105 | +- **THEN** the returned `Replay` has `status="SUCCESS"` and the |
| 106 | + data the poll handler provided |
| 107 | + |
| 108 | +### Requirement: Event factory produces canonical CFN events |
| 109 | + |
| 110 | +The library SHALL expose a `make_event` callable in `cfn_handler.testing` that returns a dict matching the documented CloudFormation custom-resource event shape, with keyword overrides for every documented field, and MUST require a non-`None` `physical_resource_id` argument when `RequestType` is `"Update"` or `"Delete"` (raising `ValueError` if not supplied). |
| 111 | + |
| 112 | +#### Scenario: Default Create event is well-formed |
| 113 | +- **WHEN** `make_event()` is called with no arguments |
| 114 | +- **THEN** the returned dict has `RequestType="Create"`, |
| 115 | + syntactically valid `StackId`, `RequestId`, `LogicalResourceId`, |
| 116 | + `ResourceType`, `ResourceProperties`, `ResponseURL`, `ServiceToken` |
| 117 | + fields, and no `PhysicalResourceId` |
| 118 | + |
| 119 | +#### Scenario: Update event requires PhysicalResourceId |
| 120 | +- **WHEN** `make_event(request_type="Update")` is called without |
| 121 | + passing `physical_resource_id` |
| 122 | +- **THEN** `ValueError` is raised with a message identifying the |
| 123 | + missing argument |
| 124 | + |
| 125 | +#### Scenario: Field overrides are applied |
| 126 | +- **WHEN** `make_event(resource_properties={"Foo": "bar"})` is |
| 127 | + called |
| 128 | +- **THEN** the returned dict has `ResourceProperties == {"Foo": "bar"}` |
| 129 | + |
| 130 | +#### Scenario: Defaults use safe placeholder values |
| 131 | +- **WHEN** `make_event()` is called with no overrides |
| 132 | +- **THEN** the `ResponseURL` host is `example.invalid` (RFC 6761 |
| 133 | + reserved name guaranteed not to resolve) and the account ID portion |
| 134 | + of `StackId` is `111111111111` (AWS-reserved example account) |
| 135 | + |
| 136 | +### Requirement: Lambda context factory satisfies the protocol |
| 137 | + |
| 138 | +The library SHALL expose a `make_context` callable in `cfn_handler.testing` that returns an object satisfying the existing `LambdaContext` protocol used by `CustomResource.__call__`, exposing `aws_request_id`, `function_name`, `invoked_function_arn`, `log_group_name`, `log_stream_name`, and `get_remaining_time_in_millis()`. |
| 139 | + |
| 140 | +#### Scenario: Context satisfies the protocol |
| 141 | +- **WHEN** `ctx = make_context()` is called and used in |
| 142 | + `resource.replay(event, ctx)` |
| 143 | +- **THEN** the call succeeds and `ctx.get_remaining_time_in_millis()` |
| 144 | + returns a positive integer |
| 145 | + |
| 146 | +#### Scenario: Remaining-time override is honoured |
| 147 | +- **WHEN** `make_context(remaining_time_ms=5000)` is called |
| 148 | +- **THEN** `ctx.get_remaining_time_in_millis()` returns `5000` |
| 149 | + |
| 150 | +### Requirement: Assertion helpers raise informative AssertionError |
| 151 | + |
| 152 | +The library SHALL expose `assert_success`, `assert_failed`, and `assert_deferred` helpers in `cfn_handler.testing`, each of which MUST raise `AssertionError` with a message identifying both the expected and actual values when the assertion fails. |
| 153 | + |
| 154 | +#### Scenario: assert_success on a SUCCESS replay passes |
| 155 | +- **WHEN** `assert_success(replay, data={"x": 1})` is called and |
| 156 | + `replay.status == "SUCCESS"` and `replay.data == {"x": 1}` |
| 157 | +- **THEN** the call returns `None` (no exception) |
| 158 | + |
| 159 | +#### Scenario: assert_success on a FAILED replay raises |
| 160 | +- **WHEN** `assert_success(replay)` is called and |
| 161 | + `replay.status == "FAILED"` with `reason="boom"` |
| 162 | +- **THEN** `AssertionError` is raised and the message contains both |
| 163 | + `"FAILED"` and `"boom"` |
| 164 | + |
| 165 | +#### Scenario: assert_failed with reason_contains matches a substring |
| 166 | +- **WHEN** `assert_failed(replay, reason_contains="boom")` is called |
| 167 | + and `replay.status == "FAILED"` with `reason="something boom happened"` |
| 168 | +- **THEN** the call returns `None` |
| 169 | + |
| 170 | +#### Scenario: assert_deferred on a SUCCESS replay raises |
| 171 | +- **WHEN** `assert_deferred(replay)` is called and |
| 172 | + `replay.status == "SUCCESS"` |
| 173 | +- **THEN** `AssertionError` is raised |
| 174 | + |
| 175 | +### Requirement: pytest fixtures auto-register via entry point |
| 176 | + |
| 177 | +The library's `pyproject.toml` SHALL declare a `pytest11` entry point named `cfn_handler` pointing at the fixtures module so that the fixtures `cfn_create_event`, `cfn_update_event`, `cfn_delete_event`, and `cfn_lambda_context` are available without any user-side `pytest_plugins` declaration. |
| 178 | + |
| 179 | +#### Scenario: Fixture is auto-discovered |
| 180 | +- **WHEN** a user with `cfn_handler` installed writes a test |
| 181 | + `def test_x(cfn_create_event): ...` in a fresh pytest project |
| 182 | + with no `conftest.py` configuration |
| 183 | +- **THEN** pytest resolves the fixture without error and passes a |
| 184 | + Create-shaped event dict |
| 185 | + |
| 186 | +#### Scenario: Each invocation gets a fresh event |
| 187 | +- **WHEN** two tests both consume `cfn_create_event` and one mutates |
| 188 | + the event dict |
| 189 | +- **THEN** the second test sees the unmutated default event (no |
| 190 | + cross-test leak) |
| 191 | + |
| 192 | +### Requirement: Replay never sends a real CFN response |
| 193 | + |
| 194 | +`CustomResource.replay` SHALL NOT, under any code path, send an HTTP request to the event's `ResponseURL` or any other URL, and the production HTTP transport MUST be replaced by an in-memory capture for the duration of the replay call and restored when the call returns or raises. |
| 195 | + |
| 196 | +#### Scenario: Replay catches a handler exception without sending HTTP |
| 197 | +- **WHEN** `replay()` is invoked with a handler that raises during |
| 198 | + execution |
| 199 | +- **THEN** the returned `Replay` has `status="FAILED"` AND no HTTP |
| 200 | + request was issued (verified via mock or instrumentation) |
| 201 | + |
| 202 | +#### Scenario: Replay restores transport after exception |
| 203 | +- **WHEN** `replay()` raises an unexpected internal exception (not a |
| 204 | + handler exception) and the same `CustomResource` instance is then |
| 205 | + invoked normally via `__call__` (with the production HTTP transport) |
| 206 | +- **THEN** the production invocation correctly issues an HTTP PUT to |
| 207 | + the event's `ResponseURL` |
0 commit comments