Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to modify DBCluster after creation when engine mode is serverless #917

Closed
jaypipes opened this issue Aug 25, 2021 · 1 comment
Closed

Unable to modify DBCluster after creation when engine mode is serverless #917

jaypipes opened this issue Aug 25, 2021 · 1 comment
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@jaypipes
Copy link
Collaborator

When regenerating the ACK controller for RDS with the latest ACK runtime and code-generator, I'm consistently getting a getting a failure when running the test_db_cluster.TestDBCluster::test_create_delete_mysql_serverless e2e test:

self = <e2e.tests.test_db_cluster.TestDBCluster object at 0x7f0f7dd5f430>, rds_client = <botocore.client.RDS object at 0x7f0f7dd93280>, master_user_pass_secret = ('default', 'dbclustersecrets', 'master_user_password')
    def test_create_delete_mysql_serverless(
            self,
            rds_client,
            master_user_pass_secret,
    ):
        db_cluster_id = "my-aurora-mysql"
        db_name = "mydb"
        mup_sec_ns, mup_sec_name, mup_sec_key = master_user_pass_secret
        replacements = REPLACEMENT_VALUES.copy()
        replacements['COPY_TAGS_TO_SNAPSHOT'] = "False"
        replacements["DB_CLUSTER_ID"] = db_cluster_id
        replacements["DB_NAME"] = db_name
        replacements["MASTER_USER_PASS_SECRET_NAMESPACE"] = mup_sec_ns
        replacements["MASTER_USER_PASS_SECRET_NAME"] = mup_sec_name
        replacements["MASTER_USER_PASS_SECRET_KEY"] = mup_sec_key
        resource_data = load_rds_resource(
            "db_cluster_mysql_serverless",
            additional_replacements=replacements,
        )
        ref = k8s.CustomResourceReference(
            CRD_GROUP, CRD_VERSION, RESOURCE_PLURAL,
            db_cluster_id, namespace="default",
        )
        k8s.create_custom_resource(ref, resource_data)
        cr = k8s.wait_resource_consumed_by_controller(ref)
        assert cr is not None
        # Let's check that the DB cluster appears in RDS
        aws_res = rds_client.describe_db_clusters(DBClusterIdentifier=db_cluster_id)
        assert aws_res is not None
        assert len(aws_res['DBClusters']) == 1
        dbc_rec = aws_res['DBClusters'][0]
        now = datetime.datetime.now()
        timeout = now + datetime.timedelta(seconds=CREATE_TIMEOUT_SECONDS)
        # TODO(jaypipes): Move this into generic AWS-side waiter
        while dbc_rec['Status'] != "available":
            if datetime.datetime.now() >= timeout:
                pytest.fail("failed to find available DBCluster before timeout")
            time.sleep(CREATE_INTERVAL_SLEEP_SECONDS)
            aws_res = rds_client.describe_db_clusters(DBClusterIdentifier=db_cluster_id)
            assert len(aws_res['DBClusters']) == 1
            dbc_rec = aws_res['DBClusters'][0]
        # We're now going to modify the CopyTagsToSnapshot field of the DB
        # instance, wait some time and verify that the RDS server-side resource
        # shows the new value of the field.
        assert dbc_rec['CopyTagsToSnapshot'] == False
        updates = {
            "spec": {"copyTagsToSnapshot": True},
        }
        k8s.patch_custom_resource(ref, updates)
        time.sleep(MODIFY_WAIT_AFTER_SECONDS)
        aws_res = rds_client.describe_db_clusters(DBClusterIdentifier=db_cluster_id)
        assert aws_res is not None
        assert len(aws_res['DBClusters']) == 1
        dbc_rec = aws_res['DBClusters'][0]
      assert dbc_rec['CopyTagsToSnapshot'] == True
E       assert False == True
E         +False
E         -True
tests/test_db_cluster.py:124: AssertionError

When looking in the rds-controller Pod logs, there are no errors since there are no failures in communicating with RDS, but you can see that the controller sets the DBCluster resource to a terminal condition after attempting to update the DBCluster when it notices that the desired state for the CopyTagsToSnapshot field has changed:

[jaypipes@thelio test-infra]$ k logs -nack-system ack-rds-controller-86f67fc976-4n7xm | grep my-aurora-mysql
2021-08-24T15:28:01.662Z	DEBUG	ackrt	> r.Sync	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:01.662Z	DEBUG	ackrt	>> rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:01.662Z	DEBUG	ackrt	>>> rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.859Z	DEBUG	ackrt	<<< rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.859Z	DEBUG	ackrt	<< rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "error": "resource not found"}
2021-08-24T15:28:02.859Z	DEBUG	ackrt	>> r.createResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.859Z	DEBUG	ackrt	>>> r.setResourceManaged	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.859Z	DEBUG	ackrt	>>>> r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.859Z	DEBUG	ackrt	>>>>> kc.Patch (metadata + spec)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.863Z	DEBUG	ackrt	<<<<< kc.Patch (metadata + spec)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.863Z	DEBUG	ackrt	patched resource metadata and spec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "latest": {}}
2021-08-24T15:28:02.863Z	DEBUG	ackrt	<<<< r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.863Z	DEBUG	ackrt	marked resource as managed	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.863Z	DEBUG	ackrt	<<< r.setResourceManaged	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.863Z	DEBUG	ackrt	>>> rm.Create	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:02.863Z	DEBUG	ackrt	>>>> rm.sdkCreate	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.629Z	DEBUG	ackrt	<<<< rm.sdkCreate	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.629Z	DEBUG	ackrt	<<< rm.Create	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.629Z	DEBUG	ackrt	>>> rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.629Z	DEBUG	ackrt	>>>> rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.840Z	DEBUG	ackrt	<<<< rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.840Z	DEBUG	ackrt	<<< rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.840Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.840Z	DEBUG	ackrt	>>>> kc.Patch (metadata + spec)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	<<<< kc.Patch (metadata + spec)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	patched resource metadata and spec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "latest": {}}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	INFO	ackrt	created new resource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	<< r.createResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	>> r.lateInitializeResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	>>> rm.LateInitialize	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	no late initialization required.	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	<<< rm.LateInitialize	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	no difference found between metadata and spec for desired and latest object.	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	<< r.lateInitializeResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	< r.Sync	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	> r.patchResourceStatus	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.844Z	DEBUG	ackrt	>> kc.Patch (status)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.848Z	DEBUG	ackrt	<< kc.Patch (status)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.848Z	DEBUG	ackrt	patched resource status	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "latest": {}}
2021-08-24T15:28:04.848Z	DEBUG	ackrt	< r.patchResourceStatus	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 1}
2021-08-24T15:28:04.848Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "dbcluster", "request": "default/my-aurora-mysql"}
2021-08-24T15:28:04.848Z	DEBUG	ackrt	> r.Sync	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.848Z	DEBUG	ackrt	>> rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.848Z	DEBUG	ackrt	>>> rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	<<< rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	<< rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	>> r.updateResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	<< r.updateResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	>> r.lateInitializeResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	>>> rm.LateInitialize	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	no late initialization required.	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	<<< rm.LateInitialize	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	>>> r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	no difference found between metadata and spec for desired and latest object.	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	<<< r.patchResourceMetadataAndSpec	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	<< r.lateInitializeResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	< r.Sync	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	> r.patchResourceStatus	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.987Z	DEBUG	ackrt	>> kc.Patch (status)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.991Z	DEBUG	ackrt	<< kc.Patch (status)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.991Z	DEBUG	ackrt	patched resource status	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "latest": {}}
2021-08-24T15:28:04.991Z	DEBUG	ackrt	< r.patchResourceStatus	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 2}
2021-08-24T15:28:04.991Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "dbcluster", "request": "default/my-aurora-mysql"}
2021-08-24T15:31:18.692Z	DEBUG	ackrt	> r.Sync	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:18.692Z	DEBUG	ackrt	>> rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:18.692Z	DEBUG	ackrt	>>> rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.281Z	DEBUG	ackrt	<<< rm.sdkFind	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.281Z	DEBUG	ackrt	<< rm.ReadOne	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.281Z	DEBUG	ackrt	>> r.updateResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.282Z	INFO	ackrt	desired resource state has changed	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "diff": [{"Path":{"Parts":["Spec","CopyTagsToSnapshot"]},"A":true,"B":false}]}
2021-08-24T15:31:19.282Z	DEBUG	ackrt	>>> rm.Update	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.282Z	DEBUG	ackrt	>>>> rm.sdkUpdate	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.387Z	DEBUG	ackrt	<<<< rm.sdkUpdate	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.387Z	DEBUG	ackrt	<<< rm.Update	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "latest": {}, "error": "resource is in terminal condition"}
2021-08-24T15:31:19.387Z	DEBUG	ackrt	<< r.updateResource	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.387Z	DEBUG	ackrt	< r.Sync	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.387Z	DEBUG	ackrt	> r.patchResourceStatus	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.387Z	DEBUG	ackrt	>> kc.Patch (status)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.391Z	DEBUG	ackrt	<< kc.Patch (status)	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.391Z	DEBUG	ackrt	patched resource status	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "latest": {}}
2021-08-24T15:31:19.391Z	DEBUG	ackrt	< r.patchResourceStatus	{"kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3, "account": "750630568209", "role": "", "region": "us-west-2", "is_adopted": false, "kind": "DBCluster", "namespace": "default", "name": "my-aurora-mysql", "generation": 3}
2021-08-24T15:31:19.391Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "dbcluster", "request": "default/my-aurora-mysql"}

If I do a kubectl describe on the DBCluster resource in question, I consistently see a Condition with type ACK.Terminal with the message "You currently can't modify BackupWindow with Aurora Serverless":

[jaypipes@thelio test-infra]$ k describe dbclusters/my-aurora-mysql
Name:         my-aurora-mysql
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  rds.services.k8s.aws/v1alpha1
Kind:         DBCluster
Metadata:
  Creation Timestamp:  2021-08-24T15:28:01Z
  Finalizers:
    finalizers.rds.services.k8s.aws/DBCluster
  Generation:        3
  Resource Version:  1522
  Self Link:         /apis/rds.services.k8s.aws/v1alpha1/namespaces/default/dbclusters/my-aurora-mysql
  UID:               591c735b-4b5e-460a-984a-fa164b9e9eaf
Spec:
  Availability Zones:
    us-west-2c
    us-west-2a
    us-west-2b
  Backup Retention Period:  1
  Copy Tags To Snapshot:    true
  Database Name:            mydb
  Db Cluster Identifier:    my-aurora-mysql
  Deletion Protection:      false
  Engine:                   aurora-mysql
  Engine Mode:              serverless
  Engine Version:           5.7.mysql_aurora.2.07.1
  Kms Key ID:               arn:aws:kms:us-west-2:750630568209:key/32b2de04-8b6f-4093-94f8-beae0f9e36fb
  Master User Password:
    Key:                         master_user_password
    Name:                        dbclustersecrets
    Namespace:                   default
  Master Username:               root
  Port:                          3306
  Preferred Backup Window:       06:38-07:08
  Preferred Maintenance Window:  mon:08:04-mon:08:34
  Storage Encrypted:             true
Status:
  Ack Resource Metadata:
    Arn:                   arn:aws:rds:us-west-2:750630568209:cluster:my-aurora-mysql
    Owner Account ID:      750630568209
  Activity Stream Status:  stopped
  Allocated Storage:       1
  Capacity:                8
  Cluster Create Time:     2021-08-24T15:28:04Z
  Conditions:
    Message:                            You currently can't modify BackupWindow with Aurora Serverless.
    Status:                             True
    Type:                               ACK.Terminal
  Cross Account Clone:                  false
  Db Cluster Parameter Group:           default.aurora-mysql5.7
  Db Cluster Resource ID:               cluster-H4QKSI3PPK6JHQAFFPY62XT7RM
  Db Subnet Group:                      default
  Earliest Restorable Time:             2021-08-24T15:29:15Z
  Endpoint:                             my-aurora-mysql.cluster-cbpwxakdx2on.us-west-2.rds.amazonaws.com
  Hosted Zone ID:                       Z1PVIF0B656C1W
  Http Endpoint Enabled:                false
  Iam Database Authentication Enabled:  false
  Latest Restorable Time:               2021-08-24T15:29:15Z
  Multi AZ:                             false
  Scaling Configuration Info:
    Auto Pause:                true
    Max Capacity:              16
    Min Capacity:              2
    Seconds Until Auto Pause:  300
    Timeout Action:            RollbackCapacityChange
  Status:                      available
  Vpc Security Groups:
    Status:                 active
    Vpc Security Group ID:  sg-c5a50d89
Events:                     <none>

Note that the e2e test is attempting to change the value of a single field of the DBCluster resource (CopyTagsToSnapshot) from the false to true. You can see in the above kubectl describe output that the desired value of Spec.CopyTagsToSnapshot is properly set to true but in the Python e2e test, we use call the RDS DescribeDBClusters API call directly:

        aws_res = rds_client.describe_db_clusters(DBClusterIdentifier=db_cluster_id)
        assert aws_res is not None
        assert len(aws_res['DBClusters']) == 1
        dbc_rec = aws_res['DBClusters'][0]
        assert dbc_rec['CopyTagsToSnapshot'] == True

and that last line fails the assertion, meaning that the DBCluster resource's CopyTagsToSnapshot field was never modified.

After Googling for the "You currently can't modify BackupWindow with Aurora Serverless" message, I ran into the following Github issues and AWS forum threads that all describe the same problem:

There appears to be no ability to change fields on a DBCluster resource once created if the engine mode is serverless?

I tried setting various fields of the DBCluster resource used in the e2e test to a non-nil value but kept getting the same error message back from the RDS ModifyDBCluster API call, with various fields in the error message:

  • "You currently can't modify BackupWindow with Aurora Serverless"
  • "You currently can't modify MaintenanceWindow with Aurora Serverless"
  • "You currently can't modify EndpointPort with Aurora Serverless"

I'm unsure how to get past this particular issue :(
@jaypipes jaypipes added kind/bug Categorizes issue or PR as related to a bug. RDS labels Aug 25, 2021
@jaypipes
Copy link
Collaborator Author

Speaking with @RobinGolden from RDS, we may need to add a custom Update implementation for DBCluster that does not set fields in the ModifyDBClusterInput shape when there is no changed value between the desired and observed fields.

jaypipes added a commit to jaypipes/ack-rds-controller that referenced this issue Aug 26, 2021
@jaypipes
bring in ACK runtime v0.12.0
961356a 
Updates the ACK runtime to v0.12.0

Comments out a part of the e2e tests that tests updating an attribute on
a DBCluster resource with engine mode 'serverless' until we've fixed
aws-controllers-k8s/community#917.
@jaypipes jaypipes mentioned this issue Aug 26, 2021
Merged
@a-hilaly a-hilaly reopened this Aug 26, 2021
jaypipes added a commit to jaypipes/ack-rds-controller that referenced this issue Aug 27, 2021
@jaypipes
custom update code path for DBCluster
16dd696 
The ModifyDBCluster RDS API call will return an InvalidParameterValue
error if the engine mode for the DBCluster is "serverless" and any of
the following fields have been set on the ModifyDBClusterInput shape:

* PreferredMaintenanceWindow
* PreferredBackupWindow
* Port

This error is returned **even if the value of the field in the
ModifyDBClusterInput shape matches the existing value of the DBCluster
resource**

To get around this issue, I've added a custom update method
implementation for the DBCluster resource that builds the
ModifyDBClusterInput shape and ONLY sets fields on that shape when the
matching Spec field is not nil AND there is a difference detected
between the latest observed value and the desired value for that field.

Fixes Issue aws-controllers-k8s/community#917
@jaypipes jaypipes mentioned this issue Aug 27, 2021
Merged
ack-bot pushed a commit to aws-controllers-k8s/rds-controller that referenced this issue Aug 27, 2021
@jaypipes
custom update code path for DBCluster (#24)
6b1e8e8 
The ModifyDBCluster RDS API call will return an InvalidParameterValue
error if the engine mode for the DBCluster is "serverless" and any of
the following fields have been set on the ModifyDBClusterInput shape:

* PreferredMaintenanceWindow
* PreferredBackupWindow
* Port

This error is returned **even if the value of the field in the
ModifyDBClusterInput shape matches the existing value of the DBCluster
resource**

To get around this issue, I've added a custom update method
implementation for the DBCluster resource that builds the
ModifyDBClusterInput shape and ONLY sets fields on that shape when the
matching Spec field is not nil AND there is a difference detected
between the latest observed value and the desired value for that field.

Fixes Issue aws-controllers-k8s/community#917

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license.
@jaypipes jaypipes mentioned this issue Oct 15, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jaypipes @a-hilaly