Add support to disable CAPZ components through a manager flag

[bryan-cox]

What type of PR is this?
/kind bug

What this PR does / why we need it:
Adds the ability to disable CAPZ components through a manager flag. Flags added for disabling ASO Secret Controller and disabling Azure JSON Machine Controller.

Which issue(s) this PR fixes:
Fixes #5472

Special notes for your reviewer:

TODOs:

• squashed commits
• includes documentation
• adds unit tests
• cherry-pick candidate

Release note:

Adds the ability to disable CAPZ components through a manager flag. Flags added for disabling ASO Secret Controller and disabling Azure JSON Machine Controller.

[bryan-cox]

/assign @nawazkh

[codecov]

Codecov Report

Attention: Patch coverage is 12.82051% with 34 lines in your changes missing coverage. Please review.

Project coverage is 53.25%. Comparing base (3a9eb5b) to head (ebc3066).

Files with missing lines	Patch %	Lines
main.go	0.00%	34 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5552      +/-   ##
==========================================
- Coverage   53.27%   53.25%   -0.02%     
==========================================
  Files         272      273       +1     
  Lines       29522    29541      +19     
==========================================
+ Hits        15727    15732       +5     
- Misses      12980    12994      +14     
  Partials      815      815              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nojnhuh]

the AzureManagedControlPlane CR which is behind the ASOAPI feature gate.

This is incorrect. The ASOAPI feature gate only enables controllers for the AzureASOManagedControlPlane and the other AzureASO... resources. This ASO secret controller is necessary for all of the resource types that CAPZ manages with ASO, including resource groups and vnets which are created for every AzureCluster and AzureManagedControlPlane. I suspect that if you disable the ASOAPI feature gate in CI, then all of the e2e tests will blow up, not just the ones exercising the AzureASO... APIs.

I think I mentioned somewhere, maybe in #5099 or in a Slack thread related to that PR, that a better solution to this general problem IMO would be a generic toggle to enable or disable individual controllers and webhooks with command line flags. Either something like that, or we explicitly disclaim all support for any installation that is not exactly equivalent to the CRDs and other manifests we publish for releases, since we generally assume the controller manager is running when all of the CRDs are installed. Changing the meaning of existing feature gates isn't a sustainable way to solve the general "I didn't install a CRD and now the controller manager is crashing" problem.

[enxebre]

that a better solution to this general problem IMO would be a generic toggle to enable or disable individual controllers and webhooks with command line flags.

I agree, fwiw created this some time ago to track that effort #5294

[nawazkh]

This is incorrect. The ASOAPI feature gate only enables controllers for the AzureASOManagedControlPlane and the other AzureASO... resources. This ASO secret controller is necessary for all of the resource types that CAPZ manages with ASO, including resource groups and vnets which are created for every AzureCluster and AzureManagedControlPlane. I suspect that if you disable the ASOAPI feature gate in CI, then all of the e2e tests will blow up, not just the ones exercising the AzureASO... APIs.

Thank you for adding more context on this Jon.

that a better solution to this general problem IMO would be a generic toggle to enable or disable individual controllers and webhooks with command line flags.

I agree, fwiw created this some time ago to track that effort #5294

I agree also. We could start with updating manager.yaml with a bunch of env variables that enable different controllers and webhooks.

---

So we essentially close out this PR @bryan-cox ?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from nawazkh. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bryan-cox]

/test all

[bryan-cox]

/test all

[bryan-cox]

@nojnhuh @nawazkh - when you have a moment, can I get another look at this PR? If we are good with moving this direction, I can follow up with some doc on how this can be used.

[nawazkh]

@nojnhuh @nawazkh - when you have a moment, can I get another look at this PR? If we are good with moving this direction, I can follow up with some doc on how this can be used.

I like the idea of being able to toggle controllers, so green flag from me on this approach.

However, we need to ensure that the Management cluster is functional despite turning off controller(maybe all in the future?), so maybe we also incorporate an e2e test to validate the functionality.
We could add that test scenario in an optional test.

@bryan-cox , what do you say ?