Introduce k8s controllers

[jewertow]

Currently, we use Helm to manage the controller deployment and its configuration, which is passed as CLI arguments. This approach offers limited flexibility, as any configuration change requires a controller restart and can lead to pod failures in case of misconfiguration.

We create required Istio resources only when necessary, driven exclusively by notifications from the FDS Client. However, this approach has a drawback - it is not reliable and does not ensure the desired state of the cluster after resources are created. Since there is no ongoing reconciliation, any drift from the intended configuration, whether due to user modifications or controller uninstallation will potentially leave the cluster in a broken state.

To address these challenges, we should implement reconciliation-based approach, where the controllers continuously monitor and enforces the desired state of the cluster.

This involves:

• introducing a declarative API to create Mesh Federation, allowing to:
  • selectively define which services should be exported
  • provide remote mesh details to connect with local instance
  • define which services should be imported from a given remote mesh
• automating cluster (and Istio) configuration to provide federation functionality
• implementing a mechanism to detect drifts and correct the configuration to have functional federated meshes
• gracefully handle uninstallation

[bartoszmajsak]

I reworked this issue to be a top-level epic serving as an umbrella for all the work related to k8s controller machinery.