Migration Assistant on K8s/EKS #1652
Description
Is your feature request related to a problem?
Migration Assistant migrations can be greatly improved, in terms of both speed and ease, by setting up the infrastructure in Kubernetes. Additionally, built-in mechanisms for scaling and orchestration in Kubernetes should pave a way for tasks such as #1072.
What solution would you like?
For this feature, the focus should be on setting up required infrastructure in Kubernetes to enable effectively performing migrations in a subsequent feature. This feature should include documentation for setting up and deploying the Migration Assistant to a Kubernetes cluster. Additionally, the Migration Assistant deployment should setup any required resources, such as a bucket for storing snapshots or tuples, or required roles that the setup process or future migration pods will need. Core functionality such as enabling persistent logging for all pods that get created as well as sending metrics with OTEL to configured targets is also expected with this feature.
What alternatives have you considered?
Some considerable time was spent trying to solve some of these problems in the existing Docker compose environment for local deployments and CDK package for AWS environment deployments, with limited success. Difficulties such as a lack of consistency between local and cloud deployments, and lack of built-in mechanisms for orchestration, scheduling, and scaling have put a large strain on this existing approach, all of which Kubernetes should be more targeted to resolve.
Do you have any additional context?
The Jira Epic for this work can be found here: https://opensearch.atlassian.net/browse/MIGRATIONS-2402