[Onboarding] Self-managed EC2 to k8s cluster (logging use case) #1686
Description
Issue Description
Please provide a detailed description of the problem or question. If you have any relevant logs, screenshots, or supporting data, include them inline or as attachments to help us assist you more effectively.
High-Level / Project Management Questions
-
Source version:
OpenSearch 2.17 -
Source environment:
self-managed on ec2 -
Target version:
OpenSearch 2.19 -
Target environment:
self-managed on EKS (k8s) -
Migration Assistant Version:
2.4.4 -
Do you have a maintenance window during which requests will not be serviced?
No
Determines the migration strategy—zero downtime vs. planned outage—and helps prevent unexpected service disruptions. If no maintenance window is available, or if the window is insufficient for a full backfill, a live capture mechanism must be included. -
Do you perform multiple updates or deletes to the same document within 10 seconds?
No
If updates occur faster than 10 seconds, additional mechanisms may be required to preserve update order and document consistency. -
Is a 30ms impact for indexing requests acceptable?
Yes
Typically, there’s a 10–30 ms delay on source cluster index requests, and sub-second delays have been observed between source and target. If this additional latency is not acceptable, live capture cannot be performed. -
What is your primary use case?
Logging
Migration strategies may vary depending on the use case.
Platform & Architecture
-
Number of nodes:
12 -
Have you identified all application dependencies on Elasticsearch/OpenSearch and the required client changes?
Yes
Ensures the migration won’t break dependent applications or integrations. -
Source authentication mechanism:
Basic Auth -
Target authentication mechanism:
Basic Auth -
[Live Capture Limitation] Do your applications generate document IDs, or do you rely on Elasticsearch/OpenSearch to generate them?
Application Generates
If document IDs are not specified by the application, source and target states may diverge during live migration. -
Do source indices contain a
_sourcefield?Yes
The_sourcefield is required for reindexing and recovery. Most migration tools, including Migration Assistant, depend on it.
Note: You can verify if_sourcehas been disabled by checking the index mappings. If_sourceis not explicitly set tofalse, it has not been disabled. -
Are you able to redirect client traffic for your migration?
Yes (client config change and deploy
To capture live traffic, source traffic must be redirected at the start of the migration and again when it completes.
Data & Cluster Metrics
-
How much data is stored in the cluster?
50 TB -
Do you have any indices created more than one major version ago that have not been reindexed?
No
Upgrades require contiguous major versions. For example, if an index was created in OpenSearch 1 and you're migrating to OpenSearch 3, it must first be reindexed into OpenSearch 2. -
What is the average and peak throughput of the cluster (in MBps)?
hundreds of MBps
Ensures the cluster operates within expected Migration Assistant limits. -
What is your indexing rate (writes per second)?
-
How many queries per second?
-
How many shards exist, and what is the maximum shard size?
Shard count and size impact performance and migration scalability. This also helps identify potential shard size limitations.
Tip: Use the following request to retrieve shard data:GET _cat/shards?v=true&s=index
Plugins
-
Have you reviewed plugin compatibility between your source and target environments?
Yes
Not all plugins are compatible across versions. Some may require alternatives or custom integrations. -
Are you using Kibana or any other visualization tools?
Yes - OpenSearch Dashboards
Ensures dashboards and visualizations are preserved or rebuilt post-migration.