[BUG] ObjectNode.get(String) invocation error because "this.rawJson" is null

[davidbaco]

What is the bug?

migration-console is throwing an error when running the console metadata evaluate command. I am migrating from ES 7.17.10 to OS 3.2.0 through an snapshot. The error is Cannot invoke "com.fasterxml.jackson.databind.node.ObjectNode.get(String)" because "this.rawJson" is null. I need help to troubleshoot further as the error itself is a bit generic and I do not understand what to do further. I have tried to specify only an index, but I get the same message. Live migration is not needed in this environment.

What are your migration environments?

• Source version: Elasticsearch 7.17.10
• Source environment: self-managed on-premises
• Target version: OpenSearch 3.2.0
• Target environment: self-managed on-premises
• Migration Assistant Version: 2.5.5

How can one reproduce the bug?

1. Compile the project and setup the K8s environment.
2. Mount the snapshot volume through a NFS share into the migration console Pod.
3. Access the migration console (kubectl -n ma exec -it deploy/ma-migration-console -c console -- /bin/bash)
4. Create a new snapshot in that mounted path (console snapshot create).
5. Verified the parameters in /conf/migration_services.yaml
6. When the snapshot is finished, run the command console metadata evaluate.

What is the expected behavior?

It is expected that the evaluation of the metadata migration will report any issues with the indices to migrate or confirm that all is good to migrate the metadata.

Do you have any additional context?

• We are not using any AWS component, but we installed the components to make ./gradlew build work.
• We forced to use K8s 1.32.9, as one of the pods doesn't start with the latest K8s version.

[gregschohn]

The error here may be because the snapshot metadata is compressed. We don't have support for that yet. You can try disabling that with compress: false (see here).
If that isn't it, we'd need some more details to understand. You could share the /conf/migration_services.yaml configuration and any of the logs from metadata migration for more details.

Thanks for the early feedback for K8s. I'm glad to see that you've discovered that work! For general background, the K8s work in the repo is very much in flux and is part of a future release (that can be tracked here). In the meantime, notice that migrationAssistant will be deprecated for migrationAssistantWithArgo, which will have new commands to interface with a migration (see the roadmap issues for more).

You can reach out on slack to myself or our team (#migrations) for a continued discussion. I'm very interested in the pod error that you got that prevented you from using a later version of K8s - if you could please add more details here or on slack, we can look into it.

[davidbaco]

Dear Greg ( @gregschohn ),

Thanks a lot for the help. I will try to reach you in the upcoming days through Slack to have a more deep sharing and explain better. Meanwhile I share here the parameters and logs (obviously I masked some information).

Configuration

source_cluster:
  allow_insecure: true
  endpoint: https://kibana.▓▓▓▓▓▓▓▓:9201
  basic_auth:
    password: ▓▓▓▓▓▓▓▓
    username: ▓▓▓▓▓▓▓▓
  version: ES_7.17.10
target_cluster:
  allow_insecure: true
  basic_auth:
    password: ▓▓▓▓▓▓▓▓
    username: ▓▓▓▓▓▓▓▓
  endpoint: https://opensearch.▓▓▓▓▓▓▓▓:9200
  version: OS_3.2
backfill:
  reindex_from_snapshot:
    k8s:
      deployment_name: ma-bulk-document-loader
      namespace: ma
replay:
  k8s:
    deployment_name: ma-replayer
    namespace: ma
snapshot:
  fs:
    repo_path: /srv/snapshots/migration
  snapshot_name: full_20250916
metadata_migration:
  cluster_awareness_attributes: 1
  from_snapshot:
    fs:
      repo_path: /srv/snapshots/migration
    snapshot_name: full_20250916
  source_cluster_version: ES_7.17
kafka:
  broker_endpoints: captured-traffic-kafka-bootstrap.ma.svc:9092
  standard:

Logs

2025-09-24 07:47:47,907 INFO o.o.m.u.ProcessHelpers [main] getNodeInstanceName()=ma-migration-console-5d548b6d97-8j899_61918a57-7be8-4977-aeb9-f870f730a08e
2025-09-24 07:47:47,909 WARN o.o.m.t.RootOtelContext [main] Collector endpoint=null, so serviceName parameter 'metadata' is being ignored since a no-op OpenTelemetry object is being created
2025-09-24 07:47:47,948 INFO o.o.m.MetadataMigration [main] Starting Metadata Evaluation
2025-09-24 07:47:47,951 INFO o.o.m.c.Evaluate [main] Running Metadata Evaluation
2025-09-24 07:47:47,968 INFO o.o.m.c.ClusterProviderRegistry [main] Found snapshot resource reader SnapshotReader_ES_7_10 for version: ELASTICSEARCH 7.17.0
2025-09-24 07:47:49,151 INFO o.o.m.c.ClusterProviderRegistry [main] Found remote writer for version: OPENSEARCH 3.2.0
2025-09-24 07:47:49,547 INFO o.o.m.b.c.OpenSearchClientFactory [main] Checking compression on cluster
2025-09-24 07:47:49,584 INFO o.o.m.b.c.OpenSearchClientFactory [reactor-http-epoll-3] After querying target, compression=false
2025-09-24 07:47:49,588 INFO o.o.m.b.t.TransformerMapper [main] Version-mapped transformer class selected: Transformer_ES_7_10_OS_2_11
2025-09-24 07:47:49,594 INFO o.o.m.MetadataTransformationRegistry [main] Default breaking changes transform config:
[{  "JsonJSTransformerProvider":{    "initializationResourcePath":"js/es-vector-knn-metadata.js",    "bindingsObject": "{}"  }},{  "JsonJSTransformerProvider":{    "initializationResourcePath":"js/metadataUpdater.js",    "bindingsObject": "{  \"rules\": [    {      \"when\": { \"type\": \"flattened\" },      \"set\": { \"type\": \"flat_object\" },      \"remove\": [\"index\"]    }  ]}"  }}]
2025-09-24 07:47:49,603 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider@33cbfa57
2025-09-24 07:47:49,604 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonConditionalTransformerProvider@6a0659ac
2025-09-24 07:47:49,606 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider@6cf31447
2025-09-24 07:47:49,607 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJoltTransformerProvider@6f3f0fae
2025-09-24 07:47:49,607 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJSTransformerProvider@4a68135e
2025-09-24 07:47:49,619 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.JsonJMESPathTransformerProvider@6c03fb16
2025-09-24 07:47:49,620 INFO o.o.m.t.TransformationLoader [main] Adding IJsonTransfomerProvider: org.opensearch.migrations.transform.NoopTransformerProvider@3e7545e8
2025-09-24 07:47:49,621 INFO o.o.m.t.TransformationLoader [main] IJsonTransformerProviders loaded: class org.opensearch.migrations.transformation.JsonTransformerForDocumentTypeRemovalProvider; class org.opensearch.migrations.transform.JsonConditionalTransformerProvider; class org.opensearch.migrations.transform.TypeMappingSanitizationTransformerProvider; class org.opensearch.migrations.transform.JsonJoltTransformerProvider; class org.opensearch.migrations.transform.JsonJSTransformerProvider; class org.opensearch.migrations.transform.JsonJMESPathTransformerProvider; class org.opensearch.migrations.transform.NoopTransformerProvider
2025-09-24 07:47:49,639 INFO o.o.m.t.TransformationLoader [main] Creating a transformer through provider=org.opensearch.migrations.transform.JsonJSTransformerProvider@4a68135e with configuration={initializationResourcePath=js/es-vector-knn-metadata.js, bindingsObject={}}
2025-09-24 07:47:50,572 INFO o.o.m.t.TransformationLoader [main] Creating a transformer through provider=org.opensearch.migrations.transform.JsonJSTransformerProvider@4a68135e with configuration={initializationResourcePath=js/metadataUpdater.js, bindingsObject={  "rules": [    {      "when": { "type": "flattened" },      "set": { "type": "flat_object" },      "remove": ["index"]    }  ]}}
2025-09-24 07:47:50,640 INFO o.o.m.c.MigratorEvaluatorBase [main] Selected transformer composite: custom = Transformers, version = Transformer_ES_7_10_OS_2_11
2025-09-24 07:47:50,660 INFO o.o.m.b.w.MetadataRunner [main] Migrating the Templates...
[▓▓▓▓▓▓▓▓... (trimmed as this file is public)]
2025-09-24 07:47:53,142 INFO o.o.m.b.w.MetadataRunner [main] Templates migration complete
2025-09-24 07:47:53,142 INFO o.o.m.c.MigratorEvaluatorBase [main] Metadata copy complete.
2025-09-24 07:47:53,147 INFO o.o.m.b.c.OpenSearchClient [main] Starting getAwarenessAttributeSettings call to path=_cluster/settings?flat_settings&include_defaults
2025-09-24 07:47:53,160 INFO o.o.m.b.c.OpenSearchClient [main] Completed getAwarenessAttributeSettings in 13 ms with statusCode=200
2025-09-24 07:47:53,204 INFO o.o.m.b.w.IndexRunner [main] Index .ds-metrics-elastic_agent.elastic_agent-default-2024.09.05-000013 was not part of the allowlist and will not be migrated.
2025-09-24 07:47:53,212 ERROR o.o.m.c.Evaluate [main] Unexpected failure
java.lang.NullPointerException: Cannot invoke "com.fasterxml.jackson.databind.node.ObjectNode.get(String)" because "this.rawJson" is null
	at org.opensearch.migrations.bulkload.version_es_7_10.IndexMetadataData_ES_7_10.getAliases(IndexMetadataData_ES_7_10.java:39)
	at org.opensearch.migrations.bulkload.version_es_7_10.IndexMetadataData_ES_7_10.getAliases(IndexMetadataData_ES_7_10.java:13)
	at org.opensearch.migrations.bulkload.worker.IndexRunner.migrateIndices(IndexRunner.java:55)
	at org.opensearch.migrations.commands.MigratorEvaluatorBase.migrateIndices(MigratorEvaluatorBase.java:150)
	at org.opensearch.migrations.commands.MigratorEvaluatorBase.migrateAllItems(MigratorEvaluatorBase.java:116)
	at org.opensearch.migrations.commands.Evaluate.execute(Evaluate.java:31)
	at org.opensearch.migrations.MetadataMigration.runCommand(MetadataMigration.java:106)
	at org.opensearch.migrations.MetadataMigration.run(MetadataMigration.java:67)
	at org.opensearch.migrations.MetadataMigration.main(MetadataMigration.java:29)
2025-09-24 07:47:53,221 INFO o.o.m.MetadataMigration [main] Clusters:
   Source:
      Type: Snapshot (ELASTICSEARCH 7.17.0)
      Local repository: /srv/snapshots/migration

   Target:
      Type: Remote Cluster (OPENSEARCH 3.2.0)
      Uri: https://opensearch.▓▓▓▓▓▓▓▓:9200
      Protocol: HTTPS
      TLS Verification: Disabled


Transformations:
   dense_vector to knn_vector:
      Convert field data type dense_vector to OpenSearch knn_vector
   flattened to flat_object:
      Convert field data type flattened to OpenSearch flat_object
   Version Transform:
      Other transforms for source to target shape conversion

Results:
   Issue(s) detected
Issues:
   Unexpected failure: Cannot invoke "com.fasterxml.jackson.databind.node.ObjectNode.get(String)" because "this.rawJson" is null

2025-09-24 07:47:53,230 INFO o.o.m.MetadataMigration [main] Consult /root/logs/ma-migration-console-5d548b6d97-8j899/metadata/metadata2025-09-24_07-47-47.log to see detailed logs for this run
2025-09-24 07:47:53,230 INFO o.o.m.MetadataMigration [main] See transformations applied for this run at /root/logs/ma-migration-console-5d548b6d97-8j899/metadata/metadata2025-09-24_07-47-47-transforms.log

Transformation log

Default breaking changes transform config
[ {
  "JsonJSTransformerProvider" : {
    "initializationResourcePath" : "js/es-vector-knn-metadata.js",
    "bindingsObject" : "{}"
  }
}, {
  "JsonJSTransformerProvider" : {
    "initializationResourcePath" : "js/metadataUpdater.js",
    "bindingsObject" : "{  \"rules\": [    {      \"when\": { \"type\": \"flattened\" },      \"set\": { \"type\": \"flat_object\" },      \"remove\": [\"index\"]    }  ]}"
  }
} ]

[AndreKurait]

@davidbaco how was your full_20250916 taken? Did you set compress: false on the repo prior to taking the snapshot and pass include_global_metadata: true on the snapshot?

Can you send your repo settings and if that still doesn't work, use a new repo and try again?

[davidbaco]

Hi again! (sorry for the late replies)

The process to take the snapshot was as follow (if I remember correctly):

1. Created the repository migration_assistant_repo. It has compression disabled.
2. Triggered one snapshot through the DEV Tools in Elastic.
3. This snapshot (full_20250916) was created afterwards into the same repository through the migration tool.

The repository for those snapshots is a NFS Share what allows to mount wherever is necessary. It is accessible in the migration assistant, and Elastic servers using the same mount point (/srv/snapshots/migration).

I am not sure about how to check the include_global_metadata, but I think it is related to the include_global_state parameter.

The snapshot information (anonymized) is this (GET /_snapshot/migration_assistant_repo/full_20250916):

{
  "snapshots" : [
    {
      "snapshot" : "full_20250916",
      "uuid" : "GQeK-DweSye6jb8ap1kfyQ",
      "repository" : "migration_assistant_repo",
      "version_id" : 7171099,
      "version" : "7.17.10",
      "indices" : [ ▓▓▓ (removed)
      ],
      "data_streams" : [ ▓▓▓ (removed)
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2025-09-16T13:36:52.730Z",
      "start_time_in_millis" : 1758029812730,
      "end_time" : "2025-09-16T14:30:58.045Z",
      "end_time_in_millis" : 1758033058045,
      "duration_in_millis" : 3245315,
      "failures" : [ ],
      "shards" : {
        "total" : 931,
        "failed" : 0,
        "successful" : 931
      },
      "feature_states" : [ ▓▓▓ (removed)
      ]
    }
  ],
  "total" : 1,
  "remaining" : 0
}

The repository information is this (GET /_snapshot/migration_assistant_repo):

{
  "migration_assistant_repo" : {
    "type" : "fs",
    "uuid" : "BtJ3U8JXRiuP_MRmua_YZQ",
    "settings" : {
      "compress" : "false",
      "location" : "/srv/snapshots/migration"
    }
  }
}

Thanks, and again sorry for the delays...