[Bug]: AWS elasticache - Redis to Valkey migration

[bhargavtheertham-cb]

Terraform Core Version

v1.7.3

AWS Provider Version

5.84.0

Affected Resource(s)

We are working on converting AWS elasticache engine type from Redis to Valkey. AWS console allows us to update the engine to valkey and choose a new parameter group , it is a fully online operation with may be a few seconds of down. Th endpoint also remains the same so fully transparent to the application. When using terraform it seems to destroy and recreate the resource which would make us lose data and take downtime.

Expected Behavior

It should update it in place without losing data as it is done via AWS Console

Actual Behavior

-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_elasticache_cluster.cache_infra_datastores_resources_staging_redis_stage_redis_1 must be replaced <---
-/+ resource "aws_elasticache_cluster" "cache_infra_datastores_resources_staging_redis_stage_redis_1" {
      ~ arn                        = "arn:aws:elasticache:us-east-1:038351868345:cluster:datastores-resources" -> (known after apply)
      ~ availability_zone          = "us-east-1f" -> (known after apply)
      ~ az_mode                    = "single-az" -> (known after apply)
      ~ cache_nodes                = [
          - {
              - address           = "datastores-resources.pwf76b.0001.use1.cache.amazonaws.com"
              - availability_zone = "us-east-1f"
              - id                = "0001"
              - port              = 6379
                # (1 unchanged attribute hidden)
            },
        ] -> (known after apply)
      + cluster_address            = (known after apply)
      + configuration_endpoint     = (known after apply)
      ~ engine                     = "redis" -> "valkey" # forces replacement <---------
      ~ engine_version             = "6.x" -> "8.0"
      ~ engine_version_actual      = "6.2.6" -> (known after apply)
      ~ id                         = "datastores-resources" -> (known after apply)
      ~ ip_discovery               = "ipv4" -> (known after apply)
      ~ maintenance_window         = "fri:09:30-fri:10:30" -> (known after apply)
      ~ network_type               = "ipv4" -> (known after apply)
      ~ parameter_group_name       = "infra-datastores-resources-staging-redis-pg" -> "infra-datastores-resources-staging-valkey-pg"
      + preferred_outpost_arn      = (known after apply)
      + replication_group_id       = (known after apply)
      - snapshot_retention_limit   = 0 -> null
      ~ snapshot_window            = "05:00-06:00" -> (known after apply)
 

### Relevant Error/Panic Output Snippet

```shell

Terraform Configuration Files

NA

Steps to Reproduce

NA

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[Angelicvorian]

I'm trying to do the same thing and have the same request.
I'm using Terraform 1.9.8 with AWS provider 5.87.0

[Angelicvorian]

I wanted to follow-up on this.
I've been doing some testing with different Redis cluster configurations and have found that Terraform DOES support doing this.
The problem I've found is this: If resources that the replication group depends on have their name changed (maybe engine is part of your naming convention for resources) then this will force the replication group to be re-created.
If you amend the resources so this is not the case, then you can prevent the change causing the destroy/create operation and it will instead allow it to update in place instead.
I've tested this a Non-clustered redis cache and a clustered redis cache and it is allowing the engine change in-place.
In my case, I was using a var for engine in the naming in other resources, so I ended up using a lifecycle rule to ignore changes to the name of a few different resources (security group, iam_user_policy, subnet_group).
Also the engine change was ignored in the aws_elasticache_user and aws_elasticache_user_group.
And lastly, I made sure the parameter group was set to use 'create_before_destroy = true'.

[adifdwimaulana]

Hi @Angelicvorian
I tried to add lifecycle create_before_destroy = true to my aws_elasticache_parameter_group. But I've got error name already exists during terraform apply. Do you encounter similar issue? Thanks

[bengaywins]

Out of curiosity, has anyone tried to remove from state and import to see if that at least works around this while it's a problem?

[adifdwimaulana]

I actually have tried by updating the engine, engine_version, and parameter_group_name following valkey parameter and it is working seamlessly in my case. There is no data loss and only a minor downtime < 10seconds

Out of curiosity, has anyone tried to remove from state and import to see if that at least works around this while it's a problem?

[YakDriver]

This is a critical data safety issue that's affecting users migrating to AWS's new Valkey engine. I've identified some related ElastiCache issues that could be addressed together:

Core Related Issues (same effort):

• Can't Create or import valkey elasticache with cluster mode enabled with single node from terraform #43233 - Valkey cluster mode creation fails with single nodes - directly related to Valkey engine support
• [Bug]: CacheClusterNotFound: Specified snapshotting cluster does not exist #41567 - Snapshot functionality broken due to hardcoded node naming assumptions - affects ElastiCache lifecycle management

These three issues together cover the migration → creation → backup workflow for Valkey clusters and could be tackled as a coordinated ElastiCache improvement effort.

Same Resource, Separate Efforts:

• aws_elasticache_cluster missing description argument #44639 - Missing description argument in aws_elasticache_cluster - simple enhancement, good quick win
• [New Data Source]: Request for aws_elasticache_cluster"s" and aws_elasticache_replication_group"s"  #34343 - Data source requests for listing clusters - larger feature implementation

The Redis→Valkey migration issue is the highest priority due to data loss risk, but fixing the underlying ElastiCache resource handling could resolve multiple pain points simultaneously. Anyone working on this might want to consider the Valkey-specific cluster mode issue (#43233) and snapshot bug (#41567) as part of the same effort.

[sukumaar]

Summary

The aws_elasticache_cluster Terraform resource accepts "valkey" as a valid engine parameter, but the underlying AWS CreateCacheCluster API only supports "memcached" and "redis" engines. This mismatch causes destructive behavior: Terraform attempts to destroy and recreate the cluster rather than performing an in-place upgrade. To migrate from Redis to Valkey, users must use aws_elasticache_replication_group or aws_elasticache_serverless_cache, which supports Valkey through CreateReplicationGroup API and CreateServerlessCache respectively.

Root Cause: API Limitation

The AWS ElastiCache CreateCacheCluster API explicitly restricts the Engine parameter to two values: memcached and redis. According to AWS documentation on creating ElastiCache clusters, cluster mode enabled configurations (which support Valkey) cannot be created using CreateCacheCluster and must instead use CreateReplicationGroup. Terraform's aws_elasticache_cluster resource wraps the CreateCacheCluster AWS API, so it inherits this limitation.

Despite this API constraint, Terraform provider versions 5.73.0 and later incorrectly accepted "valkey" as valid input for aws_elasticache_cluster, creating the false expectation that the operation would succeed.
See :

• Terraform GitHub Issue #39972 for details.
• Terraform Github Issue #39972 Comment.

Migration Behaviour Issue

When users attempt to change an existing Redis cluster's engine to Valkey using aws_elasticache_cluster, Terraform marks the resource for destruction and recreation rather than an in-place modification. This occurs because modifying the engine parameter forces Terraform to replace the resource entirely, resulting in:

• Data loss (without manual snapshot management)
• Unexpected downtime (cluster becomes unavailable)
• Failed upgrade (the underlying API rejects the Valkey request)

In contrast, the aws_elasticache_replication_group resource uses the CreateReplicationGroup API, which fully supports Valkey as an engine option and enables true in-place engine modifications.

Solution: Migration Path

Fix

aws_elasticache_cluster currently accepts "engine" = "valkey" as a valid input. This leads to a destructive behaviour: the cluster is destroyed and then fails to upgrade to Valkey. In the linked PR 45017, "valkey" has been removed as a valid input for the engine argument in aws_elasticache_cluster. This aligns the Terraform resource behaviour with the underlying AWS API capabilities. Once this PR will get merged, terraform will stop at validation/ plan stage and will not delete the aws_elasticache_cluster.

For migrating existing Redis clusters to Valkey:

Users who previously created ElastiCache clusters using aws_elasticache_cluster and wish to migrate to Valkey should:

• Take a snapshot of the existing Redis cluster.
• Create a new Valkey cluster using either: aws_elasticache_replication_group, or aws_elasticache_serverless_cache and import the snapshot during creation to preserve data.
• Update application connection strings to point to the new replication group endpoint
• Only after verification, delete the previous elasticache resource.