caching in Distributed Data not working

[DevOnBike]

Version Information

Describe the bug
I want to prepare POC for distributed caching in akka.net cluster.
I implemented methods for set cache, get cache and remove cache but they are not working properly, when i set cache I see some warnings in logs (it looks like cache is not set properly) and later get cache results in empty value.
I'm clearly missing something but I am not sure what baseing on docs https://getakka.net/articles/clustering/distributed-data.html.

Can u help with this issue ?

To Reproduce
https://github.com/DevOnBike/webapi-playground/blob/main/Messaging/DistributedCache.cs

Expected behavior

Actual behavior
wrong cache behavior.

Screenshots
all in source code in repo.

Environment
windows

Additional context
Add any other context about the problem here.

[Aaronontheweb]

Can you please include some of the logs you're seeing?

[DevOnBike]

[DevOnBike]

WARNING][03.24.2025 17:28:34.995Z][Thread 0005][akka.ssl.tcp://devonbike-cluster@localhost:55555/system/endpointManager/reliableEndpointWriter-akka.ssl.tcp%3A%2F%2Fdevonbike-cluster%40localhost%3A51640-1/endpointWriter/endpointReader-akka.ssl.tcp%3A%2F%2Fdevonbike-cluster%40localhost%3A51640-1] Deserialization failed for message with serializer id [6] and manifest []. Transient association error (association remains live). Failed to deserialize payload object when deserializing ActorSelectionMessage with payload [SerializerId=12, Manifest=Q] addressed to [user,dd-replicator]. Serializer Id [12] is used to instantiate [Akka.DistributedData.Serialization.ReplicatorMessageSerializer, Akka.DistributedData]. You can add it by adding a fallback to your ActorSystem configuration by using [DistributedData.DefaultConfig() or ClusterSharding.DefaultConfig()]. It is also automatically injected into your configuration when you call [ClusterSharding.Get(), DistributedData.Get()].. Serializer Id [6] is used to instantiate [Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote]. You can add it by adding a fallback to your ActorSystem configuration by using [RemoteConfigFactory.Default()].
Cause: System.Runtime.Serialization.SerializationException: Failed to deserialize payload object when deserializing ActorSelectionMessage with payload [SerializerId=12, Manifest=Q] addressed to [user,dd-replicator]. Serializer Id [12] is used to instantiate [Akka.DistributedData.Serialization.ReplicatorMessageSerializer, Akka.DistributedData]. You can add it by adding a fallback to your ActorSystem configuration by using [DistributedData.DefaultConfig() or ClusterSharding.DefaultConfig()]. It is also automatically injected into your configuration when you call [ClusterSharding.Get(), DistributedData.Get()].
---> System.Runtime.Serialization.SerializationException: Cannot find serializer with id [12] (manifest [Q]). The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems. Serializer Id [12] is used to instantiate [Akka.DistributedData.Serialization.ReplicatorMessageSerializer, Akka.DistributedData]. You can add it by adding a fallback to your ActorSystem configuration by using [DistributedData.DefaultConfig() or ClusterSharding.DefaultConfig()]. It is also automatically injected into your configuration when you call [ClusterSharding.Get(), DistributedData.Get()].
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.Serialization.MessageContainerSerializer.FromBinary(Byte[] bytes, Type type)
--- End of inner exception stack trace ---
at Akka.Remote.Serialization.MessageContainerSerializer.FromBinary(Byte[] bytes, Type type)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, Payload message, IActorRef senderOption)
at Akka.Remote.EndpointReader.b__11_0(InboundPayload inbound)

[DevOnBike]

better if u run app: just start webapi and webapi2.
There is ClusterController on each to play with caching.

[DevOnBike]

any news?

[Aaronontheweb]

Looks like a simple configuration error - you're missing a serializer definition - the error message tells you how to fix it:

You can add it by adding a fallback to your ActorSystem configuration by using [DistributedData.DefaultConfig() or ClusterSharding.DefaultConfig()]. It is also automatically injected into your configuration when you call [ClusterSharding.Get(), DistributedData.Get()].. 

Serializer Id [6] is used to instantiate [Akka.Remote.Serialization.MessageContainerSerializer, Akka.Remote]. You can add it by adding a fallback to your ActorSystem configuration by using [RemoteConfigFactory.Default()].

[DevOnBike]

so after digging in akka.net code I was able to add default configuration like this:

var hocon = akkaConfig.ToHocon().WithFallback(DistributedData.DefaultConfig());

akkaBuilder.AddHocon(hocon, HoconAddMode.Replace)

i can't use your code like
var hocon = akkaConfig.ToHocon().WithFallback(RemoteConfigFactory.Default());
as RemoteConfigFactory is internal class

now I am getting this error on sending somethinh to cache:
[INFO][04.01.2025 22:17:58.774Z][Thread 0008][akka://devonbike-cluster/user/dd-replicator] Message [DeltaPropagation] from [akka.ssl.tcp://devonbike-cluster@localhost:55555/user/dd-replicator#1325076703] to [akka://devonbike-cluster/user/dd-replicator] was not delivered. [1] dead letters encountered. If this is not an expected behavior then [akka://devonbike-cluster/user/dd-replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Message content: Akka.DistributedData.Internal.DeltaPropagation

all changes are here: DevOnBike/webapi-playground@ccabdc5

[Aaronontheweb]

@DevOnBike I'll give this a look tomorrow - thanks for all of your hard work reproducing this

[DevOnBike]

@DevOnBike I'll give this a look tomorrow - thanks for all of your hard work reproducing this

no problem, happy to help :)

[Aaronontheweb]

Resolved via DevOnBike/webapi-playground#2 - although IMHO this is a bug. The if you use .WithDistributedData(options => options.Name = "dd-replicator") in Akka.Hosting that should launch the replicator automatically.