WithExpireAfter and AsAtomic

[MrSmoke]

Just a quick question as I couldn't find anything in the docs.

It seems that having a cache with WithExpireAfter and WithAtomicGetOrAdd is not supported as System.InvalidOperationException : WithExpireAfter is not compatible with AsAtomic is thrown when building the cache.

I was just wondering what the reason for this was and if its something that can be worked around?

[bitfaster]

This is a bit fiddly but I will try to explain and provide the design rationale as it evolved. The short answer is that the complete API surface will not work correctly.

WithExpireAfter is implemented internally as follows (this is from the factory that instantiates the cache for this builder method):

private static ICache<K, V> CreateExpireAfter<K, V, TP>(LruInfo<K> info, IExpiryCalculator<K, V> expiry) 
   where K : notnull where TP : struct, ITelemetryPolicy<K, V>
{
        return new ConcurrentLruCore<K, V, LongTickCountLruItem<K, V>, DiscretePolicy<K, V>, TP>(
            info.ConcurrencyLevel, info.Capacity, info.KeyComparer, new DiscretePolicy<K, V>(expiry), default);
}

The ConcurrentLruCore class contains the cache algorithm, and the generic arg DiscretePolicy determines how the core algorithm should expire items. The cached value V is wrapped in LongTickCountLruItem<K, V> when stored in the cache. This LRU item stores the time that is used to expire the item. The policy and the item must be matched, because the policy must at compile time know how to access the time property of the item. The overly complicated generics (and other tricks like making the policy a struct), enable the JIT to inline and elide policy code as if this had all been handwritten for speed, even on .NET Framework. When there is no time expiry, the item does not have a timestamp property, which shrinks the item from 32 bytes to 24 bytes. Long story short, all of this was done to absolutely squeeze performance and reduce overhead.

Atomic GetOrAdd is functionally equivalent to caching a Lazy, where the cached value is wrapped in an AtomicFactory object that guarantees atomic creation. By making this a standalone class, it can be re-used with ConcurrentLfu, or used in isolation.

When calling WithExpireAfter, effectively, the LRU item wrapper is nested inside the Atomic wrapper, so if atomic and expire after were combined by hand the generic class definition will now like this:

// Open generic looks like ConcurrentLruCore<K, V, I, P, T>

public sealed class ExpireAfterLru<K, V> : ConcurrentLruCore<
   K,                            // K = key
   V,                            // V = value
   LongTickCountLruItem<K, V>,   // I = item
   DiscretePolicy<K, V>,         // P = expiry policy (determines how to expire items)
   TelemetryPolicy<K, >>         // T = telemetry policy (enables hit counting/event handling)
{}

public sealed class ExpireAfterAtomic<K, V> : ConcurrentLruCore<
   K,                                             // K = key
   AtomicFactory<K, V>,                           // V = value (which is now wrapped via AtomicFactory)
   LongTickCountLruItem<K, AtomicFactory<K, V>>,  // I = item
   DiscretePolicy<K, AtomicFactory<K, V>>,        // P = expiry policy (determines how to expire items)
   TelemetryPolicy<K, AtomicFactory<K, V>>>       // T = telemetry policy (enables hit counting/event handling)
{}

This atomic behavior can now be layered in, without penalty to the non-atomic mode. My thinking was that atomic is anyway going to take more of a perf hit because it is always more instructions to execute, so better to compromise atomic perf than non-atomic perf.

If you take a look at the implementation of AtomicFactoryCache, you will see there are some specializations for methods like GetOrAdd. In the raw ConcurrentLruCore class, the equivalent method invokes the policy with V as input.

But with atomic enabled, in sequence we call:

1. AtomicFactoryCache.GetOrAdd
2. ConcurrentLruCore.GetOrAdd (cached value in ConcurrentLruCore is an AtomicFactory with null V)
3. AtomicFactory.GetValue (V is initialized)

Notice that further above, when we defined the closed cache type with all the generic parameters, the cache policy generic is DiscretePolicy<K, AtomicFactory<K, V>>, this means the expiry calculator must implementIExpiryCalculator<K, AtomicFactory<K, V>>. Thus, at step 2 above, when the policy is invoked with an AtomicFactory with null V, the expiry calculator will also be called with a null V. This is perhaps overly restrictive, but I did not want to expose callers to this bug because I felt it would be confusing.

In the current design, it is not possible to re-order these events so the value instantiated by AtomicFactory is passed into the cache policy.

In the longer run, I would like to change the cache policy interface so there is a way to make this work correctly without regressing performance for other cases (i.e. when not using atomic or whatever). That would be a breaking change, because the cache policy is public. My plan is to make this all internal in v3, then I can iterate on it and clean it up.