Performance loss when upgrading from Dalli 2.7.10 to 3.2.2

[matthewzinke]

We attempted to upgrade our Dalli gem from 2.7.10 to 3.2.2 recently, and noticed a performance loss associated with the upgrade. Out memcached latency went up by just under 10% and our sidekiq processes started spending 2-3x as much time on memcached. We suspect this has something to do with the change from send_multiget to pipelined_get.

We are wondering if anyone has also seen performance loss associated with the upgrade from 2 to 3, and/or if there are any configuration recommendations to preserve performance characteristics.

[jasonpenny]

send_multiget looks like it was removed in https://github.com/petergoldstein/dalli/pull/852/files?diff=unified&w=1#diff-a326ee7acf56937d92cdeeb4b920547a89162170aab2ad32391200ad1b4c3638L457

was this removed just to simplify the code to always use PipelinedGetter @petergoldstein ?

[petergoldstein]

@jasonpenny The send_multiget method was simply renamed to pipelined_get in that commit to get some naming consistency. The latter method is still there, and is now invoked by the PipelinedGetter.

It is certainly possible that there's a performance loss here (possibly in the ResponseBuffer), but I'd need to get some more detail to start debugging.

[jasonpenny]

ah, I see.

I see a slowdown with a benchmark using ActiveSupport::Cache::MemCacheStore:

root@d6ba62960da5:/# DALLI_VERSION=2.7.10 ruby bench.rb
                                            user     system      total        real
2.7.10 read_multi                       8.122310   2.830021  10.952331 ( 12.855987)
root@d6ba62960da5:/# DALLI_VERSION=2.7.10 ruby bench.rb
                                            user     system      total        real
2.7.10 read_multi                       8.037037   2.876596  10.913633 ( 12.790800)

root@d6ba62960da5:/# DALLI_VERSION=3.2.2 ruby bench.rb
                                            user     system      total        real
3.2.2 read_multi                        9.198624   2.890378  12.089002 ( 13.673892)
root@d6ba62960da5:/# DALLI_VERSION=3.2.2 ruby bench.rb
                                            user     system      total        real
3.2.2 read_multi                        9.372887   2.795796  12.168683 ( 13.765607)

benchmark setup in a docker container

docker run -it --rm "ruby:2.7.3" bash

apt-get update -qq
apt-get install -y sudo
useradd test

git clone --depth 1 --branch main https://github.com/petergoldstein/dalli dalli

MEMCACHED_VERSION=1.6.10 bash dalli/scripts/install_memcached.sh

echo "
# frozen_string_literal: true
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'activesupport', '6.1.6.1'
  gem 'dalli', ENV['DALLI_VERSION']
end

require 'benchmark'
require 'active_support/all'

Benchmark.bm(37) do |x|
  x.report(%Q{#{ENV['DALLI_VERSION']} read_multi}) do
    c = ActiveSupport::Cache::MemCacheStore.new(['127.0.0.1:23417'])

    keys = ['One', 'Two' * 8, 'Three' * 40]
    keys.each { c.write(_1, rand.to_s * _1.length) }

    100_000.times do
      c.read_multi(*keys)
    end
  end
end
" > bench.rb

memcached -p 23417 -u test &

DALLI_VERSION=2.7.10 ruby bench.rb
DALLI_VERSION=3.2.2 ruby bench.rb

[jasonpenny]

I am no expert on profiling ruby, but I tried a few tools and did not find any obvious hotspots.

Something I did notice was that the callgraph of dalli v3 is larger, and I wonder if calling extra methods/more objects would be the cause of slowness here since we're calling it so often.

capture callgraph

# frozen_string_literal: true

require 'fileutils'
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'activesupport', '6.1.6.1'
  gem 'stackprof'
  gem 'dalli', path: ENV['DALLI_PATH']
  gem 'byebug'
  gem 'ruby-prof'
end

puts "Dalli version: #{Dalli::VERSION}"

dc = Dalli::Client.new('127.0.0.1:11211')
dc.flush_all
keys = ['One', 'Two' * 8, 'Three' * 40]
keys.each { dc.set(_1, rand.to_s * _1.length) }
dc = nil

RubyProf.start
c = Dalli::Client.new('127.0.0.1:11211', socket_timeout: 5)
values = nil
10.times do
  values = c.get_multi(*keys)
end
results = RubyProf.stop
puts values

output_dir = "./#{ENV['DALLI_PATH']}_calltree" 
FileUtils.mkdir_p(output_dir)
RubyProf::CallTreePrinter.new(results).print({ path: output_dir })

Dalli 2

Dalli 3

callgrind profiles: https://gist.github.com/jasonpenny/fed81267c8e0c987e63839a135f9db4a

[taiyyu]

We also saw performance degradation. We had to disable compress to get performance matching 2.7.x while using :mem_cache_store in ruby 3.1.3, rails 6.1.4, and dalli 3.2.x.

It would be nice to re-enable compress and still be able to upgrade to 3.2.x in the future. Have not been able to root cause exactly why, but hopefully, this gives a little clue towards a fix.

[petergoldstein]

This issue is nearly 3 years old. Since then, Dalli has received significant performance improvements:

In Dalli 4.2.0:

• Buffered I/O using socket.sync = false with explicit flush to reduce syscalls
• get_multi optimizations using Set for O(1) server tracking lookups
• Raw mode optimization saving 2 bytes per request

In Dalli 4.3.0:

• Interleaved read/write for pipelined gets to prevent socket buffer deadlock
• Chunked requests for large key batches

Could you please test with Dalli 4.3.0 and report if you still see performance issues compared to 2.7.10?

If we don't hear back within 30 days, we'll close this issue. Please reopen if you can provide updated benchmark data.

[jasonpenny]

If we don't hear back within 30 days, we'll close this issue. Please reopen if you can provide updated benchmark data.

The benchmark from above still shows the latest version, 4.3.0, consistently takes more time (I just tweaked the setup for the Ruby version I'm using now (3.2.9), and changes related to that).

I think you can close this issue now, though, as I won't be able to try out new versions in production anytime soon.

# DALLI_VERSION=2.7.10 ruby bench.rb 2>/dev/null
                                            user     system      total        real
2.7.10 read_multi                       4.793105   0.918976   5.712081 (  7.795677)
# DALLI_VERSION=2.7.10 ruby bench.rb 2>/dev/null
                                            user     system      total        real
2.7.10 read_multi                       4.719885   0.992058   5.711943 (  7.763094)
# DALLI_VERSION=2.7.10 ruby bench.rb 2>/dev/null
                                            user     system      total        real
2.7.10 read_multi                       4.753013   0.928492   5.681505 (  7.754576)



# DALLI_VERSION=4.3.0 ruby bench.rb 2>/dev/null
                                            user     system      total        real
4.3.0 read_multi                        5.315420   0.883568   6.198988 (  8.377180)
# DALLI_VERSION=4.3.0 ruby bench.rb 2>/dev/null
                                            user     system      total        real
4.3.0 read_multi                        5.736776   0.822704   6.559480 (  8.745277)
# DALLI_VERSION=4.3.0 ruby bench.rb 2>/dev/null
                                            user     system      total        real
4.3.0 read_multi                        5.813757   0.768453   6.582210 (  8.781631)

---

benchmark setup in a docker container:

docker run -it --rm "ruby:3.2.9" bash

apt-get update -qq
apt-get install -y sudo
useradd test

git clone --depth 1 --branch main https://github.com/petergoldstein/dalli dalli

CFLAGS="-Wno-array-bounds" MEMCACHED_VERSION=1.6.10 bash dalli/scripts/install_memcached.sh

echo "
# frozen_string_literal: true
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'activesupport', '6.1.6.1'
  gem 'dalli', ENV['DALLI_VERSION']
end

require 'logger'
require 'benchmark'
require 'active_support/all'

Dalli.logger.level = Logger::ERROR

Benchmark.bm(37) do |x|
  x.report(%Q{#{ENV['DALLI_VERSION']} read_multi}) do
    c = ActiveSupport::Cache::MemCacheStore.new(['127.0.0.1:23417'])

    keys = ['One', 'Two' * 8, 'Three' * 40]
    keys.each { c.write(_1, rand.to_s * _1.length) }

    100_000.times do
      c.read_multi(*keys)
    end
  end
end
" > bench.rb

memcached -p 23417 -u test &

DALLI_VERSION=2.7.10 ruby bench.rb
DALLI_VERSION=4.3.0 ruby bench.rb

benchmark script split out for better highlighting / readability

# frozen_string_literal: true
require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'activesupport', '6.1.6.1'
  gem 'dalli', ENV['DALLI_VERSION']
end

require 'logger'
require 'benchmark'
require 'active_support/all'

Dalli.logger.level = Logger::ERROR

Benchmark.bm(37) do |x|
  x.report(%Q{#{ENV['DALLI_VERSION']} read_multi}) do
    c = ActiveSupport::Cache::MemCacheStore.new(['127.0.0.1:23417'])

    keys = ['One', 'Two' * 8, 'Three' * 40]
    keys.each { c.write(_1, rand.to_s * _1.length) }

    100_000.times do
      c.read_multi(*keys)
    end
  end
end

[petergoldstein]

If you're interested @jasonpenny there's an open PR - #1072 - with some proposed performance enhancements and benchmarks. There's still a degradation relative to 2.7.11, but it appears to be something of a tradeoff with maintainability.

[danmayer]

OK, I think that for some use cases this should also help if you are not using rings, also I looked at the recent work Peter has done on 5.0.x will be a huge win for many use cases.

#1077

Take a look at some of the recent work @jasonpenny and if you have good usage examples try running against the latest on main of the PRs and see if these get back to equal performance for you.

[jasonpenny]

Thanks for the performance enhancement, that seems like a great result and I completely understand choosing maintainability, but I was unable to convince decision makers here that the perf degradation that we saw when we initially rolled out an upgrade were worth it (I think we'll eventually have to upgrade to get the meta protocol, but I'm no longer on the team owning our use of Dalli).

Take a look at some of the recent work @jasonpenny and if you have good usage examples try running against the latest on main of the PRs and see if these get back to equal performance for you.

As Peter said, There's still a degradation relative to 2.7.11, and it can be seen with the benchmark above (against the latest tag, v5.0.1, from 2 days ago [with Ruby 3.3.10 since 3.2 is too old for Dalli v5])

                                            user     system      total        real
2.7.11 read_multi                       4.873815   0.970103   5.843918 (  8.029130)
2.7.11 read_multi                       4.832875   0.972767   5.805642 (  7.986168)
2.7.11 read_multi                       5.126391   0.961622   6.088013 (  8.261377)
2.7.11 read_multi                       4.935165   0.962930   5.898095 (  8.038055)

5.0.1 read_multi                        6.712107   0.876509   7.588616 ( 10.001765)
5.0.1 read_multi                        6.025294   0.841125   6.866419 (  9.326584)
5.0.1 read_multi                        6.262599   0.834268   7.096867 (  9.552737)

[mhenrixon]

We accidentally upgraded dalli a second time to 5.x and both times it just completely stops working for performance reasons. 3.2.x works like a charm though

[petergoldstein]

Updated benchmarks following the recent 5.0.x allocation work — running 3 passes each to get reliable means. Environment: Apple Silicon (arm64), Ruby 4.0.2, memcached 1.6.41. Numbers won't match prior Docker/x86 runs directly, but the relative story is what matters.

Results (mean real time, 100k iterations, single server, 3 keys):

                      get (single)   get_multi    read_multi (AS)
────────────────────────────────────────────────────────────────
2.7.11                2.495s         5.189s       7.687s
5.0.1                 2.806s (+12%)  6.097s (-17%) 8.817s (-15%)
5.0.2                 2.806s (+12%)  4.005s (+23%) 6.867s (+11%)
5.0.3                 2.816s (+12%)  4.034s (+22%) 6.796s (+12%)

(positive % = faster than 2.7.11)

The get_multi and read_multi regressions reported here are resolved as of 5.0.2. The single-server fast path added in #1077 is the entire driver — for this benchmark (which uses one server), 5.0.2 flips get_multi from 17% slower than 2.7.11 to 23% faster. The 5.0.3 double-array allocation fix in Client#perform makes a negligible additional difference on these workloads.

One regression remains: single-key get is consistently ~12% slower across all 5.x versions. This appears to be genuine meta protocol overhead — it's stable across versions and doesn't respond to the allocation fixes.

A few caveats: the 2.7.11 run-to-run variance on read_multi was higher than the 5.x versions (7.58–7.76s range), and I haven't formally verified correctness of 2.7.11 under Ruby 4.0.2. The 5.0.x results are tight (< 2% spread within each version), so those are solid.

For anyone affected by the original report: if you're on a single-server deployment, upgrading to 5.0.2+ should recover the lost performance for get_multi-heavy workloads. Multi-server deployments wouldn't benefit from the fast path and I haven't benchmarked that case.