Update to Kombu 5.5.0 stopped processing SQS messages

[mdrachuk]

Hello.

Tbh I'm confused about the debug steps for this, because we didn't have any alerts or error logs. Only the actual usage showed that the tasks from the SQS weren't processing anymore.

Downgrade to 5.4.2 solved the issue for now.

Leaving this here for anybody having similar issues to see and maybe comment on details they can find.

[mgorven]

I'm seeing the same thing. It's not completely broken, but processing is extremely slow.

[Nusnus]

Can you please share a reproduction script or steps to reproduce, I'd like to solve this ASAP.

Thank you.

[mgorven]

Our workers process tasks from four standard SQS queues. Task execution time is fast (~50ms) and we use task_acks_late=True. We run with concurrency of eight and worker_prefetch_multiplier=16. With Kombu 5.4.2 one instance can process ~100 tasks per second, with 5.5.0 it drops to ~3/s. CPU usage remains low.

[Nusnus]

Ok, found it.

Throughput: 0.99 tasks per second with urllib3

python stress_test.py
Processed 50 tasks in 50.68 seconds
Throughput: 0.99 tasks per second

Throughput: 55.25 tasks per second with pycurl

python stress_test.py
Processed 50 tasks in 0.90 seconds
Throughput: 55.25 tasks per second

In PR #2134 we have migrated from pycurl to urllib3, by @spawn-guy.
Following this change, we’ve also removed pycurl from celery in PR celery/celery#9526 by @jmsmkn.

@mgorven

Our workers process tasks from four standard SQS queues. Task execution time is fast (~50ms) and we use task_acks_late=True. We run with concurrency of eight and worker_prefetch_multiplier=16. With Kombu 5.4.2 one instance can process ~100 tasks per second, with 5.5.0 it drops to ~3/s. CPU usage remains low.

By creating a quick stress_test.py to reproduce your case, I could easily see the problem using the latest version.
Reverting Commit 07c8852 from kombu and Commit 9bf0546 from celery was enough to fix the performance, as can be seen in the logs above.

Celery Worker

export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_ENDPOINT_URL=http://localhost:4566
celery -A myapp worker -l INFO

myapp.py

from celery import Celery
import time

app = Celery(
    "myapp",
    broker="sqs://",
    result_backend="redis://",
    task_acks_late=True,
    worker_prefetch_multiplier=16,
)


@app.task
def add(x, y):
    time.sleep(0.05)  # Simulate 50ms task execution time
    return x + y


if __name__ == "__main__":
    app.start()

stress_test.py

export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_ENDPOINT_URL=http://localhost:4566
python stress_test.py

from myapp import add
from celery import group
import time

num_tasks = 50
tasks = [add.s(1, 2) for _ in range(num_tasks)]
g = group(tasks)
start_time = time.time()
result = g.apply_async()
result.get()
end_time = time.time()
total_time = end_time - start_time
throughput = num_tasks / total_time
print(f"Processed {num_tasks} tasks in {total_time:.2f} seconds")
print(f"Throughput: {throughput:.2f} tasks per second")

We have had multiple pre-releases, both in Celery and Kombu, and this is the first time we have received such a report.

—

@spawn-guy, I wonder if there’s anything we can tweak to fix this instead of reverting back to pycurl.
As this was discovered after the final v5.5.0 release, this issue is a bit urgent.
Would you mind giving it a look please?
If you’re unavailable, please let me know so I can revert the changes and issue a new release, although this is the least wanted outcome.

@auvipy I’d be glad to hear your feedback as well. Thanks.

Note to self: How come we didn’t fall on this in the CI 🤔
I might need to enable SQS in pytest-celery and add a smoke test to the CI to reproduce this case and detect performance failures.
The issue is that SQS support is currently experimental in pytest-celery, and I don’t have the capacity to fix any problems that may arise (IIRC there are issues when using SQS/Redis in the CI, potentially due to a genuine bug).

[auvipy]

the performance regressions are hard to address in short notice to be honest

[auvipy]

I think we have to stick to special implementation of pycurl until we have our homegrown faster or equal alternative of pycurl. some previous attempts also faced the serious performance regressions

[auvipy]

If we can't figure out by next 1 or 2 weeks, we should revert this #2261

[Nusnus]

@auvipy

If we can't figure out by next 1 or 2 weeks, we should revert this #2261

Unfortunately we don't have the luxury of time my friend. I will take care of everything and release a fixed version on Monday.

I just want to give it a few days for @spawn-guy to check this out, but per my script above we can confirm the issue is only due to the dep changes.

[Nusnus]

the performance regressions are hard to address in short notice to be honest

I released it before the weekend knowing it might have issues. I am prepared to respond quickly.

I take responsibility, please don't stress yourself over the weekend 🙏

[Nusnus]

I think we have to stick to special implementation of pycurl until we have our homegrown faster or equal alternative of pycurl. some previous attempts also faced the serious performance regressions

Any recommendation?

[auvipy]

the performance regressions are hard to address in short notice to be honest

I released it before the weekend knowing it might have issues. I am prepared to respond quickly.

I take responsibility, please don't stress yourself over the weekend 🙏

there is no need to feel guilty. celery code base is hard.

[auvipy]

I think we have to stick to special implementation of pycurl until we have our homegrown faster or equal alternative of pycurl. some previous attempts also faced the serious performance regressions

Any recommendation?

I have to test alternatives, right now I think we should revert back to pycurl as it was working fine as far as I know.

[spawn-guy]

the main difference between pycurl and urllib3 was is using the available urllib3.PoolManager instead of custom pycurl-pool implementation.

@mdrachuk what is your use-case? does this pr-revert fix your problem?

@Nusnus have you tried tuning the max_clients setting?

i am running my kombu fork (not the rc versions) and things seem to be processed fine, but not at the speeds mentioned

[spawn-guy]

another idea i have is about the SSL-certificates and optional cert package importing

[spawn-guy]

here are my "from-home" on Windows tests of the code here #2258 (comment)

celery -A myapp worker -l INFO -P gevent

kombu = "*"

max_clients=10

Processed 50 tasks in 4.41 seconds
Throughput: 11.35 tasks per second

Processed 50 tasks in 4.29 seconds
Throughput: 11.65 tasks per second

Processed 50 tasks in 3.77 seconds
Throughput: 13.26 tasks per second

Processed 50 tasks in 3.93 seconds
Throughput: 12.72 tasks per second

Processed 50 tasks in 4.25 seconds
Throughput: 11.78 tasks per second

FORKED_BY_MULTIPROCESSING - didn't do anything

kombu = "*"

Processed 50 tasks in 4.51 seconds
Throughput: 11.09 tasks per second

Processed 50 tasks in 4.44 seconds
Throughput: 11.26 tasks per second

Processed 50 tasks in 4.00 seconds
Throughput: 12.50 tasks per second

max_clients: int = 100

Processed 50 tasks in 3.67 seconds
Throughput: 13.64 tasks per second

Processed 50 tasks in 4.24 seconds
Throughput: 11.79 tasks per second

Processed 50 tasks in 4.56 seconds
Throughput: 10.97 tasks per second

Processed 50 tasks in 4.18 seconds
Throughput: 11.95 tasks per second

max_clients: int = 1

Processed 50 tasks in 3.87 seconds
Throughput: 12.93 tasks per second

Processed 50 tasks in 4.00 seconds
Throughput: 12.51 tasks per second

Processed 50 tasks in 4.51 seconds
Throughput: 11.08 tasks per second

kombu = "5.4.2"
celery = { extras = ["sqs", "redis"] }

max_clients: int = 10

Processed 50 tasks in 4.03 seconds
Throughput: 12.41 tasks per second

Processed 50 tasks in 4.50 seconds
Throughput: 11.12 tasks per second

Processed 50 tasks in 4.52 seconds
Throughput: 11.06 tasks per second

max_clients: int = 1

Processed 50 tasks in 4.13 seconds
Throughput: 12.11 tasks per second

Processed 50 tasks in 4.17 seconds
Throughput: 11.99 tasks per second

Processed 50 tasks in 4.21 seconds
Throughput: 11.88 tasks per second

Processed 50 tasks in 4.06 seconds
Throughput: 12.33 tasks per second

i dunno. is it Windows? is it gevent?

billard version is failing Windows with Access denied

\Lib\site-packages\billiard\pool.py", line 406, in _ensure_messages_consumed
    if self.on_ready_counter.value >= completed:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 3, in getvalue
PermissionError: [WinError 5] Access is denied

all speeds seem to be the same. with or without urllib3/pycurl on both kombu versions: latest and 5.4.2

help!

[spawn-guy]

main.py config is a bit different.

app = Celery(
    "myapp",
    broker="sqs://",
    # result_backend="redis://",
    result_backend="redis://localhost:6379/1",  # local docker
    task_acks_late=True,
    worker_prefetch_multiplier=16,
    task_default_queue="default_static",
    task_queues={
        "default_static": {
            "exchange": "default_static",
            "routing_key": "default_static",
        }
    },
    broker_transport_options={
        # # !!!! same as in AWS-SQS: Visibility timeout. match the time of the longest ETA you’re planning to use.
        # # chatGPT: Set the SQS visibility timeout on the queue to be slightly longer than the longest running task.
        # "visibility_timeout": 1200,  # seconds.
        # # Celery: seconds to sleep between unsuccessful polls
        # "polling_interval": 1,  # seconds.
        # # !!!! same as in AWS-SQS: Receive message wait time
        # "wait_time_seconds": 20,  # seconds.
        "region": "eu-west-1",
        "predefined_queues": {
            "default_static": {
                "url": 'https://sqs.eu-west-1.amazonaws.com/00000accNumber/existing-queue',
            }
        }
    }
)

aws access keys are picked up from a default location. no extra env vars are set.
python 3.11

[auvipy]

May be we should highlight these suggestions you shared for people to avoid these performance issues

[spawn-guy]

@auvipy i'd like to hear more use-case specifics from @mdrachuk and @mgorven : environments, os'es, python versions. with-or-without a proxy (as i didn't test that one out)
i'd like to figure the reason of slowdown out.
not that it was unexpected. it was undesired, but possible

i am running not-high-frequency tasks(1+ seconds) on aws elastic beanstalk py3.11 instances with the urllib3 version of kombu and worker_pool=aio_pool.pool.AsyncIOPool, in eu-west-1 region for a month now (since the PR) and didn't notice any errors

another reason of slowdown with max_clients=10 could be - creation of many dynamic queues and pool-draining. but after worker init it shouldn't be a problem
and i think urllib3.PoolManager should re-use the established sqs http connection. this should cover the message-updates/creation

maybe pycurl returns the responses earlier in the http lifecycle. maybe this needs a rewrite on kombu side for urllib3.

it's like, i can agree with slowdown,
but a complete "not working" stage - i doubt it. unless there is a proxy involved ()

maybe, some configuration issue or aws outage or WAF interference on seeing a new User-Agent instead of an old one.

[spawn-guy]

as a side-note:

i am also thinking about aiohttp+aiobotocore in urllib3+botocore place, as i heard that it is even faster than other py-clients.
but i am still waiting for a long promised out-of-the-box asyncio support.
and i'm really looking forward to the outcome of recent @Nusnus experiments ;)

[auvipy]

we will revisit our recent and old experiments for introducing native async support in v6.0. but we have to reach to a consensus for this exact issue for now with recent changes

[auvipy]

as a side-note:

i am also thinking about aiohttp+aiobotocore in urllib3+botocore place, as i heard that it is even faster than other py-clients. but i am still waiting for a long promised out-of-the-box asyncio support.

we can also try httpx and see later

[mgorven]

@spawn-guy We're using Python 3.11.11 on Debian Bookworm/12 aarch64. No proxy for SQS.

[Nusnus]

// On Topic

We will revert back to pycurl and release v5.6.
v5.7+ will be able to have an alternative if it fits better than both of these.

// Off Topic

@spawn-guy

as a side-note:

i am also thinking about aiohttp+aiobotocore in urllib3+botocore place, as i heard that it is even faster than other py-clients. but i am still waiting for a long promised out-of-the-box asyncio support. and i'm really looking forward to the outcome of recent @Nusnus experiments ;)

I am trying to be as creative as I’ve ever been in my life to solve the challenges of migrating to asyncio. It became a mission for me. It requires solving a completely different core challenge first, which makes the difficulty extremely high and multi-dimensional, but this only makes it more attractive tbh 😉

EDIT:
Grok 3, ChatGPT 4.5 and the Google one (lol) all say my plan to solve both+ challenges should work within a reasonable time frame. I’m still working on the execution plan but if they don’t lie, I might be on to something.
I’m also exploring other approaches to address it, so there’s more happening behind the scenes.
I’ll stick with whatever works first 😄

[Nusnus]

@auvipy

That's too late for celery v5.5. As it is clear that new change has introduced performance regression with default mechanisms. So It's best to revert now and reconsider for 5.7

That’s my reasoning too. Enabling both by choice with pycurl as default can be an acceptable middle ground. WDYT?

[auvipy]

Yes

[spawn-guy]

sure.
revert now -> introduce choice later.
i'll lock the version on my side (or switch back to my fork) on my environments

and will still try to reproduce the slowdown today/tomorrow. no luck on windows so far.

[millerdev]

Would it be recommended to use 5.4.2 on Python 3.13 to avoid this issue, given that 5.5 is the first version of Kombu to officially support Python 3.13? If not, would it be possible to get a 5.4.3 release with support for Python 3.13 until 5.5+ has stabilized?

[spawn-guy]

As it is clear that new change has introduced

it is not so clear to me anymore.

first: @Nusnus how many times have you tried your test?

second: "our" urllib3 client seems to be only used to FETCH and SEND messages. on DeleteMessage from the queue - the boto3.Session is used and i don't see any urllib3_client logs that i've added

i've deployed things to amazon linux 2023 and now running more iterations of the same test from y home windows(cloud-home delay).

celery -A myapp worker -l DEBUG -Q default_static
total 4 ForkPoolWorkers start.

from datetime import datetime

from myapp import add
from celery import group
import time

num_iters = 10
num_tasks = 500

all_times = []

tasks = [add.s(1, 2) for _ in range(num_tasks)]

print("start,run,time,throughput")
for i in range(num_iters):
    g = group(tasks)
    start_dtime = datetime.now()
    start_time = time.time()
    result = g.apply_async()
    result.get()
    end_time = time.time()
    total_time = end_time - start_time
    throughput = num_tasks / total_time
    all_times.append(total_time)
    print(f"{start_dtime},{i},{total_time},{throughput}")

print(f"iterations: {num_iters:.2f}")
print(f"tasks in iteration: {num_tasks:.2f}")
print(f"total tasks: {num_tasks * num_iters}")
print(f"total time: {sum(all_times):.2f} seconds")
print(f"average time of iteration: {sum(all_times) / len(all_times):.2f} seconds")
print(f"average throughput: {num_tasks * num_iters / sum(all_times):.2f} tasks per second")

testing the speed of urllib3.

start,run,time,throughput
2025-03-21 12:56:43.979081,0,29.73174262046814,16.81704319799225
2025-03-21 12:57:13.710823,1,23.604838371276855,21.18209801463485
2025-03-21 12:57:37.315662,2,23.39216923713684,21.37467435923865
2025-03-21 12:58:00.707831,3,24.549772262573242,20.36678770997248
2025-03-21 12:58:25.257603,4,24.756406545639038,20.196792255703098
2025-03-21 12:58:50.014010,5,33.352821588516235,14.99123540936506
2025-03-21 12:59:23.366831,6,42.59526753425598,11.73839322872875
2025-03-21 13:00:05.962099,7,42.99054718017578,11.630463736701747
2025-03-21 13:00:48.952646,8,43.111043214797974,11.597956410119385
2025-03-21 13:01:32.063689,9,33.36323094367981,14.986558131736276
iterations: 10.00
tasks in iteration: 500
total tasks: 5000
total time: 321.45 seconds
average time of iteration: 32.14 seconds
average throughput: 15.55 tasks per second

the numbers fluctuate too much from 11 to 21 tps

and 500 tasks seem to work faster than 50

start,run,time,throughput
2025-03-21 13:08:27.189244,0,25.04867458343506,1.99611360008108
2025-03-21 13:08:52.237918,1,3.1373138427734375,15.937200581692258
2025-03-21 13:08:55.375233,2,2.7751967906951904,18.01674035068156
2025-03-21 13:08:58.150429,3,12.57353949546814,3.976604997981787
2025-03-21 13:09:10.723969,4,3.0881454944610596,16.19094699057434
2025-03-21 13:09:13.812731,5,2.7244091033935547,18.35260348297891
2025-03-21 13:09:16.537141,6,22.50865936279297,2.2213673055379957
2025-03-21 13:09:39.045800,7,3.0558571815490723,16.36202120370496
2025-03-21 13:09:42.101657,8,12.132184267044067,4.121269418551471
2025-03-21 13:09:54.233842,9,3.0647895336151123,16.3143339702749
iterations: 10
tasks in iteration: 50
total tasks: 500
total time: 90.11 seconds
average time of iteration: 9.01 seconds
average throughput: 5.55 tasks per second

the numbers fluctuate too much from 1.9 to 18.3 tps

[spawn-guy]

As I have pycurl deployment problems again - I am thinking about the fixing strategy.

I will roll back the deletion of pycurl and dependencies.
But then.. I want to keep both options available for (me).

The best way would be to introduce choice via celery configure, like pool choice. But I don't have enough time to do this.
Instead I am going to implement a kombu fallback to urllib3 (as it is required by boto3) when pycurl is not detected (in a way similar to certifi and ssl).

But I need some advice on package dependency : we now use sqs extra, that required pycurl. And I have pycurl problems on instance deployment.
My ideas are:

• add pycurl back to sqs.txt and add sqs-urlib3 or sqs-clean as an extra without pycurl. This should cover existing users to use pycurl and me to use urllib3 instead
• or add http-pycurl extra. But this will require current users to add one extra to their kombu/celery dependency

The ci will use pycurl by default

What do you say?
I'm in favor of pycurl extra

[spawn-guy]

initial code here #2269

when pycurl is found it will be used as a http client. i can't test with pycurl as it brings my old struggle back.
i'll test urllib3-fallback tomorrow to see if it works

i've picked the 3rd route to requirements management:
to use pycurl as http client - it needs to be installed/required by user :)

[spawn-guy]

so i've deployed the "fallback urllib3" version to my Amazon Linux 2023 instances and the code seems to run fine without pycurl

@mgorven will you be so kind to test the branch with pycurl manually installed to see if it works and if there is speed difference?

[spawn-guy]

i have been thinking and commenting with @jmsmkn and..

the thing is.. i can't reliably state that pycurl-urllib3 is the problem here. 'cause my change was not the only one since 5,4,2.

on the other hand i have also enabled ssl connections for sqs which were disabled instead of "fixed" previously.
might it be that this is the actual "not working" and "slow" problem is ssl-with-optional-certifi enablement? too many requests to filesystem?!

it looks like i still need to test pycurl on aws anyway :( and also if ssl works correctly

[mgorven]

@mgorven will you be so kind to test the branch with pycurl manually installed to see if it works and if there is speed difference?

The branch doesn't work at all: https://gist.github.com/mgorven/f1689323acb1a4e981644dfc9afe87ab

This is using rev 77ca118 and pycurl 7.45.6.

[spawn-guy]

@mgorven thanks for the feedback i'll look into it today. even though this is the code form 5.4.2 😝

[spawn-guy]

@mgorven the code in PR is fixed and tested with pycurl! and it works

@auvipy @Nusnus here are the speed test results

client: is on windows11 and py3.11. so the client is urllib3
worker: AWS Elastic Beanstalk - python 3.11 - Amazon Linux 2023 == 1 spot-instance/4 workers
broker: AWS SQS
result_backend: Redis on AWS ElastiCache
all of AWS is in the same region

code is seen here: #2258 (comment)
the only difference is

worker_prefetch_multiplier = 16
num_iters = 10
num_tasks = worker_prefetch_multiplier * 8

Pipfile (truncated)

[packages]
kombu = { git = "https://github.com/spawn-guy/kombu.git", ref = "feature_optinal_pycurl" }
pycurl = { version = "*", markers = "sys_platform != 'win32'", install_command = "pip install pycurl --global-option='--with-openssl' --compile" }
celery = { extras = ["sqs", "redis"] }

pycurl tester task

@app.task
def is_pycurl_available():
    try:
        import pycurl
    except ImportError:  # pragma: no cover
        print('pycurl is not installed')
    else:
        print('pycurl is installed')
        print(f"PycURL version: {pycurl.version}")
        print(f"PycURL version info: {pycurl.version_info()}")

    if CurlClient.Curl is not None:
        print('CurlClient.Curl is not None')
    else:
        print('CurlClient.Curl is None')

    return "ok"

pycurl is installed
PycURL version: PycURL/7.45.6 libcurl/8.12.1-DEV OpenSSL/3.4.1 zlib/1.2.11 brotli/1.1.0 libssh2/1.11.1_DEV nghttp2/1.64.0
PycURL version info: (11, '8.12.1-DEV', 527361, 'Linux', 1370440605, 'OpenSSL/3.4.1', 0, '1.2.11', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'gophers', 'http', 'https', 'imap', 'imaps', 'mqtt', 'pop3', 'pop3s', 'rtsp', 'scp', 'sftp', 'smb', 'smbs', 'smtp', 'smtps', 'telnet', 'tftp'), None, 0, None)
CurlClient.Curl is not None

results

start,run,time,throughput
2025-03-31 16:24:38.225600,0,11.19954538345337,11.42903534183736
2025-03-31 16:24:49.425145,1,7.3373634815216064,17.444958304485638
2025-03-31 16:24:56.762509,2,6.7569661140441895,18.9434130406478
2025-03-31 16:25:03.519981,3,6.73758339881897,18.997909550542573
2025-03-31 16:25:10.257564,4,6.803251028060913,18.814534326610467
2025-03-31 16:25:17.060815,5,6.2250285148620605,20.562154806906346
2025-03-31 16:25:23.285844,6,6.641470670700073,19.2728397589246
2025-03-31 16:25:29.927963,7,6.143255710601807,20.83585740686364
2025-03-31 16:25:36.071219,8,7.266942024230957,17.614011447070258
2025-03-31 16:25:43.338161,9,6.863733291625977,18.64874326573339
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 71.98 seconds
average time of iteration: 7.20 seconds
average throughput: 17.78 tasks per second

start,run,time,throughput
2025-03-31 16:26:59.364233,0,11.626185894012451,11.009629569566808
2025-03-31 16:27:10.990419,1,6.945514678955078,18.42915981271197
2025-03-31 16:27:17.935934,2,6.449299097061157,19.847117969505796
2025-03-31 16:27:24.385233,3,6.562275171279907,19.505430153279423
2025-03-31 16:27:30.947508,4,6.58098578453064,19.449973634782594
2025-03-31 16:27:37.528494,5,6.307449102401733,20.29346538068147
2025-03-31 16:27:43.835943,6,6.883335113525391,18.59563683721961
2025-03-31 16:27:50.719278,7,6.225593566894531,20.560288529058177
2025-03-31 16:27:56.944872,8,6.622671842575073,19.327546803259718
2025-03-31 16:28:03.567544,9,6.621161460876465,19.331955693322758
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 70.82 seconds
average time of iteration: 7.08 seconds
average throughput: 18.07 tasks per second

withOUT pycurl

Pipenv (truncated)

[packages]
kombu = { git = "https://github.com/spawn-guy/kombu.git", ref = "feature_optinal_pycurl" }
celery = { extras = ["sqs", "redis"] }

test task output

pycurl is not installed
CurlClient.Curl is None

results

start,run,time,throughput
2025-03-31 16:47:00.252294,0,12.054689645767212,10.618274195465904
2025-03-31 16:47:12.306984,1,27.492735385894775,4.655775360412873
2025-03-31 16:47:39.799719,2,26.170573711395264,4.89098945294676
2025-03-31 16:48:05.970293,3,7.433281183242798,17.21985175114276
2025-03-31 16:48:13.403574,4,16.914153337478638,7.567626794323524
2025-03-31 16:48:30.317727,5,7.251413583755493,17.65173073105962
2025-03-31 16:48:37.569141,6,16.562063455581665,7.72850559009675
2025-03-31 16:48:54.132253,7,25.98214817047119,4.926459473642463
2025-03-31 16:49:20.114401,8,26.43418788909912,4.8422142014351195
2025-03-31 16:49:46.548589,9,17.52568507194519,7.303566135905304
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 183.82 seconds
average time of iteration: 18.38 seconds
average throughput: 6.96 tasks per second

start,run,time,throughput
2025-03-31 16:50:41.249079,0,29.07918119430542,4.401774559768769
2025-03-31 16:51:10.328260,1,7.6913745403289795,16.64201883926525
2025-03-31 16:51:18.019635,2,6.810234308242798,18.795241720989633
2025-03-31 16:51:24.829869,3,7.339831829071045,17.439091655074083
2025-03-31 16:51:32.169701,4,8.319275617599487,15.385954965744263
2025-03-31 16:51:40.488976,5,6.741177082061768,18.987781872784094
2025-03-31 16:51:47.230154,6,7.075741529464722,18.08997678433898
2025-03-31 16:51:54.305895,7,6.765643119812012,18.91911792172044
2025-03-31 16:52:01.071538,8,7.579378843307495,16.887927447118248
2025-03-31 16:52:08.650917,9,7.461513042449951,17.15469761585672
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 94.86 seconds
average time of iteration: 9.49 seconds
average throughput: 13.49 tasks per second

start,run,time,throughput
2025-03-31 16:52:32.437494,0,29.409680128097534,4.352308472668863
2025-03-31 16:53:01.847174,1,7.214996576309204,17.74082615926587
2025-03-31 16:53:09.062171,2,7.3906779289245605,17.31911486753498
2025-03-31 16:53:16.452849,3,17.222590923309326,7.432098954795633
2025-03-31 16:53:33.675440,4,7.292839527130127,17.551462571447868
2025-03-31 16:53:40.968279,5,6.901855945587158,18.54573624965896
2025-03-31 16:53:47.870135,6,8.120389223098755,15.762791226299738
2025-03-31 16:53:55.990524,7,25.75519371032715,4.969871375833427
2025-03-31 16:54:21.745718,8,28.27398705482483,4.527129468928486
2025-03-31 16:54:50.019705,9,7.444072008132935,17.19489008974592
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 145.03 seconds
average time of iteration: 14.50 seconds
average throughput: 8.83 tasks per second

[Nusnus]

@mgorven the code in PR is fixed and tested with pycurl! and it works

@auvipy @Nusnus here are the speed test results

client: is on windows11 and py3.11. so the client is urllib3
worker: AWS Elastic Beanstalk - python 3.11 - Amazon Linux 2023 == 1 spot-instance/4 workers
broker: AWS SQS
result_backend: Redis on AWS ElastiCache
all of AWS is in the same region

code is seen here: #2258 (comment)
the only difference is

worker_prefetch_multiplier = 16
num_iters = 10
num_tasks = worker_prefetch_multiplier * 8

Pipfile (truncated)

[packages]
kombu = { git = "https://github.com/spawn-guy/kombu.git", ref = "feature_optinal_pycurl" }
pycurl = { version = "*", markers = "sys_platform != 'win32'", install_command = "pip install pycurl --global-option='--with-openssl' --compile" }
celery = { extras = ["sqs", "redis"] }

pycurl tester task

@app.task
def is_pycurl_available():
    try:
        import pycurl
    except ImportError:  # pragma: no cover
        print('pycurl is not installed')
    else:
        print('pycurl is installed')
        print(f"PycURL version: {pycurl.version}")
        print(f"PycURL version info: {pycurl.version_info()}")

    if CurlClient.Curl is not None:
        print('CurlClient.Curl is not None')
    else:
        print('CurlClient.Curl is None')

    return "ok"

pycurl is installed
PycURL version: PycURL/7.45.6 libcurl/8.12.1-DEV OpenSSL/3.4.1 zlib/1.2.11 brotli/1.1.0 libssh2/1.11.1_DEV nghttp2/1.64.0
PycURL version info: (11, '8.12.1-DEV', 527361, 'Linux', 1370440605, 'OpenSSL/3.4.1', 0, '1.2.11', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'gophers', 'http', 'https', 'imap', 'imaps', 'mqtt', 'pop3', 'pop3s', 'rtsp', 'scp', 'sftp', 'smb', 'smbs', 'smtp', 'smtps', 'telnet', 'tftp'), None, 0, None)
CurlClient.Curl is not None

results

start,run,time,throughput
2025-03-31 16:24:38.225600,0,11.19954538345337,11.42903534183736
2025-03-31 16:24:49.425145,1,7.3373634815216064,17.444958304485638
2025-03-31 16:24:56.762509,2,6.7569661140441895,18.9434130406478
2025-03-31 16:25:03.519981,3,6.73758339881897,18.997909550542573
2025-03-31 16:25:10.257564,4,6.803251028060913,18.814534326610467
2025-03-31 16:25:17.060815,5,6.2250285148620605,20.562154806906346
2025-03-31 16:25:23.285844,6,6.641470670700073,19.2728397589246
2025-03-31 16:25:29.927963,7,6.143255710601807,20.83585740686364
2025-03-31 16:25:36.071219,8,7.266942024230957,17.614011447070258
2025-03-31 16:25:43.338161,9,6.863733291625977,18.64874326573339
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 71.98 seconds
average time of iteration: 7.20 seconds
average throughput: 17.78 tasks per second

start,run,time,throughput
2025-03-31 16:26:59.364233,0,11.626185894012451,11.009629569566808
2025-03-31 16:27:10.990419,1,6.945514678955078,18.42915981271197
2025-03-31 16:27:17.935934,2,6.449299097061157,19.847117969505796
2025-03-31 16:27:24.385233,3,6.562275171279907,19.505430153279423
2025-03-31 16:27:30.947508,4,6.58098578453064,19.449973634782594
2025-03-31 16:27:37.528494,5,6.307449102401733,20.29346538068147
2025-03-31 16:27:43.835943,6,6.883335113525391,18.59563683721961
2025-03-31 16:27:50.719278,7,6.225593566894531,20.560288529058177
2025-03-31 16:27:56.944872,8,6.622671842575073,19.327546803259718
2025-03-31 16:28:03.567544,9,6.621161460876465,19.331955693322758
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 70.82 seconds
average time of iteration: 7.08 seconds
average throughput: 18.07 tasks per second

withOUT pycurl

Pipenv (truncated)

[packages]
kombu = { git = "https://github.com/spawn-guy/kombu.git", ref = "feature_optinal_pycurl" }
celery = { extras = ["sqs", "redis"] }

test task output

pycurl is not installed
CurlClient.Curl is None

results

start,run,time,throughput
2025-03-31 16:47:00.252294,0,12.054689645767212,10.618274195465904
2025-03-31 16:47:12.306984,1,27.492735385894775,4.655775360412873
2025-03-31 16:47:39.799719,2,26.170573711395264,4.89098945294676
2025-03-31 16:48:05.970293,3,7.433281183242798,17.21985175114276
2025-03-31 16:48:13.403574,4,16.914153337478638,7.567626794323524
2025-03-31 16:48:30.317727,5,7.251413583755493,17.65173073105962
2025-03-31 16:48:37.569141,6,16.562063455581665,7.72850559009675
2025-03-31 16:48:54.132253,7,25.98214817047119,4.926459473642463
2025-03-31 16:49:20.114401,8,26.43418788909912,4.8422142014351195
2025-03-31 16:49:46.548589,9,17.52568507194519,7.303566135905304
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 183.82 seconds
average time of iteration: 18.38 seconds
average throughput: 6.96 tasks per second

start,run,time,throughput
2025-03-31 16:50:41.249079,0,29.07918119430542,4.401774559768769
2025-03-31 16:51:10.328260,1,7.6913745403289795,16.64201883926525
2025-03-31 16:51:18.019635,2,6.810234308242798,18.795241720989633
2025-03-31 16:51:24.829869,3,7.339831829071045,17.439091655074083
2025-03-31 16:51:32.169701,4,8.319275617599487,15.385954965744263
2025-03-31 16:51:40.488976,5,6.741177082061768,18.987781872784094
2025-03-31 16:51:47.230154,6,7.075741529464722,18.08997678433898
2025-03-31 16:51:54.305895,7,6.765643119812012,18.91911792172044
2025-03-31 16:52:01.071538,8,7.579378843307495,16.887927447118248
2025-03-31 16:52:08.650917,9,7.461513042449951,17.15469761585672
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 94.86 seconds
average time of iteration: 9.49 seconds
average throughput: 13.49 tasks per second

start,run,time,throughput
2025-03-31 16:52:32.437494,0,29.409680128097534,4.352308472668863
2025-03-31 16:53:01.847174,1,7.214996576309204,17.74082615926587
2025-03-31 16:53:09.062171,2,7.3906779289245605,17.31911486753498
2025-03-31 16:53:16.452849,3,17.222590923309326,7.432098954795633
2025-03-31 16:53:33.675440,4,7.292839527130127,17.551462571447868
2025-03-31 16:53:40.968279,5,6.901855945587158,18.54573624965896
2025-03-31 16:53:47.870135,6,8.120389223098755,15.762791226299738
2025-03-31 16:53:55.990524,7,25.75519371032715,4.969871375833427
2025-03-31 16:54:21.745718,8,28.27398705482483,4.527129468928486
2025-03-31 16:54:50.019705,9,7.444072008132935,17.19489008974592
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 145.03 seconds
average time of iteration: 14.50 seconds
average throughput: 8.83 tasks per second

Maybe then we could indeed set it for 5.6.
WDYT @auvipy , maybe better than postponing to 5.7?

[mgorven]

The feature_optinal_pycurl branch now works, and with pycurl installed performance is good.

[auvipy]

@mgorven the code in PR is fixed and tested with pycurl! and it works

@auvipy @Nusnus here are the speed test results

client: is on windows11 and py3.11. so the client is urllib3
worker: AWS Elastic Beanstalk - python 3.11 - Amazon Linux 2023 == 1 spot-instance/4 workers
broker: AWS SQS
result_backend: Redis on AWS ElastiCache
all of AWS is in the same region

code is seen here: #2258 (comment)
the only difference is

worker_prefetch_multiplier = 16
num_iters = 10
num_tasks = worker_prefetch_multiplier * 8

Pipfile (truncated)

[packages]
kombu = { git = "https://github.com/spawn-guy/kombu.git", ref = "feature_optinal_pycurl" }
pycurl = { version = "*", markers = "sys_platform != 'win32'", install_command = "pip install pycurl --global-option='--with-openssl' --compile" }
celery = { extras = ["sqs", "redis"] }

pycurl tester task

@app.task
def is_pycurl_available():
    try:
        import pycurl
    except ImportError:  # pragma: no cover
        print('pycurl is not installed')
    else:
        print('pycurl is installed')
        print(f"PycURL version: {pycurl.version}")
        print(f"PycURL version info: {pycurl.version_info()}")

    if CurlClient.Curl is not None:
        print('CurlClient.Curl is not None')
    else:
        print('CurlClient.Curl is None')

    return "ok"

pycurl is installed
PycURL version: PycURL/7.45.6 libcurl/8.12.1-DEV OpenSSL/3.4.1 zlib/1.2.11 brotli/1.1.0 libssh2/1.11.1_DEV nghttp2/1.64.0
PycURL version info: (11, '8.12.1-DEV', 527361, 'Linux', 1370440605, 'OpenSSL/3.4.1', 0, '1.2.11', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'gophers', 'http', 'https', 'imap', 'imaps', 'mqtt', 'pop3', 'pop3s', 'rtsp', 'scp', 'sftp', 'smb', 'smbs', 'smtp', 'smtps', 'telnet', 'tftp'), None, 0, None)
CurlClient.Curl is not None

results

start,run,time,throughput
2025-03-31 16:24:38.225600,0,11.19954538345337,11.42903534183736
2025-03-31 16:24:49.425145,1,7.3373634815216064,17.444958304485638
2025-03-31 16:24:56.762509,2,6.7569661140441895,18.9434130406478
2025-03-31 16:25:03.519981,3,6.73758339881897,18.997909550542573
2025-03-31 16:25:10.257564,4,6.803251028060913,18.814534326610467
2025-03-31 16:25:17.060815,5,6.2250285148620605,20.562154806906346
2025-03-31 16:25:23.285844,6,6.641470670700073,19.2728397589246
2025-03-31 16:25:29.927963,7,6.143255710601807,20.83585740686364
2025-03-31 16:25:36.071219,8,7.266942024230957,17.614011447070258
2025-03-31 16:25:43.338161,9,6.863733291625977,18.64874326573339
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 71.98 seconds
average time of iteration: 7.20 seconds
average throughput: 17.78 tasks per second

start,run,time,throughput
2025-03-31 16:26:59.364233,0,11.626185894012451,11.009629569566808
2025-03-31 16:27:10.990419,1,6.945514678955078,18.42915981271197
2025-03-31 16:27:17.935934,2,6.449299097061157,19.847117969505796
2025-03-31 16:27:24.385233,3,6.562275171279907,19.505430153279423
2025-03-31 16:27:30.947508,4,6.58098578453064,19.449973634782594
2025-03-31 16:27:37.528494,5,6.307449102401733,20.29346538068147
2025-03-31 16:27:43.835943,6,6.883335113525391,18.59563683721961
2025-03-31 16:27:50.719278,7,6.225593566894531,20.560288529058177
2025-03-31 16:27:56.944872,8,6.622671842575073,19.327546803259718
2025-03-31 16:28:03.567544,9,6.621161460876465,19.331955693322758
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 70.82 seconds
average time of iteration: 7.08 seconds
average throughput: 18.07 tasks per second

withOUT pycurl

Pipenv (truncated)

[packages]
kombu = { git = "https://github.com/spawn-guy/kombu.git", ref = "feature_optinal_pycurl" }
celery = { extras = ["sqs", "redis"] }

test task output

pycurl is not installed
CurlClient.Curl is None

results

start,run,time,throughput
2025-03-31 16:47:00.252294,0,12.054689645767212,10.618274195465904
2025-03-31 16:47:12.306984,1,27.492735385894775,4.655775360412873
2025-03-31 16:47:39.799719,2,26.170573711395264,4.89098945294676
2025-03-31 16:48:05.970293,3,7.433281183242798,17.21985175114276
2025-03-31 16:48:13.403574,4,16.914153337478638,7.567626794323524
2025-03-31 16:48:30.317727,5,7.251413583755493,17.65173073105962
2025-03-31 16:48:37.569141,6,16.562063455581665,7.72850559009675
2025-03-31 16:48:54.132253,7,25.98214817047119,4.926459473642463
2025-03-31 16:49:20.114401,8,26.43418788909912,4.8422142014351195
2025-03-31 16:49:46.548589,9,17.52568507194519,7.303566135905304
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 183.82 seconds
average time of iteration: 18.38 seconds
average throughput: 6.96 tasks per second

start,run,time,throughput
2025-03-31 16:50:41.249079,0,29.07918119430542,4.401774559768769
2025-03-31 16:51:10.328260,1,7.6913745403289795,16.64201883926525
2025-03-31 16:51:18.019635,2,6.810234308242798,18.795241720989633
2025-03-31 16:51:24.829869,3,7.339831829071045,17.439091655074083
2025-03-31 16:51:32.169701,4,8.319275617599487,15.385954965744263
2025-03-31 16:51:40.488976,5,6.741177082061768,18.987781872784094
2025-03-31 16:51:47.230154,6,7.075741529464722,18.08997678433898
2025-03-31 16:51:54.305895,7,6.765643119812012,18.91911792172044
2025-03-31 16:52:01.071538,8,7.579378843307495,16.887927447118248
2025-03-31 16:52:08.650917,9,7.461513042449951,17.15469761585672
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 94.86 seconds
average time of iteration: 9.49 seconds
average throughput: 13.49 tasks per second

start,run,time,throughput
2025-03-31 16:52:32.437494,0,29.409680128097534,4.352308472668863
2025-03-31 16:53:01.847174,1,7.214996576309204,17.74082615926587
2025-03-31 16:53:09.062171,2,7.3906779289245605,17.31911486753498
2025-03-31 16:53:16.452849,3,17.222590923309326,7.432098954795633
2025-03-31 16:53:33.675440,4,7.292839527130127,17.551462571447868
2025-03-31 16:53:40.968279,5,6.901855945587158,18.54573624965896
2025-03-31 16:53:47.870135,6,8.120389223098755,15.762791226299738
2025-03-31 16:53:55.990524,7,25.75519371032715,4.969871375833427
2025-03-31 16:54:21.745718,8,28.27398705482483,4.527129468928486
2025-03-31 16:54:50.019705,9,7.444072008132935,17.19489008974592
iterations: 10
tasks in iteration: 128
total tasks: 1280
total time: 145.03 seconds
average time of iteration: 14.50 seconds
average throughput: 8.83 tasks per second

Maybe then we could indeed set it for 5.6.
WDYT @auvipy , maybe better than postponing to 5.7?

We can include this to be released with kombu 5.6 and celery 5.5.1. Usually we Don't need a major celery version for a major kombu version. As this change is Just adjustments.

[spawn-guy]

so i figured out the testing part and.. now i'm investigating the slowdown.

for one: i did find a mis-configuration in my code. and yes, it is related to max_clients: i used num_pools instead of maxsize parameter on urllib3.PoolManager init. Gotcha is described here
so then i will need to create a hotfix PR. the question is: do i do it BEFORE or AFTER "bringing the curl back"? whaddayathink @Nusnus @auvipy

second: i am now testing limiting the number of concurrent connections to a host and reusing existing ones, as pycurl implementation does(did). instead of creating new unlimited connections but only keeping the last(maxsize) number of connections
and another improvement i want to try is preload_content=False (as pycurl version did)

twoB: AWS SQS says there is no limit to concurrent connections for Standard Queues, but 300 for FIFO type (and it can be increased) i wonder if i should increase maxsize from 10 to 100

third: it seems that i can optimize my code and use 1 PoolManager for both proxy and non-proxy connections (and also replace some code of mine to select a pool). in urllib3 there are a few layers of Pools (sic!): ProxyManager->PoolManager->ConnectionPool i need to investigate

fourth: i should look into _process_queue functionality. possibly, it is not needed

extra: the urllib3 version works at 20tps when there are 2 instances deployed. but works slow and inconsistent (5tps-10tps-20tps) when only 1 instance is working. weird

[auvipy]

Will comeback to you tomorrow as I'm on vacation

[ebk46]

Just adding that we had a similar experience, although seemingly worse with SQS, Kombu 5.5.0, and Celery 5.5.0. In our case, the full lifecycle of a message+task that normally took ~50ms was now taking a minute or more, long enough that it actually timed out in SQS before deleting the message. Downgrading Celery to 5.4.0 and Kombu to 5.4.2 fixed it for us.

[soceanainn]

This release is completely broken for anyone using SQS in production as the performance decreases so drastically.

This change in performance is not something people expect as part of a minor version release.

I would strongly urge the team to revert the change causing this to avoid more people running into real production incidents from upgrading to this bad version of Kombu.

[spawn-guy]

@ebk46 @soceanainn please try this branch with pycurl. To see 8f it works and helps with the slowdown

kombu = { git = "https://github.com/spawn-guy/kombu.git", ref = "feature_optinal_pycurl" }

pycurl = { version = "*", markers = "sys_platform != 'win32'", install_command = "pip install pycurl --global-option='--with-openssl' --compile" }

[spawn-guy]

status update: as i am struggling with the urllib3 speed-up and i can verify the significant speed decrease and some implementation mistakes and i also figured out a relatively easy way to install pycurl on amzn2023 linux:

1. shall i make the pycurl a requirement and keep urllib3_client broken in the code == continue this branch/PR
2. shall i make a different pr with reverted+fixed(bytes) pycurl implementation (and requirement) and remove the urllib3_client
3. and later make a PR to specify/monkey-patch the http client switching. possibly, giving an opportunity to implement httpx/aiohttp/requests/urllib3/etc (but then urllib3 or even http.client should become the default one instead of pycurl)
4. use more time to fix urllib3 client
   ????

the problem with 3 is the time might never come for me 😝 as now i am still re-implementing it as in 4

let's vote:
👍 - option1
❤ - option2
🎉 - option3
🚀 - option4

[Nusnus]

I prefer we revert the dep change to go “back to normal”, and release v5.6 quickly.
We can maintain v5.5.x and v5.6.x to keep patch fixes in both versions.

use more time to fix urllib3 client

Assuming we revert the change completely, we have all the time in the world to get it done right IMHO.
You’re already at it and we’d be glad to help and support the development while maintaining the standard releases reverted until we validate the given issues.

---

The motivation is that v5.5.x and v5.6.x should be exactly the same except for the pycurl change (reverted back in v5.6), so we can progress with removing python 3.8 for v5.7 and “free” main from this issue while allowing silently skipping v5.5 without losing anything at all but keeping the pycurl from v5.4 and below.

@spawn-guy Does that make sense to you?

[Nusnus]

@soceanainn

This release is completely broken for anyone using SQS in production as the performance decreases so drastically.

This change in performance is not something people expect as part of a minor version release.

I would strongly urge the team to revert the change causing this to avoid more people running into real production incidents from upgrading to this bad version of Kombu.

You’re right this is a serious issue and I understand the frustration 🙏
Please see my previous comment about our plans to completely revert the change without causing additional harm.

Also take note @spawn-guy is very active and responsive so we’re in good hands and we’ll have a proper solution soon.

EDIT:
We can expect v5.6 to be released with the reverted changes in the next few weeks, hopefully less, but no longer than the end of the month.

[auvipy]

Yes lets just revert it