Migrate to official node-redis

[nodegin]

Is your feature request related to a problem? Please describe.

ioredis is being non-actively maintained for a while, it seems will be deprecated soon.
redis/node-redis#2658 (comment)

Describe the solution you'd like
Migrate to https://github.com/redis/node-redis

Describe alternatives you've considered
n/a

Additional context
n/a

[manast]

Yeah, this is a quite a sad situation, it almost seems like Redislabs is being run by a chicken without head... most frustrating is the lack of communication with the community, leaving everybody wondering what is going on.

But from BullMQ's point of view, there is not a single issue with IORedis that we are aware of, and would such an issue arise we will fork IORedis and fix it for us. The only feature that would be relevant in terms of performance would be to have support for RESPv3 but neither IORedis nor node-redis supports it.

The APIs of IORedis and node-redis are different enough to make the work of supporting node-redis non-trivial, and there is no guarantee it will be better maintained either, in fact the number of commits seems to be quite low too...

If somebody really wants to use node-redis, one possibility would be to create a wrapper on top of node-redis that implements the IORedis specific APIs that are used by BullMQ... but I do not think this is worthy right now.

[manast]

Btw, it could be that some security advisory arises in IORedis that forces us to fork it even though there are no other issues, but lets give Redislabs a chance to make things right.

[codeagencybe]

What about moving to keydb or dragonfly? Both are decent drop-in replacements for redis and both open source.
in terms of performance, they both claim to be much faster than redis too.
Sounds like multiple wins then?

[manast]

Dragonfly is supported, as it is 100% compatible with Redis (https://bullmq.io/news/101023/dragonfly-compatibility/). KeyDB did not properly work with BullMQ last time I tested and probably does still not work but I am not sure.

In any case this discussion is about the NodeJS client library and less about the change in Redis license. I do not think the new Redis license affects regular users, but instead hosting providers which will now need to get a commercial license agreement from Redis in order to provide the newer versions. If I have to guess, these changes will just encourage new forks and competition to fill the need for a robust in-memory database, so not bad news for current Redis users in the long run.

[ivan-kleshnin]

Describe the solution you'd like
Migrate to https://github.com/redis/node-redis

It's not well supported either 😞
Look at the number of issues, at the absence of comments there, hanging PRs, look at the release history.

[manast]

@ivan-kleshnin yes, thats why I am more confident to continue using just IORedis, as over the years @luin has fixed the issues that affected BullMQ and AFAIK there are no known issues that affect us... just keeping IORedis up-to-date regarding security updates should not be a big effort, but obviously some maintainer must merge the PRs and make the releases otherwise nothing will happen.

[jamesarosen]

But from BullMQ's point of view, there is not a single issue with IORedis that we are aware of, and would such an issue arise we will fork IORedis and fix it for us.

That's true, but there are plenty of bugs in ioredis that don't directly affect BullMQ. If a job uses Redis for something else (like geodist calculations), it likely needs to load two redis clients. That's a waste of developer time, CPU time, and CO2.

[elovin]

I doubt that the official redis client will address any issues related to one of the redis forks
(e.g. https://github.com/valkey-io/valkey).
And since the usage of redis forks will increase due to the redis license change,
I do not think it is a good idea to replace ioredis with node-redis.

[elovin]

Ups I totally missed that ioredis is also part of the redis org.

[lukas-becker0]

FYI @manast , the valkey guys already forked ioredis and call it iovalkey now.

(AFAIK valkey this is now the Redis fork with the biggest budget, since a number of big companies are backing it.)

[manast]

As long as IORedis works reliably for us, I do not see any benefit in using any other client... it will just introduce more risk and more likely new issues until all edge cases are ironed out.

[sirrodgepodge]

redis/ioredis#1865

^This one is busted :/, reason enough to switch?

[manast]

@sirrodgepodge the thing is, how do we know node-redis does not have other issues that would impact us? so if we do all the migration work (which is not completely trivial), and then we discover other break-dealer issues, what then?
I think the most sensible thing to do is try to fix the issues in ioredis that affect BullMQ, worst case we could fork the project, but that is not an ideal situation either. It is quite incredible that Redis took over the maintenance of IORedis just to not maintain it. At least when the maintainer was @luin it tried its best to solve the issues...

[lukas-becker0]

@sirrodgepodge, seems like the issue you linked is "ElastiCache Serverless" specific, since people using
regular redis or dragonfly do not seem to have that issues.

[ozonep]

Instead of creating yet another fork, perhaps you could consider adopting and contributing to the iovalkey client (like mentioned above)? My team has had a very positive experience with it. We submitted a PR to the iovalkey repository with stability fixes for clusters (no new features), and it was properly reviewed and merged within a week. In contrast, our original PR on ioredis has been open since March without any comments from the maintainers and will probably never be reviewed and merged. Additionally, the iovalkey client currently doesn't introduce any breaking changes compared to ioredis — just bug fixes.

[avifenesh]

Hi,
valkey-glide maintainer, and AWS elasticache team member.
In addition to #3083,
I offer valkey-glide as a replacer, and we are willing to help with the migration.
Glide offers the best implementation of PubSub with the best ability to resubscribe on reconnection.
We offer the best fault tolerance (please test us).
We have team members from AWS elasticache/memoryDB and GCP memory store involved deeply in the development, we are part of valkey org, and we have a large community (59 contributors).
I'm the Node “leader” and I will love to see glide being integrated and packages as yours.
So if it's relevant, I offer my support for the migration, as part of the planning, and implementation.
Please let me know WDYT.

[manast]

@avifenesh hello!
This is the first time I heard about GLIDE to be honest, but I checked the documentation and it looks like an interesting project, where you have focused on reliability which is really important for BullMQ users as well.
I have been thinking for a while to support other clients such as node-redis, however in order to do this, we need to refactor the internals of BullMQ to use a generic Redis client interface that can then be implemented with IORedis, Glide or any other. It's a bit of work actually but it is doable, once this generic interface is available it should be relatively straightforward to implement it for Glide.

[avifenesh]

@manast Hi :)
Yes, we are relatively young. But the client is built over our years of supporting customers, and aggregating the data of what is the real pain. Failovers, connection storm, topology changes detections, PubSub disconnection, and so on.
So before features, we first built the core stable and reliable, knowing what was missing in all the other clients we dealt with when users had problems.

In general, I actually offer as non-generic backend :)
But as a replacement for your next major, when it's possible to replace then the (to some degree, transparent) package in the backend.
We support Redis OSS versions, so no breaking changes, but following valkey.
While for you directly, it might not be an issue since you are an OSS project, but for your users, following valkey might be crucial.
Furthermore, Valkey is probably the future of the in-memory key-value stores.
It is open source, it offers memory and performance benefits (1M TPS, AZ affinity, data-structure improvements etc.) and it's backed by the industry.
As a valkey officials, we are always updated with the last features of valkey.
For example, we offer the AZ affinity, which for users of Redis/Valkey hosted on, or managed by, cloud providers, it's a big cost reduction, and performance optimization (the performance is not deterministic but likely, the cost's reduction is, transaction in the same AZ are free), or for 8.1 which about to be released soon hashes fields TTL.

But -
Also, as a generic, it makes a lot of sense. More work, obviously, but might be worth it if you have many users, using BullMQ with different clients, where it's not transparent for them what client is behind the scene.

Anyway, in both cases, I will be happy to help. Can take some tasks and collaborate. Our users are our priority, and it's enough for us that one of them open an issue, and I believe that there's more that it is important for them :)
I don't have familiarity with the codebase itself, Valkey/Redis is basically my work, with a strong emphasis on client's world, so I can give what possible from my side.

[avifenesh]

ah, and we have a python client as well :)
Also, Java.
Go just released to public preview, and C# is next.
CPP and ruby are under community development.
So you might be able to grow beyond :)