Redis Proxy improvements to Support commands PubSub , Streams , Scripts. Info... 50+ commands

[dinesh-murugiah]

Title: Redis proxy - More classes and Design patterns to support additional commands

Description:
HI i am Working for https://www.freshworks.com/ . We have self hosted redis cluster in our organisation
we are using Envoy proxy for our clients interaction . The Envoy in its current form lacks support many commands
To make it useful for us we have made changes in envoy to support more than 50+ commands and its been running in production

We would like to contribute those changes to envoy community ,
starting this issue thread to initiate a conversation and understand is anyone else is working on such improvements

Short summary of Proposal

1. The current redis proxy code follows a chain of responsibility and observer design pattern , such that the commands received in proxy filter is linearly processed in chain and response is sent back
2. This design pattern will not allow for supporting commands like pubsub, blocking commands , which requires dedicated 1:1 connections between redis and client , rather than going via upstream connection pool
3. The proposal is to extend or transform the transaction class in to a bridge class , which will have abstractions to support implementations for pubsub , blocking , transaction commands , based on the input command appropriate implementation will be invoked with a dedicated upstream client
4. When the response is received for such commands , the regular chain of flow will not happen , we can have a mediator object in client impl which will define which flow the response should take , either directly to command filter in case of pubsub commands or regular chain for other commands

we are preparing a detailed document , before that just wanted to understand

1. is community willing to take such changes
2. is anyone already working on such changes
3. what documents are expected from us to take this as a proposal and then implementation

Below are the list of commands for which we have added support in envoy in our local repo

sunion | sdiff | sinter | sinterstore | zunionstore | zinterstore | pfmerge | georadius | georadiusbymember | xrevrange
subscribe | psubscribe | unsubscribe | punsubscribe | publish | pubsub | brpoplpush | bzpopmax | bzpopmin | blpop
brpop | script | flushall | keys | slowlog | config (set & get) | client (getname setname) | info | cluster (slots keyslot) | select
scan | hello* | XCLAIM | rename | xadd | xread | xdel | xtrim | xlen | xrange
getdel | GETEX | SORT | ZMSCORE | SDIFFSTORE | MSETNX | SUBSTR | ZRANGESTORE | ZUNION | ZDIFF
XAUTOCLAIM | XGROUP | XINFO | SUNIONSTORE | SMISMEMBER | HRANDFIELD | GEOSEARCHSTORE | XREADGROUP | ZDIFFSTORE | GEOSEARCH
RANDOMKEY | ZINTER | BLMOVE | ZRANDMEMBER | BITOP | LPOS | RENAMENX |  

[optional Relevant Links:]

[adisuissa]

Thanks!
cc'ing redis filter code-owners @msukalski @HenryYYang @mattklein123 @weisisea

[MENNAELMESALMY]

Hey @dinesh-murugiah I am interested in PubSub support, I have a question about PubSub though, if this is added per your proposal, will each client have their own dedicated connection or is there some kind of pooling that will happen?

[dinesh-murugiah]

hi @MENNAELMESALMY for pub sub connections alone it will not be pooled , dedicated upstream clients will be created for each downstream pubsub connection

[MENNAELMESALMY]

Thanks!
do you know how tricky it is to make it pooled? the main thing we used Envoy for was connection pooling as we have a lot of clients and will possibly run into concurrent clients limitation of Redis (64k) and also the performance degrades with increased client connections.

[dinesh-murugiah]

does your usecase demand many pubsub connections ? In my opinion it brings un necessary complexity to multiplex pubsub connections . In our envoy proxy changes we made sure we distributed the pubsub connections across shards , so that they are evenly distributed . You can horizontally scale the cluster if you need to accomodate more pubsub commands - Just think of it like without a proxy for Pubsub , as this command demands it

[MENNAELMESALMY]

does your usecase demand many pubsub connections ?

yes we have a lot subscribers into single-topic

In our envoy proxy changes we made sure we distributed the pubsub connections across shards , so that they are evenly distributed . You can horizontally scale the cluster if you need to accomodate more pubsub commands - Just think of it like without a proxy for Pubsub , as this command demands it

will the connection be established to a single node in the cluster? so if I were to have 120k subscribers I could have 2 replica nodes in a shard and it should theoretically be fine?

[dinesh-murugiah]

@MENNAELMESALMY

yes we have a lot subscribers into single-topic
Hope you are running redis in cluster mode currently , or is it in standalone mode?

will the connection be established to a single node in the cluster? so if I were to have 120k subscribers I could have 2 replica nodes in a shard and it should theoretically be fine?

No the connections will be spilt across multiple shards of the cluster , to balance the load.
In our current implementation we have not classified the subscribe as a read command , but it would be a good idea to do it so that client can subscribe from replicas also

[veetar]

@dinesh-murugiah , XGROUP and XREADGROUP should be working as simpleCommands or WriteCommands right ?
Can you please share your view on the same?

[dinesh-murugiah]

@veetar Yes you are right XGROUP and XREADGROUP are simple commands

[zygisa]

👋 @dinesh-murugiah is the code with the changes mentioned above available publicly? We'd be interested in trying it out if it is.

[dinesh-murugiah]

@zygisa
https://github.com/freshworks/envoy/tree/v1.28.0-dbaas-RELEASE
This is our public repo with above features
NOTE: This has some not so elegant code/Design changes which we plan to repurpose based on community traction, and also it does not have inbuilt test cases (we have external test suites) for features added , which we plan to add sooner
But the code is quality tested and solid and running in our production

PS: use the above mentioned branch only