Does AMQP_1 output support function interpolation for target_address ?

[AbelThorne]

I need to send a stream of JSON events to an AMQP cluster where the topic is determined dynamically by the message content (value of a specific field). Will target_field support "function interpolation" and a syntax like this one:

target_address: /events/${! json("device_id") }

If that's not an option, any idea on how I could achieve the same result? (the high cardinality of field device_id prevents a manual approach via switch/case 😢 )

Thanks a lot 🙏

[mihaitodor]

Hey @AbelThorne 👋 Unfortunately, that field does not support interpolation (see here)... Looking at the code, I think maintaining a large pool of sender instances as opposed to one (see here) might lead to resource starvation and the code behind session.NewSender() looks non-trivial, so I think we'd kill performance if we'd create the sender object for each message. How many device IDs do you have? Maybe try adjusting the code on your end (you can copy / paste the existing output as a custom plugin as described here) and see how it performs with the required changes.

[AbelThorne]

Hi @mihaitodor, I think I finally identified what I'm missing.

The plugin does not support Anonymous Terminus behavior.

AMQP protocol supports the creation of a Sender with a nil targetAddr. You can then use the to property in the message to give the proper destination per message.

The plugin actually supports the AMQP to property (message_property_to in rpc configuration) but without support for interpolation which makes it less useful to leverage the message fields as a destination discriminator.
Also, you can't leave the target_address empty as it is required.

What do you think?

For additional context, I need to send messages to a Solace cluster using AMQP protocol and Solace makes a heavy use of this Anonymous Terminus pattern for dynamic routing of messages in different topics via a limited number of connections.

[mihaitodor]

I see, thank you for digging into that! Sounds like it wouldn’t require a lot of work, but I don’t have any cycles to spare right now. If you are happy to play around with Go, then you can copy / paste the existing implementation into a plugin and adjust it. Also, PRs are welcome!

[AbelThorne]

I'll try to make a PR 😊

[AbelThorne]

Hi @mihaitodor, sorry to come back to this question. Did you have time to look at my PR?
I really don't want to push but I'm under some pressure to move forward on this. I'd rather use the main codebase with the PR merged but if you think that's going to take some time, I'll make a custom plugin and run it on a dedicated image until the PR is merged. I'd just like to avoid unnecessary work if possible 😊

Thanks a lot for your time and consideration

[mihaitodor]

Thank you for the PR! I don’t have permissions to merge code, so you’ll have to wait until someone from the Connect team gets a chance to look at it