Slower performance after migrating from kafka_franz output to redpanda

[danspark]

Since kafka_franz is being phased out, I recently migrated to the redpanda output, which doesn't support batching in the same way kafka_franz does. Is there some way of configuring batching similarly to the kafka_franz output?

[mmatczuk]

Thanks for reporting @danspark! As far as I can tell redpanda is missing the batching field and uses much higher max_in_flight 256 vs. 10. Could you share more information on the config options you are using?

[danspark]

Here's what I found out after some digging:

I got degraded performance on connect instances that where setting a prefetch (amqp_0_9) on the input with a higher value than the default (250 configured x 10 default).

apps that did not manually configure the prefetch

apps that DID manually configure the prefetc

Even after removing the default prefetch configuration, the baseline increased considerably. The following is a staging deployment in the following duration order (p95):

1. manual prefetch configuration (250) and redpanda output
2. default prefetch and redpanda output
3. manual prefetch configuration and kafka_franz output (with batching)

[mmatczuk]

Right, taking of kafka_franz vs redpanda could you check the batching and max_in_flight settings.

[danspark]

kafka_franz batching was previously configured like this:

• batching.count: 250
• batching.period: 100ms
• max_in_flight (unchanged): 10

redpanda output was unchanged, so 256 as max_in_flight.

[danspark]

Looks like I won't be needing to investigate this for long since batching is being added to the new output

[danspark]

After batching was added to the redpanda output performance was pretty the same