redo: row-count flush threshold increases S3 checkpoint lag

[wlwilliamx]

What did you do?

1. Run a TiCDC changefeed with redo logs stored in S3.
2. Upgrade TiCDC from v6.5.12 legacy architecture to v8.5.7 new architecture.
3. Run a recurring nightly workload containing large transactions and several
   TRUNCATE DDL statements.
4. Compare checkpoint lag and redo flush behavior before and after the upgrade.

What did you expect to see?

Checkpoint lag should remain comparable to v6.5.12 under the same recurring
workload. Redo log files should normally be flushed according to
max-log-size or flush-interval, without excessive small-object requests to
S3.

DDL events following a large transaction should not be delayed significantly
by redo log persistence.

What did you see instead?

After upgrading to v8.5.7 new architecture, checkpoint lag increases
significantly during the nightly workload. The problem did not occur with
v6.5.12 legacy architecture.

Observed behavior during the affected period:

• The new-architecture redo DML writer busy ratio remains at 100%.
• The redo flush duration sum rate remains around 0.95–0.98 seconds per second
  from 00:05 to 00:35.
• Each flush contains approximately 945–974 rows, close to the hard-coded
  1024-row threshold.
• Each redo file is only approximately 1.8–3.9 MiB, far below the configured
  64 MiB maximum log size.
• Average flush latency is approximately 142–251 ms.
• P90 flush latency is approximately 276–689 ms.
• P99 flush latency has a 1.5–4 second tail.
• DDL events are blocked behind preceding DML events in the ordered event path.

The new-architecture redo writer has a fixed row-count flush threshold near
1024 events. When this threshold is reached before max-log-size or
flush-interval, the writer synchronously waits for a small redo file to be
written to external storage.

For workloads with many relatively small row events, this serializes S3
request latency and produces many small objects. A large transaction can
require many such flushes before its callback completes, delaying the
checkpoint and subsequent barrier DDL events such as TRUNCATE.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

Not available in the current report.

Upstream TiKV version (execute tikv-server --version):

Not available in the current report.

TiCDC version (execute cdc version):

Before upgrade: v6.5.12, legacy architecture
After upgrade:  v8.5.7, new architecture