[countersyncd]: Harden and optimize HFT IPFIX processing

[Pterosaur]

What I did

This change hardens and optimizes the countersyncd HFT IPFIX processing path.

HFT decoder

• Replace the generic ipfixrw object-graph parser with a strict, fixed-width SONiC HFT decoder.
• Compile template field offsets, SAI IDs, and object-name references once at template registration.
• Decode counters directly from wire bytes in template order without per-record template clones, field hash maps, or per-field byte-vector allocations.
• Scope templates by (observation_domain_id, template_id) instead of a process-global template ID.
• Keep decoder state in IpfixActor rather than thread-local storage.

Template lifecycle

• Validate complete template updates before changing active state.
• Keep active and pending generations concurrently so old IDs continue decoding until the first valid new-ID data set arrives.
• Make pending cancellation and supersession transactional.
• Reject in-place schema changes that reuse the same scoped template ID because queued old data cannot be distinguished without an exporter generation marker.
• Tombstone deleted or canceled template IDs for the process lifetime; a restart or future exporter-generation protocol is required before reuse.
• Buffer an ordered, bounded prefix of data sets that arrive before their template and replay them without reordering later sets.

Safety and backpressure

• Reject non-progressing IPFIX lengths, malformed set boundaries, invalid padding, zero-field templates, unsupported standard IEs, variable/non-8-byte HFT fields, zero enterprise numbers, duplicate fields, and unmapped labels.
• Require exactly one 8-byte observationTimeNanoseconds field and at least one enterprise counter per HFT template.
• Bound template bytes, object metadata, active/retired template keys, deferred data sets, deferred bytes, batch counters, and SAI batch-channel capacities.
• Deliver immediately to available sinks before awaiting full sinks while retaining strict bounded backpressure.
• Propagate closed-sink errors to the supervisor.
• Send flat SAIStatsBatch messages and share object names with Arc<str>.
• Replace the shared communication-statistics mutex with fixed atomic slots.
• Deactivate stale IPFIX state when the SWSS session becomes disabled or changes type.

Why I did it

The previous hot path built a generic IPFIX object graph for every message. It cloned the expanded template per record, allocated a HashMap per record and a Vec<u8> per field, scanned fields repeatedly, cloned every object name, and sent every record as a separate channel message.

Malformed length zero could also spin forever, template deletion did not remove parser state, template identity ignored observation domains, one unknown template could discard unrelated sets, and a slow sink serialized delivery to every other sink.

Performance

Criterion workload: approximately four million counters per case, release build, swss-bookworm-master, one pinned CPU for the optimized run.

Dataset	Upstream	This change
one template, 2 counters	1.35 M counters/s	7.72 M counters/s
one template, 8,000 counters	3.50 M counters/s	35.59 M counters/s
five keys, one large template each	3.48 M counters/s	36.21 M counters/s
five keys, four large templates each	3.44 M counters/s	36.16 M counters/s
five keys, mixed templates	3.41 M counters/s	31.97 M counters/s

These numbers are not a pure parser-only A/B. The benchmark was also corrected to prebuild inputs, use a causal readiness probe, and exclude setup from the measured interval; the old benchmark generated and copied random records inside its timed path. The source-derived result that does not depend on timing is that the new decoder removes per-record template clones and field maps, per-field byte-vector allocations, and per-counter object-name allocation.

How I verified it

CI-equivalent environment:

• sonicdev-microsoft.azurecr.io:443/sonic-slave-bookworm:master-amd64
• image digest sha256:073697e9f029974f00dc9eb8f11c81d664082e2f4b802ec77f2408eb30577aa9
• latestFromBranch(master) common-lib, swss-common, and sairedis artifacts
• Redis configured like .azure-pipelines/build-template.yml
• Rust/Cargo 1.86.0

Commands:

RUSTFLAGS=-Dwarnings cargo metadata --locked --no-deps --format-version 1
RUSTFLAGS=-Dwarnings cargo check --locked -p countersyncd --all-targets
RUSTFLAGS=-Dwarnings cargo test --locked -p countersyncd
RUSTFLAGS=-Dwarnings cargo test --locked -p countersyncd -- --test-threads=1
RUSTFLAGS=-Dwarnings cargo bench --locked -p countersyncd --no-run
RUSTFLAGS=-Dwarnings cargo build --release --locked -p countersyncd

Final result: 242 tests passed in both parallel and serial runs. Focused IPFIX tests cover malformed framing, random-input panic safety, multi-domain IDs, transactional active/pending lifecycle, supersession/rollback, delete tombstones, ordered deferred replay, malformed deferred data, multiple recipients, closed/full sinks, and deterministic field order.

HFT profile limitations

This is intentionally a SONiC HFT decoder, not a general RFC 7011 collector. It currently requires:

• normal Template Set ID 2 only;
• fixed 8-byte observationTimeNanoseconds encoded as the raw SONiC HFT u64 timestamp;
• fixed 8-byte enterprise counter fields;
• the current countersyncd private enterprise-number layout (high half SAI type ID, low half SAI stat ID);
• template withdrawal through the SWSS session delete/disable control path.

Options Templates, on-wire withdrawals, variable-length fields, generic IANA information elements, and RFC NTP timestamp decoding are rejected.

Follow-up issues not solved here

• Orchagent still starts streaming before a countersyncd template-ready acknowledgement exists.
• Netlink receive polling currently processes one datagram per default timer tick and still copies payload buffers.
• Transport/exporter generation identity is not carried with the netlink payload; therefore retired scoped template IDs cannot be safely reused during one daemon lifetime.
• IPFIX sequence gaps, duplicates, and exporter resets are not tracked.
• OTEL still casts full-width u64 counters to i64 and does not enforce an encoded-byte request limit.
• The HFT HLD enterprise-number example (stat << 16 | type) conflicts with the existing countersyncd implementation (type << 16 | stat). This change preserves the existing Rust behavior and adds distinct type/stat tests; the HLD should be corrected separately.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Pterosaur]

Superseded by #4860, which contains the same final diff as one DCO-signed commit.

[Copilot]

🔵 Needs a closer look

It’s a large behavioral refactor across decoding, actor messaging, backpressure, and lifecycle management where correctness depends on subtle ordering/capacity invariants that warrant final human review.

Pull request overview

This PR hardens and substantially optimizes countersyncd’s SONiC HFT IPFIX processing path by moving to fixed-width decoding, tightening template lifecycle rules, and batching/deduplicating message flow between actors to reduce allocations and improve backpressure behavior.

Changes:

• Introduces/propagates flat SAIStatsBatch messages (record-preserving, multi-record per channel item) and switches object names to Arc<str> to reduce per-sample allocation/cloning.
• Adds stronger input validation and bounded buffering/capacity controls (including CLI parsing bounds) and improves comm-stats hot-path behavior by removing a shared mutex.
• Updates integration tests and benchmarks to use readiness barriers/probes and the new batching semantics.

File summaries

File	Description
crates/countersyncd/tests/ipfix_test_helpers.rs	Adds object-metadata generation and strengthens IPFIX length handling to prevent stalling in record generation.
crates/countersyncd/tests/ipfix_helpers_integration.rs	Refactors integration test to consume SAIStatsBatch records, adds readiness and delete-barrier probes, and updates schema-change expectations.
crates/countersyncd/tests/integration_test.rs	Updates end-to-end tests for batched SAI stats delivery and revised IPFIX record layout expectations.
crates/countersyncd/src/utilities/mod.rs	Reworks comm-stats tracking from mutex+map to fixed atomic slots and optimizes hex formatting.
crates/countersyncd/src/message/saistats.rs	Replaces per-record message type patterns with SAIStatsBatch, introduces SAIStatsRef, and centralizes enterprise-number ID decoding.
crates/countersyncd/src/message/otel.rs	Updates OTel conversion to consume borrowed SAIStatsRef/Arc<str> object names rather than owned strings.
crates/countersyncd/src/main.rs	Adds IPFIX join classification, bounds channel capacities via clap value parsers, and updates default capacities for batch channels.
crates/countersyncd/src/actor/swss.rs	Tightens HFT metadata validation and ensures disabled/non-IPFIX sessions actively deactivate prior IPFIX state via deletes.
crates/countersyncd/src/actor/stats_reporter.rs	Switches stats input to SAIStatsBatchMessage and updates aggregation paths to iterate per record.
crates/countersyncd/src/actor/otel.rs	Switches to SAIStatsBatchMessage and processes records in-order while preserving flush/backoff behavior.
crates/countersyncd/src/actor/counter_db.rs	Switches to batched stats input and updates caching/write pipeline to iterate per record.
crates/countersyncd/Cargo.toml	Removes unused dependencies (ipfixrw, binrw, rand) as the strict decoder path replaces the generic parser approach.
crates/countersyncd/benches/otel_actor_perf.rs	Updates benchmark message generation to produce batched stats and Arc<str> object names.
crates/countersyncd/benches/ipfix_bench_data.rs	Reworks dataset/template preparation, adds readiness probe template/record, and includes object metadata.
crates/countersyncd/benches/ipfix_actor_perf.rs	Updates perf benchmark to use readiness probe and batched stats accounting.
crates/countersyncd/benches/end_to_end.rs	Updates end-to-end bench to use readiness drain/verification and batched channels for all sinks.
crates/countersyncd/benches/counter_db_actor_perf.rs	Updates CounterDB benchmark to send multi-record batches and validates actor completion.
Cargo.toml	Removes workspace-level unused deps corresponding to dropped generic IPFIX parser tooling.
Cargo.lock	Drops transitive deps tied to removed crates and normalizes syn/owo-colors entries accordingly.

Review details

• Files reviewed: 18/20 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.