Context
Filed from the client-side review discussion on kubescape/node-agent#866 (kubescape/node-agent#866 (review)). node-agent is adding client-side logic to split an oversized ContainerProfile chunk (rejected by QueueManager with a plain HTTP 413) into two smaller chunks and resend both. To do that safely against the current consolidation algorithm, node-agent has to manufacture a synthetic intermediate timestamp so the two halves form a genuine two-link chain instead of two rows sharing the same reporting interval — adding real complexity (timestamp parsing with monotonic-clock-suffix stripping, non-whole-hour timezone fallback handling, a zero-time first-report edge case, etc.) purely to work around how consolidateContinuousTimeSeries walks the chain.
Problem
consolidateContinuousTimeSeries (pkg/registry/file/containerprofile_processor.go:631-659) collapses a series' time_series rows by walking them in reverse chronological order and merging row j with row i+1 only when timeSeries[j].PreviousReportTimestamp == timeSeries[i+1].ReportTimestamp. Two rows that instead share an identical (previousReportTimestamp, reportTimestamp) pair — which is a natural way to represent "this single logical report was split into two physical objects" — are not recognized as the same logical report. The chain walk sees them as a fork: len(newTimeSeries) ends up > 1, and updateProfileStatus (:672) then never reaches its Completed/expired branches for that series. The profile stays in Learning permanently once this happens.
Suggested fix
Treat two (or more) rows sharing an identical (previousReportTimestamp, reportTimestamp) pair as a single logical report for consolidation purposes — merge their data (this already happens regardless of chain shape via mergeContainerProfileTS) and count them as one link in the chain rather than two.
Why this matters beyond node-agent
Once this lands (and once node-agent can assume a minimum storage version has it — storage is deployed/versioned independently of node-agent, so this alone won't let node-agent remove its client-side workaround immediately), node-agent's chainHalves/timestamp-interposition shim (added in kubescape/node-agent#866) becomes unnecessary and can be deleted — a split could just leave both halves' timestamps identical instead of manufacturing an intermediate value. That would remove a meaningful chunk of client-side complexity that exists purely to compensate for this consolidation behavior.
Related
Context
Filed from the client-side review discussion on kubescape/node-agent#866 (kubescape/node-agent#866 (review)). node-agent is adding client-side logic to split an oversized
ContainerProfilechunk (rejected byQueueManagerwith a plain HTTP 413) into two smaller chunks and resend both. To do that safely against the current consolidation algorithm, node-agent has to manufacture a synthetic intermediate timestamp so the two halves form a genuine two-link chain instead of two rows sharing the same reporting interval — adding real complexity (timestamp parsing with monotonic-clock-suffix stripping, non-whole-hour timezone fallback handling, a zero-time first-report edge case, etc.) purely to work around howconsolidateContinuousTimeSerieswalks the chain.Problem
consolidateContinuousTimeSeries(pkg/registry/file/containerprofile_processor.go:631-659) collapses a series'time_seriesrows by walking them in reverse chronological order and merging row j with row i+1 only whentimeSeries[j].PreviousReportTimestamp == timeSeries[i+1].ReportTimestamp. Two rows that instead share an identical(previousReportTimestamp, reportTimestamp)pair — which is a natural way to represent "this single logical report was split into two physical objects" — are not recognized as the same logical report. The chain walk sees them as a fork:len(newTimeSeries)ends up> 1, andupdateProfileStatus(:672) then never reaches itsCompleted/expired branches for that series. The profile stays inLearningpermanently once this happens.Suggested fix
Treat two (or more) rows sharing an identical
(previousReportTimestamp, reportTimestamp)pair as a single logical report for consolidation purposes — merge their data (this already happens regardless of chain shape viamergeContainerProfileTS) and count them as one link in the chain rather than two.Why this matters beyond node-agent
Once this lands (and once node-agent can assume a minimum storage version has it — storage is deployed/versioned independently of node-agent, so this alone won't let node-agent remove its client-side workaround immediately), node-agent's
chainHalves/timestamp-interposition shim (added in kubescape/node-agent#866) becomes unnecessary and can be deleted — a split could just leave both halves' timestamps identical instead of manufacturing an intermediate value. That would remove a meaningful chunk of client-side complexity that exists purely to compensate for this consolidation behavior.Related