Skip to content

perf: rollup Week/Quarter from Day-level queries - #210

Merged
PMerlet merged 4 commits into
mainfrom
perf/cosmos-date-agg-rollup
Mar 18, 2026
Merged

perf: rollup Week/Quarter from Day-level queries#210
PMerlet merged 4 commits into
mainfrom
perf/cosmos-date-agg-rollup

Conversation

@PMerlet

@PMerlet PMerlet commented Mar 18, 2026

Copy link
Copy Markdown
Member

Summary

  • Week and Quarter aggregations now query at Day granularity (LEFT(field, 10)) instead of using expensive Cosmos DB computed expressions (DateTimeAdd, DateTimePart, FLOOR, CONCAT) in GROUP BY
  • Daily results are rolled up into week/quarter buckets in application-level post-processing
  • This eliminates timeouts on large datasets (the previous Week query timed out after 25s on the Xpollens Operations collection)

Before (slow, times out)

SELECT LEFT(DateTimeAdd("day", -1 * ((DateTimePart("dw", c.operationDate) + 5) % 7), c.operationDate), 10) as groupKey, COUNT(1) as aggregateValue FROM c GROUP BY ...

After (fast, ~5 RUs)

SELECT LEFT(c.operationDate, 10) as groupKey, COUNT(1) as aggregateValue FROM c GROUP BY LEFT(c.operationDate, 10)

Then weeks/quarters are computed in processAggregationResults.

Test plan

  • 47 unit tests pass, including:
    • Week rollup: daily rows merged into Monday-keyed weeks
    • Quarter rollup: daily rows merged into quarter-start dates (01-01, 04-01, 07-01, 10-01)
    • Sunday correctly rolls to previous Monday
    • All four quarters handled
    • Combined with WHERE conditions and field aggregations
  • Tested live on Xpollens Cosmos DB — Day chart returns 200 in ~850ms with 5.43 RUs

🤖 Generated with Claude Code

Note

Roll up Week/Quarter aggregations from Day-level Cosmos DB queries in post-processing

  • Previously, Week and Quarter groupings used complex Cosmos DB SQL expressions (DateTimePart, DateTimeAdd, CONCAT, FLOOR) to compute bucket boundaries, which had reliability issues.
  • Now, AggregationConverter.buildDateGroupExpression groups by day (LEFT(field, 10)) for Week and Quarter, and new rollupResults post-processes daily rows into Monday-starting weekly or quarter-start buckets.
  • Reduction semantics are handled per operation: Max/Min pick extrema, Sum/Count add, and Avg divides the accumulated sum by the number of days per bucket.
  • Behavioral Change: Week and Quarter query results are now computed in application memory rather than in Cosmos DB SQL, and weeks start on Monday.

Macroscope summarized 1a1e2b9.

…ssing

Week and Quarter aggregations now use LEFT(field, 10) (Day-level GROUP BY)
which is fast and index-friendly, then roll up daily rows into
week/quarter buckets in application code. This avoids expensive computed
expressions (DateTimeAdd, DateTimePart, FLOOR, CONCAT) in Cosmos DB
GROUP BY that caused timeouts on large datasets.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread packages/datasource-cosmos/src/utils/aggregation-converter.ts
PMerlet and others added 2 commits March 18, 2026 16:25
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rollupResults was always summing values regardless of the aggregation
operation, producing wrong results for Max, Min, and Avg. Now applies
Math.max for Max, Math.min for Min, and weighted average for Avg.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread packages/datasource-cosmos/src/utils/aggregation-converter.ts Outdated
When group field values are null or undefined, String() produces
"null"/"undefined" which corrupts getMonday/getQuarterStart date
parsing. Now skips these entries during rollup.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@PMerlet
PMerlet merged commit 9528843 into main Mar 18, 2026
2 checks passed
@PMerlet
PMerlet deleted the perf/cosmos-date-agg-rollup branch March 18, 2026 15:51
forest-bot added a commit that referenced this pull request Mar 18, 2026
forest-bot added a commit that referenced this pull request Mar 27, 2026
# [2.9.0](https://github.com/ForestAdmin/forestadmin-experimental/compare/rpc-agent@2.8.0...rpc-agent@2.9.0) (2026-03-27)

### Bug Fixes

* allow aggregation by date ([#208](#208)) ([4622350](4622350))
* use Cosmos DB compatible functions for date aggregation ([#209](#209)) ([f4be4e2](f4be4e2))

### Features

* **api chart:** allow customizing api chart with query or body parameters ([#211](#211)) ([6d3eb65](6d3eb65))

### Performance Improvements

* rollup Week/Quarter from Day-level queries ([#210](#210)) ([9528843](9528843))
forest-bot added a commit that referenced this pull request Apr 24, 2026
# [1.12.0](https://github.com/ForestAdmin/forestadmin-experimental/compare/datasource-rpc@1.11.0...datasource-rpc@1.12.0) (2026-04-24)

### Bug Fixes

* allow aggregation by date ([#208](#208)) ([4622350](4622350))
* **datasource-cosmos:** increase maxConditions default to 1000 ([#212](#212)) ([96df1f5](96df1f5))
* fix aggregation when using a reserved keyword ([#207](#207)) ([7200f0b](7200f0b))
* **rpc agent:** release ([8f532a0](8f532a0))
* **rpc datasource:** fix introspection parsing ([b3978a1](b3978a1))
* use Cosmos DB compatible functions for date aggregation ([#209](#209)) ([f4be4e2](f4be4e2))

### Features

* **api chart:** allow customizing api chart with query or body parameters ([#211](#211)) ([6d3eb65](6d3eb65))

### Performance Improvements

* rollup Week/Quarter from Day-level queries ([#210](#210)) ([9528843](9528843))
forest-bot added a commit that referenced this pull request Aug 5, 2026
# [2.9.0](https://github.com/ForestAdmin/forestadmin-experimental/compare/plugin-filtered-relationship@2.8.0...plugin-filtered-relationship@2.9.0) (2026-08-05)

### Bug Fixes

* add an interface of chart ([#152](#152)) ([3e93638](3e93638))
* add requested_id in approval request ([#172](#172)) ([0a2876c](0a2876c))
* **agent-tester:** add permission level for current user to support rails agent ([#146](#146)) ([7c50a2f](7c50a2f))
* **agent-tester:** change collection name on segment export csv ([#165](#165)) ([18a11a0](18a11a0))
* **agent-tester:** chart are not well loaded because a bad http method was used ([#147](#147)) ([4147916](4147916))
* **agent-tester:** force release ([#153](#153)) ([99864e0](99864e0))
* **agent-tester:** force the release ([#150](#150)) ([7a88a43](7a88a43))
* **agent-tester:** remove load form and load layout ([#154](#154)) ([2af8652](2af8652))
* **agent-testing:** allow null or undefined values on fill helper ([#142](#142)) ([0230844](0230844))
* **agent-testing:** fill helper on action form ([#143](#143)) ([fdf7bc7](fdf7bc7))
* allow aggregation by date ([#208](#208)) ([4622350](4622350))
* **ci:** test ci ([#178](#178)) ([4268e92](4268e92))
* **ci:** try the new setup ([#176](#176)) ([b03006f](b03006f))
* configure semantic-release npm plugin for OIDC auth ([ea6f439](ea6f439))
* cosmos nested agregation ([#162](#162)) ([a0c7c12](a0c7c12))
* **cosmos:** rename reserved aliases ([#205](#205)) ([8f72858](8f72858))
* **datasource-cosmos:** increase maxConditions default to 1000 ([#212](#212)) ([96df1f5](96df1f5))
* **datasource-cosmos:** set cosmos id as non sortable ([#174](#174)) ([29baba2](29baba2))
* **datasource-rpc:** prevent infinite loop in reconciliateRpc when using rename ([#221](#221)) ([febb11b](febb11b))
* explicitly set npm registry for publishing ([6b89230](6b89230))
* **filtered relationship:** enhance bahavior to handle more case ([#138](#138)) ([995cf9f](995cf9f))
* fix aggregation when using a reserved keyword ([#207](#207)) ([7200f0b](7200f0b))
* fix virtual object selection and edition ([#183](#183)) ([852ff32](852ff32))
* force release ([#151](#151)) ([76b93bc](76b93bc))
* implement pagination cache ([#192](#192)) ([4d80176](4d80176))
* improve nested fields support ([#164](#164)) ([8acc227](8acc227))
* improve performances ([#191](#191)) ([b798b08](b798b08))
* improve scalability ([#186](#186)) ([8f69026](8f69026))
* make database name optional ([#189](#189)) ([dc93255](dc93255))
* readability ([#190](#190)) ([95d4761](95d4761))
* release agent-rpc ([fd47a08](fd47a08))
* **rpc agent:** handle properly cross rpc relations ([#220](#220)) ([e222f02](e222f02))
* **rpc agent:** release ([8f532a0](8f532a0))
* **rpc agent:** use schema path option ([#215](#215)) ([4c700ec](4c700ec))
* **rpc datasource:** fix introspection parsing ([b3978a1](b3978a1))
* **rpc datatource:** allow connect ruby as rpc ([#219](#219)) ([5034d74](5034d74))
* rpc stack connection ([#169](#169)) ([e88a2ba](e88a2ba))
* **rpc:** handle sa schema properly with multi stack ([#218](#218)) ([c8016dc](c8016dc))
* use Cosmos DB compatible functions for date aggregation ([#209](#209)) ([f4be4e2](f4be4e2))
* use npm CLI with OIDC instead of semantic-release npm plugin ([d9e3386](d9e3386))

### Features

* **actions:** ease the field set and the retrieval ([#140](#140)) ([8a924b4](8a924b4))
* add cosmos db datasource ([#159](#159)) ([0b25187](0b25187))
* add datasource-stripe ([#198](#198)) ([cc3be30](cc3be30))
* add hasura graphql datasource ([#194](#194)) ([5db0b0a](5db0b0a))
* add introspection config ([#187](#187)) ([fc470d3](fc470d3))
* **agent-tester:** add a csv export on collection ([#156](#156)) ([fe64937](fe64937))
* **agent-tester:** add a way to override permissions from the agent tester ([#168](#168)) ([1d219cb](1d219cb))
* **agent-tester:** add charts to collection ([#149](#149)) ([3a2e8c8](3a2e8c8))
* **agent-tester:** add csv export on segment ([#157](#157)) ([0538935](0538935))
* **agent-tester:** add live query segment ([#158](#158)) ([1961202](1961202))
* **agent-testings:** split agent control and testing into separate directories ([#141](#141)) ([0bf4699](0bf4699))
* **api chart:** allow customizing api chart with query or body parameters ([#211](#211)) ([6d3eb65](6d3eb65))
* **approvals:** add a parameter in action.execute to validate approval ([#171](#171)) ([b8b4333](b8b4333))
* **cosmos:** add RU consumption logs ([#204](#204)) ([c7af33e](c7af33e))
* **datasource graphql hasura:** fix release ([a843269](a843269))
* **datasource graphql hasura:** relase datasource ([#206](#206)) ([c105758](c105758))
* **datasource-airtable:** add Airtable datasource package ([#200](#200)) ([155e6fa](155e6fa))
* **nested object:** use arrow notation ([#167](#167)) ([58a99e0](58a99e0))
* **rpc agent:** add mark collection as rpc option on add datasource ([#195](#195)) ([6372499](6372499))
* **rpc:** add introspection option ([f113580](f113580))
* **rpc:** handle SA returning file ([#217](#217)) ([08cc659](08cc659))
* support manual schema definition in cosmos ([#188](#188)) ([d819342](d819342))
* **testing:** add helper to get json field on action form ([#137](#137)) ([dfc25e9](dfc25e9))
* **testing:** introduce a rpc client ([#139](#139)) ([c2a3259](c2a3259))
* trigger release ([#185](#185)) ([1b378db](1b378db))

### Performance Improvements

* rollup Week/Quarter from Day-level queries ([#210](#210)) ([9528843](9528843))
forest-bot added a commit that referenced this pull request Sep 11, 2026
# [1.3.0](https://github.com/ForestAdmin/forestadmin-experimental/compare/plugin-define-enum@1.2.0...plugin-define-enum@1.3.0) (2026-09-11)

### Bug Fixes

* add an interface of chart ([#152](#152)) ([3e93638](3e93638))
* add requested_id in approval request ([#172](#172)) ([0a2876c](0a2876c))
* **agent-tester:** add permission level for current user to support rails agent ([#146](#146)) ([7c50a2f](7c50a2f))
* **agent-tester:** change collection name on segment export csv ([#165](#165)) ([18a11a0](18a11a0))
* **agent-tester:** chart are not well loaded because a bad http method was used ([#147](#147)) ([4147916](4147916))
* **agent-tester:** force release ([#153](#153)) ([99864e0](99864e0))
* **agent-tester:** force the release ([#150](#150)) ([7a88a43](7a88a43))
* **agent-tester:** remove load form and load layout ([#154](#154)) ([2af8652](2af8652))
* **agent-testing:** allow null or undefined values on fill helper ([#142](#142)) ([0230844](0230844))
* **agent-testing:** fill helper on action form ([#143](#143)) ([fdf7bc7](fdf7bc7))
* allow aggregation by date ([#208](#208)) ([4622350](4622350))
* **ci:** test ci ([#178](#178)) ([4268e92](4268e92))
* **ci:** try the new setup ([#176](#176)) ([b03006f](b03006f))
* configure semantic-release npm plugin for OIDC auth ([ea6f439](ea6f439))
* cosmos nested agregation ([#162](#162)) ([a0c7c12](a0c7c12))
* **cosmos:** rename reserved aliases ([#205](#205)) ([8f72858](8f72858))
* **datasource-cosmos:** increase maxConditions default to 1000 ([#212](#212)) ([96df1f5](96df1f5))
* **datasource-cosmos:** set cosmos id as non sortable ([#174](#174)) ([29baba2](29baba2))
* **datasource-rpc:** prevent infinite loop in reconciliateRpc when using rename ([#221](#221)) ([febb11b](febb11b))
* explicitly set npm registry for publishing ([6b89230](6b89230))
* **filtered relationship:** enhance bahavior to handle more case ([#138](#138)) ([995cf9f](995cf9f))
* **filtered relationship:** improve logging and replace name only if necessary ([bc595b3](bc595b3))
* fix aggregation when using a reserved keyword ([#207](#207)) ([7200f0b](7200f0b))
* fix virtual object selection and edition ([#183](#183)) ([852ff32](852ff32))
* force release ([#151](#151)) ([76b93bc](76b93bc))
* implement pagination cache ([#192](#192)) ([4d80176](4d80176))
* improve nested fields support ([#164](#164)) ([8acc227](8acc227))
* improve performances ([#191](#191)) ([b798b08](b798b08))
* improve scalability ([#186](#186)) ([8f69026](8f69026))
* make database name optional ([#189](#189)) ([dc93255](dc93255))
* readability ([#190](#190)) ([95d4761](95d4761))
* release agent-rpc ([fd47a08](fd47a08))
* **rpc agent:** handle properly cross rpc relations ([#220](#220)) ([e222f02](e222f02))
* **rpc agent:** release ([8f532a0](8f532a0))
* **rpc agent:** use schema path option ([#215](#215)) ([4c700ec](4c700ec))
* **rpc datasource:** fix introspection parsing ([b3978a1](b3978a1))
* **rpc datatource:** allow connect ruby as rpc ([#219](#219)) ([5034d74](5034d74))
* rpc stack connection ([#169](#169)) ([e88a2ba](e88a2ba))
* **rpc:** handle sa schema properly with multi stack ([#218](#218)) ([c8016dc](c8016dc))
* use Cosmos DB compatible functions for date aggregation ([#209](#209)) ([f4be4e2](f4be4e2))
* use npm CLI with OIDC instead of semantic-release npm plugin ([d9e3386](d9e3386))

### Features

* **actions:** ease the field set and the retrieval ([#140](#140)) ([8a924b4](8a924b4))
* add cosmos db datasource ([#159](#159)) ([0b25187](0b25187))
* add datasource-stripe ([#198](#198)) ([cc3be30](cc3be30))
* add hasura graphql datasource ([#194](#194)) ([5db0b0a](5db0b0a))
* add introspection config ([#187](#187)) ([fc470d3](fc470d3))
* **agent-tester:** add a csv export on collection ([#156](#156)) ([fe64937](fe64937))
* **agent-tester:** add a way to override permissions from the agent tester ([#168](#168)) ([1d219cb](1d219cb))
* **agent-tester:** add charts to collection ([#149](#149)) ([3a2e8c8](3a2e8c8))
* **agent-tester:** add csv export on segment ([#157](#157)) ([0538935](0538935))
* **agent-tester:** add live query segment ([#158](#158)) ([1961202](1961202))
* **agent-testings:** split agent control and testing into separate directories ([#141](#141)) ([0bf4699](0bf4699))
* **api chart:** allow customizing api chart with query or body parameters ([#211](#211)) ([6d3eb65](6d3eb65))
* **approvals:** add a parameter in action.execute to validate approval ([#171](#171)) ([b8b4333](b8b4333))
* **cosmos:** add RU consumption logs ([#204](#204)) ([c7af33e](c7af33e))
* **datasource graphql hasura:** fix release ([a843269](a843269))
* **datasource graphql hasura:** relase datasource ([#206](#206)) ([c105758](c105758))
* **datasource-airtable:** add Airtable datasource package ([#200](#200)) ([155e6fa](155e6fa))
* **define enum plugin:** allow filtering on the main field ([#222](#222)) ([905afca](905afca))
* **nested object:** use arrow notation ([#167](#167)) ([58a99e0](58a99e0))
* **rpc agent:** add mark collection as rpc option on add datasource ([#195](#195)) ([6372499](6372499))
* **rpc:** add introspection option ([f113580](f113580))
* **rpc:** handle SA returning file ([#217](#217)) ([08cc659](08cc659))
* support manual schema definition in cosmos ([#188](#188)) ([d819342](d819342))
* **testing:** add helper to get json field on action form ([#137](#137)) ([dfc25e9](dfc25e9))
* **testing:** introduce a rpc client ([#139](#139)) ([c2a3259](c2a3259))
* trigger release ([#185](#185)) ([1b378db](1b378db))

### Performance Improvements

* rollup Week/Quarter from Day-level queries ([#210](#210)) ([9528843](9528843))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant