Skip to content

Commit 4878d0f

Browse files
thjaecklethjaeckle
committed
provide Ditto 3.5.8 release notes;
add options to configure readJournal aggregation index options Signed-off-by: Thomas Jäckle <[email protected]>
1 parent ed88251 commit 4878d0f

File tree

7 files changed

+158
-2
lines changed

7 files changed

+158
-2
lines changed

‎deployment/helm/ditto/Chart.yaml

+2-2
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ description: |
1616
A digital twin is a virtual, cloud based, representation of his real world counterpart
1717
(real world “Things”, e.g. devices like sensors, smart heating, connected cars, smart grids, EV charging stations etc).
1818
type: application
19-
version: 3.5.7 # chart version is effectively set by release-job
20-
appVersion: 3.5.7
19+
version: 3.5.8 # chart version is effectively set by release-job
20+
appVersion: 3.5.8
2121
keywords:
2222
- iot-chart
2323
- digital-twin

‎deployment/helm/ditto/templates/connectivity-deployment.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,20 @@ spec:
219219
value: "{{ .Values.connectivity.config.cleanup.creditsPerBatch }}"
220220
- name: CLEANUP_DELETE_FINAL_DELETED_SNAPSHOT
221221
value: "{{ .Values.connectivity.config.cleanup.deleteFinalDeletedSnapshot }}"
222+
- name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID
223+
value: "{{ .Values.connectivity.config.readJournal.indexes.createSnapshotAggregationIndexPidId }}"
224+
- name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID
225+
value: "{{ .Values.connectivity.config.readJournal.indexes.createSnapshotAggregationIndexPid }}"
226+
- name: MONGODB_READ_JOURNAL_HINT_NAME_FILTER_PIDS_THAT_DOESNT_CONTAIN_TAG_IN_NEWEST_ENTRY
227+
value: "{{ .Values.connectivity.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry }}"
228+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_LATEST_JOURNAL_ENTRIES
229+
value: "{{ .Values.connectivity.config.readJournal.hints.listLatestJournalEntries }}"
230+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID_ID
231+
value: "{{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId }}"
232+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID
233+
value: "{{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid }}"
234+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_ID
235+
value: "{{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchId }}"
222236
- name: CONNECTION_SNAPSHOT_INTERVAL
223237
value: "{{ .Values.connectivity.config.persistence.snapshots.interval }}"
224238
- name: CONNECTION_SNAPSHOT_THRESHOLD

‎deployment/helm/ditto/templates/policies-deployment.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,20 @@ spec:
237237
value: "{{ .Values.policies.config.cleanup.creditsPerBatch }}"
238238
- name: CLEANUP_DELETE_FINAL_DELETED_SNAPSHOT
239239
value: "{{ .Values.policies.config.cleanup.deleteFinalDeletedSnapshot }}"
240+
- name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID
241+
value: "{{ .Values.policies.config.readJournal.indexes.createSnapshotAggregationIndexPidId }}"
242+
- name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID
243+
value: "{{ .Values.policies.config.readJournal.indexes.createSnapshotAggregationIndexPid }}"
244+
- name: MONGODB_READ_JOURNAL_HINT_NAME_FILTER_PIDS_THAT_DOESNT_CONTAIN_TAG_IN_NEWEST_ENTRY
245+
value: "{{ .Values.policies.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry }}"
246+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_LATEST_JOURNAL_ENTRIES
247+
value: "{{ .Values.policies.config.readJournal.hints.listLatestJournalEntries }}"
248+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID_ID
249+
value: "{{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId }}"
250+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID
251+
value: "{{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid }}"
252+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_ID
253+
value: "{{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchId }}"
240254
- name: POLICIES_PERSISTENCE_PING_RATE_FREQUENCY
241255
value: "{{ .Values.policies.config.persistence.pingRate.frequency }}"
242256
- name: POLICIES_PERSISTENCE_PING_RATE_ENTITIES

‎deployment/helm/ditto/templates/things-deployment.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,20 @@ spec:
238238
value: "{{ .Values.things.config.cleanup.creditsPerBatch }}"
239239
- name: CLEANUP_DELETE_FINAL_DELETED_SNAPSHOT
240240
value: "{{ .Values.things.config.cleanup.deleteFinalDeletedSnapshot }}"
241+
- name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID
242+
value: "{{ .Values.things.config.readJournal.indexes.createSnapshotAggregationIndexPidId }}"
243+
- name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID
244+
value: "{{ .Values.things.config.readJournal.indexes.createSnapshotAggregationIndexPid }}"
245+
- name: MONGODB_READ_JOURNAL_HINT_NAME_FILTER_PIDS_THAT_DOESNT_CONTAIN_TAG_IN_NEWEST_ENTRY
246+
value: "{{ .Values.things.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry }}"
247+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_LATEST_JOURNAL_ENTRIES
248+
value: "{{ .Values.things.config.readJournal.hints.listLatestJournalEntries }}"
249+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID_ID
250+
value: "{{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId }}"
251+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID
252+
value: "{{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid }}"
253+
- name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_ID
254+
value: "{{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchId }}"
241255
- name: THING_SNAPSHOT_INTERVAL
242256
value: "{{ .Values.things.config.persistence.snapshots.interval }}"
243257
- name: THING_SNAPSHOT_THRESHOLD

‎deployment/helm/ditto/values.yaml

+60
Original file line numberDiff line numberDiff line change
@@ -646,6 +646,26 @@ policies:
646646
# a revision number 1 higher than the previously deleted entity. If the final snapshot is deleted as well,
647647
# recreation of an entity with the same ID will lead to revisionNumber=1 after its recreation.
648648
deleteFinalDeletedSnapshot: true
649+
# readJournal holds configuration regarding the MongoReadJournal and e.g. the aggregation queries which are performed in it
650+
readJournal:
651+
# indexes contains configuration about additional indexes to create
652+
indexes:
653+
# createSnapshotAggregationIndexPidId whether to create the "pid"+"_id" compound index on the snapshot collection
654+
createSnapshotAggregationIndexPidId: false
655+
# createSnapshotAggregationIndexPid whether to create the "pid" compound index on the snapshot collection
656+
createSnapshotAggregationIndexPid: false
657+
# hints contains hint names to configure for different aggregation calls done in MongoReadJournal
658+
hints:
659+
# filterPidsThatDoesntContainTagInNewestEntry contains the hint name to use in the aggregation query
660+
filterPidsThatDoesntContainTagInNewestEntry: null
661+
# listLatestJournalEntries contains the hint name to use in the aggregation query
662+
listLatestJournalEntries: null
663+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains both "pid" and "_id"
664+
listNewestActiveSnapshotsByBatchPidId: null
665+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains only "pid"
666+
listNewestActiveSnapshotsByBatchPid: null
667+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains only "_id"
668+
listNewestActiveSnapshotsByBatchId: null
649669
# persistence holds configuration regarding (pekko) persistence of policies (event journal and snapshots)
650670
persistence:
651671
# activityCheckInterval configures to keep policies for that amount of time in memory when no other use did happen:
@@ -855,6 +875,26 @@ things:
855875
# a revision number 1 higher than the previously deleted entity. If the final snapshot is deleted as well,
856876
# recreation of an entity with the same ID will lead to revisionNumber=1 after its recreation.
857877
deleteFinalDeletedSnapshot: true
878+
# readJournal holds configuration regarding the MongoReadJournal and e.g. the aggregation queries which are performed in it
879+
readJournal:
880+
# indexes contains configuration about additional indexes to create
881+
indexes:
882+
# createSnapshotAggregationIndexPidId whether to create the "pid"+"_id" compound index on the snapshot collection
883+
createSnapshotAggregationIndexPidId: false
884+
# createSnapshotAggregationIndexPid whether to create the "pid" compound index on the snapshot collection
885+
createSnapshotAggregationIndexPid: false
886+
# hints contains hint names to configure for different aggregation calls done in MongoReadJournal
887+
hints:
888+
# filterPidsThatDoesntContainTagInNewestEntry contains the hint name to use in the aggregation query
889+
filterPidsThatDoesntContainTagInNewestEntry: null
890+
# listLatestJournalEntries contains the hint name to use in the aggregation query
891+
listLatestJournalEntries: null
892+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains both "pid" and "_id"
893+
listNewestActiveSnapshotsByBatchPidId: null
894+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains only "pid"
895+
listNewestActiveSnapshotsByBatchPid: null
896+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains only "_id"
897+
listNewestActiveSnapshotsByBatchId: null
858898
# persistence holds configuration regarding (pekko) persistence of things (event journal and snapshots)
859899
persistence:
860900
# activityCheckInterval configures to keep things for that amount of time in memory when no other use did happen
@@ -1313,6 +1353,26 @@ connectivity:
13131353
# a revision number 1 higher than the previously deleted entity. If the final snapshot is deleted as well,
13141354
# recreation of an entity with the same ID will lead to revisionNumber=1 after its recreation.
13151355
deleteFinalDeletedSnapshot: true
1356+
# readJournal holds configuration regarding the MongoReadJournal and e.g. the aggregation queries which are performed in it
1357+
readJournal:
1358+
# indexes contains configuration about additional indexes to create
1359+
indexes:
1360+
# createSnapshotAggregationIndexPidId whether to create the "pid"+"_id" compound index on the snapshot collection
1361+
createSnapshotAggregationIndexPidId: false
1362+
# createSnapshotAggregationIndexPid whether to create the "pid" compound index on the snapshot collection
1363+
createSnapshotAggregationIndexPid: false
1364+
# hints contains hint names to configure for different aggregation calls done in MongoReadJournal
1365+
hints:
1366+
# filterPidsThatDoesntContainTagInNewestEntry contains the hint name to use in the aggregation query
1367+
filterPidsThatDoesntContainTagInNewestEntry: null
1368+
# listLatestJournalEntries contains the hint name to use in the aggregation query
1369+
listLatestJournalEntries: null
1370+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains both "pid" and "_id"
1371+
listNewestActiveSnapshotsByBatchPidId: null
1372+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains only "pid"
1373+
listNewestActiveSnapshotsByBatchPid: null
1374+
# listNewestActiveSnapshotsByBatchPidId contains the hint name to use in the aggregation query where the $match contains only "_id"
1375+
listNewestActiveSnapshotsByBatchId: null
13161376
# persistence holds configuration regarding (akka) persistence of connections (event journal and snapshots)
13171377
persistence:
13181378
# keep closed, inactive connections for that amount of time in memory when no other use did happen:

‎documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml

+3
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ entries:
2323
- title: Release Notes
2424
output: web
2525
folderitems:
26+
- title: 3.5.8
27+
url: /release_notes_358.html
28+
output: web
2629
- title: 3.5.7
2730
url: /release_notes_357.html
2831
output: web

‎documentation/src/main/resources/pages/ditto/release_notes_358.md

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: Release notes 3.5.8
3+
tags: [release_notes]
4+
published: true
5+
keywords: release notes, announcements, changelog
6+
summary: "Version 3.5.8 of Eclipse Ditto, released on 12.06.2024"
7+
permalink: release_notes_358.html
8+
---
9+
10+
This is a bugfix release, no new features since [3.5.7](release_notes_357.html) were added.
11+
12+
## Changelog
13+
14+
Compared to the latest release [3.5.7](release_notes_357.html), the following changes and bugfixes were added.
15+
16+
### Bugfixes
17+
18+
This is a complete list of the
19+
[merged pull requests](https://github.com/eclipse-ditto/ditto/pulls?q=is%3Apr+milestone%3A3.5.8).
20+
21+
#### Fix generation of WoT Thing Descriptions wrong base and href paths
22+
23+
Reported issue [#1959](https://github.com/eclipse-ditto/ditto/issues/1959) correctly identified a bug in the
24+
[WoT (Web of Things) Thing Description generation](basic-wot-integration.html#thing-description-generation).
25+
The generated `base` was not ending with a slash and the relative `href` in the TDs were beginning with a slash which
26+
leads to wrongly interpreted absolute paths, e.g. when using the browser `URL` object.
27+
28+
This was fixed in PR [#1962](https://github.com/eclipse-ditto/ditto/pull/1962).
29+
30+
#### Fix performance regression issue when running against MongoDB 6 - part 2
31+
32+
In Ditto [3.5.7](release_notes_357.html) a MongoDB `aggregation` query done by Ditto as part of background deletion
33+
was aimed to be optimized.
34+
This fix provided in PR [#1956](https://github.com/eclipse-ditto/ditto/pull/1956) however could lead to side effects resulting again in very long-running
35+
aggregation queries.
36+
37+
With PR [#1961](https://github.com/eclipse-ditto/ditto/pull/1961) provided in this bugfix release, the options to
38+
configure the index `hint`s and even to create additional indexes were enhanced and the defaults changed so that no `hint`
39+
is provided by default, but the MongoDB query planner decides on which index to use.
40+
The option to configure a `hint` however was enhanced, depending on which fields were present in the `$match` aggregation step.
41+
42+
43+
### Helm Chart
44+
45+
The [Ditto Helm Chart](https://github.com/eclipse-ditto/ditto/tree/master/deployment/helm) contained some configuration
46+
bugs and enhancements which are also addressed with this bugfix release.
47+
48+
#### Provided configuration for MongoReadJournal aggregation index hinting
49+
50+
Provide Helm value options to configure settings for the above-mentioned PR
51+
[#1961](https://github.com/eclipse-ditto/ditto/pull/1961).

0 commit comments

Comments
 (0)