You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* [dynamodb] Update to SDKv2 Enhanced Client
In addition, introduce new more simple table layout, having only one
table for all items and with more efficient data encoding (saves some read capacity).
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Time To Live (TTL) support with new table schema
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Support QuantityType
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] suppress null warnings in tests
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Optimized query performance
Similar to openhab#8938,
avoid calling Item.getUnit() repeatedly when querying data.
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Support for Group items
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Update copyright to 2021
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Removing TODO comments and add javadoc
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] javadoc
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Readability improved in TableCreatingPutItem
Also documenting the full retry logic.
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] verify fixes
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Remove slf4j from explicit dependencies
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Remove jackson from pom.xml, add as feature dep
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] bnd.importpackage tuned
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] abort query() immediately if not configured to avoid NPE
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] less chatty diagnostics
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] xml formatting
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] corrected logger class
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] null checks
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] netty client configured
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] bnd not to filter out importpackage org.slf4j.impl
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] cfg bundle group id
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Remove usage of org.apache.commons
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Remove extra prints from test
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Reducing @SupressWarnings with generics
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] README extra space removed
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] spotless
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Removed unnecessary logging
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] encapsulation
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] removed unnecessary NonNullByDefault({}) ctr-injected field
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] null annotations
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] less verbose logging in tests
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Prefer Collections.emptyList over List.of()
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] less verbose call
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Visitor to return values (simplifies the code)
Less warnings suppressed
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] comments for remaining warning supressions
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] README tuning, typo fixing
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Using less verbose syntax
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] simplified logging on errors
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Code review comments
Avoiding null checker while having more compact code
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] Null safety
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] configuration label and description formatting
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] xml indentation with tabs
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] @nullable 1-line annotation with class fields
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] No need to override credentials per request
Client has the credentials set on build time
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] set API timeouts no matter what
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] adding exception message
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] static logger
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] dependency
- comments clarifying the logic of properties
- adding netty to dep.noembedding to ensure it is not compiled in
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] ensure correct jackson and netty versions using dependencyMgt
Specifically for development and testing
See openhab@051c764
for further discussion why this is needed.
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] avoid google collections
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] jackson-dataformat-cbor not jackson-cbor
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] also restrict netty-transport-native-epoll linux-x86_64 version
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] refering dynamodb.cfg similar to other bundles
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] bnd.importpackage to excl. reactivestreams and typesafe.netty
These are compiled-in dependencies, and thus we do not want to have them in
OSGi Import-Package.
Signed-off-by: Sami Salonen <[email protected]>
* Update bundles/org.openhab.persistence.dynamodb/src/main/resources/OH-INF/config/config.xml
Co-authored-by: Fabian Wolter <[email protected]>
Signed-off-by: Sami Salonen <[email protected]>
* Update bundles/org.openhab.persistence.dynamodb/src/main/resources/OH-INF/config/config.xml
Signed-off-by: Sami Salonen <[email protected]>
Co-authored-by: Fabian Wolter <[email protected]>
* [dynamodb] remove netty-codec-http2 as it is included in tp-netty
See openhab/openhab-core#2257
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] removed duplicate in bnd.importpackage
Signed-off-by: Sami Salonen <[email protected]>
* [dynamodb] slf4j-api marked as provided to remove dep errors in runtime
Signed-off-by: Sami Salonen <[email protected]>
Co-authored-by: Fabian Wolter <[email protected]>
@@ -55,17 +43,61 @@ Please also note possible [Free Tier](https://aws.amazon.com/free/) benefits.
55
43
56
44
## Configuration
57
45
58
-
This service can be configured in the file `services/dynamodb.cfg`.
46
+
This service can be configured using the MainUI or using persistence configuration file `services/dynamodb.cfg`.
47
+
48
+
In order to configure the persistence service, you need to configure two things:
49
+
50
+
1. Table schema revision to use
51
+
2. AWS credentials to access DynamoDB
52
+
53
+
### Table schema
54
+
55
+
The DynamoDB persistence addon provides two different table schemas: "new" and "legacy".
56
+
As the name implies, "legacy" is offered for backwards-compatibility purpose for old users who like to access the data that is already stored in DynamoDB.
57
+
All users are advised to transition to "new" table schema, which is more optimized.
58
+
59
+
At this moment there is no supported way to migrate data from old format to new.
60
+
61
+
#### New table schema
62
+
63
+
Configure the addon to use new schema by setting `table` parameter (name of the table).
64
+
65
+
Only one table will be created for all data. The table will have the following fields
|`s`| String | Yes | State of the item, stored as DynamoDB string. |
73
+
|`n`| Number | Yes | State of the item, stored as DynamoDB number. |
74
+
|`exp`| Number | Yes | Expiry date for item, in epoch seconds |
75
+
76
+
Other notes
77
+
78
+
-`i` and `t` forms the composite primary key (partition key, sort key) for the table
79
+
- Only one of `s` or `n` attributes are specified, not both. Most items are converted to number type for most compact representation.
80
+
- Compared to legacy format, data overhead is minimizing by using short attribute names, number timestamps and having only single table.
81
+
-`exp` attribute is used with DynamoDB Time To Live (TTL) feature to automatically delete old data
82
+
83
+
#### Legacy schema
59
84
60
-
### Basic configuration
85
+
Configure the addon to use legacy schema by setting `tablePrefix` parameter.
86
+
87
+
- When an item is persisted via this service, a table is created (if necessary).
88
+
- The service will create at most two tables for different item types.
89
+
- The tables will be named `<tablePrefix><item-type>`, where the `<item-type>` is either `bigdecimal` (numeric items) or `string` (string and complex items).
90
+
- Each table will have three columns: `itemname` (item name), `timeutc` (in ISO 8601 format with millisecond accuracy), and `itemstate` (either a number or string representing item state).
91
+
92
+
### Credentials Configuration Using Access Key and Secret Key
| accessKey || Yes | access key as shown in [Setting up Amazon account](#setting-up-an-amazon-account). |
65
97
| secretKey || Yes | secret key as shown in [Setting up Amazon account](#setting-up-an-amazon-account). |
66
98
| region || Yes | AWS region ID as described in [Setting up Amazon account](#setting-up-an-amazon-account). The region needs to match the region that was used to create the user. |
67
99
68
-
### Configuration Using Credentials File
100
+
### Credentials Configuration Using Credentials File
69
101
70
102
Alternatively, instead of specifying `accessKey` and `secretKey`, one can configure a configuration profile file.
| expireDays | (null) | No | Expire time for data in days (relative to stored timestamp) |
133
+
| readCapacityUnits | 1 | No | read capacity for the created tables |
134
+
| writeCapacityUnits | 1 | No | write capacity for the created tables |
107
135
108
136
Refer to Amazon documentation on [provisioned throughput](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html) for details on read/write capacity.
137
+
DynamoDB Time to Live (TTL) setting is configured using `expireDays`.
109
138
110
139
All item- and event-related configuration is done in the file `persistence/dynamodb.persist`.
111
140
112
141
## Details
113
142
114
-
### Tables Creation
115
-
116
-
When an item is persisted via this service, a table is created (if necessary).
117
-
Currently, the service will create at most two tables for different item types.
118
-
The tables will be named `<tablePrefix><item-type>`, where the `<item-type>` is either `bigdecimal` (numeric items) or `string` (string and complex items).
119
-
120
-
Each table will have three columns: `itemname` (item name), `timeutc` (in ISO 8601 format with millisecond accuracy), and `itemstate` (either a number or string representing item state).
121
-
122
-
## Buffering
123
-
124
-
By default, the service is asynchronous which means that data is not written immediately to DynamoDB but instead buffered in-memory.
125
-
The size of the buffer, in terms of datapoints, can be configured with `bufferSize`.
126
-
Every `bufferCommitIntervalMillis` the whole buffer of data is flushed to DynamoDB.
127
-
128
-
It is recommended to have the buffering enabled since the synchronous behaviour (writing data immediately) might have adverse impact to the whole system when there is many items persisted at the same time.
129
-
The buffering can be disabled by setting `bufferSize` to zero.
130
-
131
-
The defaults should be suitable in many use cases.
132
-
133
143
### Caveats
134
144
135
145
When the tables are created, the read/write capacity is configured according to configuration.
136
146
However, the service does not modify the capacity of existing tables.
137
147
As a workaround, you can modify the read/write capacity of existing tables using the [Amazon console](https://aws.amazon.com/console/).
138
148
149
+
Similar caveat applies for DynamoDB Time to Live (TTL) setting `expireDays`.
150
+
139
151
## Developer Notes
140
152
141
153
### Updating Amazon SDK
142
154
143
155
1. Clean `lib/*`
144
156
2. Update SDK version in `scripts/fetch_sdk_pom.xml`. You can use the [maven online repository browser](https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dynamodb) to find the latest version available online.
145
157
3.`scripts/fetch_sdk.sh`
146
-
4. Copy `scripts/target/site/dependencies.html` and `scripts/target/dependency/*.jar` to `lib/`
0 commit comments