Skip to content

Commit b675160

Browse files
ssalonenfwolter
andauthored
[dynamodb] Dynamodb refactor (openhab#9937)
* [dynamodb] Update to SDKv2 Enhanced Client In addition, introduce new more simple table layout, having only one table for all items and with more efficient data encoding (saves some read capacity). Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Time To Live (TTL) support with new table schema Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Support QuantityType Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] suppress null warnings in tests Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Optimized query performance Similar to openhab#8938, avoid calling Item.getUnit() repeatedly when querying data. Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Support for Group items Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Update copyright to 2021 Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Removing TODO comments and add javadoc Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] javadoc Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Readability improved in TableCreatingPutItem Also documenting the full retry logic. Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] verify fixes Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Remove slf4j from explicit dependencies Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Remove jackson from pom.xml, add as feature dep Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] bnd.importpackage tuned Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] abort query() immediately if not configured to avoid NPE Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] less chatty diagnostics Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] xml formatting Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] corrected logger class Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] null checks Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] netty client configured Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] bnd not to filter out importpackage org.slf4j.impl Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] cfg bundle group id Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Remove usage of org.apache.commons Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Remove extra prints from test Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Reducing @SupressWarnings with generics Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] README extra space removed Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] spotless Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Removed unnecessary logging Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] encapsulation Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] removed unnecessary NonNullByDefault({}) ctr-injected field Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] null annotations Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] less verbose logging in tests Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Prefer Collections.emptyList over List.of() Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] less verbose call Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Visitor to return values (simplifies the code) Less warnings suppressed Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] comments for remaining warning supressions Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] README tuning, typo fixing Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Using less verbose syntax Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] simplified logging on errors Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Code review comments Avoiding null checker while having more compact code Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] Null safety Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] configuration label and description formatting Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] xml indentation with tabs Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] @nullable 1-line annotation with class fields Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] No need to override credentials per request Client has the credentials set on build time Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] set API timeouts no matter what Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] adding exception message Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] static logger Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] dependency - comments clarifying the logic of properties - adding netty to dep.noembedding to ensure it is not compiled in Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] ensure correct jackson and netty versions using dependencyMgt Specifically for development and testing See openhab@051c764 for further discussion why this is needed. Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] avoid google collections Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] jackson-dataformat-cbor not jackson-cbor Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] also restrict netty-transport-native-epoll linux-x86_64 version Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] refering dynamodb.cfg similar to other bundles Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] bnd.importpackage to excl. reactivestreams and typesafe.netty These are compiled-in dependencies, and thus we do not want to have them in OSGi Import-Package. Signed-off-by: Sami Salonen <[email protected]> * Update bundles/org.openhab.persistence.dynamodb/src/main/resources/OH-INF/config/config.xml Co-authored-by: Fabian Wolter <[email protected]> Signed-off-by: Sami Salonen <[email protected]> * Update bundles/org.openhab.persistence.dynamodb/src/main/resources/OH-INF/config/config.xml Signed-off-by: Sami Salonen <[email protected]> Co-authored-by: Fabian Wolter <[email protected]> * [dynamodb] remove netty-codec-http2 as it is included in tp-netty See openhab/openhab-core#2257 Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] removed duplicate in bnd.importpackage Signed-off-by: Sami Salonen <[email protected]> * [dynamodb] slf4j-api marked as provided to remove dep errors in runtime Signed-off-by: Sami Salonen <[email protected]> Co-authored-by: Fabian Wolter <[email protected]>
1 parent 08602c0 commit b675160

File tree

58 files changed

+4387
-1572
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+4387
-1572
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/src/test/resources/native-libs/

bundles/org.openhab.persistence.dynamodb/README.md

+63-59
Original file line numberDiff line numberDiff line change
@@ -15,22 +15,10 @@ This service is provided "AS IS", and the user takes full responsibility of any
1515

1616
## Table of Contents
1717

18-
<!-- Using MarkdownTOC plugin for Sublime Text to update the table of contents (TOC) -->
19-
<!-- MarkdownTOC depth=3 autolink=true bracket=round -->
20-
21-
- [Prerequisites](#prerequisites)
22-
- [Setting Up an Amazon Account](#setting-up-an-amazon-account)
23-
- [Configuration](#configuration)
24-
- [Basic configuration](#basic-configuration)
25-
- [Configuration Using Credentials File](#configuration-using-credentials-file)
26-
- [Advanced Configuration](#advanced-configuration)
27-
- [Details](#details)
28-
- [Tables Creation](#tables-creation)
29-
- [Caveats](#caveats)
30-
- [Developer Notes](#developer-notes)
31-
- [Updating Amazon SDK](#updating-amazon-sdk)
32-
33-
<!-- /MarkdownTOC -->
18+
{::options toc_levels="2..4"/}
19+
20+
- TOC
21+
{:toc}
3422

3523
## Prerequisites
3624

@@ -55,17 +43,61 @@ Please also note possible [Free Tier](https://aws.amazon.com/free/) benefits.
5543

5644
## Configuration
5745

58-
This service can be configured in the file `services/dynamodb.cfg`.
46+
This service can be configured using the MainUI or using persistence configuration file `services/dynamodb.cfg`.
47+
48+
In order to configure the persistence service, you need to configure two things:
49+
50+
1. Table schema revision to use
51+
2. AWS credentials to access DynamoDB
52+
53+
### Table schema
54+
55+
The DynamoDB persistence addon provides two different table schemas: "new" and "legacy".
56+
As the name implies, "legacy" is offered for backwards-compatibility purpose for old users who like to access the data that is already stored in DynamoDB.
57+
All users are advised to transition to "new" table schema, which is more optimized.
58+
59+
At this moment there is no supported way to migrate data from old format to new.
60+
61+
#### New table schema
62+
63+
Configure the addon to use new schema by setting `table` parameter (name of the table).
64+
65+
Only one table will be created for all data. The table will have the following fields
66+
67+
68+
| Attribute | Type | Data type | Description |
69+
| --------- | ------ | --------- | --------------------------------------------- |
70+
| `i` | String | Yes | Item name |
71+
| `t` | Number | Yes | Timestamp in milliepoch |
72+
| `s` | String | Yes | State of the item, stored as DynamoDB string. |
73+
| `n` | Number | Yes | State of the item, stored as DynamoDB number. |
74+
| `exp` | Number | Yes | Expiry date for item, in epoch seconds |
75+
76+
Other notes
77+
78+
- `i` and `t` forms the composite primary key (partition key, sort key) for the table
79+
- Only one of `s` or `n` attributes are specified, not both. Most items are converted to number type for most compact representation.
80+
- Compared to legacy format, data overhead is minimizing by using short attribute names, number timestamps and having only single table.
81+
- `exp` attribute is used with DynamoDB Time To Live (TTL) feature to automatically delete old data
82+
83+
#### Legacy schema
5984

60-
### Basic configuration
85+
Configure the addon to use legacy schema by setting `tablePrefix` parameter.
86+
87+
- When an item is persisted via this service, a table is created (if necessary).
88+
- The service will create at most two tables for different item types.
89+
- The tables will be named `<tablePrefix><item-type>`, where the `<item-type>` is either `bigdecimal` (numeric items) or `string` (string and complex items).
90+
- Each table will have three columns: `itemname` (item name), `timeutc` (in ISO 8601 format with millisecond accuracy), and `itemstate` (either a number or string representing item state).
91+
92+
### Credentials Configuration Using Access Key and Secret Key
6193

6294
| Property | Default | Required | Description |
6395
| --------- | ------- | :------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
6496
| accessKey | | Yes | access key as shown in [Setting up Amazon account](#setting-up-an-amazon-account). |
6597
| secretKey | | Yes | secret key as shown in [Setting up Amazon account](#setting-up-an-amazon-account). |
6698
| region | | Yes | AWS region ID as described in [Setting up Amazon account](#setting-up-an-amazon-account). The region needs to match the region that was used to create the user. |
6799

68-
### Configuration Using Credentials File
100+
### Credentials Configuration Using Credentials File
69101

70102
Alternatively, instead of specifying `accessKey` and `secretKey`, one can configure a configuration profile file.
71103

@@ -95,68 +127,43 @@ aws_secret_access_key=testSecretKey
95127

96128
In addition to the configuration properties above, the following are also available:
97129

98-
| Property | Default | Required | Description |
99-
| -------------------------- | ---------- | :------: | -------------------------------------------------------------------------------------------------- |
100-
| readCapacityUnits | 1 | No | read capacity for the created tables |
101-
| writeCapacityUnits | 1 | No | write capacity for the created tables |
102-
| tablePrefix | `openhab-` | No | table prefix used in the name of created tables |
103-
| bufferCommitIntervalMillis | 1000 | No | Interval to commit (write) buffered data. In milliseconds. |
104-
| bufferSize | 1000 | No | Internal buffer size in datapoints which is used to batch writes to DynamoDB every `bufferCommitIntervalMillis`. |
105-
106-
Typically you should not need to modify parameters related to buffering.
130+
| Property | Default | Required | Description |
131+
| ------------------ | ------- | :------: | ----------------------------------------------------------- |
132+
| expireDays | (null) | No | Expire time for data in days (relative to stored timestamp) |
133+
| readCapacityUnits | 1 | No | read capacity for the created tables |
134+
| writeCapacityUnits | 1 | No | write capacity for the created tables |
107135

108136
Refer to Amazon documentation on [provisioned throughput](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html) for details on read/write capacity.
137+
DynamoDB Time to Live (TTL) setting is configured using `expireDays`.
109138

110139
All item- and event-related configuration is done in the file `persistence/dynamodb.persist`.
111140

112141
## Details
113142

114-
### Tables Creation
115-
116-
When an item is persisted via this service, a table is created (if necessary).
117-
Currently, the service will create at most two tables for different item types.
118-
The tables will be named `<tablePrefix><item-type>`, where the `<item-type>` is either `bigdecimal` (numeric items) or `string` (string and complex items).
119-
120-
Each table will have three columns: `itemname` (item name), `timeutc` (in ISO 8601 format with millisecond accuracy), and `itemstate` (either a number or string representing item state).
121-
122-
## Buffering
123-
124-
By default, the service is asynchronous which means that data is not written immediately to DynamoDB but instead buffered in-memory.
125-
The size of the buffer, in terms of datapoints, can be configured with `bufferSize`.
126-
Every `bufferCommitIntervalMillis` the whole buffer of data is flushed to DynamoDB.
127-
128-
It is recommended to have the buffering enabled since the synchronous behaviour (writing data immediately) might have adverse impact to the whole system when there is many items persisted at the same time.
129-
The buffering can be disabled by setting `bufferSize` to zero.
130-
131-
The defaults should be suitable in many use cases.
132-
133143
### Caveats
134144

135145
When the tables are created, the read/write capacity is configured according to configuration.
136146
However, the service does not modify the capacity of existing tables.
137147
As a workaround, you can modify the read/write capacity of existing tables using the [Amazon console](https://aws.amazon.com/console/).
138148

149+
Similar caveat applies for DynamoDB Time to Live (TTL) setting `expireDays`.
150+
139151
## Developer Notes
140152

141153
### Updating Amazon SDK
142154

143155
1. Clean `lib/*`
144156
2. Update SDK version in `scripts/fetch_sdk_pom.xml`. You can use the [maven online repository browser](https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dynamodb) to find the latest version available online.
145157
3. `scripts/fetch_sdk.sh`
146-
4. Copy `scripts/target/site/dependencies.html` and `scripts/target/dependency/*.jar` to `lib/`
147-
5. Generate `build.properties` entries
148-
`ls lib/*.jar | python -c "import sys; print(' ' + ',\\\\\\n '.join(map(str.strip, sys.stdin.readlines())))"`
149-
6. Generate `META-INF/MANIFEST.MF` `Bundle-ClassPath` entries
150-
`ls lib/*.jar | python -c "import sys; print(' ' + ',\\n '.join(map(str.strip, sys.stdin.readlines())))"`
151-
7. Generate `.classpath` entries
152-
`ls lib/*.jar | python -c "import sys;pre='<classpathentry exported=\"true\" kind=\"lib\" path=\"';post='\"/>'; print('\\t' + pre + (post + '\\n\\t' + pre).join(map(str.strip, sys.stdin.readlines())) + post)"`
158+
4. Copy printed dependencies to `pom.xml`
153159

154160
After these changes, it's good practice to run integration tests (against live AWS DynamoDB) in `org.openhab.persistence.dynamodb.test` bundle.
155161
See README.md in the test bundle for more information how to execute the tests.
156162

157163
### Running integration tests
158164

159-
To run integration tests, one needs to provide AWS credentials.
165+
When running integration tests, local temporary DynamoDB server is used, emulating the real AWS DynamoDB API.
166+
One can configure AWS credentials to run the test against real AWS DynamoDB for most realistic tests.
160167

161168
Eclipse instructions
162169

@@ -168,7 +175,4 @@ Eclipse instructions
168175
-DDYNAMODBTEST_REGION=REGION-ID
169176
-DDYNAMODBTEST_ACCESS=ACCESS-KEY
170177
-DDYNAMODBTEST_SECRET=SECRET
171-
````
172-
173-
The tests will create tables with prefix `dynamodb-integration-tests-`.
174-
Note that when tests are begun, all data is removed from that table!
178+
````

0 commit comments

Comments
 (0)