diff --git a/src/UserGuide/Master/Table/Tools-System/Benchmark.md b/src/UserGuide/Master/Table/Tools-System/Benchmark.md index b8dc8c4ff..ea262493f 100644 --- a/src/UserGuide/Master/Table/Tools-System/Benchmark.md +++ b/src/UserGuide/Master/Table/Tools-System/Benchmark.md @@ -40,50 +40,51 @@ Figure 1-2 *IoT-benchmark Modular Design* Currently, IoT-benchmark supports the following time series databases, versions and connection methods: -| Database | Version | Connection mmethod | -| :-------------- | :-------------- | :------------------------------------------------------- | -| InfluxDB | v1.x v2.0 | SDK | -| TimescaleDB | -- | JDBC | -| OpenTSDB | -- | HTTP Request | -| QuestDB | v6.0.7 | JDBC | -| TDengine | v2.2.0.2 | JDBC | -| VictoriaMetrics | v1.64.0 | HTTP Request | -| KairosDB | -- | HTTP Request | -| IoTDB | v2.0 v1.x v0.13 | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | +| Database | Version | Connection mmethod | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | JDBC | +| OpenTSDB | -- | HTTP Request | +| QuestDB | v6.0.7 | JDBC | +| TDengine | v2.2.0.2 | JDBC | +| VictoriaMetrics | v1.64.0 | HTTP Request | +| KairosDB | -- | HTTP Request | + ## 2. **Installation and Operation** -#### **Prerequisites** +### 2.1 **Prerequisites** 1. Java 8 2. Maven 3.6+ 3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 -#### **How to Obtain** +### 2.2 **How to Obtain** - **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. - - **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** +- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** - - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.1 and run the following command in the root directory to compile the latest IoTDB Session package: + - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: - ```Bash - mvn clean package install -pl session -am -DskipTests - ``` + ```Bash + mvn clean package install -pl session -am -DskipTests + ``` - - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. + - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. - ```Bash - mvn clean package install -pl iotdb-2.0 -am -DskipTests - ``` + ```Bash + mvn clean package install -pl iotdb-2.0 -am -DskipTests + ``` - - The compiled test package will be located at: + - The compiled test package will be located at: - ```Bash - ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 - ``` + ```Bash + ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 + ``` -#### **Test Package Structure** +### 2.3 **Test Package Structure** The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. @@ -113,7 +114,7 @@ drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib -#### **Execution** **of** **Tests** +### 2.4 **Execution** **of** **Tests** 1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: @@ -127,9 +128,7 @@ drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib 4. Upon completion, review the results and analyze the test process. -#### - -#### **Results Interpretation** +### 2.5 **Results Interpretation** All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: @@ -147,7 +146,7 @@ All test log files are stored in the `logs` folder, while test results are saved ## 3. **Main** **Parameters** -#### IoTDB Service Model +### 3.1 IoTDB Service Model The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. @@ -155,8 +154,6 @@ The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The d - **IoTDB_DIALECT_MODE = table:** - The number of devices must be an integer multiple of the number of tables. - The number of tables must be an integer multiple of the number of databases. -- **IoTDB_DIALECT_MODE = tree:** - - The number of devices must be greater than or equal to the number of databases. Key Parameters for IoTDB Service Model @@ -167,7 +164,7 @@ Key Parameters for IoTDB Service Model | SENSOR_NUMBER | Integer | `10` | Controls the number of attribute columns in the table model. | | IoTDB_TABLE_NUMBER | Integer | `1` | Specifies the number of tables when using the table model. | -#### **Working** **M****ode** +### 3.2 **Working** **Mode** The `BENCHMARK_WORK_MODE` parameter supports four operational modes: @@ -185,7 +182,7 @@ Mode configurations are shown in the following below: | Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | | Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | -#### **Server** **Connection** **Information** +### 3.3 **Server** **Connection** **Information** Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: @@ -199,7 +196,7 @@ Once the working mode is specified, the following parameters must be configured | DB_NAME | String | `test` | Name of the target time-series database. | | TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | -#### **Write Scenario Parameters** +### 3.4 **Write Scenario Parameters** | **Parameter** | **Type** | **Example** | D**escription** | | :------------------------- | :-------------------- | :-------------------------- | :----------------------------------------------------------- | @@ -217,7 +214,7 @@ Once the working mode is specified, the following parameters must be configured | LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | | OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | -#### **Query Scenario Parameters** +### 3.5 **Query Scenario Parameters** | Parameter | Type | Example | Description | | :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | @@ -231,7 +228,7 @@ Once the working mode is specified, the following parameters must be configured | LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | | OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | -#### **Query Types and Example SQL** +### 3.6 **Query Types and Example SQL** | Number | Query Type | IoTDB Sample SQL | | :----- | :----------------------------- | :----------------------------------------------------------- | @@ -246,7 +243,7 @@ Once the working mode is specified, the following parameters must be configured | Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | | Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | -#### **Test process and test result persistence** +### 3.7 **Test process and test result persistence** IoT-benchmark currently supports persisting the test process and test results through configuration parameters. @@ -279,9 +276,9 @@ IoT-benchmark currently supports persisting the test process and test results th 2. Stores the test results after test completion. 3. Created if it does not exist. -#### Automation Script +### 3.8 Automation Script -##### One-Click Script Startup +#### One-Click Script Startup The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. @@ -298,7 +295,7 @@ The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark 1. Check test-related logs in the `logs` folder. 2. Check monitoring-related logs in the `server-logs` folder. -##### Automatic Execution of Multiple Tests +#### Automatic Execution of Multiple Tests Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. @@ -328,13 +325,13 @@ Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in se - Changed parameters persist across subsequent tests unless explicitly reset. -1. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command +2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command ```Bash > ./rep-benchmark.sh ``` -2. Test results will be displayed in the terminal. +Test results will be displayed in the terminal. **Important Notes:** diff --git a/src/UserGuide/Master/Tree/Tools-System/Benchmark.md b/src/UserGuide/Master/Tree/Tools-System/Benchmark.md index 8172be84c..4802594ed 100644 --- a/src/UserGuide/Master/Tree/Tools-System/Benchmark.md +++ b/src/UserGuide/Master/Tree/Tools-System/Benchmark.md @@ -21,189 +21,335 @@ # Benchmark Tool -IoT-benchmark is a time-series database benchmarking tool based on Java and big data environment, developed and open sourced by School of Software Tsinghua University. It is easy to use, supports multiple writing and query methods, supports storing test information and results for further query or analysis, and supports integration with Tableau to visualize test results. +## 1. **Basic Overview** -Figure 1-1 below includes the test benchmark process and other extended functions. These processes can be unified by IoT-benchmark. IoT Benchmark supports a variety of workloads, including **pure write, pure query, write query mixed**, etc., supports **software and hardware system monitoring, test metric measurement** and other monitoring functions, and also realizes **initializing the database automatically, test data analysis and system parameter optimization** functions. +IoT-benchmark is a time-series database benchmarking tool developed in Java for big data environments. It was developed and open-sourced by the School of Software, Tsinghua University. The tool is user-friendly, supports various write and query methods, allows storing test information and results for further queries or analysis, and integrates with Tableau for visualizing test results. -![img](/img/benchmark-English1.png) +Figure 1-1 illustrates the test benchmark process and its extended functionalities, all of which can be streamlined by IoT-benchmark. It supports a variety of workloads, including write-only, read-only, and mixed write-and-read operations. Additionally, it offers software and hardware system monitoring, performance metric measurement, automated database initialization, test data analysis, and system parameter optimization. +![](/img/benchmark-English1.png) -Figure 1-1 +Figure 1-1 *IoT-benchmark Test Benchmark Process* -Referring to the YCSB test tool's design idea of separating the three components of workload generation, performance metric measurement and database interface, the modular design of IoT-benchmark is shown in Figure 1-2. Different from the YCSB-based test tool system, IoT-benchmark adds a system monitoring module to support the persistence of test data and system monitoring data. In addition, some special load testing functions especially designed for time series data scenarios have been added, such as supporting batch writing and multiple out-of-sequence data writing modes for IoT scenarios. +IoT-benchmark adopts the modular design concept of the YCSB test tool, which separates workload generation, performance measurement, and database interface components. Its modular structure is illustrated in Figure 1-2. Unlike YCSB-based testing tools, IoT-benchmark introduces a system monitoring module that supports the persistence of both test data and system metrics. It also includes load-testing functionalities specifically designed for time-series data scenarios, such as batch writes and multiple out-of-order data insertion modes for IoT environments. -![img](/img/benchmark-%20English2.png) +![](/img/benchmark-%20English2.png) +Figure 1-2 *IoT-benchmark Modular Design* -Figure 1-2 +**Supported Databases** -Currently IoT-benchmark supports the following time series databases, versions and connection methods: +Currently, IoT-benchmark supports the following time series databases, versions and connection methods: -| Database | Version | Connection mmethod | -| --------------- | ------- | -------------------------------------------------------- | -| InfluxDB | v1.x
v2.0 | SDK | | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | -| IoTDB | v1.x
v0.13 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| Database | Version | Connection mmethod | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | JDBC | +| OpenTSDB | -- | HTTP Request | +| QuestDB | v6.0.7 | JDBC | +| TDengine | v2.2.0.2 | JDBC | +| VictoriaMetrics | v1.64.0 | HTTP Request | +| KairosDB | -- | HTTP Request | -Table 1-1 Comparison of big data test benchmarks -## 1. Software Installation and Environment Setup +## 2. **Installation and Operation** -### 1.1 Prerequisites +### 2.1 **Prerequisites** 1. Java 8 2. Maven 3.6+ -3. The corresponding appropriate version of the database, such as Apache IoTDB 1.0 +3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 -### 1.2 How to Get IoT Benchmark +### 2.2 **How to Obtain** -- **Get the binary package**: Enter https://github.com/thulab/iot-benchmark/releases to download the required installation package. Download it as a compressed file, select a folder to decompress and use it. -- Compiled from source (can be tested with Apache IoTDB 1.0): - - The first step (compile the latest IoTDB Session package): Enter the official website https://github.com/apache/iotdb/tree/rel/1.0 to download the IoTDB source code, and run the command `mvn clean package install -pl session -am -DskipTests` in the root directory to compiles the latest package for IoTDB Session. - - The second step (compile the IoTDB Benchmark test package): Enter the official website https://github.com/thulab/iot-benchmark to download the source code, run `mvn clean package install -pl iotdb-1.0 -am -DskipTests` in the root directory to compile Apache IoTDB version 1.0 test package. The relative path between the test package and the root directory is `./iotdb-1.0/target/iotdb-1.0-0.0.1/iotdb-1.0-0.0.1`. +- **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. -### 1.3 IoT Benchmark's Test Package Structure +- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** -The directory structure of the test package is shown in Figure 1-3 below. The test configuration file is conf/config.properties, and the test startup scripts are benchmark\.sh (Linux & MacOS) and benchmark.bat (Windows). The detailed usage of the files is shown in Table 1-2. + - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: -![img](/img/bm3.png) + ```Bash + mvn clean package install -pl session -am -DskipTests + ``` + - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. -Figure 1-3 List of files and folders + ```Bash + mvn clean package install -pl iotdb-2.0 -am -DskipTests + ``` -| Name | File | Usage | -| ---------------- | ----------------- | -------------------------------- | -| benchmark.bat | - | Startup script on Windows | -| benchmark\.sh | - | Startup script on Linux/Mac | -| conf | config.properties | Test scenario configuration file | -| logback.xml | - | Log output configuration file | -| lib | - | Dependency library | -| LICENSE | - | License file | -| bin | startup\.sh | Init script folder | -| ser-benchmark\.sh | - | Monitor mode startup script | + - The compiled test package will be located at: -Table 1-2 Usage list of files and folders + ```Bash + ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 + ``` -### 1.4 IoT Benchmark Execution Test +### 2.3 **Test Package Structure** -1. Modify the configuration file according to the test requirements. For the main parameters, see next chapter. The corresponding configuration file is conf/config.properties. For example, to test Apache IoTDB 1.0, you need to modify DB_SWITCH=IoTDB-100-SESSION_BY_TABLET. -2. Start the time series database under test. -3. Running. -4. Start IoT-benchmark to execute the test. Observe the status of the time series database and IoT-benchmark under test during execution, and view the results and analyze the test process after execution. +The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. -### 1.5 IoT Benchmark Results Interpretation +```Shell +-rw-r--r--. 1 root root 2881 Jan 10 01:36 benchmark.bat +-rwxr-xr-x. 1 root root 314 Jan 10 01:36 benchmark.sh +drwxr-xr-x. 2 root root 24 Jan 10 01:36 bin +-rwxr-xr-x. 1 root root 1140 Jan 10 01:36 cli-benchmark.sh +drwxr-xr-x. 2 root root 107 Jan 10 01:36 conf +drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib +-rw-r--r--. 1 root root 11357 Jan 10 01:36 LICENSE +-rwxr-xr-x. 1 root root 939 Jan 10 01:36 rep-benchmark.sh +-rw-r--r--. 1 root root 14 Jan 10 01:36 routine +``` -All the log files of the test are stored in the logs folder, and the test results are stored in the data/csvOutput folder after the test is completed. For example, after the test, we get the following result matrix: +| Name | File | Usage | +| :--------------- | :---------------- | :-------------------------------------------------- | +| benchmark.bat | - | Startup script on Windows | +| benchmark.sh | - | Startup script on Linux/Mac | +| bin | startup.sh | Initialization script folder | +| conf | config.properties | Test scenario configuration file | +| lib | - | Dependency library | +| LICENSE | - | License file | +| cli-benchmark.sh | - | One-click startup script | +| routine | - | Automatic execution of multiple test configurations | +| rep-benchmark.sh | - | Automatic execution of multiple test scripts | -![img](/img/bm4.png) -- Result Matrix - - OkOperation: successful operations - - OkPoint: For write operations, it is the number of points successfully written; for query operations, it is the number of points successfully queried. - - FailOperation: failed operations - - FailPoint: For write operations, it is the number of write failure points -- Latency(mx) Matrix - - AVG: average operation time - - MIN: minimum operation time - - Pn: the quantile value of the overall distribution of operations, for example, P25 is the lower quartile. +### 2.4 **Execution** **of** **Tests** -## 2. Main Parameters +1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: -This chapter mainly explains the purpose and configuration method of the main parameters. + ```Bash + DB_SWITCH=IoTDB-200-SESSION_BY_TABLET + ``` -### 2.1 Working Mode and Operation Proportion +2. Ensure the target time-series database is running. -- The working mode parameter "BENCHMARK_WORK_MODE" can be selected as "default mode" and "server monitoring"; the "server monitoring" mode can be started directly by executing the ser-benchmark\.sh script, and the script will automatically modify this parameter. "Default mode" is a commonly used test mode, combined with the configuration of the OPERATION_PROPORTION parameter to achieve the definition of test operation proportions of "pure write", "pure query" and "read-write mix". +3. Start IoT-benchmark to execute the test. Monitor the status of both the target database and IoT-benchmark during execution. -- When running ServerMode to monitor the operating environment of the time series database under test, IoT-benchmark relies on sysstat software related commands; if MySQL or IoTDB is selected for persistent test process data, this type of database needs to be installed; the recording mode of ServerMode and CSV can only be used in the Linux system to record relevant system information during the test. Therefore, we recommend using MacOs or Linux system. This article uses Linux (Centos7) system as an example. If you use Windows system, you can use the benchmark.bat script in the conf folder to start IoT-benchmark. +4. Upon completion, review the results and analyze the test process. -Table 1-3 Test mode +### 2.5 **Results Interpretation** -| Mode Name | BENCHMARK_WORK_MODE | Description | -| ------------ | ------------------- | ------------------------------------------------------------ | -| default mode | testWithDefaultPath | Supports mixed workloads with multiple read and write operations | -| server mode | serverMODE | Server resource usage monitoring mode (running in this mode is started by the ser-benchmark\.sh script, no need to manually configure this parameter) | +All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: -### 2.2 Server Connection Information +![](/img/bm4.png) -After the working mode is specified, how to inform IoT-benchmark of the information of the time series database under test? Currently, the type of the time-series database under test is informed through "DB_SWITCH"; the network address of the time-series database under test is informed through "HOST"; the network port of the time-series database under test is informed through "PORT"; the login user name of the time-series database under test is informed through "USERNAME"; "PASSWORD" informs the password of the login user of the time series database under test; informs the name of the time series database under test through "DB_NAME"; informs the connection authentication token of the time series database under test through "TOKEN" (used by InfluxDB 2.0). +- **Result Matrix:** + - OkOperation: Number of successful operations. + - OkPoint: Number of successfully written points (for write operations) or successfully queried points (for query operations). + - FailOperation: Number of failed operations. + - FailPoint: Number of failed write points. +- **Latency (ms) Matrix:** + - AVG: Average operation latency. + - MIN: Minimum operation latency. + - Pn: Quantile values of the overall operation distribution (e.g., P25 represents the 25th percentile, or lower quartile). -### 2.3 Write Scene Setup Parameters +## 3. **Main** **Parameters** + +### 3.1 IoTDB Service Model -Table 1-4 Write scene setup parameters +The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. -| Parameter Name | Type | Example | Description | -| -------------------------- | --------- | ------------------------- | ------------------------------------------------------------ | -| CLIENT_NUMBER | Integer | 100 | Total number of clients | -| GROUP_NUMBER | Integer | 20 | Number of storage groups; only for IoTDB. | -| DEVICE_NUMBER | Integer | 100 | Total number of devices | -| SENSOR_NUMBER | Integer | 300 | Total number of sensors per device | -| INSERT_DATATYPE_PROPORTION | String | 1:1:1:1:1:1 | the data type proportion of the device, BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | Integer | 1000 | Timestamp interval, that is, the fixed length between two timestamps of generated data. | -| OP_MIN_INTERVAL | Integer | 0 | Minimum operation execution interval: if the operation time is greater than this value, execute the next one immediately, otherwise wait (OP_MIN_INTERVAL-actual execution time) ms; if it is 0, the parameter will not take effect; if it is -1, its value is consistent with POINT_STEP. | -| IS_OUT_OF_ORDER | Boolean | false | Whether to write out of order | -| OUT_OF_ORDER_RATIO | Float | 0.3 | Ratio of data written out of order | -| BATCH_SIZE_PER_WRITE | Integer | 1 | Number of data rows written in batches (how many rows of data are written at a time) | -| START_TIME | Timestamp | 2022-10-30T00:00:00+08:00 | The start timestamp of writing data; use this timestamp as the starting point to start the simulation to create the data timestamp. | -| LOOP | Integer | 86400 | Total number of operations: Each type of operation will be divided according to the ratio defined by OPERATION_PROPORTION | -| OPERATION_PROPORTION | String | 1:0:0:0:0:0:0:0:0:0:0 | The ratio of each operation. Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, please note the use of English colons. Each term in the scale is an integer. | +- **For IoTDB 2.0 and later versions**, the `IoTDB_DIALECT_MODE` parameter must be specified, and only one mode can be set for each IoTDB instance. +- **IoTDB_DIALECT_MODE = tree:** + - The number of devices must be greater than or equal to the number of databases. + +### 3.2 **Working** **Mode** + +The `BENCHMARK_WORK_MODE` parameter supports four operational modes: + +1. **General Test Mode (****`testWithDefaultPath`****):** Configured via the `OPERATION_PROPORTION` parameter to support write-only, read-only, and mixed read-write operations. +2. **Data Generation Mode (****`generateDataMode`****):** Generates a reusable dataset, which is saved to `FILE_PATH` for subsequent use in the correctness write and correctness query modes. +3. **Single Database Correctness Write Mode (****`verificationWriteMode`****):** Verifies the correctness of dataset writing by writing the dataset generated in data generation mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. +4. **Single Database Correctness Query Mode (****`verificationQueryMode`****):** Verifies the correctness of dataset queries after using the correctness write mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. + +Mode configurations are shown in the following below: + +| **Mode name** | **BENCHMARK_WORK_MODE** | Description | Required Configuration | +| :------------------------------------- | :---------------------- | :------------------------------------------------------ | :------------------------- | +| General test mode | testWithDefaultPath | Supports multiple read and write mixed load operations. | `OPERATION_PROPORTION` | +| Generate data mode | generateDataMode | Generates datasets recognizable by IoT-benchmark. | `FILE_PATH` and `DATA_SET` | +| Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | +| Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | + +### 3.3 **Server** **Connection** **Information** + +Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: + +| **Parameter** | **Type** | **Example** | D**escription** | +| :------------ | :------- | :---------------------------- | :----------------------------------------------------- | +| DB_SWITCH | String | `IoTDB-200-SESSION_BY_TABLET` | Specifies the type of time-series database under test. | +| HOST | String | `127.0.0.1` | Network address of the target time-series database. | +| PORT | Integer | `6667` | Network port of the target time-series database. | +| USERNAME | String | `root` | Login username for the time-series database. | +| PASSWORD | String | `root` | Password for the database login user. | +| DB_NAME | String | `test` | Name of the target time-series database. | +| TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | + +### 3.4 **Write Scenario Parameters** + +| **Parameter** | **Type** | **Example** | D**escription** | +| :------------------------- | :-------------------- | :-------------------------- | :----------------------------------------------------------- | +| CLIENT_NUMBER | Integer | `100` | Total number of clients used for writing. | +| GROUP_NUMBER | Integer | `20` | Number of databases (only applicable for IoTDB). | +| DEVICE_NUMBER | Integer | `100` | Total number of devices. | +| SENSOR_NUMBER | Integer | `300` | Total number of sensors per device. (Control the number of attribute columns if you use the IoTDB table model) | +| INSERT_DATATYPE_PROPORTION | String | `1:1:1:1:1:1:0:0:0:0` | Ratio of data types: `BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE`. | +| POINT_STEP | Integer | `1000` | Time interval (in ms) between generated data points. | +| OP_MIN_INTERVAL | Integer | `0` | Minimum execution interval for operations (ms): if the operation takes more than the value, the next one will be executed immediately, otherwise wait (OP_MIN_INTERVAL - actual execution time) ms; if it is 0, the parameter is not effective; if it is -1, its value is consistent with POINT_STEP | +| IS_OUT_OF_ORDER | Boolean | `false` | Specifies whether to write data out of order. | +| OUT_OF_ORDER_RATIO | Floating point number | `0.3` | Proportion of out-of-order data. | +| BATCH_SIZE_PER_WRITE | Integer | `1` | Number of data rows written per batch. | +| START_TIME | Time | `2022-10-30T00:00:00+08:00` | Start timestamp for data generation. | +| LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | +| OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | -According to the configuration parameters in Table 1-4, the test scenario can be described as follows: write 30,000 (100 devices, 300 sensors for each device) time series sequential data for a day on October 30, 2022 to the time series database under test, in total 2.592 billion data points. The 300 sensor data types of each device are 50 Booleans, 50 integers, 50 long integers, 50 floats, 50 doubles, and 50 characters. If we change the value of IS_OUT_OF_ORDER in the table to true, then the scenario is: write 30,000 time series data on October 30, 2022 to the measured time series database, and there are 30% out of order data ( arrives in the time series database later than other data points whose generation time is later than itself). +### 3.5 **Query Scenario Parameters** -### 2.4 Query Scene Setup Parameters +| Parameter | Type | Example | Description | +| :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | +| QUERY_DEVICE_NUM | Integer | `2` | Number of devices involved in each query statement. | +| QUERY_SENSOR_NUM | Integer | `2` | Number of sensors involved in each query statement. | +| QUERY_AGGREGATE_FUN | Character | `count` | Aggregate functions used in queries (`COUNT`, `AVG`, `SUM`, etc.). | +| STEP_SIZE | Integer | `1` | Time interval step for time filter conditions. | +| QUERY_INTERVAL | Integer | `250000` | Time interval between query start and end times. | +| QUERY_LOWER_VALUE | Integer | `-5` | Threshold for conditional queries (`WHERE value > QUERY_LOWER_VALUE`). | +| GROUP_BY_TIME_UNIT | Integer | `20000` | The size of the group in the `GROUP BY` statement | +| LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | +| OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | -Table 1-5 Query scene setup parameters +### 3.6 **Query Types and Example SQL** -| Parameter Name | Type | Example | Description | -| -------------------- | ------- | --------------------- | ------------------------------------------------------------ | -| QUERY_DEVICE_NUM | Integer | 2 | The number of devices involved in the query in each query statement. | -| QUERY_SENSOR_NUM | Integer | 2 | The number of sensors involved in the query in each query statement. | -| QUERY_AGGREGATE_FUN | String | count | Aggregate functions used in aggregate queries, such as count, avg, sum, max_time, etc. | -| STEP_SIZE | Integer | 1 | The change step of the starting time point of the time filter condition, if set to 0, the time filter condition of each query is the same, unit: POINT_STEP. | -| QUERY_INTERVAL | Integer | 250000 | The time interval between the start time and the end time in the start and end time query, and the time interval in Group By. | -| QUERY_LOWER_VALUE | Integer | -5 | Parameters for conditional query clauses, where xxx > QUERY_LOWER_VALUE. | -| GROUP_BY_TIME_UNIT | Integer | 20000 | The size of the group in the Group By statement. | -| LOOP | Integer | 10 | Total number of operations. Each type of operation will be divided according to the ratio defined by OPERATION_PROPORTION. | -| OPERATION_PROPORTION | String | 0:0:0:0:0:0:0:0:0:0:1 | Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | +| Number | Query Type | IoTDB Sample SQL | +| :----- | :----------------------------- | :----------------------------------------------------------- | +| Q1 | Precise Point Query | `select v1 from root.db.d1 where time = ?` | +| Q2 | Time Range Query | `select v1 from root.db.d1 where time > ? and time < ?` | +| Q3 | Time Range with Value Filter | `select v1 from root.db.d1 where time > ? and time < ? and v1 > ?` | +| Q4 | Time Range Aggregation Query | `select count(v1) from root.db.d1 where and time > ? and time < ?` | +| Q5 | Full-Time Range with Filtering | `select count(v1) from root.db.d1 where v1 > ?` | +| Q6 | Range Aggregation with Filter | `select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ?` | +| Q7 | Time Grouping Aggregation | `select count(v1) from root.db.d1 group by ([?, ?), ?, ?)` | +| Q8 | Latest Point Query | `select last v1 from root.db.d1` | +| Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | +| Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | -Table 1-6 Query types and example SQL +### 3.7 **Test process and test result persistence** -| Id | Query Type | IoTDB Example SQL | -| ---- | ---------------------------------------------------- | ------------------------------------------------------------ | -| Q1 | exact point query | select v1 from root.db.d1 where time = ? | -| Q2 | time range query | select v1 from root.db.d1 where time > ? and time < ? | -| Q3 | time range query with value filtering | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | -| Q4 | time range aggregation query | select count(v1) from root.db.d1 where and time > ? and time < ? | -| Q5 | full time range aggregate query with value filtering | select count(v1) from root.db.d1 where v1 > ? | -| Q6 | time range aggregation query with value filtering | select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ? | -| Q7 | time grouping aggregation query | select count(v1) from root.db.d1 group by ([?, ?), ?, ?) | -| Q8 | latest point query | select last v1 from root.db.d1 | -| Q9 | reverse order time range query | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | -| Q10 | reverse order time range query with value filtering | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +IoT-benchmark currently supports persisting the test process and test results through configuration parameters. -According to the configuration parameters in Table 1-5, the test scenario can be described as follows: Execute 10 reverse order time range queries with value filtering for 2 devices and 2 sensors from the time series database under test. The SQL statement is: `select s_0,s_31from data where time >2022-10-30T00:00:00+08:00 and time < 2022-10-30T00:04:10+08:00 and s_0 > -5 and device in d_21,d_46 order by time desc`. +| **Parameter** | **Type** | **Example** | D**escription** | +| :-------------------- | :------- | :---------- | :----------------------------------------------------------- | +| TEST_DATA_PERSISTENCE | String | `None` | Specifies the result persistence method. Options: `None`, `IoTDB`, `MySQL`, `CSV`. | +| RECORD_SPLIT | Boolean | `true` | Whether to split results into multiple records. (Not supported by IoTDB currently.) | +| RECORD_SPLIT_MAX_LINE | Integer | `10000000` | Maximum number of rows per record (10 million rows per database table or CSV file). | +| TEST_DATA_STORE_IP | String | `127.0.0.1` | IP address of the database for result storage. | +| TEST_DATA_STORE_PORT | Integer | `6667` | Port number of the output database. | +| TEST_DATA_STORE_DB | String | `result` | Name of the output database. | +| TEST_DATA_STORE_USER | String | `root` | Username for accessing the output database. | +| TEST_DATA_STORE_PW | String | `root` | Password for accessing the output database. | -### 2.5 Persistence of Test Process and Test Results +**Result Persistence Details** -IoT-benchmark currently supports persisting the test process and test results to IoTDB, MySQL, and CSV through the configuration parameter "TEST_DATA_PERSISTENCE"; writing to MySQL and CSV can define the upper limit of the number of rows in the sub-database and sub-table, such as "RECORD_SPLIT=true, RECORD_SPLIT_MAX_LINE=10000000" means that each database table or CSV file is divided and stored according to the total number of 10 million rows; if the records are recorded to MySQL or IoTDB, database link information needs to be provided, including "TEST_DATA_STORE_IP" the IP address of the database, "TEST_DATA_STORE_PORT" the port number of the database, "TEST_DATA_STORE_DB" the name of the database, "TEST_DATA_STORE_USER" the database user name, and "TEST_DATA_STORE_PW" the database user password. +- **CSV Mode:** If `TEST_DATA_PERSISTENCE` is set to `CSV`, a `data` folder is generated in the IoT-benchmark root directory during and after test execution. This folder contains: + - `csv` folder: Records the test process. + - `csvOutput` folder: Stores the test results. +- **MySQL Mode:** If `TEST_DATA_PERSISTENCE` is set to `MySQL`, IoT-benchmark creates the following tables in the specified MySQL database: + - **Test Process Table:** + 1. Created before the test starts. + 2. Named as: `testWithDefaultPath___`. + - **Configuration Table:** + 1. Named `CONFIG`. + 2. Stores the test configuration. + 3. Created if it does not exist. + - **Final Result Table:** + 1. Named `FINAL_RESULT`. + 2. Stores the test results after test completion. + 3. Created if it does not exist. -If we set "TEST_DATA_PERSISTENCE=CSV", we can see the newly generated data folder under the IoT-benchmark root directory during and after the test execution, which contains the csv folder to record the test process; the csvOutput folder to record the test results . If we set "TEST_DATA_PERSISTENCE=MySQL", it will create a data table named "testWithDefaultPath_tested database name_remarks_test start time" in the specified MySQL database before the test starts to record the test process; it will record the test process in the "CONFIG" data table (create the table if it does not exist), write the configuration information of this test; when the test is completed, the result of this test will be written in the data table named "FINAL_RESULT" (create the table if it does not exist). +### 3.8 Automation Script -## 3. Use Case +#### One-Click Script Startup + +The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. + +**Steps to Run:** + +1. Edit the `IOTDB_HOME` parameter in `cli-benchmark.sh` to the local IoTDB directory. +2. Start the test by running the following command: + +```Bash +> ./cli-benchmark.sh +``` + +1. After the test completes: +1. Check test-related logs in the `logs` folder. +2. Check monitoring-related logs in the `server-logs` folder. + +#### Automatic Execution of Multiple Tests + +Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. + +1. **Routine Configuration:** Each line in the `routine` file specifies the parameters that change for each test. For example: + + ```Plain + LOOP=10 DEVICE_NUMBER=100 TEST + LOOP=20 DEVICE_NUMBER=50 TEST + LOOP=50 DEVICE_NUMBER=20 TEST + ``` + +In this example, three tests will run sequentially with `LOOP` values of 10, 20, and 50. + +Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in sequence. + +**Important Notes:** + +- Multiple parameters can be changed in each test using the format: + + ```Bash + LOOP=20 DEVICE_NUMBER=10 TEST + ``` + +- Avoid unnecessary spaces. + +- The `TEST` keyword marks the start of a new test. + +- Changed parameters persist across subsequent tests unless explicitly reset. + +2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command + + ```Bash + > ./rep-benchmark.sh + ``` + +Test results will be displayed in the terminal. + +**Important Notes:** + +- Closing the terminal or losing the client connection will terminate the test process. + +- To run the test as a background daemon, execute: + + ```Bash + > ./rep-benchmark.sh > /dev/null 2>&1 & + ``` + +- To monitor progress, check the logs: + + ```Bash + > cd ./logs + > tail -f log_info.log + ``` + +## 4. Use Case We take the application of CRRC Qingdao Sifang Vehicle Research Institute Co., Ltd. as an example, and refer to the scene described in "Apache IoTDB in Intelligent Operation and Maintenance Platform Storage" for practical operation instructions. Test objective: Simulate the actual needs of switching time series databases in the scene of CRRC Qingdao Sifang Institute, and compare the performance of the expected IoTDB and KairosDB used by the original system. -Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 2-1. The specific usage of each virtual machine is shown in Table 2-2. +Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 4-1. The specific usage of each virtual machine is shown in Table 4-2. -Table 2-1 Virtual machine configuration information +Table 4-1 Virtual machine configuration information | Hardware Configuration Information | Value | | ---------------------------------- | ------- | @@ -213,7 +359,7 @@ Table 2-1 Virtual machine configuration information | hard disk | 200G | | network | Gigabit | -Table 2-2 Virtual machine usage +Table 4-2 Virtual machine usage | IP | Usage | | ---------- | ------------- | @@ -222,11 +368,11 @@ Table 2-2 Virtual machine usage | 172.21.4.4 | KaiosDB | | 172.21.4.5 | MySQL | -### 3.1 Write Test +### 4.1 Write Test -Scenario description: Create 100 clients to simulate 100 trains, each train has 3000 sensors, the data type is DOUBLE, the data time interval is 500ms (2Hz), and they are sent sequentially. Referring to the above requirements, we need to modify the IoT-benchmark configuration parameters as listed in Table 2-3. +Scenario description: Create 100 clients to simulate 100 trains, each train has 3000 sensors, the data type is DOUBLE, the data time interval is 500ms (2Hz), and they are sent sequentially. Referring to the above requirements, we need to modify the IoT-benchmark configuration parameters as listed in Table 4-3. -Table 2-3 Configuration parameter information +Table 4-3 Configuration parameter information | Parameter Name | IoTDB Value | KairosDB Value | | -------------------------- | --------------------------- | -------------- | @@ -253,40 +399,40 @@ Table 2-3 Configuration parameter information | TEST_DATA_STORE_PW | admin | | | REMARK | demo | | -First, start the tested time series databases Apache-IoTDB and KairosDB on 172.21.4.3 and 172.21.4.4 respectively, and then start server resource monitoring through the ser-benchamrk\.sh script on 172.21.4.2, 172.21.4.3 and 172.21.4.4 (Figure 2-1). Then modify the conf/config.properties files in the iotdb-0.13-0.0.1 and kairosdb-0.0.1 folders in 172.21.4.2 according to Table 2-3 to meet the test requirements. Use benchmark\.sh to start the writing test of Apache-IoTDB and KairosDB successively. +First, start the tested time series databases Apache-IoTDB and KairosDB on 172.21.4.3 and 172.21.4.4 respectively, and then start server resource monitoring through the ser-benchamrk\.sh script on 172.21.4.2, 172.21.4.3 and 172.21.4.4 (Figure 4-1). Then modify the conf/config.properties files in the iotdb-0.13-0.0.1 and kairosdb-0.0.1 folders in 172.21.4.2 according to Table 4-3 to meet the test requirements. Use benchmark\.sh to start the writing test of Apache-IoTDB and KairosDB successively. ![img](/img/bm5.png) -Figure 2-1 Server monitoring tasks +Figure 4-1 Server monitoring tasks -For example, if we first start the test on KairosDB, IoT-benchmark will create a CONFIG data table in the MySQL database to store the configuration information of this test (Figure 2-2), and there will be a log output of the current test progress during the test execution (Figure 2-3) . When the test is completed, the test result will be output (Figure 2-3), and the result will be written into the FINAL_RESULT data table (Figure 2-4). +For example, if we first start the test on KairosDB, IoT-benchmark will create a CONFIG data table in the MySQL database to store the configuration information of this test (Figure 4-2), and there will be a log output of the current test progress during the test execution (Figure 4-3) . When the test is completed, the test result will be output (Figure 4-3), and the result will be written into the FINAL_RESULT data table (Figure 4-4). ![img](/img/bm6.png) -Figure 2-2 Test configuration information table +Figure 4-2 Test configuration information table ![img](/img/bm7.png) ![img](/img/bm8.png) ![img](/img/bm9.png) ![img](/img/bm10.png) -Figure 2-3 Test progress and results +Figure 4-3 Test progress and results ![img](/img/bm11.png) -Figure 2-4 Test result table +Figure 4-4 Test result table Afterwards, we will start the test on Apache-IoTDB. The same IoT-benchmark will write the test configuration information in the MySQL database CONFIG data table. During the test execution, there will be a log to output the current test progress. When the test is completed, the test result will be output, and the result will be written into the FINAL_RESULT data table. -According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 2-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc. +According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 4-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc. ![img](/img/bm12.png) -Figure 2-5 Disk usage +Figure 4-5 Disk usage So what is the resource usage of each server during the test? What is the specific performance of each write operation? At this time, we can visualize the data in the server monitoring table and test process recording table by installing and using Tableau. The use of Tableau will not be introduced in this article. After connecting to the data table for test data persistence, the specific results are as follows (taking Apache-IoTDB as an example): @@ -295,13 +441,13 @@ So what is the resource usage of each server during the test? What is the specif ![img](/img/bm14.png) -Figure 2-6 Visualization of testing process in Tableau +Figure 4-6 Visualization of testing process in Tableau -### 3.2 Query Test +### 4.2 Query Test Scenario description: In the writing test scenario, 10 clients are simulated to perform all types of query tasks on the data stored in the time series database Apache-IoTDB. The configuration is as follows. -Table 2-4 Configuration parameter information +Table 4-4 Configuration parameter information | Parameter Name | Example | | -------------------- | --------------------- | @@ -320,13 +466,13 @@ Results: ![img](/img/bm15.png) -Figure 2-7 Query test results +Figure 4-7 Query test results -### 3.3 Description of Other Parameters +### 4.3 Description of Other Parameters In the previous chapters, the write performance comparison between Apache-IoTDB and KairosDB was performed, but if the user wants to perform a simulated real write rate test, how to configure it? How to control if the test time is too long? Are there any regularities in the generated simulated data? If the IoT-Benchmark server configuration is low, can multiple machines be used to simulate pressure output? -Table 2-5 Configuration parameter information +Table 4-5 Configuration parameter information | Scenario | Parameter | Value | Notes | | ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | @@ -341,4 +487,4 @@ Table 2-5 Configuration parameter information | STRING_LENGTH | 10 | String length | | | DOUBLE_LENGTH | 2 | Decimal places | | | Three machines simulate data writing of 300 devices | BENCHMARK_CLUSTER | true | Enable multi-benchmark mode | -| BENCHMARK_INDEX | 0, 1, 3 | Take the writing parameters in the [write test](./Benchmark.md#write-test) as an example: No. 0 is responsible for writing data of device numbers 0-99; No. 1 is responsible for writing data of device numbers 100-199; No. 2 is responsible for writing data of device numbers 200-299. | | \ No newline at end of file +| BENCHMARK_INDEX | 0, 1, 3 | Take the writing parameters in the [write test](./Benchmark.md#_4-1-write-test) as an example: No. 0 is responsible for writing data of device numbers 0-99; No. 1 is responsible for writing data of device numbers 100-199; No. 2 is responsible for writing data of device numbers 200-299. | | \ No newline at end of file diff --git a/src/UserGuide/latest-Table/Tools-System/Benchmark.md b/src/UserGuide/latest-Table/Tools-System/Benchmark.md index b8dc8c4ff..bf32467f3 100644 --- a/src/UserGuide/latest-Table/Tools-System/Benchmark.md +++ b/src/UserGuide/latest-Table/Tools-System/Benchmark.md @@ -40,50 +40,51 @@ Figure 1-2 *IoT-benchmark Modular Design* Currently, IoT-benchmark supports the following time series databases, versions and connection methods: -| Database | Version | Connection mmethod | -| :-------------- | :-------------- | :------------------------------------------------------- | -| InfluxDB | v1.x v2.0 | SDK | -| TimescaleDB | -- | JDBC | -| OpenTSDB | -- | HTTP Request | -| QuestDB | v6.0.7 | JDBC | -| TDengine | v2.2.0.2 | JDBC | -| VictoriaMetrics | v1.64.0 | HTTP Request | -| KairosDB | -- | HTTP Request | -| IoTDB | v2.0 v1.x v0.13 | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | +| Database | Version | Connection mmethod | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | JDBC | +| OpenTSDB | -- | HTTP Request | +| QuestDB | v6.0.7 | JDBC | +| TDengine | v2.2.0.2 | JDBC | +| VictoriaMetrics | v1.64.0 | HTTP Request | +| KairosDB | -- | HTTP Request | + ## 2. **Installation and Operation** -#### **Prerequisites** +### 2.1 **Prerequisites** 1. Java 8 2. Maven 3.6+ 3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 -#### **How to Obtain** +### 2.2 **How to Obtain** - **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. - - **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** +- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** - - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.1 and run the following command in the root directory to compile the latest IoTDB Session package: + - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: - ```Bash - mvn clean package install -pl session -am -DskipTests - ``` + ```Bash + mvn clean package install -pl session -am -DskipTests + ``` - - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. + - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. - ```Bash - mvn clean package install -pl iotdb-2.0 -am -DskipTests - ``` + ```Bash + mvn clean package install -pl iotdb-2.0 -am -DskipTests + ``` - - The compiled test package will be located at: + - The compiled test package will be located at: - ```Bash - ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 - ``` + ```Bash + ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 + ``` -#### **Test Package Structure** +### 2.3 **Test Package Structure** The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. @@ -113,7 +114,7 @@ drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib -#### **Execution** **of** **Tests** +### 2.4 **Execution** **of** **Tests** 1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: @@ -127,9 +128,7 @@ drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib 4. Upon completion, review the results and analyze the test process. -#### - -#### **Results Interpretation** +### 2.5 **Results Interpretation** All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: @@ -147,7 +146,7 @@ All test log files are stored in the `logs` folder, while test results are saved ## 3. **Main** **Parameters** -#### IoTDB Service Model +### 3.1 IoTDB Service Model The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. @@ -155,8 +154,6 @@ The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The d - **IoTDB_DIALECT_MODE = table:** - The number of devices must be an integer multiple of the number of tables. - The number of tables must be an integer multiple of the number of databases. -- **IoTDB_DIALECT_MODE = tree:** - - The number of devices must be greater than or equal to the number of databases. Key Parameters for IoTDB Service Model @@ -167,7 +164,7 @@ Key Parameters for IoTDB Service Model | SENSOR_NUMBER | Integer | `10` | Controls the number of attribute columns in the table model. | | IoTDB_TABLE_NUMBER | Integer | `1` | Specifies the number of tables when using the table model. | -#### **Working** **M****ode** +### 3.2 **Working** **Mode** The `BENCHMARK_WORK_MODE` parameter supports four operational modes: @@ -185,7 +182,7 @@ Mode configurations are shown in the following below: | Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | | Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | -#### **Server** **Connection** **Information** +### 3.3 **Server** **Connection** **Information** Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: @@ -199,7 +196,7 @@ Once the working mode is specified, the following parameters must be configured | DB_NAME | String | `test` | Name of the target time-series database. | | TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | -#### **Write Scenario Parameters** +### 3.4 **Write Scenario Parameters** | **Parameter** | **Type** | **Example** | D**escription** | | :------------------------- | :-------------------- | :-------------------------- | :----------------------------------------------------------- | @@ -217,7 +214,7 @@ Once the working mode is specified, the following parameters must be configured | LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | | OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | -#### **Query Scenario Parameters** +### 3.5 **Query Scenario Parameters** | Parameter | Type | Example | Description | | :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | @@ -231,7 +228,7 @@ Once the working mode is specified, the following parameters must be configured | LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | | OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | -#### **Query Types and Example SQL** +### 3.6 **Query Types and Example SQL** | Number | Query Type | IoTDB Sample SQL | | :----- | :----------------------------- | :----------------------------------------------------------- | @@ -246,7 +243,7 @@ Once the working mode is specified, the following parameters must be configured | Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | | Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | -#### **Test process and test result persistence** +### 3.7 **Test process and test result persistence** IoT-benchmark currently supports persisting the test process and test results through configuration parameters. @@ -279,9 +276,9 @@ IoT-benchmark currently supports persisting the test process and test results th 2. Stores the test results after test completion. 3. Created if it does not exist. -#### Automation Script +### 3.8 Automation Script -##### One-Click Script Startup +#### One-Click Script Startup The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. @@ -298,7 +295,7 @@ The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark 1. Check test-related logs in the `logs` folder. 2. Check monitoring-related logs in the `server-logs` folder. -##### Automatic Execution of Multiple Tests +#### Automatic Execution of Multiple Tests Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. @@ -328,13 +325,13 @@ Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in se - Changed parameters persist across subsequent tests unless explicitly reset. -1. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command +2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command ```Bash > ./rep-benchmark.sh ``` -2. Test results will be displayed in the terminal. +Test results will be displayed in the terminal. **Important Notes:** diff --git a/src/UserGuide/latest/Tools-System/Benchmark.md b/src/UserGuide/latest/Tools-System/Benchmark.md index 8172be84c..2f6bd86d6 100644 --- a/src/UserGuide/latest/Tools-System/Benchmark.md +++ b/src/UserGuide/latest/Tools-System/Benchmark.md @@ -21,189 +21,335 @@ # Benchmark Tool -IoT-benchmark is a time-series database benchmarking tool based on Java and big data environment, developed and open sourced by School of Software Tsinghua University. It is easy to use, supports multiple writing and query methods, supports storing test information and results for further query or analysis, and supports integration with Tableau to visualize test results. +## 1. **Basic Overview** -Figure 1-1 below includes the test benchmark process and other extended functions. These processes can be unified by IoT-benchmark. IoT Benchmark supports a variety of workloads, including **pure write, pure query, write query mixed**, etc., supports **software and hardware system monitoring, test metric measurement** and other monitoring functions, and also realizes **initializing the database automatically, test data analysis and system parameter optimization** functions. +IoT-benchmark is a time-series database benchmarking tool developed in Java for big data environments. It was developed and open-sourced by the School of Software, Tsinghua University. The tool is user-friendly, supports various write and query methods, allows storing test information and results for further queries or analysis, and integrates with Tableau for visualizing test results. -![img](/img/benchmark-English1.png) +Figure 1-1 illustrates the test benchmark process and its extended functionalities, all of which can be streamlined by IoT-benchmark. It supports a variety of workloads, including write-only, read-only, and mixed write-and-read operations. Additionally, it offers software and hardware system monitoring, performance metric measurement, automated database initialization, test data analysis, and system parameter optimization. +![](/img/benchmark-English1.png) -Figure 1-1 +Figure 1-1 *IoT-benchmark Test Benchmark Process* -Referring to the YCSB test tool's design idea of separating the three components of workload generation, performance metric measurement and database interface, the modular design of IoT-benchmark is shown in Figure 1-2. Different from the YCSB-based test tool system, IoT-benchmark adds a system monitoring module to support the persistence of test data and system monitoring data. In addition, some special load testing functions especially designed for time series data scenarios have been added, such as supporting batch writing and multiple out-of-sequence data writing modes for IoT scenarios. +IoT-benchmark adopts the modular design concept of the YCSB test tool, which separates workload generation, performance measurement, and database interface components. Its modular structure is illustrated in Figure 1-2. Unlike YCSB-based testing tools, IoT-benchmark introduces a system monitoring module that supports the persistence of both test data and system metrics. It also includes load-testing functionalities specifically designed for time-series data scenarios, such as batch writes and multiple out-of-order data insertion modes for IoT environments. -![img](/img/benchmark-%20English2.png) +![](/img/benchmark-%20English2.png) +Figure 1-2 *IoT-benchmark Modular Design* -Figure 1-2 +**Supported Databases** -Currently IoT-benchmark supports the following time series databases, versions and connection methods: +Currently, IoT-benchmark supports the following time series databases, versions and connection methods: -| Database | Version | Connection mmethod | -| --------------- | ------- | -------------------------------------------------------- | -| InfluxDB | v1.x
v2.0 | SDK | | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | -| IoTDB | v1.x
v0.13 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| Database | Version | Connection mmethod | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | JDBC, SessionByTablet, SessionByRecord, SessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | JDBC | +| OpenTSDB | -- | HTTP Request | +| QuestDB | v6.0.7 | JDBC | +| TDengine | v2.2.0.2 | JDBC | +| VictoriaMetrics | v1.64.0 | HTTP Request | +| KairosDB | -- | HTTP Request | -Table 1-1 Comparison of big data test benchmarks -## 1. Software Installation and Environment Setup +## 2. **Installation and Operation** -### 1.1 Prerequisites +### 2.1 **Prerequisites** 1. Java 8 2. Maven 3.6+ -3. The corresponding appropriate version of the database, such as Apache IoTDB 1.0 +3. The corresponding appropriate version of the database, such as Apache IoTDB 2.0 -### 1.2 How to Get IoT Benchmark +### 2.2 **How to Obtain** -- **Get the binary package**: Enter https://github.com/thulab/iot-benchmark/releases to download the required installation package. Download it as a compressed file, select a folder to decompress and use it. -- Compiled from source (can be tested with Apache IoTDB 1.0): - - The first step (compile the latest IoTDB Session package): Enter the official website https://github.com/apache/iotdb/tree/rel/1.0 to download the IoTDB source code, and run the command `mvn clean package install -pl session -am -DskipTests` in the root directory to compiles the latest package for IoTDB Session. - - The second step (compile the IoTDB Benchmark test package): Enter the official website https://github.com/thulab/iot-benchmark to download the source code, run `mvn clean package install -pl iotdb-1.0 -am -DskipTests` in the root directory to compile Apache IoTDB version 1.0 test package. The relative path between the test package and the root directory is `./iotdb-1.0/target/iotdb-1.0-0.0.1/iotdb-1.0-0.0.1`. +- **B****inary package****:** Visit https://github.com/thulab/iot-benchmark/releases to download the installation package. Extract the compressed file into a desired folder for use. -### 1.3 IoT Benchmark's Test Package Structure +- **Source Code** **Compilation (for** **Apache** **IoTDB 2.0 testing):** -The directory structure of the test package is shown in Figure 1-3 below. The test configuration file is conf/config.properties, and the test startup scripts are benchmark\.sh (Linux & MacOS) and benchmark.bat (Windows). The detailed usage of the files is shown in Table 1-2. + - **Compile the latest IoTDB Session package:** Download the IoTDB source code from https://github.com/apache/iotdb/tree/rc/2.0.5 and run the following command in the root directory to compile the latest IoTDB Session package: -![img](/img/bm3.png) + ```Bash + mvn clean package install -pl session -am -DskipTests + ``` + - **Compile the IoT-benchmark test package:** Download the source code from https://github.com/thulab/iot-benchmark and run the following command in the root directory to compile the Apache IoTDB 2.0 test package:. -Figure 1-3 List of files and folders + ```Bash + mvn clean package install -pl iotdb-2.0 -am -DskipTests + ``` -| Name | File | Usage | -| ---------------- | ----------------- | -------------------------------- | -| benchmark.bat | - | Startup script on Windows | -| benchmark\.sh | - | Startup script on Linux/Mac | -| conf | config.properties | Test scenario configuration file | -| logback.xml | - | Log output configuration file | -| lib | - | Dependency library | -| LICENSE | - | License file | -| bin | startup\.sh | Init script folder | -| ser-benchmark\.sh | - | Monitor mode startup script | + - The compiled test package will be located at: -Table 1-2 Usage list of files and folders + ```Bash + ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 + ``` -### 1.4 IoT Benchmark Execution Test +### 2.3 **Test Package Structure** -1. Modify the configuration file according to the test requirements. For the main parameters, see next chapter. The corresponding configuration file is conf/config.properties. For example, to test Apache IoTDB 1.0, you need to modify DB_SWITCH=IoTDB-100-SESSION_BY_TABLET. -2. Start the time series database under test. -3. Running. -4. Start IoT-benchmark to execute the test. Observe the status of the time series database and IoT-benchmark under test during execution, and view the results and analyze the test process after execution. +The directory structure of the test package is shown below. The test configuration file is `conf/config.properties`, and the test startup scripts are `benchmark.sh` (Linux & MacOS) and `benchmark.bat` (Windows). The detailed usage of the files is shown in the table below. -### 1.5 IoT Benchmark Results Interpretation +```Shell +-rw-r--r--. 1 root root 2881 Jan 10 01:36 benchmark.bat +-rwxr-xr-x. 1 root root 314 Jan 10 01:36 benchmark.sh +drwxr-xr-x. 2 root root 24 Jan 10 01:36 bin +-rwxr-xr-x. 1 root root 1140 Jan 10 01:36 cli-benchmark.sh +drwxr-xr-x. 2 root root 107 Jan 10 01:36 conf +drwxr-xr-x. 2 root root 4096 Jan 10 01:38 lib +-rw-r--r--. 1 root root 11357 Jan 10 01:36 LICENSE +-rwxr-xr-x. 1 root root 939 Jan 10 01:36 rep-benchmark.sh +-rw-r--r--. 1 root root 14 Jan 10 01:36 routine +``` -All the log files of the test are stored in the logs folder, and the test results are stored in the data/csvOutput folder after the test is completed. For example, after the test, we get the following result matrix: +| Name | File | Usage | +| :--------------- | :---------------- | :-------------------------------------------------- | +| benchmark.bat | - | Startup script on Windows | +| benchmark.sh | - | Startup script on Linux/Mac | +| bin | startup.sh | Initialization script folder | +| conf | config.properties | Test scenario configuration file | +| lib | - | Dependency library | +| LICENSE | - | License file | +| cli-benchmark.sh | - | One-click startup script | +| routine | - | Automatic execution of multiple test configurations | +| rep-benchmark.sh | - | Automatic execution of multiple test scripts | -![img](/img/bm4.png) -- Result Matrix - - OkOperation: successful operations - - OkPoint: For write operations, it is the number of points successfully written; for query operations, it is the number of points successfully queried. - - FailOperation: failed operations - - FailPoint: For write operations, it is the number of write failure points -- Latency(mx) Matrix - - AVG: average operation time - - MIN: minimum operation time - - Pn: the quantile value of the overall distribution of operations, for example, P25 is the lower quartile. +### 2.4 **Execution** **of** **Tests** -## 2. Main Parameters +1. Modify the configuration file (conf/config.properties) according to test requirements. For example, to test Apache IoTDB 2.0, set the following parameter: -This chapter mainly explains the purpose and configuration method of the main parameters. + ```Bash + DB_SWITCH=IoTDB-200-SESSION_BY_TABLET + ``` -### 2.1 Working Mode and Operation Proportion +2. Ensure the target time-series database is running. -- The working mode parameter "BENCHMARK_WORK_MODE" can be selected as "default mode" and "server monitoring"; the "server monitoring" mode can be started directly by executing the ser-benchmark\.sh script, and the script will automatically modify this parameter. "Default mode" is a commonly used test mode, combined with the configuration of the OPERATION_PROPORTION parameter to achieve the definition of test operation proportions of "pure write", "pure query" and "read-write mix". +3. Start IoT-benchmark to execute the test. Monitor the status of both the target database and IoT-benchmark during execution. -- When running ServerMode to monitor the operating environment of the time series database under test, IoT-benchmark relies on sysstat software related commands; if MySQL or IoTDB is selected for persistent test process data, this type of database needs to be installed; the recording mode of ServerMode and CSV can only be used in the Linux system to record relevant system information during the test. Therefore, we recommend using MacOs or Linux system. This article uses Linux (Centos7) system as an example. If you use Windows system, you can use the benchmark.bat script in the conf folder to start IoT-benchmark. +4. Upon completion, review the results and analyze the test process. -Table 1-3 Test mode +### 2.5 **Results Interpretation** -| Mode Name | BENCHMARK_WORK_MODE | Description | -| ------------ | ------------------- | ------------------------------------------------------------ | -| default mode | testWithDefaultPath | Supports mixed workloads with multiple read and write operations | -| server mode | serverMODE | Server resource usage monitoring mode (running in this mode is started by the ser-benchmark\.sh script, no need to manually configure this parameter) | +All test log files are stored in the `logs` folder, while test results are saved in the `data/csvOutput` folder. For example, the following result matrix illustrates the test outcome: -### 2.2 Server Connection Information +![](/img/bm4.png) -After the working mode is specified, how to inform IoT-benchmark of the information of the time series database under test? Currently, the type of the time-series database under test is informed through "DB_SWITCH"; the network address of the time-series database under test is informed through "HOST"; the network port of the time-series database under test is informed through "PORT"; the login user name of the time-series database under test is informed through "USERNAME"; "PASSWORD" informs the password of the login user of the time series database under test; informs the name of the time series database under test through "DB_NAME"; informs the connection authentication token of the time series database under test through "TOKEN" (used by InfluxDB 2.0). +- **Result Matrix:** + - OkOperation: Number of successful operations. + - OkPoint: Number of successfully written points (for write operations) or successfully queried points (for query operations). + - FailOperation: Number of failed operations. + - FailPoint: Number of failed write points. +- **Latency (ms) Matrix:** + - AVG: Average operation latency. + - MIN: Minimum operation latency. + - Pn: Quantile values of the overall operation distribution (e.g., P25 represents the 25th percentile, or lower quartile). -### 2.3 Write Scene Setup Parameters +## 3. **Main** **Parameters** + +### 3.1 IoTDB Service Model -Table 1-4 Write scene setup parameters +The `IoTDB_DIALECT_MODE` parameter supports two modes: `tree` and `table`. The default value is `tree`. -| Parameter Name | Type | Example | Description | -| -------------------------- | --------- | ------------------------- | ------------------------------------------------------------ | -| CLIENT_NUMBER | Integer | 100 | Total number of clients | -| GROUP_NUMBER | Integer | 20 | Number of storage groups; only for IoTDB. | -| DEVICE_NUMBER | Integer | 100 | Total number of devices | -| SENSOR_NUMBER | Integer | 300 | Total number of sensors per device | -| INSERT_DATATYPE_PROPORTION | String | 1:1:1:1:1:1 | the data type proportion of the device, BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | Integer | 1000 | Timestamp interval, that is, the fixed length between two timestamps of generated data. | -| OP_MIN_INTERVAL | Integer | 0 | Minimum operation execution interval: if the operation time is greater than this value, execute the next one immediately, otherwise wait (OP_MIN_INTERVAL-actual execution time) ms; if it is 0, the parameter will not take effect; if it is -1, its value is consistent with POINT_STEP. | -| IS_OUT_OF_ORDER | Boolean | false | Whether to write out of order | -| OUT_OF_ORDER_RATIO | Float | 0.3 | Ratio of data written out of order | -| BATCH_SIZE_PER_WRITE | Integer | 1 | Number of data rows written in batches (how many rows of data are written at a time) | -| START_TIME | Timestamp | 2022-10-30T00:00:00+08:00 | The start timestamp of writing data; use this timestamp as the starting point to start the simulation to create the data timestamp. | -| LOOP | Integer | 86400 | Total number of operations: Each type of operation will be divided according to the ratio defined by OPERATION_PROPORTION | -| OPERATION_PROPORTION | String | 1:0:0:0:0:0:0:0:0:0:0 | The ratio of each operation. Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, please note the use of English colons. Each term in the scale is an integer. | +- **For IoTDB 2.0 and later versions**, the `IoTDB_DIALECT_MODE` parameter must be specified, and only one mode can be set for each IoTDB instance. +- **IoTDB_DIALECT_MODE = tree:** + - The number of devices must be greater than or equal to the number of databases. + +### 3.2 **Working** **Mode** + +The `BENCHMARK_WORK_MODE` parameter supports four operational modes: + +1. **General Test Mode (****`testWithDefaultPath`****):** Configured via the `OPERATION_PROPORTION` parameter to support write-only, read-only, and mixed read-write operations. +2. **Data Generation Mode (****`generateDataMode`****):** Generates a reusable dataset, which is saved to `FILE_PATH` for subsequent use in the correctness write and correctness query modes. +3. **Single Database Correctness Write Mode (****`verificationWriteMode`****):** Verifies the correctness of dataset writing by writing the dataset generated in data generation mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. +4. **Single Database Correctness Query Mode (****`verificationQueryMode`****):** Verifies the correctness of dataset queries after using the correctness write mode. This mode supports only IoTDB v1.0+ and InfluxDB v1.x. + +Mode configurations are shown in the following below: + +| **Mode name** | **BENCHMARK_WORK_MODE** | Description | Required Configuration | +| :------------------------------------- | :---------------------- | :------------------------------------------------------ | :------------------------- | +| General test mode | testWithDefaultPath | Supports multiple read and write mixed load operations. | `OPERATION_PROPORTION` | +| Generate data mode | generateDataMode | Generates datasets recognizable by IoT-benchmark. | `FILE_PATH` and `DATA_SET` | +| Single database correctness write mode | verificationWriteMode | Writes datasets for correctness verification. | `FILE_PATH` and `DATA_SET` | +| Single database correctness query mode | verificationQueryMode | Queries datasets to verify correctness. | `FILE_PATH` and `DATA_SET` | + +### 3.3 **Server** **Connection** **Information** + +Once the working mode is specified, the following parameters must be configured to inform IoT-benchmark of the target time-series database: + +| **Parameter** | **Type** | **Example** | D**escription** | +| :------------ | :------- | :---------------------------- | :----------------------------------------------------- | +| DB_SWITCH | String | `IoTDB-200-SESSION_BY_TABLET` | Specifies the type of time-series database under test. | +| HOST | String | `127.0.0.1` | Network address of the target time-series database. | +| PORT | Integer | `6667` | Network port of the target time-series database. | +| USERNAME | String | `root` | Login username for the time-series database. | +| PASSWORD | String | `root` | Password for the database login user. | +| DB_NAME | String | `test` | Name of the target time-series database. | +| TOKEN | String | - | Authentication token (used for InfluxDB 2.0). | + +### 3.4 **Write Scenario Parameters** + +| **Parameter** | **Type** | **Example** | D**escription** | +| :------------------------- | :-------------------- | :-------------------------- | :----------------------------------------------------------- | +| CLIENT_NUMBER | Integer | `100` | Total number of clients used for writing. | +| GROUP_NUMBER | Integer | `20` | Number of databases (only applicable for IoTDB). | +| DEVICE_NUMBER | Integer | `100` | Total number of devices. | +| SENSOR_NUMBER | Integer | `300` | Total number of sensors per device. (Control the number of attribute columns if you use the IoTDB table model) | +| INSERT_DATATYPE_PROPORTION | String | `1:1:1:1:1:1:0:0:0:0` | Ratio of data types: `BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT:STRING:BLOB:TIMESTAMP:DATE`. | +| POINT_STEP | Integer | `1000` | Time interval (in ms) between generated data points. | +| OP_MIN_INTERVAL | Integer | `0` | Minimum execution interval for operations (ms): if the operation takes more than the value, the next one will be executed immediately, otherwise wait (OP_MIN_INTERVAL - actual execution time) ms; if it is 0, the parameter is not effective; if it is -1, its value is consistent with POINT_STEP | +| IS_OUT_OF_ORDER | Boolean | `false` | Specifies whether to write data out of order. | +| OUT_OF_ORDER_RATIO | Floating point number | `0.3` | Proportion of out-of-order data. | +| BATCH_SIZE_PER_WRITE | Integer | `1` | Number of data rows written per batch. | +| START_TIME | Time | `2022-10-30T00:00:00+08:00` | Start timestamp for data generation. | +| LOOP | Integer | `86400` | Total number of write operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | +| OPERATION_PROPORTION | Character | `1:0:0:0:0:0:0:0:0:0:0` | Ratio of operation types (write:Q1:Q2:...:Q10). | -According to the configuration parameters in Table 1-4, the test scenario can be described as follows: write 30,000 (100 devices, 300 sensors for each device) time series sequential data for a day on October 30, 2022 to the time series database under test, in total 2.592 billion data points. The 300 sensor data types of each device are 50 Booleans, 50 integers, 50 long integers, 50 floats, 50 doubles, and 50 characters. If we change the value of IS_OUT_OF_ORDER in the table to true, then the scenario is: write 30,000 time series data on October 30, 2022 to the measured time series database, and there are 30% out of order data ( arrives in the time series database later than other data points whose generation time is later than itself). +### 3.5 **Query Scenario Parameters** -### 2.4 Query Scene Setup Parameters +| Parameter | Type | Example | Description | +| :------------------- | :-------- | :---------------------- | :----------------------------------------------------------- | +| QUERY_DEVICE_NUM | Integer | `2` | Number of devices involved in each query statement. | +| QUERY_SENSOR_NUM | Integer | `2` | Number of sensors involved in each query statement. | +| QUERY_AGGREGATE_FUN | Character | `count` | Aggregate functions used in queries (`COUNT`, `AVG`, `SUM`, etc.). | +| STEP_SIZE | Integer | `1` | Time interval step for time filter conditions. | +| QUERY_INTERVAL | Integer | `250000` | Time interval between query start and end times. | +| QUERY_LOWER_VALUE | Integer | `-5` | Threshold for conditional queries (`WHERE value > QUERY_LOWER_VALUE`). | +| GROUP_BY_TIME_UNIT | Integer | `20000` | The size of the group in the `GROUP BY` statement | +| LOOP | Integer | `10` | Total number of query operations: Each type of operation will be divided according to the proportion defined by `OPERATION_PROPORTION` | +| OPERATION_PROPORTION | Character | `0:0:0:0:0:0:0:0:0:0:1` | Ratio of operation types (`write:Q1:Q2:...:Q10`). | -Table 1-5 Query scene setup parameters +### 3.6 **Query Types and Example SQL** -| Parameter Name | Type | Example | Description | -| -------------------- | ------- | --------------------- | ------------------------------------------------------------ | -| QUERY_DEVICE_NUM | Integer | 2 | The number of devices involved in the query in each query statement. | -| QUERY_SENSOR_NUM | Integer | 2 | The number of sensors involved in the query in each query statement. | -| QUERY_AGGREGATE_FUN | String | count | Aggregate functions used in aggregate queries, such as count, avg, sum, max_time, etc. | -| STEP_SIZE | Integer | 1 | The change step of the starting time point of the time filter condition, if set to 0, the time filter condition of each query is the same, unit: POINT_STEP. | -| QUERY_INTERVAL | Integer | 250000 | The time interval between the start time and the end time in the start and end time query, and the time interval in Group By. | -| QUERY_LOWER_VALUE | Integer | -5 | Parameters for conditional query clauses, where xxx > QUERY_LOWER_VALUE. | -| GROUP_BY_TIME_UNIT | Integer | 20000 | The size of the group in the Group By statement. | -| LOOP | Integer | 10 | Total number of operations. Each type of operation will be divided according to the ratio defined by OPERATION_PROPORTION. | -| OPERATION_PROPORTION | String | 0:0:0:0:0:0:0:0:0:0:1 | Write:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | +| Number | Query Type | IoTDB Sample SQL | +| :----- | :----------------------------- | :----------------------------------------------------------- | +| Q1 | Precise Point Query | `select v1 from root.db.d1 where time = ?` | +| Q2 | Time Range Query | `select v1 from root.db.d1 where time > ? and time < ?` | +| Q3 | Time Range with Value Filter | `select v1 from root.db.d1 where time > ? and time < ? and v1 > ?` | +| Q4 | Time Range Aggregation Query | `select count(v1) from root.db.d1 where and time > ? and time < ?` | +| Q5 | Full-Time Range with Filtering | `select count(v1) from root.db.d1 where v1 > ?` | +| Q6 | Range Aggregation with Filter | `select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ?` | +| Q7 | Time Grouping Aggregation | `select count(v1) from root.db.d1 group by ([?, ?), ?, ?)` | +| Q8 | Latest Point Query | `select last v1 from root.db.d1` | +| Q9 | Descending Range Query | `select v1 from root.sg.d1 where time > ? and time < ? order by time desc` | +| Q10 | Descending Range with Filter | `select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc` | -Table 1-6 Query types and example SQL +### 3.7 **Test process and test result persistence** -| Id | Query Type | IoTDB Example SQL | -| ---- | ---------------------------------------------------- | ------------------------------------------------------------ | -| Q1 | exact point query | select v1 from root.db.d1 where time = ? | -| Q2 | time range query | select v1 from root.db.d1 where time > ? and time < ? | -| Q3 | time range query with value filtering | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | -| Q4 | time range aggregation query | select count(v1) from root.db.d1 where and time > ? and time < ? | -| Q5 | full time range aggregate query with value filtering | select count(v1) from root.db.d1 where v1 > ? | -| Q6 | time range aggregation query with value filtering | select count(v1) from root.db.d1 where v1 > ? and time > ? and time < ? | -| Q7 | time grouping aggregation query | select count(v1) from root.db.d1 group by ([?, ?), ?, ?) | -| Q8 | latest point query | select last v1 from root.db.d1 | -| Q9 | reverse order time range query | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | -| Q10 | reverse order time range query with value filtering | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +IoT-benchmark currently supports persisting the test process and test results through configuration parameters. -According to the configuration parameters in Table 1-5, the test scenario can be described as follows: Execute 10 reverse order time range queries with value filtering for 2 devices and 2 sensors from the time series database under test. The SQL statement is: `select s_0,s_31from data where time >2022-10-30T00:00:00+08:00 and time < 2022-10-30T00:04:10+08:00 and s_0 > -5 and device in d_21,d_46 order by time desc`. +| **Parameter** | **Type** | **Example** | D**escription** | +| :-------------------- | :------- | :---------- | :----------------------------------------------------------- | +| TEST_DATA_PERSISTENCE | String | `None` | Specifies the result persistence method. Options: `None`, `IoTDB`, `MySQL`, `CSV`. | +| RECORD_SPLIT | Boolean | `true` | Whether to split results into multiple records. (Not supported by IoTDB currently.) | +| RECORD_SPLIT_MAX_LINE | Integer | `10000000` | Maximum number of rows per record (10 million rows per database table or CSV file). | +| TEST_DATA_STORE_IP | String | `127.0.0.1` | IP address of the database for result storage. | +| TEST_DATA_STORE_PORT | Integer | `6667` | Port number of the output database. | +| TEST_DATA_STORE_DB | String | `result` | Name of the output database. | +| TEST_DATA_STORE_USER | String | `root` | Username for accessing the output database. | +| TEST_DATA_STORE_PW | String | `root` | Password for accessing the output database. | -### 2.5 Persistence of Test Process and Test Results +**Result Persistence Details** -IoT-benchmark currently supports persisting the test process and test results to IoTDB, MySQL, and CSV through the configuration parameter "TEST_DATA_PERSISTENCE"; writing to MySQL and CSV can define the upper limit of the number of rows in the sub-database and sub-table, such as "RECORD_SPLIT=true, RECORD_SPLIT_MAX_LINE=10000000" means that each database table or CSV file is divided and stored according to the total number of 10 million rows; if the records are recorded to MySQL or IoTDB, database link information needs to be provided, including "TEST_DATA_STORE_IP" the IP address of the database, "TEST_DATA_STORE_PORT" the port number of the database, "TEST_DATA_STORE_DB" the name of the database, "TEST_DATA_STORE_USER" the database user name, and "TEST_DATA_STORE_PW" the database user password. +- **CSV Mode:** If `TEST_DATA_PERSISTENCE` is set to `CSV`, a `data` folder is generated in the IoT-benchmark root directory during and after test execution. This folder contains: + - `csv` folder: Records the test process. + - `csvOutput` folder: Stores the test results. +- **MySQL Mode:** If `TEST_DATA_PERSISTENCE` is set to `MySQL`, IoT-benchmark creates the following tables in the specified MySQL database: + - **Test Process Table:** + 1. Created before the test starts. + 2. Named as: `testWithDefaultPath___`. + - **Configuration Table:** + 1. Named `CONFIG`. + 2. Stores the test configuration. + 3. Created if it does not exist. + - **Final Result Table:** + 1. Named `FINAL_RESULT`. + 2. Stores the test results after test completion. + 3. Created if it does not exist. -If we set "TEST_DATA_PERSISTENCE=CSV", we can see the newly generated data folder under the IoT-benchmark root directory during and after the test execution, which contains the csv folder to record the test process; the csvOutput folder to record the test results . If we set "TEST_DATA_PERSISTENCE=MySQL", it will create a data table named "testWithDefaultPath_tested database name_remarks_test start time" in the specified MySQL database before the test starts to record the test process; it will record the test process in the "CONFIG" data table (create the table if it does not exist), write the configuration information of this test; when the test is completed, the result of this test will be written in the data table named "FINAL_RESULT" (create the table if it does not exist). +### 3.8 Automation Script -## 3. Use Case +#### One-Click Script Startup + +The `cli-benchmark.sh` script allows one-click startup of IoTDB, IoTDB Benchmark monitoring, and IoTDB Benchmark testing. However, please note that this script will clear all existing data in IoTDB during startup, so use it with caution. + +**Steps to Run:** + +1. Edit the `IOTDB_HOME` parameter in `cli-benchmark.sh` to the local IoTDB directory. +2. Start the test by running the following command: + +```Bash +> ./cli-benchmark.sh +``` + +1. After the test completes: + 1. Check test-related logs in the `logs` folder. + 2. Check monitoring-related logs in the `server-logs` folder. + +#### Automatic Execution of Multiple Tests + +Single tests are often insufficient without comparative results. Therefore, IoT-benchmark provides an interface for executing multiple tests in sequence. + +1. **Routine Configuration:** Each line in the `routine` file specifies the parameters that change for each test. For example: + + ```Plain + LOOP=10 DEVICE_NUMBER=100 TEST + LOOP=20 DEVICE_NUMBER=50 TEST + LOOP=50 DEVICE_NUMBER=20 TEST + ``` + +In this example, three tests will run sequentially with `LOOP` values of 10, 20, and 50. + +Then the test process with 3 LOOP parameters of 10, 20, and 50 is executed in sequence. + +**Important Notes:** + +- Multiple parameters can be changed in each test using the format: + + ```Bash + LOOP=20 DEVICE_NUMBER=10 TEST + ``` + +- Avoid unnecessary spaces. + +- The `TEST` keyword marks the start of a new test. + +- Changed parameters persist across subsequent tests unless explicitly reset. + +2. **Start the Test:** After configuring the `routine` file, start multi-test execution using the following command + + ```Bash + > ./rep-benchmark.sh + ``` + +Test results will be displayed in the terminal. + +**Important Notes:** + +- Closing the terminal or losing the client connection will terminate the test process. + +- To run the test as a background daemon, execute: + + ```Bash + > ./rep-benchmark.sh > /dev/null 2>&1 & + ``` + +- To monitor progress, check the logs: + + ```Bash + > cd ./logs + > tail -f log_info.log + ``` + +## 4. Use Case We take the application of CRRC Qingdao Sifang Vehicle Research Institute Co., Ltd. as an example, and refer to the scene described in "Apache IoTDB in Intelligent Operation and Maintenance Platform Storage" for practical operation instructions. Test objective: Simulate the actual needs of switching time series databases in the scene of CRRC Qingdao Sifang Institute, and compare the performance of the expected IoTDB and KairosDB used by the original system. -Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 2-1. The specific usage of each virtual machine is shown in Table 2-2. +Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 4-1. The specific usage of each virtual machine is shown in Table 4-2. -Table 2-1 Virtual machine configuration information +Table 4-1 Virtual machine configuration information | Hardware Configuration Information | Value | | ---------------------------------- | ------- | @@ -213,7 +359,7 @@ Table 2-1 Virtual machine configuration information | hard disk | 200G | | network | Gigabit | -Table 2-2 Virtual machine usage +Table 4-2 Virtual machine usage | IP | Usage | | ---------- | ------------- | @@ -222,11 +368,11 @@ Table 2-2 Virtual machine usage | 172.21.4.4 | KaiosDB | | 172.21.4.5 | MySQL | -### 3.1 Write Test +### 4.1 Write Test -Scenario description: Create 100 clients to simulate 100 trains, each train has 3000 sensors, the data type is DOUBLE, the data time interval is 500ms (2Hz), and they are sent sequentially. Referring to the above requirements, we need to modify the IoT-benchmark configuration parameters as listed in Table 2-3. +Scenario description: Create 100 clients to simulate 100 trains, each train has 3000 sensors, the data type is DOUBLE, the data time interval is 500ms (2Hz), and they are sent sequentially. Referring to the above requirements, we need to modify the IoT-benchmark configuration parameters as listed in Table 4-3. -Table 2-3 Configuration parameter information +Table 4-3 Configuration parameter information | Parameter Name | IoTDB Value | KairosDB Value | | -------------------------- | --------------------------- | -------------- | @@ -253,40 +399,40 @@ Table 2-3 Configuration parameter information | TEST_DATA_STORE_PW | admin | | | REMARK | demo | | -First, start the tested time series databases Apache-IoTDB and KairosDB on 172.21.4.3 and 172.21.4.4 respectively, and then start server resource monitoring through the ser-benchamrk\.sh script on 172.21.4.2, 172.21.4.3 and 172.21.4.4 (Figure 2-1). Then modify the conf/config.properties files in the iotdb-0.13-0.0.1 and kairosdb-0.0.1 folders in 172.21.4.2 according to Table 2-3 to meet the test requirements. Use benchmark\.sh to start the writing test of Apache-IoTDB and KairosDB successively. +First, start the tested time series databases Apache-IoTDB and KairosDB on 172.21.4.3 and 172.21.4.4 respectively, and then start server resource monitoring through the ser-benchamrk\.sh script on 172.21.4.2, 172.21.4.3 and 172.21.4.4 (Figure 4-1). Then modify the conf/config.properties files in the iotdb-0.13-0.0.1 and kairosdb-0.0.1 folders in 172.21.4.2 according to Table 4-3 to meet the test requirements. Use benchmark\.sh to start the writing test of Apache-IoTDB and KairosDB successively. ![img](/img/bm5.png) -Figure 2-1 Server monitoring tasks +Figure 4-1 Server monitoring tasks -For example, if we first start the test on KairosDB, IoT-benchmark will create a CONFIG data table in the MySQL database to store the configuration information of this test (Figure 2-2), and there will be a log output of the current test progress during the test execution (Figure 2-3) . When the test is completed, the test result will be output (Figure 2-3), and the result will be written into the FINAL_RESULT data table (Figure 2-4). +For example, if we first start the test on KairosDB, IoT-benchmark will create a CONFIG data table in the MySQL database to store the configuration information of this test (Figure 4-2), and there will be a log output of the current test progress during the test execution (Figure 4-3) . When the test is completed, the test result will be output (Figure 4-3), and the result will be written into the FINAL_RESULT data table (Figure 4-4). ![img](/img/bm6.png) -Figure 2-2 Test configuration information table +Figure 4-2 Test configuration information table ![img](/img/bm7.png) ![img](/img/bm8.png) ![img](/img/bm9.png) ![img](/img/bm10.png) -Figure 2-3 Test progress and results +Figure 4-3 Test progress and results ![img](/img/bm11.png) -Figure 2-4 Test result table +Figure 4-4 Test result table Afterwards, we will start the test on Apache-IoTDB. The same IoT-benchmark will write the test configuration information in the MySQL database CONFIG data table. During the test execution, there will be a log to output the current test progress. When the test is completed, the test result will be output, and the result will be written into the FINAL_RESULT data table. -According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 2-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc. +According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 4-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc. ![img](/img/bm12.png) -Figure 2-5 Disk usage +Figure 4-5 Disk usage So what is the resource usage of each server during the test? What is the specific performance of each write operation? At this time, we can visualize the data in the server monitoring table and test process recording table by installing and using Tableau. The use of Tableau will not be introduced in this article. After connecting to the data table for test data persistence, the specific results are as follows (taking Apache-IoTDB as an example): @@ -295,13 +441,13 @@ So what is the resource usage of each server during the test? What is the specif ![img](/img/bm14.png) -Figure 2-6 Visualization of testing process in Tableau +Figure 4-6 Visualization of testing process in Tableau -### 3.2 Query Test +### 4.2 Query Test Scenario description: In the writing test scenario, 10 clients are simulated to perform all types of query tasks on the data stored in the time series database Apache-IoTDB. The configuration is as follows. -Table 2-4 Configuration parameter information +Table 4-4 Configuration parameter information | Parameter Name | Example | | -------------------- | --------------------- | @@ -320,13 +466,13 @@ Results: ![img](/img/bm15.png) -Figure 2-7 Query test results +Figure 4-7 Query test results -### 3.3 Description of Other Parameters +### 4.3 Description of Other Parameters In the previous chapters, the write performance comparison between Apache-IoTDB and KairosDB was performed, but if the user wants to perform a simulated real write rate test, how to configure it? How to control if the test time is too long? Are there any regularities in the generated simulated data? If the IoT-Benchmark server configuration is low, can multiple machines be used to simulate pressure output? -Table 2-5 Configuration parameter information +Table 4-5 Configuration parameter information | Scenario | Parameter | Value | Notes | | ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | @@ -341,4 +487,4 @@ Table 2-5 Configuration parameter information | STRING_LENGTH | 10 | String length | | | DOUBLE_LENGTH | 2 | Decimal places | | | Three machines simulate data writing of 300 devices | BENCHMARK_CLUSTER | true | Enable multi-benchmark mode | -| BENCHMARK_INDEX | 0, 1, 3 | Take the writing parameters in the [write test](./Benchmark.md#write-test) as an example: No. 0 is responsible for writing data of device numbers 0-99; No. 1 is responsible for writing data of device numbers 100-199; No. 2 is responsible for writing data of device numbers 200-299. | | \ No newline at end of file +| BENCHMARK_INDEX | 0, 1, 3 | Take the writing parameters in the [write test](./Benchmark.md#_4-1-write-test) as an example: No. 0 is responsible for writing data of device numbers 0-99; No. 1 is responsible for writing data of device numbers 100-199; No. 2 is responsible for writing data of device numbers 200-299. | | \ No newline at end of file diff --git a/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md b/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md index a0323e753..8d7d578ef 100644 --- a/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/Master/Table/Tools-System/Benchmark.md @@ -40,16 +40,17 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： -| 数据库 | 版本 | 连接方式 | -| :-------------- | :-------------- | :------------------------------------------------------- | -| InfluxDB | v1.x v2.0 | SDK | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | -| IoTDB | v2.0 v1.x v0.13 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| 数据库 | 版本 | 连接方式 | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | jdbc | +| OpenTSDB | -- | Http Request | +| QuestDB | v6.0.7 | jdbc | +| TDengine | v2.2.0.2 | jdbc | +| VictoriaMetrics | v1.64.0 | Http Request | +| KairosDB | -- | Http Request | + 表1-1大数据测试基准对比 @@ -59,15 +60,15 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 1. Java 8 2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 1.0 +3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 ### 2.2 获取方式 - 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用户 Apache IoTDB 2.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.1)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 +- 源代码编译（可用 Apache IoTDB 2.0 的测试）： + - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 @@ -99,11 +100,11 @@ drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib | routine | | 多项测试配置文件 | | rep-benchmark.sh | | 多项测试启动脚本 | -表1-2文件和文件夹列表用途 +表2-1文件和文件夹列表用途 ### 2.4 执行测试 -1. 按照测试需求修改配置文件，主要参数介绍见 1.2 节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 1.0，则需要修改 DB_SWITCH=IoTDB-100-SESSION_BY_TABLET** +1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** 2. 启动被测时间序列数据库 3. 通过运行 4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 @@ -135,7 +136,6 @@ drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib - 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 - sql_dialect等于table时，要满足：device数为table数的整数倍，table数为database数的整数倍 -- sql_dialect等于tree时，要满足：device数量 >= database数量 - 表模型模式下，调整参数如下： | **参数名称** | **类型** | **示例** | **系统描述** | diff --git a/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md b/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md index d3c5026dc..6a4e9095f 100644 --- a/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/Master/Tree/Tools-System/Benchmark.md @@ -21,7 +21,7 @@ # 测试工具 -## 1. 概述 +## 1. 基本概述 IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测试工具，由清华大学软件学院研发并开源。它使用方便，支持多种写入以及查询方式，支持存储测试信息和结果以供进一步查询或分析，支持与 Tableau 集成以可视化测试结果。 @@ -38,70 +38,78 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 图1-2 - 目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： -| 数据库 | 版本 | 连接方式 | -| --------------- | ------- | -------------------------------------------------------- | -| InfluxDB | v1.x
v2.0 | SDK | | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | -| IoTDB | v1.x
v0.13 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| 数据库 | 版本 | 连接方式 | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | jdbc | +| OpenTSDB | -- | Http Request | +| QuestDB | v6.0.7 | jdbc | +| TDengine | v2.2.0.2 | jdbc | +| VictoriaMetrics | v1.64.0 | Http Request | +| KairosDB | -- | Http Request | + 表1-1大数据测试基准对比 -### 1.1 软件安装与环境搭建 +## 2. 安装运行 -#### IoT Benchmark 运行的前置条件 +### 2.1 前置条件 1. Java 8 2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 1.0 - - +3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 -#### IoT Benchmark 的获取方式 -- **获取二进制包**：进入https://github.com/thulab/iot-benchmark/releases 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用户 Apache IoTDB 1.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入官网 https://github.com/apache/iotdb/tree/rel/1.0 下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - - 第二步（编译 IoTDB Benchmark 测试包）：进入官网 https://github.com/thulab/iot-benchmark 下载源码，在根目录下运行 mvn clean package install -pl iotdb-1.0 -am -DskipTests 编译测试 Apache IoTDB 1.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-1.0/target/iotdb-1.0-0.0.1/iotdb-1.0-0.0.1 +### 2.2 获取方式 +- 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 +- 源代码编译（可用 Apache IoTDB 2.0 的测试）： + - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 + - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 -#### IoT Benchmark 的测试包结构 -测试包的目录结构如下图1-3所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见表1-2所示。 +### 2.3 测试包结构 -![](/img/bm3.png) +测试包的目录结构如下所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见下表所示。 -图1-3文件和文件夹列表 +```Shell +-rw-r--r--. 1 root root 2881 1月 10 01:36 benchmark.bat +-rwxr-xr-x. 1 root root 314 1月 10 01:36 benchmark.sh +drwxr-xr-x. 2 root root 24 1月 10 01:36 bin +-rwxr-xr-x. 1 root root 1140 1月 10 01:36 cli-benchmark.sh +drwxr-xr-x. 2 root root 107 1月 10 01:36 conf +drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib +-rw-r--r--. 1 root root 11357 1月 10 01:36 LICENSE +-rwxr-xr-x. 1 root root 939 1月 10 01:36 rep-benchmark.sh +-rw-r--r--. 1 root root 14 1月 10 01:36 routine +``` | 名称 | 子文件 | 用途 | -| ---------------- | ----------------- | ------------------------- | +| :--------------- | :---------------- | :------------------------ | | benchmark.bat | - | Windows环境运行启动脚本 | -| benchmark\.sh | - | Linux/Mac环境运行启动脚本 | +| benchmark.sh | - | Linux/Mac环境运行启动脚本 | +| bin | startup.sh | 初始化脚本文件夹 | | conf | config.properties | 测试场景配置文件 | -| logback.xml | 日志输出配置文件 | | | lib | - | 依赖库文件 | | LICENSE | - | 许可文件 | -| bin | startup\.sh | 初始化脚本文件夹 | -| ser-benchmark\.sh | - | 监控模式启动脚本 | +| cli-benchmark.sh | - | 一键化启动脚本 | +| routine | | 多项测试配置文件 | +| rep-benchmark.sh | | 多项测试启动脚本 | -表1-2文件和文件夹列表用途 +表2-1文件和文件夹列表用途 -#### IoT Benchmark 执行测试 +### 2.4 执行测试 -1. 按照测试需求修改配置文件，主要参数介绍见 1.2 节，对应配置文件为conf/config.properties，**比如测试Apache** **IoTDB 1.0，则需要修改 DB_SWITCH=IoTDB-100-SESSION_BY_TABLET** +1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** 2. 启动被测时间序列数据库 3. 通过运行 4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 -#### IoT Benchmark 结果说明 +### 2.5 结果说明 测试的所有日志文件被存放于 logs 文件夹下，测试的结果在测试完成后被存放到 data/csvOutput 文件夹下，例如测试后我们得到了如下的结果矩阵： @@ -119,55 +127,69 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 -### 1.2 主要参数介绍 +## 3. 主要参数 -本节重点解释说明了主要参数的用途和配置方法。 -#### 工作模式和操作比例 +### 3.1 IoTDB服务模型 -- 工作模式参数“BENCHMARK_WORK_MODE”可选项为“默认模式”和“服务器监控”；其中“服务器监控”模式可直接通过执行ser-benchmark.sh脚本启动，脚本会自动修改该参数。“默认模式”为常用测试模式，结合配置OPERATION_PROPORTION参数达到“纯写入”、“纯查询”和“读写混合”的测试操作比例定义 +参数`IoTDB_DIALECT_MODE`支持tree、table,默认值为tree。 -- 当运行ServerMode来执行被测时序数据库运行环境监控时IoT-benchmark依赖sysstat软件相关命令；如果需要持久化测试过程数据时选择MySQL或IoTDB，则需要安装该类数据库；ServerMode和CSV的记录模式只能在Linux系统中使用，记录测试过程中的相关系统信息。因此我们建议使用MacOs或Linux系统，本文以Linux（Centos7）系统为例，如果使用Windows系统，可以使用conf文件夹下的benchmark.bat脚本启动IoT-benchmark。 +- 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 +- sql_dialect等于tree时，要满足：device数量 >= database数量 - 表1-3测试模式 +### 3.2 工作模式 -| 模式名称 | BENCHMARK_WORK_MODE | 模式内容 | -| ---------------------- | ------------------- | ------------------------------------------------------------ | -| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | -| 服务器资源使用监控模式 | serverMODE | 服务器资源使用监控模式（该模式下运行通过ser-benchmark.sh脚本启动，无需手动配置该参数 | +工作模式参数“`BENCHMARK_WORK_MODE`”可选项有如下四种模式： -#### 服务器连接信息 +- 常用测试模式：结合配置`OPERATION_PROPORTION`参数达到“纯写入”、“纯查询”和“读写混合”的测试操作。 +- 生成数据模式：为了生成可以重复使用的数据集，iot-benchmark提供生成数据集的模式，生成数据集到FILE_PATH，以供后续使用正确性写入模式和正确性查询模式使用。 +- 单数据库正确性写入模式：为了验证数据集写入的正确性，您可以使用该模式写入生成数据模式中生成的数据集，目前该模式仅支持IoTDB v1.0 及更新的版本和InfluxDB v1.x。 +- 单数据库正确性查询模式：在运行这个模式之前需要先使用正确性写入模式写入数据到数据库。为了验证数据集写入的正确性，您可以使用该模式查询写入到数据库中的数据集，目前该模式仅支持IoTDB v1.0 和 InfluxDB v1.x。 -工作模式指定后，被测时序数据库的信息要如何告知IoT-benchmark呢？当前通过“DB_SWITCH”告知被测时序数据库类型；通过“HOST”告知被测时序数据库网络地址；通过“PORT”告知被测时序数据库网络端口；通过“USERNAME”告知被测时序数据库登录用户名；通过“PASSWORD”告知被测时序数据库登录用户的密码；通过“DB_NAME”告知被测时序数据库名称；通过“TOKEN”告知被测时序数据库连接认证Token（InfluxDB 2.0使用）； +| **模式名称** | **BENCHMARK_WORK_MODE** | **模式内容** | +| :--------------------- | :---------------------- | :------------------------------- | +| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | +| 生成数据模式 | generateDataMode | 生成Benchmark本身识别的数据 | +| 单数据库正确性写入模式 | verificationWriteMode | 需要配置 FILE_PATH 以及 DATA_SET | +| 单数据库正确性查询模式 | verificationQueryMode | 需要配置 FILE_PATH 以及 DATA_SET | -#### 写入场景构建参数 +### 3.3 服务器连接信息 -表1-4写入场景构建参数 +工作模式指定后，被测时序数据库的信息会通过如下参数告知IoT-benchmark -| 参数名称 | 类型 | 示例 | 系统描述 | -| -------------------------- | ------ | ------------------------- | ------------------------------------------------------------ | -| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | -| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | -| DEVICE_NUMBER | 整数 | 100 | 设备总数 | -| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数 | -| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | -| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | -| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | -| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | -| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | -| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | -| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | +| **参数名称** | **类型** | **示例** | **系统描述** | +| :----------- | :------- | :-------------------------- | :---------------------------------------------- | +| DB_SWITCH | 字符串 | IoTDB-200-SESSION_BY_TABLET | 被测时序数据库类型 | +| HOST | 字符串 | 127.0.0.1 | 被测时序数据库网络地址 | +| PORT | 整数 | 6667 | 被测时序数据库网络端口 | +| USERNAME | 字符串 | root | 被测时序数据库登录用户名 | +| PASSWORD | 字符串 | root | 被测时序数据库登录用户的密码 | +| DB_NAME | 字符串 | test | 被测时序数据库名称 | +| TOKEN | 字符串 | | 被测时序数据库连接认证Token（InfluxDB 2.0使用） | -按照表1-4配置参数启动可描述测试场景为：向被测时序数据库压力写入30000个（100个设备，每个设备300个传感器）时间序列2022年10月30日一天的顺序数据，总计25.92亿个数据点。其中每个设备的300个传感器数据类型分别为50个布尔、50个整数、50个长整数、50个浮点、50个双精度、50个字符。如果我们将表格中IS_OUT_OF_ORDER的值改为true，那么他表示的场景为：向被测时序数据库压力写入30000个时间序列2022年10月30日一天的数据，其中存在30%的乱序数据（到达时序数据库时间晚于其他生成时间晚于自身的数据点）。 +### 3.4 写入场景 -#### 查询场景构建参数 +| **参数名称** | **类型** | **示例** | **系统描述** | +| :------------------------- | :------- | :------------------------ | :----------------------------------------------------------- | +| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | +| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | +| DEVICE_NUMBER | 整数 | 100 | 设备总数 | +| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数； **如果使用 IoTDB 表模型，则控制属性列数量** | +| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | +| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | +| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | +| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | +| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | +| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | +| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | +| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | +| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | -表1-5查询场景构建参数 + +### 3.5 查询场景 | 参数名称 | 类型 | 示例 | 系统描述 | -| -------------------- | ---- | --------------------- | ------------------------------------------------------------ | +| :------------------- | :--- | :-------------------- | :----------------------------------------------------------- | | QUERY_DEVICE_NUM | 整数 | 2 | 每条查询语句中查询涉及到的设备数量 | | QUERY_SENSOR_NUM | 整数 | 2 | 每条查询语句中查询涉及到的传感器数量 | | QUERY_AGGREGATE_FUN | 字符 | count | 在聚集查询中使用的聚集函数，比如count、avg、sum、max_time等 | @@ -178,10 +200,11 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 | LOOP | 整数 | 10 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | | OPERATION_PROPORTION | 字符 | 0:0:0:0:0:0:0:0:0:0:1 | 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | -表1-6查询类型及示例 SQL + +### 3.6 操作比例 | 编号 | 查询类型 | IoTDB 示例 SQL | -| ---- | ---------------------------- | ------------------------------------------------------------ | +| :--- | :--------------------------- | :----------------------------------------------------------- | | Q1 | 精确点查询 | select v1 from root.db.d1 where time = ? | | Q2 | 时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? | | Q3 | 带值过滤的时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | @@ -193,27 +216,96 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 | Q9 | 倒序范围查询 | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | | Q10 | 倒序带值过滤的范围查询 | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +### 3.7 测试过程和测试结果持久化 + +IoT-benchmark目前支持通过配置参数将测试过程和测试结果持久化: + +| **参数名称** | **类型** | **示例** | **系统描述** | +| :-------------------- | :------- | :-------- | :----------------------------------------------------------- | +| TEST_DATA_PERSISTENCE | 字符串 | None | 结果持久化选择，支持None，IoTDB，MySQL和CSV | +| RECORD_SPLIT | 布尔 | true | 是否将结果划分后输出到多个记录, IoTDB 暂时不支持 | +| RECORD_SPLIT_MAX_LINE | 整数 | 10000000 | 记录行数的上限（每个数据库表或CSV文件按照总行数为1千万切分存放） | +| TEST_DATA_STORE_IP | 字符串 | 127.0.0.1 | 输出数据库的IP地址 | +| TEST_DATA_STORE_PORT | 整数 | 6667 | 输出数据库的端口号 | +| TEST_DATA_STORE_DB | 字符串 | result | 输出数据库的名称 | +| TEST_DATA_STORE_USER | 字符串 | root | 输出数据库的用户名 | +| TEST_DATA_STORE_PW | 字符串 | root | 输出数据库的用户密码 | + +- 如果我们设置“`TEST_DATA_PERSISTENCE=CSV`”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的`data`文件夹，其下包含`csv`文件夹记录测试过程；`csvOutput`文件夹记录测试结果。 +- 如果我们设置“`TEST_DATA_PERSISTENCE=MySQL`”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 + +### 3.8 自动化脚本 + +#### 一键化启动脚本 + +您可以通过`cli-benchmark.sh`脚本一键化启动IoTDB、监控的IoTDB Benchmark和测试的IoTDB Benchmark，但需要注意该脚本启动时会清理IoTDB中的**所有数据**，请谨慎使用。 + +首先，您需要修改`cli-benchmark.sh`中的`IOTDB_HOME`参数为您本地的IoTDB所在的文件夹。 + +然后您可以使用脚本启动测试 + +```Bash +> ./cli-benchmark.sh +``` + +测试完成后您可以在`logs`文件夹中查看测试相关日志，在`server-logs`文件夹中查看监控相关日志。 + +#### 自动执行多项测试 + +通常，除非与其他测试结果进行比较，否则单个测试是没有意义的。因此，我们提供了一个接口来通过一次启动执行多个测试。 + +- 配置 routine + +这个文件的每一行应该是每个测试过程会改变的参数（否则就变成复制测试）。例如，"例程"文件是： + +```Plain +LOOP=10 DEVICE_NUMBER=100 TEST +LOOP=20 DEVICE_NUMBER=50 TEST +LOOP=50 DEVICE_NUMBER=20 TEST +``` + +然后依次执行3个LOOP参数分别为10、20、50的测试过程。 + +> 注意： +> +> 您可以使用“LOOP=20 DEVICE_NUMBER=10 TEST”等格式更改每个测试中的多个参数，不允许使用不必要的空间。 关键字"TEST"意味着新的测试开始。如果您更改不同的参数，更改后的参数将保留在下一次测试中。 + +- 开始测试 + +配置文件routine后，您可以通过启动脚本启动多测试任务： +```Bash +> ./rep-benchmark.sh +``` +然后测试信息将显示在终端中。 +> 注意： +> +> 如果您关闭终端或失去与客户端机器的连接，测试过程将终止。 如果输出传输到终端，则与任何其他情况相同。 -按照表1-5配置参数启动可描述测试场景为：从被测时序数据库执行10次2个设备2个传感器的倒序带值过滤的范围查询，SQL语句为：select s_0,s_31from data where time >2022-10-30T00:00:00+08:00 and time < 2022-10-30T00:04:10+08:00 and s_0 > -5 and device in d_21,d_46 order by time desc。 +使用此接口通常需要很长时间，您可能希望将测试过程作为守护程序执行。这样，您可以通过启动脚本将测试任务作为守护程序启动： -#### 测试过程和测试结果持久化 +```Bash +> ./rep-benchmark.sh > /dev/null 2>&1 & +``` -IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试过程和测试结果持久化到IoTDB、MySQL和CSV；其中写入到MySQL和CSV可以定义分库分表的行数上限，例如“RECORD_SPLIT=true、RECORD_SPLIT_MAX_LINE=10000000”表示每个数据库表或CSV文件按照总行数为1千万切分存放；如果记录到MySQL或IoTDB需要提供数据库链接信息，分别包括“TEST_DATA_STORE_IP”数据库的IP地址、“TEST_DATA_STORE_PORT”数据库的端口号、“TEST_DATA_STORE_DB”数据库的名称、“TEST_DATA_STORE_USER”数据库用户名、“TEST_DATA_STORE_PW”数据库用户密码。 +在这种情况下，如果您想知道发生了什么，可以通过以下命令查看日志信息： -如果我们设置“TEST_DATA_PERSISTENCE=CSV”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的data文件夹，其下包含csv文件夹记录测试过程；csvOutput文件夹记录测试结果。如果我们设置“TEST_DATA_PERSISTENCE=MySQL”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 +```Bash +> cd ./logs +> tail -f log_info.log +``` -## 2. 实际案例 +## 4. 实际案例 我们以中车青岛四方车辆研究所有限公司应用为例，参考《ＡｐａｃｈｅＩｏＴＤＢ在智能运维平台存储中的应用》中描述的场景进行实际操作说明。 测试目标：模拟中车青岛四方所场景因切换时间序列数据库实际需求，对比预期使用的IoTDB和原有系统使用的KairosDB性能。 -测试环境：为了保证在实验过程中消除其他无关服务与进程对数据库性能的影响，以及不同数据库之间的相互影响，本实验中的本地数据库均部署并运行在资源配置相同的多个独立的虚拟机上。因此，本实验搭建了 4 台 Linux( CentOS7 /x86) 虚拟机，并分别在上面部署了IoT-benchmark、 IoTDB数据库、KairosDB数据库、MySQL数据库。每一台虚拟机的具体资源配置如表2-1所示。每一台虚拟机的具体用途如表2-2所示。 +测试环境：为了保证在实验过程中消除其他无关服务与进程对数据库性能的影响，以及不同数据库之间的相互影响，本实验中的本地数据库均部署并运行在资源配置相同的多个独立的虚拟机上。因此，本实验搭建了 4 台 Linux( CentOS7 /x86) 虚拟机，并分别在上面部署了IoT-benchmark、 IoTDB数据库、KairosDB数据库、MySQL数据库。每一台虚拟机的具体资源配置如表4-1所示。每一台虚拟机的具体用途如表4-2所示。 -表2-1虚拟机配置信息 +表4-1虚拟机配置信息 | 硬件配置信息 | 系统描述 | | ------------ | -------- | @@ -225,7 +317,7 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 -表2-2虚拟机用途 +表4-2虚拟机用途 | IP | 用途 | | ---------- | ------------- | @@ -234,11 +326,11 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 | 172.21.4.4 | KaiosDB | | 172.21.4.5 | MySQL | -### 2.1 写入测试 +### 4.1 写入测试 -场景描述：创建100个客户端来模拟100列车、每列车3000个传感器、数据类型为DOUBLE类型、数据时间间隔为500ms（2Hz）、顺序发送。参考以上需求我们需要修改IoT-benchmark配置参数如表2-3中所列。 +场景描述：创建100个客户端来模拟100列车、每列车3000个传感器、数据类型为DOUBLE类型、数据时间间隔为500ms（2Hz）、顺序发送。参考以上需求我们需要修改IoT-benchmark配置参数如表4-3中所列。 -表2-3配置参数信息 +表4-3配置参数信息 | 参数名称 | IoTDB值 | KairosDB值 | | -------------------------- | --------------------------- | ---------- | @@ -265,51 +357,51 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 | TEST_DATA_STORE_PW | admin | | | REMARK | demo | | -首先在172.21.4.3和172.21.4.4上分别启动被测时间序列数据库Apache-IoTDB和KairosDB，之后在172.21.4.2、172.21.4.3和172.21.4.4上通过ser-benchamrk.sh脚本启动服务器资源监控（图2-1）。然后按照表2-3在172.21.4.2分别修改iotdb-0.13-0.0.1和kairosdb-0.0.1文件夹内的conf/config.properties文件满足测试需求。先后使用benchmark.sh启动对Apache-IoTDB和KairosDB的写入测试。 +首先在172.21.4.3和172.21.4.4上分别启动被测时间序列数据库Apache-IoTDB和KairosDB，之后在172.21.4.2、172.21.4.3和172.21.4.4上通过ser-benchamrk.sh脚本启动服务器资源监控（图4-1）。然后按照表4-3在172.21.4.2分别修改iotdb-0.13-0.0.1和kairosdb-0.0.1文件夹内的conf/config.properties文件满足测试需求。先后使用benchmark.sh启动对Apache-IoTDB和KairosDB的写入测试。 ![img](/img/bm5.png) -图2-1服务器监控任务 +图4-1服务器监控任务 -​ 例如我们首先启动对KairosDB的测试，IoT-benchmark会在MySQL数据库中创建CONFIG数据表存放本次测试配置信息（图2-2），测试执行中会有日志输出当前测试进度（图2-3）。测试完成时会输出本次测试结果（图2-3），同时将结果写入FINAL_RESULT数据表中（图2-4）。 +​ 例如我们首先启动对KairosDB的测试，IoT-benchmark会在MySQL数据库中创建CONFIG数据表存放本次测试配置信息（图4-2），测试执行中会有日志输出当前测试进度（图4-3）。测试完成时会输出本次测试结果（图4-3），同时将结果写入FINAL_RESULT数据表中（图4-4）。 ![](/img/bm6.png) -图2-2测试配置信息表 +图4-2测试配置信息表 ![](/img/bm7.png) ![](/img/bm8.png) ![](/img/bm9.png) ![](/img/bm10.png) -图2-3测试进度和结果 +图4-3测试进度和结果 ![](/img/bm11.png) -图2-4测试结果表 +图4-4测试结果表 之后我们再启动对Apache-IoTDB的测试，同样的IoT-benchmark会在MySQL数据库CONFIG数据表中写入本次测试配置信息，测试执行中会有日志输出当前测试进度。测试完成时会输出本次测试结果，同时将结果写入FINAL_RESULT数据表中。 -依照测试结果信息我们知道同样的配置写入Apache-IoTDB和KairosDB写入延时时间分别为：55.98ms和1324.45ms；写入吞吐分别为：5,125,600.86点/秒和224,819.01点/秒；测试分别执行了585.30秒和11777.99秒。并且KairosDB有写入失败出现，排查后发现是数据磁盘使用率已达到100%，无磁盘空间继续接收数据。而Apache-IoTDB无写入失败现象，全部数据写入完毕后占用磁盘空间仅为4.7G（如图2-5所示）；从写入吞吐和磁盘占用情况上看Apache-IoTDB均优于KairosDB。当然后续还有其他测试来从多方面观察和对比，比如查询性能、文件压缩比、数据安全性等。 +依照测试结果信息我们知道同样的配置写入Apache-IoTDB和KairosDB写入延时时间分别为：55.98ms和1324.45ms；写入吞吐分别为：5,125,600.86点/秒和224,819.01点/秒；测试分别执行了585.30秒和11777.99秒。并且KairosDB有写入失败出现，排查后发现是数据磁盘使用率已达到100%，无磁盘空间继续接收数据。而Apache-IoTDB无写入失败现象，全部数据写入完毕后占用磁盘空间仅为4.7G（如图4-5所示）；从写入吞吐和磁盘占用情况上看Apache-IoTDB均优于KairosDB。当然后续还有其他测试来从多方面观察和对比，比如查询性能、文件压缩比、数据安全性等。 ![](/img/bm12.png) -图2-5磁盘使用情况 +图4-5磁盘使用情况 那么测试过程中各个服务器资源使用情况如何呢？每个写操作具体的表现如何呢？这个时候我们就可以通过安装和使用Tableau来可视化服务器监控表和测试过程记录表内的数据了。Tableau的使用本文不展开介绍，通过它连接测试数据持久化的数据表后具体结果下如图（以Apache-IoTDB为例）： ![](/img/bm13.png) ![](/img/bm14.png) -图2-6Tableau可视化测试过程 +图4-6Tableau可视化测试过程 -### 2.2 询测试 +### 4.2 查询测试 场景描述：在写入测试场景下模拟10个客户端对时序数据库Apache-IoTDB内存放的数据进行全类型查询任务。配置如下： -表2-4配置参数信息 +表4-4配置参数信息 | 参数名称 | 示例 | | -------------------- | --------------------- | @@ -328,25 +420,25 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 ![](/img/bm15.png) -图2-7查询测试结果 +图4-7查询测试结果 -### 2.3 其他参数说明 +### 4.3 其他参数说明 之前章节中针对Apache-IoTDB和KairosDB进行写入性能对比，但是用户如果要执行模拟真实写入速率测试该如何配置？测试时间过长该如何控制呢？生成的模拟数据有哪些规律吗？如果IoT-Benchmark服务器配置较低，可以使用多台机器模拟压力输出吗？ -表2-5配置参数信息 - -| 场景 | 参数 | 值 | 说明 | -| ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | --------------------------------- | -| 模拟真实写入速率 | OP_INTERVAL | -1 | 也可输入整数控制操作间隔 | -| 指定测试时长（1小时） | TEST_MAX_TIME | 3600000 | 单位 ms；需要LOOP执行时间大于该值 | -| 定义模拟数据规律：支持全部数据类型，数量平均分类；支持五种数据分布，数量平均分布；字符串长度为10；小数位数为2 | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | 数据类型分布比率 | -| LINE_RATIO | 1 | 线性 | | -| SIN_RATIO | 1 | 傅里叶函数 | | -| SQUARE_RATIO | 1 | 方波 | | -| RANDOM_RATIO | 1 | 随机数 | | -| CONSTANT_RATIO | 1 | 常数 | | -| STRING_LENGTH | 10 | 字符串长度 | | -| DOUBLE_LENGTH | 2 | 小数位数 | | -| 三台机器模拟300台设备数据写入 | BENCHMARK_CLUSTER | true | 开启多benchmark模式 | -| BENCHMARK_INDEX | 0、1、3 | 以[写入测试](./Benchmark.md#写入测试)写入参数为例：0号负责设备编号0-99数据写入；1号负责设备编号100-199数据写入；2号负责设备编号200-299数据写入； | | \ No newline at end of file +表4-5配置参数信息 + +| 场景 | 参数 | 值 | 说明 | +| ------------------------------------------------------------ | -------------------------- |---------------------------------------------------------------------------------------------------| --------------------------------- | +| 模拟真实写入速率 | OP_INTERVAL | -1 | 也可输入整数控制操作间隔 | +| 指定测试时长（1小时） | TEST_MAX_TIME | 3600000 | 单位 ms；需要LOOP执行时间大于该值 | +| 定义模拟数据规律：支持全部数据类型，数量平均分类；支持五种数据分布，数量平均分布；字符串长度为10；小数位数为2 | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | 数据类型分布比率 | +| LINE_RATIO | 1 | 线性 | | +| SIN_RATIO | 1 | 傅里叶函数 | | +| SQUARE_RATIO | 1 | 方波 | | +| RANDOM_RATIO | 1 | 随机数 | | +| CONSTANT_RATIO | 1 | 常数 | | +| STRING_LENGTH | 10 | 字符串长度 | | +| DOUBLE_LENGTH | 2 | 小数位数 | | +| 三台机器模拟300台设备数据写入 | BENCHMARK_CLUSTER | true | 开启多benchmark模式 | +| BENCHMARK_INDEX | 0、1、3 | 以[写入测试](./Benchmark.md#_4-1-写入测试)写入参数为例：0号负责设备编号0-99数据写入；1号负责设备编号100-199数据写入；2号负责设备编号200-299数据写入； | | \ No newline at end of file diff --git a/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md b/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md index a0323e753..8d7d578ef 100644 --- a/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/latest-Table/Tools-System/Benchmark.md @@ -40,16 +40,17 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： -| 数据库 | 版本 | 连接方式 | -| :-------------- | :-------------- | :------------------------------------------------------- | -| InfluxDB | v1.x v2.0 | SDK | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | -| IoTDB | v2.0 v1.x v0.13 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| 数据库 | 版本 | 连接方式 | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | jdbc | +| OpenTSDB | -- | Http Request | +| QuestDB | v6.0.7 | jdbc | +| TDengine | v2.2.0.2 | jdbc | +| VictoriaMetrics | v1.64.0 | Http Request | +| KairosDB | -- | Http Request | + 表1-1大数据测试基准对比 @@ -59,15 +60,15 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 1. Java 8 2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 1.0 +3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 ### 2.2 获取方式 - 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用户 Apache IoTDB 2.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.1)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 +- 源代码编译（可用 Apache IoTDB 2.0 的测试）： + - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 @@ -99,11 +100,11 @@ drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib | routine | | 多项测试配置文件 | | rep-benchmark.sh | | 多项测试启动脚本 | -表1-2文件和文件夹列表用途 +表2-1文件和文件夹列表用途 ### 2.4 执行测试 -1. 按照测试需求修改配置文件，主要参数介绍见 1.2 节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 1.0，则需要修改 DB_SWITCH=IoTDB-100-SESSION_BY_TABLET** +1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** 2. 启动被测时间序列数据库 3. 通过运行 4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 @@ -135,7 +136,6 @@ drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib - 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 - sql_dialect等于table时，要满足：device数为table数的整数倍，table数为database数的整数倍 -- sql_dialect等于tree时，要满足：device数量 >= database数量 - 表模型模式下，调整参数如下： | **参数名称** | **类型** | **示例** | **系统描述** | diff --git a/src/zh/UserGuide/latest/Tools-System/Benchmark.md b/src/zh/UserGuide/latest/Tools-System/Benchmark.md index d3c5026dc..6a4e9095f 100644 --- a/src/zh/UserGuide/latest/Tools-System/Benchmark.md +++ b/src/zh/UserGuide/latest/Tools-System/Benchmark.md @@ -21,7 +21,7 @@ # 测试工具 -## 1. 概述 +## 1. 基本概述 IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测试工具，由清华大学软件学院研发并开源。它使用方便，支持多种写入以及查询方式，支持存储测试信息和结果以供进一步查询或分析，支持与 Tableau 集成以可视化测试结果。 @@ -38,70 +38,78 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 图1-2 - 目前 IoT-benchmark 支持如下时间序列数据库、版本和连接方式： -| 数据库 | 版本 | 连接方式 | -| --------------- | ------- | -------------------------------------------------------- | -| InfluxDB | v1.x
v2.0 | SDK | | -| TimescaleDB | -- | jdbc | -| OpenTSDB | -- | Http Request | -| QuestDB | v6.0.7 | jdbc | -| TDengine | v2.2.0.2 | jdbc | -| VictoriaMetrics | v1.64.0 | Http Request | -| KairosDB | -- | Http Request | -| IoTDB | v1.x
v0.13 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| 数据库 | 版本 | 连接方式 | +| :-------------- |:-----------| :------------------------------------------------------- | +| IoTDB | v1.x v2.x | jdbc、sessionByTablet、sessionByRecord、sessionByRecords | +| InfluxDB | v1.x v2.x | SDK | +| TimescaleDB | -- | jdbc | +| OpenTSDB | -- | Http Request | +| QuestDB | v6.0.7 | jdbc | +| TDengine | v2.2.0.2 | jdbc | +| VictoriaMetrics | v1.64.0 | Http Request | +| KairosDB | -- | Http Request | + 表1-1大数据测试基准对比 -### 1.1 软件安装与环境搭建 +## 2. 安装运行 -#### IoT Benchmark 运行的前置条件 +### 2.1 前置条件 1. Java 8 2. Maven 3.6+ -3. 对应的合适版本的数据库，如 Apache IoTDB 1.0 - - +3. 对应的合适版本的数据库，如 Apache IoTDB 2.0 -#### IoT Benchmark 的获取方式 -- **获取二进制包**：进入https://github.com/thulab/iot-benchmark/releases 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 -- 源代码编译（可用户 Apache IoTDB 1.0 的测试）： - - 第一步（编译 IoTDB Session 最新包）：进入官网 https://github.com/apache/iotdb/tree/rel/1.0 下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 - - 第二步（编译 IoTDB Benchmark 测试包）：进入官网 https://github.com/thulab/iot-benchmark 下载源码，在根目录下运行 mvn clean package install -pl iotdb-1.0 -am -DskipTests 编译测试 Apache IoTDB 1.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-1.0/target/iotdb-1.0-0.0.1/iotdb-1.0-0.0.1 +### 2.2 获取方式 +- 获取二进制包：进入[这里](https://github.com/thulab/iot-benchmark/releases) 下载需要的安装包。下载下来为一个压缩文件，选择文件夹解压即可使用。 +- 源代码编译（可用 Apache IoTDB 2.0 的测试）： + - 第一步（编译 IoTDB Session 最新包）：进入[官网](https://github.com/apache/iotdb/tree/rc/2.0.5)下载 IoTDB 源码，在根目录下运行命令 mvn clean package install -pl session -am -DskipTests 编译 IoTDB Session 的最新包。 + - 第二步（编译 IoTDB Benchmark 测试包）：进入[官网](https://github.com/thulab/iot-benchmark)下载源码，在根目录下运行 mvn clean package install -pl iotdb-2.0 -am -DskipTests 编译测试 Apache IoTDB 2.0版本的测试包，测试包位置与根目录的相对路径为 ./iotdb-2.0/target/iotdb-2.0-0.0.1/iotdb-2.0-0.0.1 -#### IoT Benchmark 的测试包结构 -测试包的目录结构如下图1-3所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见表1-2所示。 +### 2.3 测试包结构 -![](/img/bm3.png) +测试包的目录结构如下所示。其中测试配置文件为conf/config.properties，测试启动脚本为benchmark\.sh (Linux & MacOS) 和 benchmark.bat (Windows)，详细文件用途见下表所示。 -图1-3文件和文件夹列表 +```Shell +-rw-r--r--. 1 root root 2881 1月 10 01:36 benchmark.bat +-rwxr-xr-x. 1 root root 314 1月 10 01:36 benchmark.sh +drwxr-xr-x. 2 root root 24 1月 10 01:36 bin +-rwxr-xr-x. 1 root root 1140 1月 10 01:36 cli-benchmark.sh +drwxr-xr-x. 2 root root 107 1月 10 01:36 conf +drwxr-xr-x. 2 root root 4096 1月 10 01:38 lib +-rw-r--r--. 1 root root 11357 1月 10 01:36 LICENSE +-rwxr-xr-x. 1 root root 939 1月 10 01:36 rep-benchmark.sh +-rw-r--r--. 1 root root 14 1月 10 01:36 routine +``` | 名称 | 子文件 | 用途 | -| ---------------- | ----------------- | ------------------------- | +| :--------------- | :---------------- | :------------------------ | | benchmark.bat | - | Windows环境运行启动脚本 | -| benchmark\.sh | - | Linux/Mac环境运行启动脚本 | +| benchmark.sh | - | Linux/Mac环境运行启动脚本 | +| bin | startup.sh | 初始化脚本文件夹 | | conf | config.properties | 测试场景配置文件 | -| logback.xml | 日志输出配置文件 | | | lib | - | 依赖库文件 | | LICENSE | - | 许可文件 | -| bin | startup\.sh | 初始化脚本文件夹 | -| ser-benchmark\.sh | - | 监控模式启动脚本 | +| cli-benchmark.sh | - | 一键化启动脚本 | +| routine | | 多项测试配置文件 | +| rep-benchmark.sh | | 多项测试启动脚本 | -表1-2文件和文件夹列表用途 +表2-1文件和文件夹列表用途 -#### IoT Benchmark 执行测试 +### 2.4 执行测试 -1. 按照测试需求修改配置文件，主要参数介绍见 1.2 节，对应配置文件为conf/config.properties，**比如测试Apache** **IoTDB 1.0，则需要修改 DB_SWITCH=IoTDB-100-SESSION_BY_TABLET** +1. 按照测试需求修改配置文件，主要参数介绍见第3节，对应配置文件为conf/config.properties，**比如测试Apache IoTDB 2.0，则需要修改 DB_SWITCH=IoTDB-200-SESSION_BY_TABLET** 2. 启动被测时间序列数据库 3. 通过运行 4. 启动IoT-benchmark执行测试。执行中观测被测时间序列数据库和IoT-benchmark状态，执行完毕后查看结果和分析测试过程。 -#### IoT Benchmark 结果说明 +### 2.5 结果说明 测试的所有日志文件被存放于 logs 文件夹下，测试的结果在测试完成后被存放到 data/csvOutput 文件夹下，例如测试后我们得到了如下的结果矩阵： @@ -119,55 +127,69 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 -### 1.2 主要参数介绍 +## 3. 主要参数 -本节重点解释说明了主要参数的用途和配置方法。 -#### 工作模式和操作比例 +### 3.1 IoTDB服务模型 -- 工作模式参数“BENCHMARK_WORK_MODE”可选项为“默认模式”和“服务器监控”；其中“服务器监控”模式可直接通过执行ser-benchmark.sh脚本启动，脚本会自动修改该参数。“默认模式”为常用测试模式，结合配置OPERATION_PROPORTION参数达到“纯写入”、“纯查询”和“读写混合”的测试操作比例定义 +参数`IoTDB_DIALECT_MODE`支持tree、table,默认值为tree。 -- 当运行ServerMode来执行被测时序数据库运行环境监控时IoT-benchmark依赖sysstat软件相关命令；如果需要持久化测试过程数据时选择MySQL或IoTDB，则需要安装该类数据库；ServerMode和CSV的记录模式只能在Linux系统中使用，记录测试过程中的相关系统信息。因此我们建议使用MacOs或Linux系统，本文以Linux（Centos7）系统为例，如果使用Windows系统，可以使用conf文件夹下的benchmark.bat脚本启动IoT-benchmark。 +- 当被测数据库为IoTDB-2.0及以上版本时需指定sql_dialect, 并且一个IoTDB只能指定一种。 +- sql_dialect等于tree时，要满足：device数量 >= database数量 - 表1-3测试模式 +### 3.2 工作模式 -| 模式名称 | BENCHMARK_WORK_MODE | 模式内容 | -| ---------------------- | ------------------- | ------------------------------------------------------------ | -| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | -| 服务器资源使用监控模式 | serverMODE | 服务器资源使用监控模式（该模式下运行通过ser-benchmark.sh脚本启动，无需手动配置该参数 | +工作模式参数“`BENCHMARK_WORK_MODE`”可选项有如下四种模式： -#### 服务器连接信息 +- 常用测试模式：结合配置`OPERATION_PROPORTION`参数达到“纯写入”、“纯查询”和“读写混合”的测试操作。 +- 生成数据模式：为了生成可以重复使用的数据集，iot-benchmark提供生成数据集的模式，生成数据集到FILE_PATH，以供后续使用正确性写入模式和正确性查询模式使用。 +- 单数据库正确性写入模式：为了验证数据集写入的正确性，您可以使用该模式写入生成数据模式中生成的数据集，目前该模式仅支持IoTDB v1.0 及更新的版本和InfluxDB v1.x。 +- 单数据库正确性查询模式：在运行这个模式之前需要先使用正确性写入模式写入数据到数据库。为了验证数据集写入的正确性，您可以使用该模式查询写入到数据库中的数据集，目前该模式仅支持IoTDB v1.0 和 InfluxDB v1.x。 -工作模式指定后，被测时序数据库的信息要如何告知IoT-benchmark呢？当前通过“DB_SWITCH”告知被测时序数据库类型；通过“HOST”告知被测时序数据库网络地址；通过“PORT”告知被测时序数据库网络端口；通过“USERNAME”告知被测时序数据库登录用户名；通过“PASSWORD”告知被测时序数据库登录用户的密码；通过“DB_NAME”告知被测时序数据库名称；通过“TOKEN”告知被测时序数据库连接认证Token（InfluxDB 2.0使用）； +| **模式名称** | **BENCHMARK_WORK_MODE** | **模式内容** | +| :--------------------- | :---------------------- | :------------------------------- | +| 常规测试模式 | testWithDefaultPath | 支持多种读和写操作的混合负载 | +| 生成数据模式 | generateDataMode | 生成Benchmark本身识别的数据 | +| 单数据库正确性写入模式 | verificationWriteMode | 需要配置 FILE_PATH 以及 DATA_SET | +| 单数据库正确性查询模式 | verificationQueryMode | 需要配置 FILE_PATH 以及 DATA_SET | -#### 写入场景构建参数 +### 3.3 服务器连接信息 -表1-4写入场景构建参数 +工作模式指定后，被测时序数据库的信息会通过如下参数告知IoT-benchmark -| 参数名称 | 类型 | 示例 | 系统描述 | -| -------------------------- | ------ | ------------------------- | ------------------------------------------------------------ | -| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | -| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | -| DEVICE_NUMBER | 整数 | 100 | 设备总数 | -| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数 | -| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | -| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | -| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | -| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | -| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | -| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | -| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | -| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | -| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | +| **参数名称** | **类型** | **示例** | **系统描述** | +| :----------- | :------- | :-------------------------- | :---------------------------------------------- | +| DB_SWITCH | 字符串 | IoTDB-200-SESSION_BY_TABLET | 被测时序数据库类型 | +| HOST | 字符串 | 127.0.0.1 | 被测时序数据库网络地址 | +| PORT | 整数 | 6667 | 被测时序数据库网络端口 | +| USERNAME | 字符串 | root | 被测时序数据库登录用户名 | +| PASSWORD | 字符串 | root | 被测时序数据库登录用户的密码 | +| DB_NAME | 字符串 | test | 被测时序数据库名称 | +| TOKEN | 字符串 | | 被测时序数据库连接认证Token（InfluxDB 2.0使用） | -按照表1-4配置参数启动可描述测试场景为：向被测时序数据库压力写入30000个（100个设备，每个设备300个传感器）时间序列2022年10月30日一天的顺序数据，总计25.92亿个数据点。其中每个设备的300个传感器数据类型分别为50个布尔、50个整数、50个长整数、50个浮点、50个双精度、50个字符。如果我们将表格中IS_OUT_OF_ORDER的值改为true，那么他表示的场景为：向被测时序数据库压力写入30000个时间序列2022年10月30日一天的数据，其中存在30%的乱序数据（到达时序数据库时间晚于其他生成时间晚于自身的数据点）。 +### 3.4 写入场景 -#### 查询场景构建参数 +| **参数名称** | **类型** | **示例** | **系统描述** | +| :------------------------- | :------- | :------------------------ | :----------------------------------------------------------- | +| CLIENT_NUMBER | 整数 | 100 | 客户端总数 | +| GROUP_NUMBER | 整数 | 20 | 数据库的数量；仅针对IoTDB。 | +| DEVICE_NUMBER | 整数 | 100 | 设备总数 | +| SENSOR_NUMBER | 整数 | 300 | 每个设备的传感器总数； **如果使用 IoTDB 表模型，则控制属性列数量** | +| INSERT_DATATYPE_PROPORTION | 字符串 | 1:1:1:1:1:1 | 设备的数据类型比例，BOOLEAN:INT32:INT64:FLOAT:DOUBLE:TEXT | +| POINT_STEP | 整数 | 1000 | 数据间时间戳间隔，即生成的数据两个时间戳之间的固定长度。 | +| OP_MIN_INTERVAL | 整数 | 0 | 操作最小执行间隔：若操作耗时大于该值则立即执行下一个，否则等待 (OP_MIN_INTERVAL-实际执行时间) ms；如果为0，则参数不生效；如果为-1，则其值和POINT_STEP一致 | +| IS_OUT_OF_ORDER | 布尔 | false | 是否乱序写入 | +| OUT_OF_ORDER_RATIO | 浮点数 | 0.3 | 乱序写入的数据比例 | +| BATCH_SIZE_PER_WRITE | 整数 | 1 | 批写入数据行数（一次写入多少行数据） | +| START_TIME | 时间 | 2022-10-30T00:00:00+08:00 | 写入数据的开始时间戳；以该时间戳为起点开始模拟创建数据时间戳。 | +| LOOP | 整数 | 86400 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | +| OPERATION_PROPORTION | 字符 | 1:0:0:0:0:0:0:0:0:0:0 | # 各操作的比例,按照顺序为 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10, 请注意使用英文冒号。比例中的每一项是整数。 | -表1-5查询场景构建参数 + +### 3.5 查询场景 | 参数名称 | 类型 | 示例 | 系统描述 | -| -------------------- | ---- | --------------------- | ------------------------------------------------------------ | +| :------------------- | :--- | :-------------------- | :----------------------------------------------------------- | | QUERY_DEVICE_NUM | 整数 | 2 | 每条查询语句中查询涉及到的设备数量 | | QUERY_SENSOR_NUM | 整数 | 2 | 每条查询语句中查询涉及到的传感器数量 | | QUERY_AGGREGATE_FUN | 字符 | count | 在聚集查询中使用的聚集函数，比如count、avg、sum、max_time等 | @@ -178,10 +200,11 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 | LOOP | 整数 | 10 | 总操作次数：具体每种类型操作会按OPERATION_PROPORTION定义的比例划分 | | OPERATION_PROPORTION | 字符 | 0:0:0:0:0:0:0:0:0:0:1 | 写入:Q1:Q2:Q3:Q4:Q5:Q6:Q7:Q8:Q9:Q10 | -表1-6查询类型及示例 SQL + +### 3.6 操作比例 | 编号 | 查询类型 | IoTDB 示例 SQL | -| ---- | ---------------------------- | ------------------------------------------------------------ | +| :--- | :--------------------------- | :----------------------------------------------------------- | | Q1 | 精确点查询 | select v1 from root.db.d1 where time = ? | | Q2 | 时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? | | Q3 | 带值过滤的时间范围查询 | select v1 from root.db.d1 where time > ? and time < ? and v1 > ? | @@ -193,27 +216,96 @@ IoT-benchmark 是基于 Java 和大数据环境开发的时序数据库基准测 | Q9 | 倒序范围查询 | select v1 from root.sg.d1 where time > ? and time < ? order by time desc | | Q10 | 倒序带值过滤的范围查询 | select v1 from root.sg.d1 where time > ? and time < ? and v1 > ? order by time desc | +### 3.7 测试过程和测试结果持久化 + +IoT-benchmark目前支持通过配置参数将测试过程和测试结果持久化: + +| **参数名称** | **类型** | **示例** | **系统描述** | +| :-------------------- | :------- | :-------- | :----------------------------------------------------------- | +| TEST_DATA_PERSISTENCE | 字符串 | None | 结果持久化选择，支持None，IoTDB，MySQL和CSV | +| RECORD_SPLIT | 布尔 | true | 是否将结果划分后输出到多个记录, IoTDB 暂时不支持 | +| RECORD_SPLIT_MAX_LINE | 整数 | 10000000 | 记录行数的上限（每个数据库表或CSV文件按照总行数为1千万切分存放） | +| TEST_DATA_STORE_IP | 字符串 | 127.0.0.1 | 输出数据库的IP地址 | +| TEST_DATA_STORE_PORT | 整数 | 6667 | 输出数据库的端口号 | +| TEST_DATA_STORE_DB | 字符串 | result | 输出数据库的名称 | +| TEST_DATA_STORE_USER | 字符串 | root | 输出数据库的用户名 | +| TEST_DATA_STORE_PW | 字符串 | root | 输出数据库的用户密码 | + +- 如果我们设置“`TEST_DATA_PERSISTENCE=CSV`”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的`data`文件夹，其下包含`csv`文件夹记录测试过程；`csvOutput`文件夹记录测试结果。 +- 如果我们设置“`TEST_DATA_PERSISTENCE=MySQL`”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 + +### 3.8 自动化脚本 + +#### 一键化启动脚本 + +您可以通过`cli-benchmark.sh`脚本一键化启动IoTDB、监控的IoTDB Benchmark和测试的IoTDB Benchmark，但需要注意该脚本启动时会清理IoTDB中的**所有数据**，请谨慎使用。 + +首先，您需要修改`cli-benchmark.sh`中的`IOTDB_HOME`参数为您本地的IoTDB所在的文件夹。 + +然后您可以使用脚本启动测试 + +```Bash +> ./cli-benchmark.sh +``` + +测试完成后您可以在`logs`文件夹中查看测试相关日志，在`server-logs`文件夹中查看监控相关日志。 + +#### 自动执行多项测试 + +通常，除非与其他测试结果进行比较，否则单个测试是没有意义的。因此，我们提供了一个接口来通过一次启动执行多个测试。 + +- 配置 routine + +这个文件的每一行应该是每个测试过程会改变的参数（否则就变成复制测试）。例如，"例程"文件是： + +```Plain +LOOP=10 DEVICE_NUMBER=100 TEST +LOOP=20 DEVICE_NUMBER=50 TEST +LOOP=50 DEVICE_NUMBER=20 TEST +``` + +然后依次执行3个LOOP参数分别为10、20、50的测试过程。 + +> 注意： +> +> 您可以使用“LOOP=20 DEVICE_NUMBER=10 TEST”等格式更改每个测试中的多个参数，不允许使用不必要的空间。 关键字"TEST"意味着新的测试开始。如果您更改不同的参数，更改后的参数将保留在下一次测试中。 + +- 开始测试 + +配置文件routine后，您可以通过启动脚本启动多测试任务： +```Bash +> ./rep-benchmark.sh +``` +然后测试信息将显示在终端中。 +> 注意： +> +> 如果您关闭终端或失去与客户端机器的连接，测试过程将终止。 如果输出传输到终端，则与任何其他情况相同。 -按照表1-5配置参数启动可描述测试场景为：从被测时序数据库执行10次2个设备2个传感器的倒序带值过滤的范围查询，SQL语句为：select s_0,s_31from data where time >2022-10-30T00:00:00+08:00 and time < 2022-10-30T00:04:10+08:00 and s_0 > -5 and device in d_21,d_46 order by time desc。 +使用此接口通常需要很长时间，您可能希望将测试过程作为守护程序执行。这样，您可以通过启动脚本将测试任务作为守护程序启动： -#### 测试过程和测试结果持久化 +```Bash +> ./rep-benchmark.sh > /dev/null 2>&1 & +``` -IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试过程和测试结果持久化到IoTDB、MySQL和CSV；其中写入到MySQL和CSV可以定义分库分表的行数上限，例如“RECORD_SPLIT=true、RECORD_SPLIT_MAX_LINE=10000000”表示每个数据库表或CSV文件按照总行数为1千万切分存放；如果记录到MySQL或IoTDB需要提供数据库链接信息，分别包括“TEST_DATA_STORE_IP”数据库的IP地址、“TEST_DATA_STORE_PORT”数据库的端口号、“TEST_DATA_STORE_DB”数据库的名称、“TEST_DATA_STORE_USER”数据库用户名、“TEST_DATA_STORE_PW”数据库用户密码。 +在这种情况下，如果您想知道发生了什么，可以通过以下命令查看日志信息： -如果我们设置“TEST_DATA_PERSISTENCE=CSV”，测试执行时和执行完毕后我们可以在IoT-benchmark根目录下看到新生成的data文件夹，其下包含csv文件夹记录测试过程；csvOutput文件夹记录测试结果。如果我们设置“TEST_DATA_PERSISTENCE=MySQL”，它会在测试开始前在指定的MySQL数据库中创建命名如“testWithDefaultPath_被测数据库名称_备注_测试启动时间”的数据表记录测试过程；会在名为“CONFIG”的数据表（如果不存在则创建该表），写入本次测试的配置信息；当测试完成时会在名为“FINAL_RESULT”的数据表（如果不存在则创建该表）中写入本次测试结果。 +```Bash +> cd ./logs +> tail -f log_info.log +``` -## 2. 实际案例 +## 4. 实际案例 我们以中车青岛四方车辆研究所有限公司应用为例，参考《ＡｐａｃｈｅＩｏＴＤＢ在智能运维平台存储中的应用》中描述的场景进行实际操作说明。 测试目标：模拟中车青岛四方所场景因切换时间序列数据库实际需求，对比预期使用的IoTDB和原有系统使用的KairosDB性能。 -测试环境：为了保证在实验过程中消除其他无关服务与进程对数据库性能的影响，以及不同数据库之间的相互影响，本实验中的本地数据库均部署并运行在资源配置相同的多个独立的虚拟机上。因此，本实验搭建了 4 台 Linux( CentOS7 /x86) 虚拟机，并分别在上面部署了IoT-benchmark、 IoTDB数据库、KairosDB数据库、MySQL数据库。每一台虚拟机的具体资源配置如表2-1所示。每一台虚拟机的具体用途如表2-2所示。 +测试环境：为了保证在实验过程中消除其他无关服务与进程对数据库性能的影响，以及不同数据库之间的相互影响，本实验中的本地数据库均部署并运行在资源配置相同的多个独立的虚拟机上。因此，本实验搭建了 4 台 Linux( CentOS7 /x86) 虚拟机，并分别在上面部署了IoT-benchmark、 IoTDB数据库、KairosDB数据库、MySQL数据库。每一台虚拟机的具体资源配置如表4-1所示。每一台虚拟机的具体用途如表4-2所示。 -表2-1虚拟机配置信息 +表4-1虚拟机配置信息 | 硬件配置信息 | 系统描述 | | ------------ | -------- | @@ -225,7 +317,7 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 -表2-2虚拟机用途 +表4-2虚拟机用途 | IP | 用途 | | ---------- | ------------- | @@ -234,11 +326,11 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 | 172.21.4.4 | KaiosDB | | 172.21.4.5 | MySQL | -### 2.1 写入测试 +### 4.1 写入测试 -场景描述：创建100个客户端来模拟100列车、每列车3000个传感器、数据类型为DOUBLE类型、数据时间间隔为500ms（2Hz）、顺序发送。参考以上需求我们需要修改IoT-benchmark配置参数如表2-3中所列。 +场景描述：创建100个客户端来模拟100列车、每列车3000个传感器、数据类型为DOUBLE类型、数据时间间隔为500ms（2Hz）、顺序发送。参考以上需求我们需要修改IoT-benchmark配置参数如表4-3中所列。 -表2-3配置参数信息 +表4-3配置参数信息 | 参数名称 | IoTDB值 | KairosDB值 | | -------------------------- | --------------------------- | ---------- | @@ -265,51 +357,51 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 | TEST_DATA_STORE_PW | admin | | | REMARK | demo | | -首先在172.21.4.3和172.21.4.4上分别启动被测时间序列数据库Apache-IoTDB和KairosDB，之后在172.21.4.2、172.21.4.3和172.21.4.4上通过ser-benchamrk.sh脚本启动服务器资源监控（图2-1）。然后按照表2-3在172.21.4.2分别修改iotdb-0.13-0.0.1和kairosdb-0.0.1文件夹内的conf/config.properties文件满足测试需求。先后使用benchmark.sh启动对Apache-IoTDB和KairosDB的写入测试。 +首先在172.21.4.3和172.21.4.4上分别启动被测时间序列数据库Apache-IoTDB和KairosDB，之后在172.21.4.2、172.21.4.3和172.21.4.4上通过ser-benchamrk.sh脚本启动服务器资源监控（图4-1）。然后按照表4-3在172.21.4.2分别修改iotdb-0.13-0.0.1和kairosdb-0.0.1文件夹内的conf/config.properties文件满足测试需求。先后使用benchmark.sh启动对Apache-IoTDB和KairosDB的写入测试。 ![img](/img/bm5.png) -图2-1服务器监控任务 +图4-1服务器监控任务 -​ 例如我们首先启动对KairosDB的测试，IoT-benchmark会在MySQL数据库中创建CONFIG数据表存放本次测试配置信息（图2-2），测试执行中会有日志输出当前测试进度（图2-3）。测试完成时会输出本次测试结果（图2-3），同时将结果写入FINAL_RESULT数据表中（图2-4）。 +​ 例如我们首先启动对KairosDB的测试，IoT-benchmark会在MySQL数据库中创建CONFIG数据表存放本次测试配置信息（图4-2），测试执行中会有日志输出当前测试进度（图4-3）。测试完成时会输出本次测试结果（图4-3），同时将结果写入FINAL_RESULT数据表中（图4-4）。 ![](/img/bm6.png) -图2-2测试配置信息表 +图4-2测试配置信息表 ![](/img/bm7.png) ![](/img/bm8.png) ![](/img/bm9.png) ![](/img/bm10.png) -图2-3测试进度和结果 +图4-3测试进度和结果 ![](/img/bm11.png) -图2-4测试结果表 +图4-4测试结果表 之后我们再启动对Apache-IoTDB的测试，同样的IoT-benchmark会在MySQL数据库CONFIG数据表中写入本次测试配置信息，测试执行中会有日志输出当前测试进度。测试完成时会输出本次测试结果，同时将结果写入FINAL_RESULT数据表中。 -依照测试结果信息我们知道同样的配置写入Apache-IoTDB和KairosDB写入延时时间分别为：55.98ms和1324.45ms；写入吞吐分别为：5,125,600.86点/秒和224,819.01点/秒；测试分别执行了585.30秒和11777.99秒。并且KairosDB有写入失败出现，排查后发现是数据磁盘使用率已达到100%，无磁盘空间继续接收数据。而Apache-IoTDB无写入失败现象，全部数据写入完毕后占用磁盘空间仅为4.7G（如图2-5所示）；从写入吞吐和磁盘占用情况上看Apache-IoTDB均优于KairosDB。当然后续还有其他测试来从多方面观察和对比，比如查询性能、文件压缩比、数据安全性等。 +依照测试结果信息我们知道同样的配置写入Apache-IoTDB和KairosDB写入延时时间分别为：55.98ms和1324.45ms；写入吞吐分别为：5,125,600.86点/秒和224,819.01点/秒；测试分别执行了585.30秒和11777.99秒。并且KairosDB有写入失败出现，排查后发现是数据磁盘使用率已达到100%，无磁盘空间继续接收数据。而Apache-IoTDB无写入失败现象，全部数据写入完毕后占用磁盘空间仅为4.7G（如图4-5所示）；从写入吞吐和磁盘占用情况上看Apache-IoTDB均优于KairosDB。当然后续还有其他测试来从多方面观察和对比，比如查询性能、文件压缩比、数据安全性等。 ![](/img/bm12.png) -图2-5磁盘使用情况 +图4-5磁盘使用情况 那么测试过程中各个服务器资源使用情况如何呢？每个写操作具体的表现如何呢？这个时候我们就可以通过安装和使用Tableau来可视化服务器监控表和测试过程记录表内的数据了。Tableau的使用本文不展开介绍，通过它连接测试数据持久化的数据表后具体结果下如图（以Apache-IoTDB为例）： ![](/img/bm13.png) ![](/img/bm14.png) -图2-6Tableau可视化测试过程 +图4-6Tableau可视化测试过程 -### 2.2 询测试 +### 4.2 查询测试 场景描述：在写入测试场景下模拟10个客户端对时序数据库Apache-IoTDB内存放的数据进行全类型查询任务。配置如下： -表2-4配置参数信息 +表4-4配置参数信息 | 参数名称 | 示例 | | -------------------- | --------------------- | @@ -328,25 +420,25 @@ IoT-benchmark目前支持通过配置参数“TEST_DATA_PERSISTENCE”将测试 ![](/img/bm15.png) -图2-7查询测试结果 +图4-7查询测试结果 -### 2.3 其他参数说明 +### 4.3 其他参数说明 之前章节中针对Apache-IoTDB和KairosDB进行写入性能对比，但是用户如果要执行模拟真实写入速率测试该如何配置？测试时间过长该如何控制呢？生成的模拟数据有哪些规律吗？如果IoT-Benchmark服务器配置较低，可以使用多台机器模拟压力输出吗？ -表2-5配置参数信息 - -| 场景 | 参数 | 值 | 说明 | -| ------------------------------------------------------------ | -------------------------- | ------------------------------------------------------------ | --------------------------------- | -| 模拟真实写入速率 | OP_INTERVAL | -1 | 也可输入整数控制操作间隔 | -| 指定测试时长（1小时） | TEST_MAX_TIME | 3600000 | 单位 ms；需要LOOP执行时间大于该值 | -| 定义模拟数据规律：支持全部数据类型，数量平均分类；支持五种数据分布，数量平均分布；字符串长度为10；小数位数为2 | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | 数据类型分布比率 | -| LINE_RATIO | 1 | 线性 | | -| SIN_RATIO | 1 | 傅里叶函数 | | -| SQUARE_RATIO | 1 | 方波 | | -| RANDOM_RATIO | 1 | 随机数 | | -| CONSTANT_RATIO | 1 | 常数 | | -| STRING_LENGTH | 10 | 字符串长度 | | -| DOUBLE_LENGTH | 2 | 小数位数 | | -| 三台机器模拟300台设备数据写入 | BENCHMARK_CLUSTER | true | 开启多benchmark模式 | -| BENCHMARK_INDEX | 0、1、3 | 以[写入测试](./Benchmark.md#写入测试)写入参数为例：0号负责设备编号0-99数据写入；1号负责设备编号100-199数据写入；2号负责设备编号200-299数据写入； | | \ No newline at end of file +表4-5配置参数信息 + +| 场景 | 参数 | 值 | 说明 | +| ------------------------------------------------------------ | -------------------------- |---------------------------------------------------------------------------------------------------| --------------------------------- | +| 模拟真实写入速率 | OP_INTERVAL | -1 | 也可输入整数控制操作间隔 | +| 指定测试时长（1小时） | TEST_MAX_TIME | 3600000 | 单位 ms；需要LOOP执行时间大于该值 | +| 定义模拟数据规律：支持全部数据类型，数量平均分类；支持五种数据分布，数量平均分布；字符串长度为10；小数位数为2 | INSERT_DATATYPE_PROPORTION | 1:1:1:1:1:1 | 数据类型分布比率 | +| LINE_RATIO | 1 | 线性 | | +| SIN_RATIO | 1 | 傅里叶函数 | | +| SQUARE_RATIO | 1 | 方波 | | +| RANDOM_RATIO | 1 | 随机数 | | +| CONSTANT_RATIO | 1 | 常数 | | +| STRING_LENGTH | 10 | 字符串长度 | | +| DOUBLE_LENGTH | 2 | 小数位数 | | +| 三台机器模拟300台设备数据写入 | BENCHMARK_CLUSTER | true | 开启多benchmark模式 | +| BENCHMARK_INDEX | 0、1、3 | 以[写入测试](./Benchmark.md#_4-1-写入测试)写入参数为例：0号负责设备编号0-99数据写入；1号负责设备编号100-199数据写入；2号负责设备编号200-299数据写入； | | \ No newline at end of file