Skip to content

Commit 1ed9701

Browse files
dai-chendai-chen
authored
Reorganize development docs (#1200)
* Add new dev doc index page Signed-off-by: Chen Dai <[email protected]> * Rename all docs in consistent naming convention Signed-off-by: Chen Dai <[email protected]> * Convert feature PR to new doc Signed-off-by: Chen Dai <[email protected]> * Update index doc Signed-off-by: Chen Dai <[email protected]> * Add link in root readme Signed-off-by: Chen Dai <[email protected]> Signed-off-by: Chen Dai <[email protected]>
1 parent b29f4c2 commit 1ed9701

18 files changed

+1357
-4
lines changed

‎README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -122,11 +122,11 @@ The following projects have been merged into this repository as separate folders
122122

123123
Besides basic filtering and aggregation, OpenSearch SQL also supports complex queries, such as querying semi-structured data, JOINs, set operations, sub-queries etc. Beyond the standard functions, OpenSearch functions are provided for better analytics and visualization. Please check our [documentation](#documentation) for more details.
124124

125-
Recently we have been actively improving our query engine primarily for better correctness and extensibility. Behind the scene, the new enhanced engine has already supported both SQL and Piped Processing Language. Please find more details in [SQL Engine V2 - Release Notes](./docs/dev/NewSQLEngine.md).
125+
Recently we have been actively improving our query engine primarily for better correctness and extensibility. Behind the scene, the new enhanced engine has already supported both SQL and Piped Processing Language. Please find more details in [SQL Engine V2 - Release Notes](./docs/dev/intro-v2-engine.md).
126126

127127
## Documentation
128128

129-
Please refer to the [SQL Language Reference Manual](./docs/user/index.rst), [Piped Processing Language (PPL) Reference Manual](./docs/user/ppl/index.rst) and [Technical Documentation](https://opensearch.org/docs/latest/search-plugins/sql/index/) for detailed information on installing and configuring plugin.
129+
Please refer to the [SQL Language Reference Manual](./docs/user/index.rst), [Piped Processing Language (PPL) Reference Manual](./docs/user/ppl/index.rst), [OpenSearch SQL/PPL Engine Development Manual](./docs/dev/index.md) and [Technical Documentation](https://opensearch.org/docs/latest/search-plugins/sql/index/) for detailed information on installing and configuring plugin.
130130

131131
## Forum
132132

‎docs/dev/datasource-prometheus.md

+331
Large diffs are not rendered by default.

‎docs/dev/datasource-query-s3.md

+252
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
## 1.Overview
2+
3+
In this document, we will propose a solution in OpenSearch Observability to query log data stored in S3.
4+
5+
### 1.1.Problem Statements
6+
7+
Currently, OpenSearch Observability is collection of plugins and applications that let you visualize data-driven events by using Piped Processing Language to explore, discover, and query data stored in OpenSearch. The major requirements we heard from customer are
8+
9+
* **cost**, regarding to hardware cost of setup an OpenSearch cluster.
10+
* **ingestion performance,** it is not easy to supporting high throughput raw log ingestion.
11+
* **flexibility,** OpenSearch index required user know their query pattern before ingest data. Which is not flexible.
12+
13+
Can build a new solution for OpenSearch observably uses and leverage S3 as storage. The benefits are
14+
15+
* **cost efficiency**, comparing with OpenSearch, S3 is cheap.
16+
* **high ingestion throughput**, S3 is by design to support high throughput write.
17+
* **flexibility**, user do not need to worry about index mapping define and reindex. everything could be define at query tier.
18+
* **scalability**, user do not need to optimize their OpenSearch cluster for write. S3 is auto scale.
19+
* **data durability**, S3 provide 11s 9 of data durability.
20+
21+
With all these benefits, are there any concerns? The **ability to query S3 in OpenSearch and query performance** are the major concerns. In this doc, we will provide the solution to solve these two major concerns.
22+
23+
## 2.Terminology
24+
25+
* **Catalog**. OpenSearch access external data source through catalog. For example, S3 catalog. Each catalog has attributes, most important attributes is data source access credentials.
26+
* **Table**: To access external data source. User should create external table to describe the schema and location. Table is the virtual concept which does not mapping to OpenSearch index.
27+
* **Materialized View**: User could create view from existing tables. Each view is 1:1 mapping to OpenSearch index. There are two types of views
28+
* (1) Permanent view (default) which is fully managed by user.
29+
* (2) Transient View which is maintained by Maximus automatically. user could decide when to drop the transient view.
30+
31+
## 3.Requirements
32+
33+
### 3.1.Use Cases
34+
35+
#### Use Case - 1: pre-build and query metrics from log on s3
36+
37+
**_Create_**
38+
39+
* User could ingest the log or events directly to s3 with existing ingestion tools (e.g. [fluentbit](https://docs.fluentbit.io/manual/pipeline/outputs/s3)). The ingested log should be partitioned. e.g. _s3://my-raw-httplog/region/yyyy/mm/dd_.
40+
* User configure s3 Catalog in OpenSearch. By default, s3 connector use [ProfileCredentialsProvider](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/profile/ProfileCredentialsProvider.html).
41+
42+
```
43+
settings.s3.access_key: xxxxx
44+
settings.s3.secret_key: xxxxx
45+
```
46+
47+
* User create table in OpenSearch of their log data on s3. Maximus will create table httplog in the s3 catalog
48+
49+
```
50+
CREATE EXTERNAL TABLE `s3`.`httplog`(
51+
@timestamp timestamp,
52+
clientip string,
53+
request string,
54+
state integer,
55+
size long
56+
)
57+
ROW FORMAT SERDE
58+
'json'
59+
'grok', <timestamp> <className>
60+
PARTITION BY
61+
${region}/${year}/${month}/${day}/
62+
LOCATION
63+
's3://my-raw-httplog/';
64+
```
65+
66+
* User create the view *failEventsMetrics*. Note: User could only create view from schema-defined table in Maximus
67+
68+
```
69+
CREATE MATERIALIZED VIEW failEventsMetrics (
70+
cnt long,
71+
time timestamp,
72+
status string
73+
)
74+
AS source=`s3`.`httpLog` status != 200 | status count() as cnt by span(5mins), status
75+
WITH (
76+
REFRESH=AUTO
77+
)
78+
```
79+
80+
* Maximus will (1) create view *failEventsMetrics* in the default catalog. (2) create index *failEventsMetrics* in the OpenSearch cluster. (3) refresh *failEventsMetrics* index with logs on s3*.* _Notes: Create view return success when operation (1) and (2) success. Refresh view is an async task will be executed in background asynchronously._
81+
* User could describe the view to monitor the status.
82+
83+
```
84+
DESCRIBE/SHOW MATERIALIZED VIEW failEventsMetrics
85+
86+
# Return
87+
status: INIT | IN_PROGRESS | READY
88+
```
89+
90+
_**Query**_
91+
92+
* User could query the view.
93+
94+
```
95+
source=failEventsMetrics time in last 7 days
96+
```
97+
98+
* User could query the table, Thunder will rewrite table query as view query when optimizing the query.
99+
100+
```
101+
source=`s3`.`httpLog` status != 200 | status count() as cnt by span(1hour), status
102+
```
103+
104+
_**Drop**_
105+
106+
107+
* User could drop the table. Maximus will delete httpLog metadata from catalog and associated view drop.
108+
109+
```
110+
DROP TABLE `s3`.`httpLog`
111+
```
112+
113+
* User could drop the view. Maximus will delete failEventsMetrics metadata from catalog and delete failEventsMetrics indices.
114+
115+
```
116+
DROP MATERIALIZED VIEW failEventsMetrics
117+
```
118+
119+
**Access Control**
120+
121+
* User could not configure any permission on table.
122+
* User could not directly configure any permission on view. But user could configure permission on index associated with the view.
123+
![image1](https://user-images.githubusercontent.com/2969395/182239505-a8541ec1-f46f-4b91-882a-8a4be36f5aea.png)
124+
125+
#### Use Case - 2: Ad-hoc s3 query in OpenSearch
126+
127+
**_Create_**
128+
129+
* Similar as previous example, User could ingest the log or events directly to s3 with existing ingestion tools. create catalog and table from data on s3.
130+
131+
_**Query**_
132+
133+
* User could query table without create view. At runtime, Maximus will create the transient view and populate the view with required data.
134+
135+
```
136+
source=`s3`.`httpLog` status != 200 | status count() as cnt by span(5mins), status
137+
```
138+
139+
_**List**_
140+
141+
* User could list all the view of a table no matter how the view is created. User could query/drop/describe the temp view create by Maximus, as same as user created view.
142+
143+
```
144+
LIST VIEW on `s3`.`httpLog`
145+
146+
# return
147+
httplog-tempView-xxxx
148+
```
149+
150+
**Access Control**
151+
152+
* User could not configure any permission on table. During query time, Maximus will use the credentials to access s3 data. It is user’s ownership to configure the permission on s3 data.
153+
* User could not directly configure any permission on view. But user could configure permission on index associated with the view.
154+
![image2](https://user-images.githubusercontent.com/2969395/182239672-72b2cfc6-c22e-4279-b33e-67a85ee6a778.png)
155+
156+
### 3.2.Function Requirements
157+
158+
#### Query S3
159+
160+
* User could query time series data on S3 by using PPL in OpenSearch.
161+
* For querying time series data in S3, user must create a **Catalog** for S3 with required setting.
162+
* access key and secret key
163+
* endpoint
164+
* For querying time series data in S3, user must create a **Table** of time series data on S3.
165+
166+
#### View
167+
168+
* Support create materialized view from time series data on S3.
169+
* Support fully materialized view refresh
170+
* Support manually materialized view incrementally refresh
171+
* Support automatically materialized view incrementally refresh
172+
* Support hybrid scan
173+
* Support drop materialized view
174+
* Support show materialized view
175+
176+
#### Query Optimization
177+
178+
* Inject optimizer rule to rewrite the query with MV to avoid S3 scan
179+
180+
#### Automatic query acceleration
181+
182+
* Automatically select view candidate based on OpenSearch - S3 query execution metrics
183+
* Store workload and selected view info for visualization and user intervention
184+
* Automatically create/refresh/drop view.
185+
186+
#### S3 data format
187+
188+
* The time series data could be compressed with gzip.
189+
* The time series data file format could be.
190+
191+
* JSON
192+
* TSV
193+
194+
* If the time series data should be partitioned and have snapshot id. Query engine could support automatically incremental refresh and hybrid scan.
195+
196+
#### Resource Management
197+
198+
* Support circuit breaker based resource control when executing a query.
199+
* Support task based resource manager
200+
201+
#### Fault Tolerant
202+
203+
* For fast query process, we scarify Fault Tolerant. Support query fast failure in case of hardware failure.
204+
205+
#### Setting
206+
207+
* Support configure of x% of disk automatically create view should used.
208+
209+
### 3.3.Non Function Requirements
210+
211+
#### Security:
212+
213+
There are three types of privileges that are related to materialized views
214+
215+
* Privileges directly on the materialized view itself
216+
* Privileges on the objects (e.g. table) that the materialized view accesses.
217+
218+
*_materialized view itself_*
219+
220+
* User could not directly configure any access control on view. But user could configure any access control on index associated with the view. In the other words, materialized inherits all the access control on the index associated with view.
221+
* When **automatically** **refresh view**, Maximus will use the backend role. It required the backend role has permission to index data.
222+
223+
*_objects (e.g. table) that the materialized view accesses_*
224+
225+
* As with non-materialized views, a user who wishes to access a materialized view needs privileges only on the view, not on the underlying object(s) that the view references.
226+
227+
*_table_*
228+
229+
* User could not configure any privileges on table. *_Note: because the table query could be rewrite as view query If the user do not have required permission to access the view. User could still get no permission exception._*
230+
231+
*_objects (e.g. table) that the table refer_*
232+
233+
* The s3 access control is only evaluated during s3 access. if the table access is rewrite as view access. The s3 access control will not take effect.
234+
235+
*_Encryption_*
236+
237+
* Materialized view data will be encrypted at rest.
238+
239+
#### Others
240+
241+
* Easy to use: the solution should be designed easy to use. It should just work out of the box and provide good performance with minimal setup.
242+
* **Performance**: Use **http_log** dataset to benchmark with OpenSearch cluster.
243+
* Scalability: The solution should be scale horizontally.
244+
* **Serverless**: The solution should be designed easily deployed as Serverless infra.
245+
* **Multi-tenant**: The solution should support multi tenant use case.
246+
* **Metrics**: Todo
247+
* **Log:** Todo
248+
249+
## 4.What is, and what is not
250+
251+
* We design and optimize for observability use cases only. Not OLTP and OLAP.
252+
* We only support time series log data on S3. We do not support query arbitrary data on S3.

‎docs/dev/index.md

+68
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
2+
# OpenSearch SQL/PPL Engine Development Manual
3+
4+
## Introduction
5+
6+
+ [Architecture](intro-architecture.md): a quick overview of architecture
7+
+ [V2 Engine](intro-v2-engine.md): introduces why we developed new V2 engine
8+
+ Concepts
9+
+ Quickstart
10+
11+
---
12+
## Clients
13+
14+
+ **CLI**
15+
+ **JDBC Driver**
16+
+ **ODBC Driver**
17+
+ **Query Workbench**
18+
19+
---
20+
## Deployment
21+
22+
+ **Standalone Mode**
23+
+ **OpenSearch Cluster**
24+
25+
---
26+
## Programming Guides
27+
28+
+ **API**
29+
+ **JavaDoc**
30+
31+
---
32+
## Development Guides
33+
34+
### Language Processing
35+
36+
+ **SQL**
37+
+ [Aggregate Window Function](sql-aggregate-window-function.md): aggregate window function support
38+
+ **Piped Processing Language**
39+
40+
### Query Processing
41+
42+
+ **Query Analyzing**
43+
+ [Semantic Analysis](query-semantic-analysis.md): performs semantic analysis to ensure semantic correctness
44+
+ [Type Conversion](query-type-conversion.md): implement implicit data type conversion
45+
+ **Query Planning**
46+
+ [Logical Optimization](query-optimizier-improvement.md): improvement on logical optimizer and physical implementer
47+
+ **Query Execution**
48+
+ [Query Manager](query-manager.md): query management
49+
+ **Query Acceleration**
50+
+ [Automatic Acceleration](query-automatic-acceleration.md): workload based automatic query acceleration proposal
51+
52+
### Data Sources
53+
54+
+ **OpenSearch**
55+
+ [Relevancy Search](opensearch-relevancy-search.md): OpenSearch relevancy search functions
56+
+ [Sub Queries](opensearch-nested-field-subquery.md): support sub queries on OpenSearch nested field
57+
+ [Pagination](opensearch-pagination.md): pagination implementation by OpenSearch scroll API
58+
+ [Prometheus](datasource-prometheus.md): Prometheus query federation
59+
+ **File System**
60+
+ [Querying S3](datasource-query-s3.md): S3 query federation proposal
61+
62+
---
63+
## Other Documents
64+
65+
+ **Test Framework**
66+
+ [Doc Test](testing-doctest.md): makes our doc live and runnable to ensure documentation correctness
67+
+ [Comparison Test](testing-comparison-test.md): compares with other databases to ensure functional correctness
68+
+ **Operation Tools**

‎docs/dev/Architecture.md ‎docs/dev/intro-architecture.md

File renamed without changes.

‎docs/dev/NewSQLEngine.md ‎docs/dev/intro-v2-engine.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ With the architecture and extensibility improved significantly, the following SQ
3131
* [Semi-structured data query](../../docs/user/beyond/partiql.rst#example-2-selecting-deeper-levels): support querying OpenSearch object fields on arbitrary level
3232
* OpenSearch multi-field: handled automatically and users won't have the access, ex. `text` is converted to `text.keyword` if it’s a multi-field
3333

34-
As for correctness, besides full coverage of unit and integration test, we developed a new comparison test framework to ensure correctness by comparing with other databases. Please find more details in [Testing](./Testing.md).
34+
As for correctness, besides full coverage of unit and integration test, we developed a new comparison test framework to ensure correctness by comparing with other databases. Please find more details in [Testing](./testing-comparison-test.md).
3535

3636

3737
---
@@ -63,7 +63,7 @@ You can find all the limitations in [Limitations](../../docs/user/limitations/li
6363
---
6464
## 4.How it's Implemented
6565

66-
If you're interested in the new query engine, please find more details in [Developer Guide](../../DEVELOPER_GUIDE.rst), [Architecture](./Architecture.md) and other docs in the dev folder.
66+
If you're interested in the new query engine, please find more details in [Developer Guide](../../DEVELOPER_GUIDE.rst), [Architecture](./intro-architecture.md) and other docs in the dev folder.
6767

6868

6969
---

‎docs/dev/SubQuery.md ‎docs/dev/opensearch-nested-field-subquery.md

File renamed without changes.

‎docs/dev/Pagination.md ‎docs/dev/opensearch-pagination.md

File renamed without changes.

0 commit comments

Comments
 (0)