Doris Roadmap 2024 #30669
Comments
morningman
added
kind/community
|
Currently, I have completed the development and testing of the JDBC catalog for Apache Druid. If possible, I would like to contribute this feature. PR:#27270 |
|
Flink-connector-doris will use FlinkCDC to synchronize multiple tables or the entire database from MongoDB and DB2 to Doris. |
|
typo "Mutlt cluster support" => "Multi cluster support" |
|
Looking forward to version 2.1 |
me too |
|
Walking with innovators |
|
Support generating columns |
HI @vinlee19 , thanks for your contribution. I'm not sure if it is suitable for Druid to using JDBC as data connector? I'm concerning the performance issue. But indeed Trino is using JDBC connect Druid. |
me too |
1 similar comment
me too |
|
why there is no 'Support building mv from Paimon table' |
|
|
It will be supported |
|
|
Support transactional multi table DELETE INSERT for adapting aggregate tables in dbt's Incremental models |
|
Look forward to supporting DataGrip and kettle! |
|
The correct url for |
|
binlog，binlog，binlog! |
|
Which version of Doris is preparing to support DB2？ |
|
On the "Index Overview" page in the docs, I see that Inverted Indexes have |
|
HI, |
|
According to this roadmap, inverted indexes are not yet "working with separation of storage and computation", but is there an issue tracking this? |
…al/hdfs/s3 (#41080) ## Proposed changes Issue Number: #30669 <!--Describe your changes.--> This change supports reading the contents of external file tables from rcbinary, rctext, and sequence files via the JNI connector. todo-lists: - [x] Support read rc_binary files using local tvf - [x] Support read rc_text/sequence files using local tvf - [x] Support using s3/hdfs tvf Example: **sequence file:** input: ``` mysql select * from local( "file_path" = "test/test.seq", "format" = "sequence", "backend_id" = "10011", "hive_schema"="k1:tinyint;k2:smallint;k3:int;k4:bigint;k5:float;k6:double;k7:decimal(10,2);k8:string;k9:char(10);k10:varchar(20);k11:boolean;k12:timestamp;k13:date;k14:array<string>;k15:map<string,int>;k16:struct<name:string,age:int>"); ``` output: ``` +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | k1 | k2 | k3 | k4 | k5 | k6 | k7 | k8 | k9 | k10 | k11 | k12 | k13 | k14 | k15 | k16 | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | 7 | 13 | 74 | 13000000000 | 6.15 | 4.376 | 57.30 | world | Char | Varchar | 1 | 2022-01-01 10:00:00 | 2022-01-01 | ["A", "B", "C"] | {"key2":2, "key1":1} | {"name":"John", "age":30} | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ 1 row in set (0.07 sec) ``` **rc_binary file:** input: ```mysql select * from local( "file_path" = "test/test.rcbinary", "format" = "rc_binary", "backend_id" = "10011", "hive_schema"="k1:tinyint;k2:smallint;k3:int;k4:bigint;k5:float;k6:double;k7:decimal(10,2);k8:string;k9:char(10);k10:varchar(20);k11:boolean;k12:timestamp;k13:date;k14:array<string>;k15:m ap<string,int>;k16:struct<name:string,age:int>"); ``` output: ``` +------+------+------+-------------+------+------+--------+------+------------+-----------+------+---------------------+------------+-----------------+------------------+-------------------------------+ | k1 | k2 | k3 | k4 | k5 | k6 | k7 | k8 | k9 | k10 | k11 | k12 | k13 | k14 | k15 | k16 | +------+------+------+-------------+------+------+--------+------+------------+-----------+------+---------------------+------------+-----------------+------------------+-------------------------------+ | 1 | 2 | 3 | 10000000000 | 1.23 | 3.14 | 100.50 | you | are | beautiful | 0 | 2023-10-29 02:00:00 | 2023-10-29 | ["D", "E", "F"] | {"k2":5, "k1":3} | {"name":"chandler", "age":54} | +------+------+------+-------------+------+------+--------+------+------------+-----------+------+---------------------+------------+-----------------+------------------+-------------------------------+ 1 row in set (0.12 sec) ``` **rc_text file:** input: ``` mysql select * from local( "file_path" = "test/test.rctext", "format" = "rc_text", "backend_id" = "10011", "hive_schema"="k1:tiny int;k2:smallint;k3:int;k4:bigint;k5:float;k6:double;k7:decimal(10,2);k8:string;k9:char(10);k10:varchar(20);k11:boolean;k12:timestamp;k13:date;k14:array<string>;k15: map<string,int>;k16:struct<name:string,age:int>"); ``` output: ``` +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | k1 | k2 | k3 | k4 | k5 | k6 | k7 | k8 | k9 | k10 | k11 | k12 | k13 | k14 | k15 | k16 | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | 7 | 13 | 74 | 13000000000 | 6.15 | 4.376 | 57.30 | world | Char | Varchar | 1 | 2022-01-01 10:00:00 | 2022-01-01 | ["A", "B", "C"] | {"key2":2, "key1":1} | {"name":"John", "age":30} | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ 1 row in set (0.06 sec) ```
…al/hdfs/s3 (#41080) Issue Number: #30669 <!--Describe your changes.--> This change supports reading the contents of external file tables from rcbinary, rctext, and sequence files via the JNI connector. todo-lists: - [x] Support read rc_binary files using local tvf - [x] Support read rc_text/sequence files using local tvf - [x] Support using s3/hdfs tvf Example: **sequence file:** input: ``` mysql select * from local( "file_path" = "test/test.seq", "format" = "sequence", "backend_id" = "10011", "hive_schema"="k1:tinyint;k2:smallint;k3:int;k4:bigint;k5:float;k6:double;k7:decimal(10,2);k8:string;k9:char(10);k10:varchar(20);k11:boolean;k12:timestamp;k13:date;k14:array<string>;k15:map<string,int>;k16:struct<name:string,age:int>"); ``` output: ``` +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | k1 | k2 | k3 | k4 | k5 | k6 | k7 | k8 | k9 | k10 | k11 | k12 | k13 | k14 | k15 | k16 | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | 7 | 13 | 74 | 13000000000 | 6.15 | 4.376 | 57.30 | world | Char | Varchar | 1 | 2022-01-01 10:00:00 | 2022-01-01 | ["A", "B", "C"] | {"key2":2, "key1":1} | {"name":"John", "age":30} | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ 1 row in set (0.07 sec) ``` **rc_binary file:** input: ```mysql select * from local( "file_path" = "test/test.rcbinary", "format" = "rc_binary", "backend_id" = "10011", "hive_schema"="k1:tinyint;k2:smallint;k3:int;k4:bigint;k5:float;k6:double;k7:decimal(10,2);k8:string;k9:char(10);k10:varchar(20);k11:boolean;k12:timestamp;k13:date;k14:array<string>;k15:m ap<string,int>;k16:struct<name:string,age:int>"); ``` output: ``` +------+------+------+-------------+------+------+--------+------+------------+-----------+------+---------------------+------------+-----------------+------------------+-------------------------------+ | k1 | k2 | k3 | k4 | k5 | k6 | k7 | k8 | k9 | k10 | k11 | k12 | k13 | k14 | k15 | k16 | +------+------+------+-------------+------+------+--------+------+------------+-----------+------+---------------------+------------+-----------------+------------------+-------------------------------+ | 1 | 2 | 3 | 10000000000 | 1.23 | 3.14 | 100.50 | you | are | beautiful | 0 | 2023-10-29 02:00:00 | 2023-10-29 | ["D", "E", "F"] | {"k2":5, "k1":3} | {"name":"chandler", "age":54} | +------+------+------+-------------+------+------+--------+------+------------+-----------+------+---------------------+------------+-----------------+------------------+-------------------------------+ 1 row in set (0.12 sec) ``` **rc_text file:** input: ``` mysql select * from local( "file_path" = "test/test.rctext", "format" = "rc_text", "backend_id" = "10011", "hive_schema"="k1:tiny int;k2:smallint;k3:int;k4:bigint;k5:float;k6:double;k7:decimal(10,2);k8:string;k9:char(10);k10:varchar(20);k11:boolean;k12:timestamp;k13:date;k14:array<string>;k15: map<string,int>;k16:struct<name:string,age:int>"); ``` output: ``` +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | k1 | k2 | k3 | k4 | k5 | k6 | k7 | k8 | k9 | k10 | k11 | k12 | k13 | k14 | k15 | k16 | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ | 7 | 13 | 74 | 13000000000 | 6.15 | 4.376 | 57.30 | world | Char | Varchar | 1 | 2022-01-01 10:00:00 | 2022-01-01 | ["A", "B", "C"] | {"key2":2, "key1":1} | {"name":"John", "age":30} | +------+------+------+-------------+------+-------+-------+-------+------------+---------+------+---------------------+------------+-----------------+----------------------+---------------------------+ 1 row in set (0.06 sec) ```
|
Stateless FE feature looks interesting. What does Stateless FE mean? Storing no local states/contexts? Like session states, user authentication states? Another FE can takeover in between query execution? Can you please elaborate on what is planned for this at the high level? |
and others
Roadmap 2023
Roadmap 2022
Separation of Storage and Computation
Async Materialized View
Build materialized view
REPLACEoperationTransparent Rewriting
Materialized view management
Semi Structure Data Analysis
Inverted Index
VARIANT data type
VARIANTdata typeQuery Optimizer
Basic framework
Planning quality
DataLake Analysis
Support more file format
Support more lake format
Trino Connector compatibility
Datalake write back
INSERT OVERWRITEINSERTEnhanced JDBC Catalog
Enhanced file analysis
Enhanced file cache
Integrate with Apache Ranger
SQL dialect support
Query Processing
Storage Engine
INSERT INTOMERGE INTOCLUSTER BYEcosystem & Tools
The text was updated successfully, but these errors were encountered: