Fulltext index: Initial version

[kunga]

Make a working prototype of fulltext index (WITH (layout=flat, tokenizer=whitespace))

Internal design doc

Vector index history: azevaykin, MBkkt

Syntax

ALTER TABLE texts
ADD INDEX index
GLOBAL USING fulltext
ON (text)
WITH (
    layout = "flat"
    tokenizer = "whitespace",
    use_filter_lowercase = true
)

Steps

• Add public api (see Create vector index in SchemeShard #4967)
  • Initial public api for fulltext index #23381
  • Support multiple columns in fulltext index #24439
  • Extend support of multiple columns in fulltext index #25337
• Add scheme objects (see Create vector index in SchemeShard #4967)
  • Add fulltext index feature flag #23701
  • Support fulltext index on SchemeShard #23739
  • Test fulltext index backup #28095
• Add base helpers
  • Add base fulltext helpers #24589
  • Add fulltext length filter #26991
  • Support fulltext ngrams #28082
• Support CREATE / ALTER TABLE ADD INDEX fulltext (see ADD Vector index USING syntax #6602)
  • Pass index settings as is
  • Pass index settings as is #24606
  • Handle fulltext index type in YQL
  • Handle fulltext index type in CREATE / ALTER TABLE #25670
  • Test fulltext scheme queries with QueryService #26112
• Add index build on an existing table
  • DataShard fulltext index build scan #25028
  • Build fulltext index on SchemeShard #25811
• Show fulltext index parameters in UI
  • feat(ui): support fulltext index ydb-embedded-ui#2962
• Make simple INSERT/UPDATE/DELETE statements work
  • Add fulltext KQP query tests #26566
  • Add insert rows functionality for fulltext indexed tables #26670
  • Add delete rows functionality for fulltext indexed tables #27345
  • Add upsert, update, and replace rows functionality for fulltext indexed tables #27566
• Make simple SELECT statements work
  • Add example select queries from fulltext index #27754
  • Add FulltextContains YQL function
  • Implement FulltextContains on KQP
• Add SDK support (see Create vector index in SchemeShard #4967)
  • Add fulltext index C++ SDK support #27727
• Add UTF8 support
  • Support UTF8 in fulltext analyzers #26850
  • Support Utf8 as a fulltext column #27892
• Implement SHOW CREATE TABLE (see create_table_formatter.cpp)
  • Show create table supports fulltext index #27956
• Support multiple key columns
  • Support filtered fulltext index
  • Support multiple fulltext columns
• Support languages
  • Support stopwords
  • Support morphology