0.22.7

We are pleased to announce Dolt 0.22.7.

This release focuses on bug fixes, and performance improvements in SQL. In particular delivered huge performance improvements in our SQL implementation. You can find the scope of these performance improvements detailed on our benchmarks page.

Merged PRs

• 1116: partition ranges, covering indexes, smarter iterators
• 1111: README quotes changed bugfix for windows terminal
  On the README there are instructions on how to add values into a table. The values in one portion have single quotes on the outside and have double-quotes for any string. While that format works in a Unix terminal, it doesn't work in
• 1109: Attempt to add default decimal type to FromKind
• 1108: Fixed dolt status output incorrectly displayed for staged files
  The function printStagedDiffs always returned 0, even when there were diffs not staged. This return was also causing it to print in printStatus "nothing to commit, working tree clean". This was not the case.
  I changed printStagedDiffs to return the number of the staged tables plus the number of staged docs instead. This prevents it from entering the if statement with the print also.
• 1107: go/libraries/doltcore/{row, sqle, table}: Generalize TableReader
  Created table.SqlTableReader as a replacement for directly reading from table maps. Used it to replace types.MapIterator in sqle.doltTableRowIter
• 1106: Added verify-constraints command
• 1105: /MySQLDockerfile: peg version to match Gemfile.lock BUNDLED WITH
• 1103: /go/cmd/dolt: added feature flag for keyless schemas
• 1102: go/libraries/doltcore/{doltdb,table}: remove row access methods from doltdb.Table
  Removed:
  • Table.GetRowByPKVals()
  • Table.GetRow()
  • Table.GetRows()
    Had do to some refactoring along the way to fix dependency cycles.
    Reversed dependency rowconv -> pipeline to pipeline -> rowconv
• 1101: Ensure that MERGE() works properly with fast forward.
  Added test case as well.
• 1098: Export NewJSONReader to use in dolthubapi
• 1097: Fixed table import allowing NULLs in the primary key
  Fixes #1096
• 1093: go/libraries/doltcore/table/editor: Convert TableEditor to interface
• 1090: Add --author, -m to COMMIT. Add --author to MERGE()
  COMMIT('-m', 'hi', '--author', 'John Doe [email protected]')
  MERGE('feature-branch', '--author', 'John Doe [email protected]')
• 1089: Add the Dolt mascot to README
• 1088: fixed rand seed
• 1087: increase query parallelism from the default of 2 to 8
• 1085: split TableEditors and IndexEditor to their own package
• 1084: bats/: keyless spec
  This is a set of skipped BATS tests that provide a spec for keyless tables.
• 1082: Fixed internal index comparisons considering unnecessary parameters
  Fixes #1081
• 1080: Fixed shell error loop on UNIQUE violation
  Fixes #1079
• 1078: Upgraded to latest go-mysql-server with support for indexed joins on any number of tables
• 1075: Support for CURRENT_USER SQL function without ()
• 1074: Add dolt_commit error check when autocommit is off
  Fails loudly when autocommit is off for dolt_commit.
• 245: Fixed tuple comparisons
• 240: Enginetests for Keyless tables
• 239: naked functions
  Fix for naked CURRENT_USER function call was in vitess, this just adds tests.

Closed Issues

• 1099: MERGE() is creating a new commit on FFs.
• 1096: Table import can allow NULLs in the primary key
• 1081: "string is too large for column"
• 1079: Indefinitely errors in SQL shell once a UNIQUE constraint has been violated
• 1071: Throw error in DOLT_COMMIT if autocommit is not true
• 241: expression.Tuple is uncomparable

0.22.6

Merged PRs

• 1068: Add -a flag to cli and DOLT_COMMIT
  This pr adds a -a flag to dolt commit and DOLT_COMMIT. It stages all tables.
  It also cleans up some of the previous work done in #1056 by removing all method handlers in repo_state and moving them to the RepoStateReader and RepoStateWriter
  It does not refactor the RSR/RSW interfaces in environment.go. This will be done in a subsequent pr.

Closed Issues

• 1067: Support dolt commit -a

0.22.5

This is a patch release that adds no new features or bug fixes.

Merged PRs

• 1065: fix typo in GA for Homebrew
• 1064: Fix brew formula

0.22.3

Merged PRs

• 1062: Updated go-mysql-server with a patch to fix failing function expressions
• 238: Zachmu/funcs
  Got rid of all embedded function fields in function types, since they make it impossible for analyzer to finish (function fields are not comparable with reflect.DeepEquals, which the analyzer uses to decide if the query plan has settled).

0.22.2

Merged PRs

• 1059: pass in-memory gc gen when we conjoin
  On the conjoin path, we're not passing "garbage collection generation" when we update the manifest. NomsBlockStore interprets this as the c
  onjoin having been preempted by and out-of-process write and blocks the write.
• 1052: Vinai/dolt commit author no config
  Add a bats test that models the following behavior.
  1. Unsets name and user.
  2. Makes a sql change
  3. Add a commit with --author

0.22.1

Merged PRs

• 1045: Conslidated benchmark directory
• 1043: Vinai/1034 add author info
  This adds the --author tag to dolt commit
• 1042: Vinai/clean up tags
  Cleans up some of the comments left on #1041
• 1041: Vinai/1023 remove tag info
  Fixes #1022 and #1023
• 1040: go/libraries/doltcore/remotestorage: Add hedged download requests to remotestorage/chunk_store.
• 1039: go/libraries/doltcore/remotestorage: Refactor urlAndRanges a bit.
• 1038: go/libraries/doltcore/remotestorage: Simplify concurrentExec implementation with errgroup.
• 1037: proto: Add StreamDownloadLocations to ChunkStoreService.
• 1036: go/cmd/dolt/commands: added write queries and ancestor commit to filter-branch
• 1033: Temporary parallelism implementation on indexes
• 1031: reset --hard
• 1029: Added dolt_version()
  As version() is used to emulate the target MySQL version, I've added dolt_version() so that one may specifically query the dolt version.
• 1026: Increased the default sql server timeout to 8 hours
• 1025: go/libraries/doltcore/remotestorage: chunk_store.go: Small improvements to GetDownloadLocations batch size and HTTP GET error logging.
• 1024: dolt filter-branch
• 1022: Skipping tests broken by recent changes to info schema (EXTRA)
• 1021: != operator now uses indexes

Closed Issues

• 1034: Add --author option to dolt commit
• 1023: Remove tag info from EXTRA in dolt SQL schema

0.22.0

We are excited to announce the minor version release of Dolt 0.22.0.

SQL Tables

We continue to expand the SQL tables that surface information about the commit graph, in this release we added:

• dolt_commits
• dolt_commit_ancestors
• dolt_commit_diffs_<table>

SQL

We added support for prepared statements to our SQL server.

Merged PRs

• 1019: Fix dolt ls --all
• 1018: mysql-client-tests: Add some simple client connector tests for prepared statements.
• 1016: Rewrote the README
• 1015: go/go.mod: Bump go-mysql-server; support prepared statements.
• 1014: Added bats test for index merging from branch without index
• 1013: dolt_commits and dolt commit_ancestors tables
• 1012: added reset_hard() sql function
• 1011: Bh/commit diff
• 1009: Richer commit message for Dolt Homebrew bump
• 1008: Mergeable Indexes Pt. 2
  Tests for mergeable indexes
• 1002: s/liquidata-inc/dolthub/ for ishell and mmap-go
• 1001: NewCreatingWriter breaks dolthubapi with recent changes
  There might be a better fix for this, but dolthubapi uses NewCreatingWriter which breaks with Andy's recent changes (it's being used in dolthubapi here)
• 233: Reorder Master
• 232: Indexes search for exact match rather than submatches
• 231: added 'auto_increment' to EXTRA field for DESCRIBE table;
• 229: Add support for prepared statements.

Closed Issues

• 1007: dolt push does not seem to push correctly in Windows Powershell
• 1003: AUTO_INCREMENT column info does not display in describe table output

0.21.4

Merged PRs

• 999: Another fix to brew bump job
• 997: Fix typo in brew bump
  From failed run on most recent release:
  

0.21.2

Merged PRs

• 995: support for ALTER TABLE AUTO_INCREMENT
• 994: Updated namespace for sqllogictest
• 993: Added WSL notice to README
• 990: mysql auto increment semantics
• 989: Fix a few docs typos
• 988: {bats, go}: Some fixes to InferSchema and add bats test
• 987: Turbine Import Fix
• 985: go/**/*.go: Update copyright headers for company name change.
• 982: go/libraries/utils/async: Have ActionExecutor use sync.WaitGroup.
• 981: Attempt to clean up error signaling in diff summary.
• 980: In prettyPrint, defer closing the iterator before doing anything else
  We were missing close() when an UPDATE or INSERT etc. had an error during cursor iteration, therefore leaving a server process running. Also save sql history file before executing the query, so it gets saved even if the user interrupts execution.
• 976: /.github/workflows/ci-go-tests.yaml: run go tests only when go/ changes
  I think this might be a good addition... will only run go tests when there are go changes
• 975: Extract some import logic to be used in dolthubapi
  In reference to this comment https://github.com/dolthub/ld/pull/5262#discussion_r514465176
  I had some duplicate logic in dolthubapi for the import table api. I extracted some logic so that I can use InferSchema and MoveDataToRoot to root to reduce some of the duplications
• 974: Skipped two newly added test queries that don't work in dolt yet
• 973: Support for COM_LIST_FIELDS, fixed SHOW INDEXES
• 972: Update README.md
  Removed errant Liquidata reference
• 971: Added GitHub workflow tests for race conditions
  Will fail until #967 is merged into master, however the workflow only works when the PR is based against master. Therefore this PR does not target the aforementioned PR's branch.
• 970: Memory fix for CREATE INDEX
  Used a pre-existing 16 million row repo to test CREATE INDEX memory usage on.
  Before:
  72.47GB RAM Usage
  18min 48sec
  After:
  1.88GB RAM Usage
  2min 2sec
  Copied the same strategy as used in table_editor.go to periodically flush the contents once some arbitrary amount of operations have been performed.
• 967: go: Make all tests pass under -race.
• 966: go/store/types/edits: Rework AsyncSortedEdits to use errgroup, and a transient goroutine for each work item.
• 965: dolt merge --no-ff
• 225: Andy/mysql auto increment
• 224: Zachmu/xx
  Use xxhash everywhere, and standardize the construction of hash keys.
• 223: Zachmu/in subquery
  Implemented hashed lookups for IN (SELECT ... ) expressions. This is about 5x faster than using indexed lookups into the subquery table in tests.
  In a followup I'm going to replace the existing CRC64 hashing with xxhash everywhere it's used, so we're back to a single hash function.
• 221: Fixed bug in delete and update caused by indexes being pushed down to tables
• 220: Support for COM_LIST_FIELDS, fixed SHOW INDEXES
• 219: Zachmu/turbine perf
  1. Do pushdown analysis within subqueries
  2. Push index lookups down to tables in subqueries
• 218: Fix unit tests to run with -race.
• 217: validate auto_increment on in-line and out-of-line PR defs

Closed Issues

• 978: Support UNIQUE in CREATE TABLE statements, not just in CREATE INDEX statements
• 962: Index creation must not be limited by working memory
• 961: UNIQUE does not work on field level
• 959: Cannot create UNIQUE index on FK fields Dolt considers it duplicate

0.21.1

We are excited to announce the release of Dolt 0.21.1, a patch release with functionality and performance improvements.

Benchmarks

A significant new aspect of the Dolt release process will be providing SQL benchmarks. You can read a blog about our approach to benchmarking using sysbench here, and you can find the benchmarking tools here. By way of example the benchmarks for this release were created with the following command:

./run_benchmarks.sh bulk_insert oscarbatori v0.21.0 v0.22.1

This produced the following result, which we host on DoltHub:

Merged PRs

• 957: go/store/{datas,util/tempfiles}: Fix some races in map writes. One effects clones, one effects only tests.
• 953: create auto_increment tables with out-of-line PR defs
• 952: go/libraries/doltcore/sqle: Add support for UPDATE and DELETE using table indexes.
• 949: auto increment
• 947: don't drop column values on column drop
• 946: go/cmd/dolt: commands/sql: Small improvement to only call rowIter.Close() once on sql results iterators.
• 945: Use docker-compose for orchestrating benchmarking
• 944: go/store/types: value_store: Optimize GC to work in parallel and use less memory.
• 942: feature gating
• 941: Upgraded to latest go-mysql-server and re-enabled query plan tests
• 939: Added new indexes overwriting auto-generated indexes
• 938: go/store/{nbs,chunks}: Convert some core methods to provide results in callbacks. Convert some functions to use errgroup.
• 937: Update README.md with the latest dolt commands
• 934: Add go routine to clone
  I parallelized the table file writing process by using go routines. Specifically, I made use of the "golang.org/x/sync/errgroup" package which allows for convenient error management across a waitgroup.
  A couple of benchmarks I tested this on were
  1. Dolt-benchmarks-test: No difference in speed really
  2. Coronavirus: Original ~30sec. Current 15sec
  3. Tatoeba Sentence Translation: Original: ~17mins Current: 10mins
• 933: /go/libraries/doltcore/diff: Ignore NULLs in cell-wise diff
  fix for #899
  The from root in this repo has NULL values written to the map which causes erroneous diffs.
  https://www.dolthub.com/repositories/dolthub/us-supreme-court-cases/compare/master/hb502v6tf3uj43ijfhot6dopmgdm1muk
• 932: /go/cmd/dolt/commands: Help Text Fix
• 216: Updated sql.MergeableIndexLookup interface
• 215: memory: *_index.go: Construct sql equality evaluations with accurate types in the literals.
• 214: auto increment
• 213: triggers bugfix
  Fixed bug in insert triggers, which couldn't handle out-of-order column insertions.
  Fixes #950
• 212: sql/analyzer: pushdown.go: Allow pushdown on Update, RowUpdateAccumulator and DeleteFrom plan nodes.
• 211: join bugs
• 210: sql/plan: {update,insert,update,process}.go: Fix some potential issues with context lifecycle and reuse.
  • insert, update, delete: Only call underlying table editors with our captured
    context once when we are Close(). Return a nil error after that.
  • process: Change to only call onDone when the rowTrackingIter is Closed.
  • process: Change to call childIter.Close() before onDone is called. Child
    iterators have a right to Close() before the context in which they are
    running is canceled.
• 208: Create UNIQUE index if present in column definition
• 207: Pushdown and plan decoration
  Two major changes:
  1. Changes to pushdown operation, to push table predicates below join nodes and to fix many bugs and deficiencies. Also large refactoring.
  2. Added DecoratedNodes to query plans to illustrate when indexes are being used to access tables outside the context of a join