Skip to content

Commit 4b22260

Browse files
kyleconroyclaude
kyleconroy
and
claude
authored
Add MySQL to goldeneye: generate relations.jsonl and check the analyze cases against a live server (#4616)
* goldeneye: add MySQL, generating relations.jsonl and checking the analyze cases Add a mysql package to goldeneye that reads a live server named by MYSQL_SERVER_URI. MySQL keeps no catalog of its types, functions or operators, so those files stay hand-written; what the server does describe is its data dictionary, so relations.jsonl is generated from information_schema, with names in lower case since MySQL matches them in any case and sqlc's parser lowercases every identifier. The other system schemas are tables rather than views, which the analysis core would hand codegen as models, so they are left out for now. The dialect lives under internal/engine/dolphin, so the command gains a dialect directory distinct from the engine name. The package also checks the analyze_*/mysql cases against the server. Result columns come from the result set's metadata as go-sql-driver reports it; provenance and parameters come from the optimizer trace's expanded_query, which prints each block after resolution and before optimisation, and from the note EXPLAIN leaves for the statements the trace does not expand. Views and derived tables are kept unmerged so a column read through one is still its column. A column read from a table is spelled by its declaration, since the driver hides the length that tells the sizes of TEXT apart. Making sqlc agree: MySQL reports its aggregates as nullable, so the aggregates that are NULL over no rows are marked so in functions.jsonl, which changes the committed analyze_select output; "bigint unsigned" and the other unsigned spellings become aliases of their types so the generated relations resolve; and an analyze_system_catalog case covers querying information_schema. The shared placeholder rewriter learns sqlc.slice and the second count of LIMIT ?, ?. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012DTeySVan5NcA3RS6xxdUn * goldeneye: generate the MySQL dialect from MySQL 26 The gen workflow runs the mysql:26.7 image, so the pinned major is 26 and relations.jsonl is regenerated from 26.7.0, which adds the JSON duality view and library views to information_schema. EXPLAIN defaults to the tree format since MySQL 26, which leaves no rewritten statement behind, so the analysis check asks for the traditional format. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012DTeySVan5NcA3RS6xxdUn * Standardize the test databases on MySQL 26.7 sqlc-test-setup pins the release it installs in one place, so the deb bundle it downloads, the version it accepts as already installed, and the docker-compose service all name MySQL 26.7, the same release goldeneye generates the dialect from and its gen workflow runs. The installed-version check compared the first digit of the major release, which would have called 26 older than 9; it compares the number now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012DTeySVan5NcA3RS6xxdUn --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4d12e22 commit 4b22260

23 files changed

Lines changed: 2081 additions & 64 deletions

File tree

‎.github/workflows/gen.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,35 @@ jobs:
3434
- name: Fail if the committed dialect differs
3535
run: git add -N internal/engine/postgresql && git diff --exit-code --stat -- internal/engine/postgresql
3636

37+
mysql:
38+
name: generate mysql dialect
39+
runs-on: ubuntu-24.04
40+
services:
41+
mysql:
42+
image: mysql:26.7
43+
env:
44+
MYSQL_ROOT_PASSWORD: mysecretpassword
45+
ports:
46+
- 3306:3306
47+
options: --health-cmd "mysqladmin ping -h 127.0.0.1 -pmysecretpassword" --health-interval 10s --health-timeout 5s --health-retries 5
48+
steps:
49+
- uses: actions/checkout@v7
50+
- uses: actions/setup-go@v7
51+
with:
52+
go-version-file: internal/goldeneye/go.mod
53+
check-latest: true
54+
- run: go run ./cmd/goldeneye generate mysql
55+
working-directory: internal/goldeneye
56+
env:
57+
MYSQL_SERVER_URI: root:mysecretpassword@tcp(localhost:${{ job.services.mysql.ports['3306'] }})/mysql
58+
- name: Save results
59+
uses: actions/upload-artifact@v7
60+
with:
61+
name: dialect-mysql
62+
path: internal/engine/dolphin/dialect
63+
- name: Fail if the committed dialect differs
64+
run: git add -N internal/engine/dolphin && git diff --exit-code --stat -- internal/engine/dolphin
65+
3766
clickhouse:
3867
name: generate clickhouse dialect
3968
runs-on: ubuntu-24.04

‎CLAUDE.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ go run ./cmd/sqlc-test-setup install
2323
This will:
2424
- Configure the apt proxy (if `http_proxy` is set, e.g. in Claude Code remote environments)
2525
- Install PostgreSQL via apt
26-
- Download and install MySQL 9 from Oracle's deb bundle
26+
- Download and install MySQL 26.7 from Oracle's deb bundle
2727
- Resolve all dependencies automatically
2828
- Skip anything already installed
2929

@@ -150,14 +150,15 @@ from a live database by `/internal/goldeneye`, a nested module, and its tests
150150
verify the committed files against one byte for byte. The same module checks
151151
the `analyze_*` cases under `/internal/endtoend/testdata/` against what the
152152
database itself reports for them, so a `fixture.sql` next to a case's schema
153-
gives the queries rows to run against. ClickHouse and SQLite have the check
154-
today; engines whose database is not available skip.
153+
gives the queries rows to run against. ClickHouse, MySQL and SQLite have the
154+
check today; engines whose database is not available skip.
155155

156156
```bash
157157
cd internal/goldeneye
158158
go run ./cmd/goldeneye install clickhouse # download the pinned clickhouse binary once
159159
go run ./cmd/goldeneye install sqlite # build the pinned sqlite3 shells once; needs a C compiler
160-
POSTGRESQL_SERVER_URI="postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable" go test ./...
160+
POSTGRESQL_SERVER_URI="postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable" \
161+
MYSQL_SERVER_URI="root:mysecretpassword@tcp(127.0.0.1:3306)/mysql" go test ./...
161162
go run ./cmd/goldeneye generate postgresql # rewrite the files after a change
162163
```
163164

@@ -180,7 +181,7 @@ The `docker-compose.yml` provides test databases:
180181
- Password: `mysecretpassword`
181182
- Database: `postgres`
182183

183-
- **MySQL 9** - Port 3306
184+
- **MySQL 26.7** - Port 3306
184185
- User: `root`
185186
- Password: `mysecretpassword`
186187
- Database: `dinotest`

‎cmd/sqlc-test-setup/main.go‎

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import (
1111
"os/exec"
1212
"path/filepath"
1313
"runtime"
14+
"strconv"
1415
"strings"
1516
"time"
1617
)
@@ -86,7 +87,16 @@ func commandExists(name string) bool {
8687
return err == nil
8788
}
8889

89-
// isMySQLVersionOK checks if the mysqld --version output indicates MySQL 9+.
90+
// The MySQL release the tests run against. Everything that names a MySQL
91+
// version — the docker-compose service, the goldeneye dialect generator and
92+
// its gen workflow — is pinned to the same release.
93+
const (
94+
mysqlMajor = 26
95+
mysqlVersion = "26.7.0"
96+
)
97+
98+
// isMySQLVersionOK checks if the mysqld --version output indicates the
99+
// pinned major release or a later one.
90100
// Example version string: "/usr/sbin/mysqld Ver 8.0.44-0ubuntu0.24.04.2 ..."
91101
func isMySQLVersionOK(versionOutput string) bool {
92102
// Look for "Ver X.Y.Z" pattern
@@ -95,11 +105,11 @@ func isMySQLVersionOK(versionOutput string) bool {
95105
if strings.EqualFold(f, "Ver") && i+1 < len(fields) {
96106
ver := strings.Split(fields[i+1], ".")
97107
if len(ver) > 0 {
98-
major := strings.TrimLeft(ver[0], "0")
99-
if major == "" {
108+
major, err := strconv.Atoi(ver[0])
109+
if err != nil {
100110
return false
101111
}
102-
return major[0] >= '9'
112+
return major >= mysqlMajor
103113
}
104114
}
105115
}
@@ -302,18 +312,18 @@ func sha256File(path string) (string, error) {
302312
}
303313

304314
func installMySQL() error {
305-
log.Println("--- Installing MySQL 9 ---")
315+
log.Printf("--- Installing MySQL %s ---", mysqlVersion)
306316

307317
if commandExists("mysqld") {
308318
out, err := runOutput("mysqld", "--version")
309319
if err == nil {
310320
version := strings.TrimSpace(out)
311321
log.Printf("mysql is already installed: %s", version)
312322
if isMySQLVersionOK(version) {
313-
log.Println("mysql version is 9+, skipping installation")
323+
log.Printf("mysql version is %d+, skipping installation", mysqlMajor)
314324
return nil
315325
}
316-
log.Println("mysql version is too old, upgrading to MySQL 9")
326+
log.Printf("mysql version is too old, upgrading to MySQL %s", mysqlVersion)
317327
// Stop existing MySQL before upgrading
318328
_ = exec.Command("sudo", "service", "mysql", "stop").Run()
319329
_ = exec.Command("sudo", "pkill", "-f", "mysqld").Run()
@@ -322,20 +332,22 @@ func installMySQL() error {
322332
log.Println("removing old mysql packages")
323333
_ = run("sudo", "apt-get", "remove", "-y", "mysql-server", "mysql-client", "mysql-common",
324334
"mysql-server-core-*", "mysql-client-core-*")
325-
// Clear old data directory so MySQL 9 can initialize fresh
335+
// Clear old data directory so the new release can initialize fresh
326336
log.Println("clearing old mysql data directory")
327337
_ = run("sudo", "rm", "-rf", "/var/lib/mysql")
328338
_ = run("sudo", "mkdir", "-p", "/var/lib/mysql")
329339
_ = run("sudo", "chown", "mysql:mysql", "/var/lib/mysql")
330340
}
331341
}
332342

333-
bundleURL := "https://dev.mysql.com/get/Downloads/MySQL-9.1/mysql-server_9.1.0-1ubuntu24.04_amd64.deb-bundle.tar"
334-
bundleTar := "/tmp/mysql-server-bundle.tar"
335-
extractDir := "/tmp/mysql9"
343+
major, minor, _ := strings.Cut(mysqlVersion, ".")
344+
minor, _, _ = strings.Cut(minor, ".")
345+
bundleURL := fmt.Sprintf("https://dev.mysql.com/get/Downloads/MySQL-%s.%s/mysql-server_%s-1ubuntu24.04_amd64.deb-bundle.tar", major, minor, mysqlVersion)
346+
bundleTar := fmt.Sprintf("/tmp/mysql-server-%s-bundle.tar", mysqlVersion)
347+
extractDir := "/tmp/mysql-server-" + mysqlVersion
336348

337349
if _, err := os.Stat(bundleTar); err != nil {
338-
log.Printf("downloading MySQL 9 bundle from %s", bundleURL)
350+
log.Printf("downloading MySQL %s bundle from %s", mysqlVersion, bundleURL)
339351
if err := run("curl", "-L", "-o", bundleTar, bundleURL); err != nil {
340352
return fmt.Errorf("downloading mysql bundle: %w", err)
341353
}
@@ -378,7 +390,7 @@ func installMySQL() error {
378390
return fmt.Errorf("apt-get install -f: %w", err)
379391
}
380392

381-
log.Println("mysql 9 installed successfully")
393+
log.Printf("mysql %s installed successfully", mysqlVersion)
382394
return nil
383395
}
384396

‎docker-compose.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
version: "3.8"
22
services:
33
mysql:
4-
image: "mysql:9"
4+
image: "mysql:26.7"
55
ports:
66
- "3306:3306"
77
restart: always

‎internal/endtoend/testdata/analyze_select/mysql/stdout.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,8 @@
153153
{
154154
"name": "created",
155155
"type": {
156-
"name": "datetime"
156+
"name": "datetime",
157+
"nullable": true
157158
}
158159
}
159160
],

‎internal/endtoend/testdata/analyze_system_catalog/mysql/exec.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"command": "analyze",
3+
"args": ["--dialect", "mysql", "--schema", "schema.sql", "query.sql"],
4+
"contexts": ["base"]
5+
}

‎internal/endtoend/testdata/analyze_system_catalog/mysql/query.sql‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
-- name: ListColumns :many
2+
SELECT table_name, column_name, data_type
3+
FROM information_schema.columns
4+
WHERE table_schema = ?;
5+
6+
-- name: CountTables :one
7+
SELECT count(*) AS total FROM information_schema.tables WHERE table_type = ?;

‎internal/endtoend/testdata/analyze_system_catalog/mysql/schema.sql‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
CREATE TABLE authors (
2+
id BIGINT PRIMARY KEY,
3+
name VARCHAR(255) NOT NULL
4+
);

‎internal/endtoend/testdata/analyze_system_catalog/mysql/stdout.json‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
[
2+
{
3+
"name": "ListColumns",
4+
"cmd": ":many",
5+
"columns": [
6+
{
7+
"name": "table_name",
8+
"type": {
9+
"name": "varchar"
10+
},
11+
"table": "columns"
12+
},
13+
{
14+
"name": "column_name",
15+
"type": {
16+
"name": "varchar",
17+
"nullable": true
18+
},
19+
"table": "columns"
20+
},
21+
{
22+
"name": "data_type",
23+
"type": {
24+
"name": "longtext",
25+
"nullable": true
26+
},
27+
"table": "columns"
28+
}
29+
],
30+
"params": [
31+
{
32+
"number": 1,
33+
"column": {
34+
"name": "table_schema",
35+
"type": {
36+
"name": "varchar"
37+
},
38+
"table": "columns"
39+
}
40+
}
41+
]
42+
},
43+
{
44+
"name": "CountTables",
45+
"cmd": ":one",
46+
"columns": [
47+
{
48+
"name": "total",
49+
"type": {
50+
"name": "bigint"
51+
}
52+
}
53+
],
54+
"params": [
55+
{
56+
"number": 1,
57+
"column": {
58+
"name": "table_type",
59+
"type": {
60+
"name": "enum"
61+
},
62+
"table": "tables"
63+
}
64+
}
65+
]
66+
}
67+
]

‎internal/engine/dolphin/dialect/functions.jsonl‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
{"name":"ASIN","args":[{"type":"double precision"}],"returns":"double precision"}
1818
{"name":"ATAN","args":[{"type":"double precision"}],"returns":"double precision"}
1919
{"name":"ATAN2","args":[{"type":"double precision"}],"returns":"double precision"}
20-
{"name":"AVG","args":[{"type":"any"}],"returns":"any"}
20+
{"name":"AVG","args":[{"type":"any"}],"returns":"any","nullable":true}
2121
{"name":"BENCHMARK","args":[{"type":"int"},{"type":"any"}],"returns":"int"}
2222
{"name":"BIN","args":[{"type":"int"}],"returns":"text"}
2323
{"name":"BIN_TO_UUID","args":[{"type":"binary"}],"returns":"text"}
@@ -130,7 +130,7 @@
130130
{"name":"IS_USED_LOCK","args":[{"type":"text"}],"returns":"bool"}
131131
{"name":"IS_UUID","args":[{"type":"text"}],"returns":"bool"}
132132
{"name":"JSON_ARRAY","args":[{"type":"any","mode":"v"}],"returns":"json"}
133-
{"name":"JSON_ARRAYAGG","args":[{"type":"any"}],"returns":"json"}
133+
{"name":"JSON_ARRAYAGG","args":[{"type":"any"}],"returns":"json","nullable":true}
134134
{"name":"JSON_ARRAY_APPEND","args":[{"type":"text"},{"type":"text"},{"type":"any"},{"type":"any","mode":"v"}],"returns":"json"}
135135
{"name":"JSON_ARRAY_INSERT","args":[{"type":"text"},{"type":"text"},{"type":"any"},{"type":"any","mode":"v"}],"returns":"json"}
136136
{"name":"JSON_CONTAINS","args":[{"type":"text"},{"type":"text"}],"returns":"bool"}
@@ -147,7 +147,7 @@
147147
{"name":"JSON_MERGE_PATCH","args":[{"type":"text"},{"type":"text"},{"type":"text","mode":"v"}],"returns":"json"}
148148
{"name":"JSON_MERGE_PRESERVE","args":[{"type":"text"},{"type":"text"},{"type":"text","mode":"v"}],"returns":"json"}
149149
{"name":"JSON_OBJECT","args":[{"type":"any","mode":"v"}],"returns":"json"}
150-
{"name":"JSON_OBJECTAGG","args":[{"type":"any"},{"type":"any"}],"returns":"json"}
150+
{"name":"JSON_OBJECTAGG","args":[{"type":"any"},{"type":"any"}],"returns":"json","nullable":true}
151151
{"name":"JSON_OVERLAPS","args":[{"type":"text"},{"type":"text"}],"returns":"bool"}
152152
{"name":"JSON_PRETTY","args":[{"type":"text"}],"returns":"any"}
153153
{"name":"JSON_QUOTE","args":[{"type":"text"}],"returns":"text"}
@@ -196,7 +196,7 @@
196196
{"name":"MASTER_POS_WAIT","args":[{"type":"text"},{"type":"int"}],"returns":"int"}
197197
{"name":"MASTER_POS_WAIT","args":[{"type":"text"},{"type":"int"},{"type":"int"}],"returns":"int"}
198198
{"name":"MASTER_POS_WAIT","args":[{"type":"text"},{"type":"int"},{"type":"int"},{"type":"text"}],"returns":"int"}
199-
{"name":"MAX","args":[{"type":"any"}],"returns":"any"}
199+
{"name":"MAX","args":[{"type":"any"}],"returns":"any","nullable":true}
200200
{"name":"MBRCONTAINS","args":[{"type":"any"},{"type":"any"}],"returns":"bool"}
201201
{"name":"MBRCOVEREDBY","args":[{"type":"any"},{"type":"any"}],"returns":"bool"}
202202
{"name":"MBRCOVERS","args":[{"type":"any"},{"type":"any"}],"returns":"bool"}
@@ -209,7 +209,7 @@
209209
{"name":"MD5","args":[{"type":"text"}],"returns":"text"}
210210
{"name":"MICROSECOND","args":[{"type":"time"}],"returns":"int"}
211211
{"name":"MID","args":[{"type":"text"},{"type":"int"},{"type":"int"}],"returns":"text"}
212-
{"name":"MIN","args":[{"type":"any"}],"returns":"any"}
212+
{"name":"MIN","args":[{"type":"any"}],"returns":"any","nullable":true}
213213
{"name":"MINUTE","args":[{"type":"time"}],"returns":"int"}
214214
{"name":"MOD","args":[{"type":"int"},{"type":"int"}],"returns":"int"}
215215
{"name":"MONTH","args":[{"type":"date"}],"returns":"int"}
@@ -277,10 +277,10 @@
277277
{"name":"SQRT","args":[{"type":"double precision"}],"returns":"double precision"}
278278
{"name":"STATEMENT_DIGEST","args":[{"type":"text"}],"returns":"text"}
279279
{"name":"STATEMENT_DIGEST_TEXT","args":[{"type":"text"}],"returns":"text"}
280-
{"name":"STD","args":[{"type":"any"}],"returns":"any"}
281-
{"name":"STDDEV","args":[{"type":"any"}],"returns":"any"}
282-
{"name":"STDDEV_POP","args":[{"type":"any"}],"returns":"any"}
283-
{"name":"STDDEV_SAMP","args":[{"type":"any"}],"returns":"any"}
280+
{"name":"STD","args":[{"type":"any"}],"returns":"any","nullable":true}
281+
{"name":"STDDEV","args":[{"type":"any"}],"returns":"any","nullable":true}
282+
{"name":"STDDEV_POP","args":[{"type":"any"}],"returns":"any","nullable":true}
283+
{"name":"STDDEV_SAMP","args":[{"type":"any"}],"returns":"any","nullable":true}
284284
{"name":"STRCMP","args":[{"type":"text"},{"type":"text"}],"returns":"tinyint"}
285285
{"name":"STR_TO_DATE","args":[{"type":"text"},{"type":"text"}],"returns":"datetime"}
286286
{"name":"ST_AREA","args":[{"type":"any"}],"returns":"double precision"}
@@ -468,7 +468,7 @@
468468
{"name":"SUBSTRING","args":[{"type":"text"},{"type":"int"},{"type":"int"}],"returns":"text"}
469469
{"name":"SUBSTRING_INDEX","args":[{"type":"text"},{"type":"int"},{"type":"int"}],"returns":"text"}
470470
{"name":"SUBTIME","args":[{"type":"time"},{"type":"time"}],"returns":"time"}
471-
{"name":"SUM","args":[{"type":"any"}],"returns":"any"}
471+
{"name":"SUM","args":[{"type":"any"}],"returns":"any","nullable":true}
472472
{"name":"SYSDATE","returns":"datetime"}
473473
{"name":"SYSDATE","args":[{"type":"int"}],"returns":"datetime"}
474474
{"name":"SYSTEM_USER","returns":"text"}
@@ -505,9 +505,9 @@
505505
{"name":"UUID_TO_BIN","args":[{"type":"text"},{"type":"tinyint"}],"returns":"binary"}
506506
{"name":"VALIDATE_PASSWORD_STRENGTH","args":[{"type":"text"}],"returns":"int"}
507507
{"name":"VALUES","args":[{"type":"any"}],"returns":"any"}
508-
{"name":"VARIANCE","args":[{"type":"any"}],"returns":"any"}
509-
{"name":"VAR_POP","args":[{"type":"any"}],"returns":"any"}
510-
{"name":"VAR_SAMP","args":[{"type":"any"}],"returns":"any"}
508+
{"name":"VARIANCE","args":[{"type":"any"}],"returns":"any","nullable":true}
509+
{"name":"VAR_POP","args":[{"type":"any"}],"returns":"any","nullable":true}
510+
{"name":"VAR_SAMP","args":[{"type":"any"}],"returns":"any","nullable":true}
511511
{"name":"VERSION","returns":"text"}
512512
{"name":"WAIT_FOR_EXECUTED_GTID_SET","args":[{"type":"text"},{"type":"int","has_default":true}],"returns":"bool"}
513513
{"name":"WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS","args":[{"type":"text"},{"type":"int","has_default":true},{"type":"text","has_default":true}],"returns":"bool"}

0 commit comments

Comments
 (0)