Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] v18.x #125

Open
wants to merge 24 commits into
base: release-18.0
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6474aae
add new lock syntax for mysql8
patrickcarnahan Aug 7, 2023
ffd4555
Add shutdown state in MySQL server plugin
davidpiegza Sep 25, 2023
85e0bb3
Expose the `--tablet_types_to_wait` flag in `vtcombo`.
arthurschreiber Feb 19, 2024
856c299
Add hotfix to make vitess compile under go 1.23
andyedison Aug 26, 2024
7792591
Only run on larger runners when running inside the `vitessio` organiz…
arthurschreiber Oct 9, 2024
03bf1e5
Merge branch 'release-18.0' of https://github.com/vitessio/vitess int…
arthurschreiber Oct 14, 2024
3cfefd4
logstats: do not allocate memory while logging (#15539)
vmg Mar 25, 2024
2aa50d2
Merge branch 'release-18.0' of https://github.com/github/vitess-gh in…
arthurschreiber Oct 17, 2024
20fd074
Merge branch 'release-18.0' of https://github.com/vitessio/vitess int…
arthurschreiber Oct 28, 2024
d112b3f
fix: use related semtable for each operator to sql builder conversion
harshit-gangal Oct 28, 2024
f30abe0
Merge branch 'release-18.0' of https://github.com/vitessio/vitess int…
arthurschreiber Oct 29, 2024
d18e303
Flakes: De-flake TestGatewayBufferingWhenPrimarySwitchesServingState …
mattlord Jan 17, 2024
5d2abca
VReplication: Improve query buffering behavior during MoveTables traf…
mattlord Apr 25, 2024
286e1ea
Filter by keyspace earlier in `tabletgateway`s `WaitForTablets(...)` …
timvaillancourt Feb 27, 2024
7d79407
Fix race condition that prevents queries from being buffered after vt…
GuptaManan100 Aug 29, 2024
e24e5d9
test: add a test for premature buffering
GuptaManan100 Oct 19, 2024
01124f2
test: update test to reproduce the buffering problem
GuptaManan100 Oct 21, 2024
21a8ad2
test: uncomment lines that were accidentally commented
GuptaManan100 Oct 21, 2024
0c87ea7
feat: fix the problem by adding a new field in the shard state
GuptaManan100 Oct 25, 2024
517f519
test: fix test to also send a higher primary timestamp
GuptaManan100 Oct 25, 2024
d77fae7
feat: add tests for processing of health checks and also fix a bug
GuptaManan100 Oct 25, 2024
b0e2f64
test: add more unit tests
GuptaManan100 Oct 25, 2024
fd8c3cb
Appease the linter.
arthurschreiber Nov 8, 2024
eb53017
Merge pull request #128 from github/arthur/buffering-backports
arthurschreiber Nov 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/cluster_endtoend_onlineddl_vrepl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl) mysql57
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl_stress)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl_stress) mysql57
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl_stress_suite)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl_stress_suite) mysql57
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl_suite)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (onlineddl_vrepl_suite) mysql57
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/cluster_endtoend_vreplication_basic.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (vreplication_basic)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (vreplication_migrate_vdiff2_convert_tz)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/docker_build_base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ permissions: read-all
jobs:
build_and_push_base:
name: Build and push vitess/base Docker images
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
if: github.repository == 'vitessio/vitess'

strategy:
Expand Down Expand Up @@ -77,7 +77,7 @@ jobs:
build_and_push_k8s:
needs: build_and_push_base
name: Build and push vitess/k8s image
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
if: github.repository == 'vitessio/vitess'

strategy:
Expand Down Expand Up @@ -170,7 +170,7 @@ jobs:
build_and_push_components:
needs: build_and_push_k8s
name: Build and push vitess components Docker images
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
if: github.repository == 'vitessio/vitess'

strategy:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/docker_build_lite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ permissions: read-all
jobs:
build_and_push:
name: Build and push vitess/lite Docker images
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
if: github.repository == 'vitessio/vitess'

strategy:
Expand Down Expand Up @@ -70,4 +70,4 @@ jobs:
context: .
file: ${{ env.DOCKERFILE }}
push: true
tags: ${{ env.DOCKER_TAG }}
tags: ${{ env.DOCKER_TAG }}
4 changes: 2 additions & 2 deletions .github/workflows/docker_build_vttestserver.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ permissions: read-all
jobs:
build_and_push:
name: Build and push vitess/vttestserver Docker images
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
if: github.repository == 'vitessio/vitess'

strategy:
Expand Down Expand Up @@ -62,4 +62,4 @@ jobs:
context: .
file: ${{ env.DOCKERFILE }}
push: true
tags: ${{ env.DOCKER_TAG }}
tags: ${{ env.DOCKER_TAG }}
4 changes: 2 additions & 2 deletions .github/workflows/local_example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ jobs:

build:
name: Local example using ${{ matrix.topo }} on ubuntu-22.04
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
strategy:
matrix:
topo: [consul,etcd,zk2]
Expand Down Expand Up @@ -81,7 +81,7 @@ jobs:
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
sudo apt-get update

# Install everything else we need, and configure
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/region_example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ jobs:

build:
name: Region Sharding example using ${{ matrix.topo }} on ubuntu-22.04
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
strategy:
matrix:
topo: [etcd]
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unit_race.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:

build:
name: Unit Test (Race)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}
steps:
- name: Skip CI
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/upgrade_downgrade_test_backups_e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
upgrade_downgrade_test_e2e:
timeout-minutes: 60
name: Run Upgrade Downgrade Test - Backups - E2E
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
upgrade_downgrade_test_e2e:
timeout-minutes: 60
name: Run Upgrade Downgrade Test - Backups - E2E - Next Release
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
upgrade_downgrade_test_manual:
timeout-minutes: 40
name: Run Upgrade Downgrade Test - Backups - Manual
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
upgrade_downgrade_test_manual:
timeout-minutes: 40
name: Run Upgrade Downgrade Test - Backups - Manual - Next Release
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Query Serving (Queries)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Query Serving (Queries) Next Release
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Query Serving (Schema)
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Query Serving (Schema) Next Release
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Reparent New Vtctl
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Reparent New VTTablet
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Reparent Old Vtctl
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:

upgrade_downgrade_test:
name: Run Upgrade Downgrade Test - Reparent Old VTTablet
runs-on: gh-hosted-runners-16cores-1
runs-on: ${{ github.repository_owner == 'vitessio' && 'gh-hosted-runners-16cores-1' || 'ubuntu-latest' }}

steps:
- name: Skip CI
Expand Down
30 changes: 22 additions & 8 deletions go/cmd/vtcombo/cli/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ import (
"vitess.io/vitess/go/vt/srvtopo"
"vitess.io/vitess/go/vt/topo"
"vitess.io/vitess/go/vt/topo/memorytopo"
"vitess.io/vitess/go/vt/topo/topoproto"
"vitess.io/vitess/go/vt/topotools"
"vitess.io/vitess/go/vt/vtcombo"
"vitess.io/vitess/go/vt/vtctld"
Expand Down Expand Up @@ -76,9 +77,10 @@ In particular, it contains:
plannerName string
vschemaPersistenceDir string

tpb vttestpb.VTTestTopology
ts *topo.Server
resilientServer *srvtopo.ResilientServer
tpb vttestpb.VTTestTopology
ts *topo.Server
resilientServer *srvtopo.ResilientServer
tabletTypesToWait []topodatapb.TabletType
)

func init() {
Expand Down Expand Up @@ -110,6 +112,7 @@ func init() {

Main.Flags().Var(vttest.TextTopoData(&tpb), "proto_topo", "vttest proto definition of the topology, encoded in compact text format. See vttest.proto for more information.")
Main.Flags().Var(vttest.JSONTopoData(&tpb), "json_topo", "vttest proto definition of the topology, encoded in json format. See vttest.proto for more information.")
Main.Flags().Var((*topoproto.TabletTypeListFlag)(&tabletTypesToWait), "tablet_types_to_wait", "Wait till connected for specified tablet types during Gateway initialization. Should be provided as a comma-separated set of tablet types.")

// We're going to force the value later, so don't even bother letting the
// user know about this flag.
Expand Down Expand Up @@ -273,18 +276,29 @@ func run(cmd *cobra.Command, args []string) (err error) {

// vtgate configuration and init
resilientServer = srvtopo.NewResilientServer(context.Background(), ts, "ResilientSrvTopoServer")
tabletTypesToWait := []topodatapb.TabletType{
topodatapb.TabletType_PRIMARY,
topodatapb.TabletType_REPLICA,
topodatapb.TabletType_RDONLY,

tabletTypes := make([]topodatapb.TabletType, 0, 1)
if len(tabletTypesToWait) != 0 {
for _, tt := range tabletTypesToWait {
if topoproto.IsServingType(tt) {
tabletTypes = append(tabletTypes, tt)
}
}

if len(tabletTypes) == 0 {
log.Exitf("tablet_types_to_wait should contain at least one serving tablet type")
}
} else {
tabletTypes = append(tabletTypes, topodatapb.TabletType_PRIMARY, topodatapb.TabletType_REPLICA, topodatapb.TabletType_RDONLY)
}

plannerVersion, _ := plancontext.PlannerNameToVersion(plannerName)

vtgate.QueryLogHandler = "/debug/vtgate/querylog"
vtgate.QueryLogzHandler = "/debug/vtgate/querylogz"
vtgate.QueryzHandler = "/debug/vtgate/queryz"
// pass nil for healthcheck, it will get created
vtg := vtgate.Init(context.Background(), nil, resilientServer, tpb.Cells[0], tabletTypesToWait, plannerVersion)
vtg := vtgate.Init(context.Background(), nil, resilientServer, tpb.Cells[0], tabletTypes, plannerVersion)

// vtctld configuration and init
err = vtctld.InitVtctld(ts)
Expand Down
1 change: 1 addition & 0 deletions go/flags/endtoend/vtcombo.txt
Original file line number Diff line number Diff line change
Expand Up @@ -348,6 +348,7 @@ Flags:
--tablet_manager_protocol string Protocol to use to make tabletmanager RPCs to vttablets. (default "grpc")
--tablet_refresh_interval duration Tablet refresh interval. (default 1m0s)
--tablet_refresh_known_tablets Whether to reload the tablet's address/port map from topo in case they change. (default true)
--tablet_types_to_wait strings Wait till connected for specified tablet types during Gateway initialization. Should be provided as a comma-separated set of tablet types.
--tablet_url_template string Format string describing debug tablet url formatting. See getTabletDebugURL() for how to customize this. (default "http://{{ "{{.GetTabletHostPort}}" }}")
--throttle_tablet_types string Comma separated VTTablet types to be considered by the throttler. default: 'replica'. example: 'replica,rdonly'. 'replica' aways implicitly included (default "replica")
--topo_consul_lock_delay duration LockDelay for consul session. (default 15s)
Expand Down
Loading
Loading