Getting "failed to switch Protocols 403" error during tsh login

[vivekyad4v]

Hosting - Azure AKS + Gloo edge API gateway
Teleport version - 16.4.3

Error logs -

Enter an OTP code from a device:
2024-10-22T02:45:54Z DEBU [KEYSTORE]  Teleport TLS certificate valid until "2024-10-22 05:59:23 +0000 UTC". client/client_store.go:118
2024-10-22T02:45:54Z DEBU [KEYAGENT]  Deleting obsolete stored key with index {ProxyHost:=foo.bar Username:vivek ClusterName:foo.bar}. client
/keyagent.go:555
2024-10-22T02:45:55Z DEBU [KEYSTORE]  Adding known host tpt.uat.kodo.com with proxy foo.bar client/trusted_certs_store.go:395
2024-10-22T02:45:55Z INFO [KEYAGENT]  Loading SSH key for user "vivek" and cluster "foo.bar". client/keyagent.go:198
2024-10-22T02:45:55Z DEBU [KEYSTORE]  Teleport TLS certificate valid until "2024-10-22 14:45:54 +0000 UTC". client/client_store.go:118
2024-10-22T02:45:55Z DEBU [KEYSTORE]  Teleport TLS certificate valid until "2024-10-22 14:45:54 +0000 UTC". client/client_store.go:118
2024-10-22T02:45:55Z DEBU [KEYSTORE]  Teleport TLS certificate valid until "2024-10-22 14:45:54 +0000 UTC". client/client_store.go:118
2024-10-22T02:45:55Z DEBU [KEYSTORE]  Teleport TLS certificate valid until "2024-10-22 14:45:54 +0000 UTC". client/client_store.go:118
2024-10-22T02:45:55Z DEBU [KEYSTORE]  Teleport TLS certificate valid until "2024-10-22 14:45:54 +0000 UTC". client/client_store.go:118

ERROR REPORT:
Original Error: trace.aggregate connection error: desc = "transport: Error while dialing: failed to dial: failed to switch Protocols 403"
Stack Trace:
github.com/gravitational/teleport/lib/client/api.go:4158 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToRootCluster
github.com/gravitational/teleport/tool/tsh/common/tsh.go:1964 github.com/gravitational/teleport/tool/tsh/common.onLogin
github.com/gravitational/teleport/tool/tsh/common/tsh.go:1443 github.com/gravitational/teleport/tool/tsh/common.Run
github.com/gravitational/teleport/tool/tsh/common/tsh.go:614 github.com/gravitational/teleport/tool/tsh/common.Main
github.com/gravitational/teleport/tool/tsh/main.go:26 main.main
runtime/proc.go:271 runtime.main
runtime/asm_amd64.s:1695 runtime.goexit
User Message: connection error: desc = "transport: Error while dialing: failed to dial: failed to switch Protocols 403"

Helm config -

clusterName: "foor.bar"
publicAddr: [foo.bar:443,foo.bar]
proxyListenerMode: "multiplex"
acme: false
acmeEmail: "[email protected]"
persistence:
  enabled: true
  volumeSize: 30Gi
highAvailability:
  replicaCount: 1 ## Multi-attach vol error if replica increased
serviceAccount:
  name: "p2p"
ingress:
  enabled: false
  useExisting: true
service:
  type: "ClusterIP"
proxyProtocol: "off"
log:
  level: "DEBUG"
extraArgs:
  - "--insecure"
proxy:
  teleportConfig:
    proxy_service:
      https_keypairs_reload_interval: 0s #12h
      https_keypairs: []
    auth_service:
      proxy_listener_mode: "multiplex"
operator:
  enabled: true

Normal API calls just work fine.

[webvictim]

It looks you have some kind of load balancer or proxy between your machine and the Teleport cluster which is refusing to allow websocket upgrades.

https://goteleport.com/docs/reference/architecture/tls-routing/#working-with-layer-7-load-balancers-or-reverse-proxies

[vivekyad4v]

Yes, i had to create a way without API gateway. This solved the problem for now.