Error while dialing: failed to dial: unable to proxy connection #47012

mailman78 · 2024-10-01T09:19:08Z

mailman78
Oct 1, 2024

Hi guys,

This is my architecture:

VM in the local network (Linux machine - Ubuntu, on which I have installed teleport)
On Cloudflare I have set-up DNS entry with external IP address (teleport.sandbox.skidata.com)
On FW I have created a rule which NAT that external IP into the local IP of the Linux machine, where teleport is running.

I have configured teleport:

root@teleport1:/var/lib/teleport# sudo teleport configure -o file
--cluster-name=teleport.sandbox.skidata.com
--public-addr=teleport.sandbox.skidata.com:443
--cert-file=/var/lib/teleport/fullchain.pem
--key-file=/var/lib/teleport/privkey.pem
WARNING: The data directory /var/lib/teleport is not empty and may contain existing cluster state. Running this configuration is likely a mistake. To join a new cluster, specify an alternate --data-dir or clear the /var/lib/teleport directory.

A Teleport configuration file has been created at "/etc/teleport.yaml".
To start Teleport with this configuration file, run:

teleport start --config="/etc/teleport.yaml"

Happy Teleporting!

But, I have an error:

oot@teleport1:/var/lib/teleport# journalctl -fu teleport
Sep 23 14:10:52 teleport1.test.skidata.com teleport[15906]:         github.com/gravitational/teleport/lib/service/service.go:2371 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initProxyEndpoint.(*TeleportProcess).OnExit.func56
Sep 23 14:10:52 teleport1.test.skidata.com teleport[15906]:         github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
Sep 23 14:10:52 teleport1.test.skidata.com teleport[15906]:         github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
Sep 23 14:10:52 teleport1.test.skidata.com teleport[15906]:         runtime/asm_amd64.s:1695 runtime.goexit
Sep 23 14:10:52 teleport1.test.skidata.com teleport[15906]: User Message: EOF] service/service.go:5180
Sep 23 14:10:52 teleport1.test.skidata.com teleport[15906]: 2024-09-23T14:10:52+02:00 INFO [PROXY:SER] Exited. pid:15906.1 service/service.go:5193
Sep 23 14:10:52 teleport1.test.skidata.com teleport[15906]: 2024-09-23T14:10:52+02:00 INFO [PROC:1]    All services stopped or timeout passed, exiting immediately. pid:15906.1 service/signals.go:120
Sep 23 14:10:52 teleport1.test.skidata.com systemd[1]: teleport.service: Deactivated successfully.
Sep 23 14:10:52 teleport1.test.skidata.com systemd[1]: Stopped Teleport Service.
Sep 23 14:10:52 teleport1.test.skidata.com systemd[1]: teleport.service: Consumed 8.533s CPU time.
q^C
root@teleport1:/var/lib/teleport# teleport start --config="/etc/teleport.yaml"
2024-09-23T15:02:06+02:00 INFO  Starting Teleport with a config file version:16.4.0 config_file:/etc/teleport.yaml common/teleport.go:756
2024-09-23T15:02:06+02:00 INFO [PROC:1]    Generating new host UUID pid:17004.1 host_uuid:d1c662de-b469-4c41-b7b8-8005bb3f511f service/service.go:6295
2024-09-23T15:02:06+02:00 INFO [PROC:1]    Service is creating new listener. pid:17004.1 type:debug address:/var/lib/teleport/debug.sock service/signals.go:249
2024-09-23T15:02:07+02:00 INFO [AUTH]      Updating cluster configuration: StaticTokens([]). auth/init.go:467
2024-09-23T15:02:07+02:00 INFO [AUTH]      Creating namespace: "default". auth/init.go:474
2024-09-23T15:02:07+02:00 INFO [AUTH]      Creating session recording config: Kind:"session_recording_config" Version:"v2" Metadata:<Name:"session-recording-config" Namespace:"default" Labels:<key:"teleport.dev/origin" value:"defaults" > > Spec:<Mode:"node" ProxyChecksHostKeys:"\010\001" > . auth/init.go:859
2024-09-23T15:02:07+02:00 INFO [AUTH]      Creating access graph settings: kind:"access_graph_settings" version:"v1" metadata:{name:"access-graph-settings"} spec:{secrets_scan_config:ACCESS_GRAPH_SECRETS_SCAN_CONFIG_DISABLED}. auth/init.go:897
2024-09-23T15:02:07+02:00 INFO [AUTH]      Creating cluster auth preference: AuthPreference(Type="local",SecondFactor="otp"). auth/init.go:777
2024-09-23T15:02:07+02:00 INFO [AUTH]      Creating cluster networking configuration: Kind:"cluster_networking_config" Version:"v2" Metadata:<Name:"cluster-networking-config" Namespace:"default" Labels:<key:"teleport.dev/origin" value:"config-file" > > Spec:<KeepAliveInterval:300000000000 KeepAliveCountMax:3 ProxyListenerMode:Multiplex TunnelStrategy:<AgentMesh:<> > > . auth/init.go:819
2024-09-23T15:02:07+02:00 INFO [AUTH]      Starting migration 1 create_db_cas migration/migration.go:123
2024-09-23T15:02:07+02:00 INFO [AUTH]      Completed migration 1 create_db_cas migration/migration.go:143
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating spiffe certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating oidc_idp certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating saml_idp certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating db certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating host certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating jwt certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating user certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating db_client certificate authority. auth/init.go:635
2024-09-23T15:02:07+02:00 INFO [AUTH]      First start: generating openssh certificate authority. auth/init.go:635
2024-09-23T15:02:09+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:terraform-provider time:2024-09-23T13:02:09.343Z uid:45d03b27-fae7-49ce-95d6-4cb86bdb0e36 user:system events/emitter.go:288
2024-09-23T15:02:09+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:editor time:2024-09-23T13:02:09.35Z uid:2b12e63b-0d10-4db6-90ac-09c0fa82fb23 user:system events/emitter.go:288
2024-09-23T15:02:09+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:access time:2024-09-23T13:02:09.356Z uid:0d9ab2c1-6ecd-40ca-a33e-fc65af6709a9 user:system events/emitter.go:288
2024-09-23T15:02:09+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:auditor time:2024-09-23T13:02:09.362Z uid:520754d3-b4d2-4b63-bf15-5f18f5b64b18 user:system events/emitter.go:288
2024-09-23T15:02:09+02:00 INFO [AUTH]      Auth server is running periodic operations. auth/init.go:574
2024-09-23T15:02:09+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:17004.1 identity:Admin service/connect.go:531
2024-09-23T15:02:09+02:00 INFO [PROC:1]    The process successfully wrote the credentials and state to the disk. pid:17004.1 identity:Admin service/connect.go:566
2024-09-23T15:02:09+02:00 INFO [AUTH:COMP] upload completer will run every 5m0s events/complete.go:167
2024-09-23T15:02:09+02:00 INFO [AUTH:1:CA] Cache "auth" first init succeeded. cache/cache.go:1035
2024-09-23T15:02:09+02:00 INFO [PROC:1]    Service is creating new listener. pid:17004.1 type:auth address:0.0.0.0:3025 service/signals.go:249
2024-09-23T15:02:09+02:00 WARN [AUTH:1]    'proxy_protocol' unspecified. Starting Auth service with external PROXY protocol support, but IP pinned connection affected by PROXY headers will not be allowed. Set 'proxy_protocol: on' in 'auth_service' config if Auth service runs behind L4 load balancer with enabled PROXY protocol, or set 'proxy_protocol: off' otherwise pid:17004.1 service/service.go:2187
2024-09-23T15:02:09+02:00 WARN [AUTH:1]    Configuration setting auth_service/advertise_ip is not set, using inferred address pid:17004.1 address:10.14.91.110:3025 service/service.go:2274
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  starting upload completer service pid:17004.1 service/service.go:3136
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log/upload service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log/upload/streaming service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log/upload/streaming/default service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log/upload service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log/upload/corrupted service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1]  Creating directory. pid:17004.1 directory:/var/lib/teleport/log/upload/corrupted/default service/service.go:3152
2024-09-23T15:02:09+02:00 INFO [AUTH:1]    Auth service is starting. pid:17004.1 version:16.4.0 git_ref:v16.4.0-0-ge08d4e0 listen_address:10.14.91.110:3025 service/service.go:2222
2024-09-23T15:02:09+02:00 INFO [UPLOAD]    uploader will scan /var/lib/teleport/log/upload/streaming/default every 5s filesessions/fileasync.go:196
2024-09-23T15:02:09+02:00 INFO [UPLOAD:1:] upload completer will run every 5m0s events/complete.go:167
2024-09-23T15:02:09+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:17004.1 identity:Instance service/connect.go:531
2024-09-23T15:02:09+02:00 INFO [AUTH:SPIF] Obtained lock, SPIFFEFederation syncer is starting pid:17004.1 machineidv1/spiffe_federation_syncer.go:209
2024-09-23T15:02:09+02:00 ERRO [PROC:1]    Failed to establish connection to cluster. pid:17004.1 identity:Instance error:[
ERROR REPORT:
Original Error: trace.aggregate connection error: desc = &[#34](https://support.goteleport.com/hc/requests/34);transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden&[#34](https://support.goteleport.com/hc/requests/34);
Stack Trace:
        github.com/gravitational/teleport/lib/service/connect.go:1407 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClientDirect
        github.com/gravitational/teleport/lib/service/connect.go:1242 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient.func1
        github.com/gravitational/teleport/lib/service/connect.go:1311 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient
        github.com/gravitational/teleport/lib/service/connect.go:1133 github.com/gravitational/teleport/lib/service.(*TeleportProcess).getConnector
        github.com/gravitational/teleport/lib/service/connect.go:539 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect
        github.com/gravitational/teleport/lib/service/connect.go:218 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
        github.com/gravitational/teleport/lib/service/connect.go:191 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
        github.com/gravitational/teleport/lib/service/connect.go:84 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
        github.com/gravitational/teleport/lib/service/service.go:3051 github.com/gravitational/teleport/lib/service.(*TeleportProcess).RegisterWithAuthServer.func1
        github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: connection error: desc = &[#34](https://support.goteleport.com/hc/requests/34);transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden&[#34](https://support.goteleport.com/hc/requests/34);] service/connect.go:100
2024-09-23T15:02:09+02:00 ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:17004.1 service/connect.go:102
2024-09-23T15:02:09+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:17004.1 identity:Node service/connect.go:531
2024-09-23T15:02:09+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:17004.1 identity:Proxy service/connect.go:531
2024-09-23T15:02:12+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:17004.1 identity:Instance service/connect.go:531
2024-09-23T15:02:12+02:00 ERRO [PROC:1]    Failed to establish connection to cluster. pid:17004.1 identity:Instance error:[
ERROR REPORT:
Original Error: trace.aggregate connection error: desc = &[#34](https://support.goteleport.com/hc/requests/34);transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden&[#34](https://support.goteleport.com/hc/requests/34);
Stack Trace:
        github.com/gravitational/teleport/lib/service/connect.go:1407 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClientDirect
        github.com/gravitational/teleport/lib/service/connect.go:1242 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient.func1
        github.com/gravitational/teleport/lib/service/connect.go:1311 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient
        github.com/gravitational/teleport/lib/service/connect.go:1133 github.com/gravitational/teleport/lib/service.(*TeleportProcess).getConnector
        github.com/gravitational/teleport/lib/service/connect.go:539 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect
        github.com/gravitational/teleport/lib/service/connect.go:218 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
        github.com/gravitational/teleport/lib/service/connect.go:191 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
        github.com/gravitational/teleport/lib/service/connect.go:84 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
        github.com/gravitational/teleport/lib/service/service.go:3051 github.com/gravitational/teleport/lib/service.(*TeleportProcess).RegisterWithAuthServer.func1
        github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: connection error: desc = &[#34](https://support.goteleport.com/hc/requests/34);transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden&[#34](https://support.goteleport.com/hc/requests/34);] service/connect.go:100
2024-09-23T15:02:12+02:00 ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:17004.1 service/connect.go:102
q2024-09-23T15:02:36+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:17004.1 identity:Instance service/connect.go:531
2024-09-23T15:02:36+02:00 ERRO [PROC:1]    Failed to establish connection to cluster. pid:17004.1 identity:Instance error:[
ERROR REPORT:
Original Error: trace.aggregate connection error: desc = &[#34](https://support.goteleport.com/hc/requests/34);transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden&[#34](https://support.goteleport.com/hc/requests/34);
Stack Trace:
        github.com/gravitational/teleport/lib/service/connect.go:1407 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClientDirect
        github.com/gravitational/teleport/lib/service/connect.go:1242 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient.func1
        github.com/gravitational/teleport/lib/service/connect.go:1311 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient
        github.com/gravitational/teleport/lib/service/connect.go:1133 github.com/gravitational/teleport/lib/service.(*TeleportProcess).getConnector
        github.com/gravitational/teleport/lib/service/connect.go:539 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect
        github.com/gravitational/teleport/lib/service/connect.go:218 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
        github.com/gravitational/teleport/lib/service/connect.go:191 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
        github.com/gravitational/teleport/lib/service/connect.go:84 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
        github.com/gravitational/teleport/lib/service/service.go:3051 github.com/gravitational/teleport/lib/service.(*TeleportProcess).RegisterWithAuthServer.func1
        github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: connection error: desc = &[#34](https://support.goteleport.com/hc/requests/34);transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden&[#34](https://support.goteleport.com/hc/requests/34);] service/connect.go:100
2024-09-23T15:02:36+02:00 ERRO [PROC:1]    Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:17004.1 service/connect.go:102
^C2024-09-23T15:02:42+02:00 INFO [PROC:1]    Got shutdown signal, exiting within timeout. pid:17004.1 signal:interrupt timeout:3 service/signals.go:103
2024-09-23T15:02:42+02:00 WARN [DEBUG:1]   Debug server exited with error. pid:17004.1 error:[accept unix /var/lib/teleport/debug.sock: use of closed network connection] service/service.go:3488
2024-09-23T15:02:45+02:00 WARN [PROC:1]    Failed sending inventory goodbye during shutdown pid:17004.1 error:[
ERROR REPORT:
Original Error: context.deadlineExceededError context deadline exceeded
Stack Trace:
        github.com/gravitational/teleport/lib/inventory/inventory.go:365 github.com/gravitational/teleport/lib/inventory.(*downstreamHandle).SendGoodbye
        github.com/gravitational/teleport/lib/service/service.go:5989 github.com/gravitational/teleport/lib/service.(*TeleportProcess).StartShutdown
        github.com/gravitational/teleport/lib/service/service.go:6029 github.com/gravitational/teleport/lib/service.(*TeleportProcess).Shutdown
        github.com/gravitational/teleport/lib/service/signals.go:112 github.com/gravitational/teleport/lib/service.(*TeleportProcess).WaitForSignals.func1
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: context deadline exceeded] service/service.go:5990
2024-09-23T15:02:45+02:00 INFO [PROC:1]    All services stopped or timeout passed, exiting immediately. pid:17004.1 service/signals.go:120
2024-09-23T15:02:45+02:00 INFO [PROC:1]    Stopping connection attempts, teleport is shutting down. pid:17004.1 identity:Instance service/connect.go:120

So, can anybody tell me what is wrong here and how to re-solved it?

Thanks in advance,
Dusko

mailman78 · 2024-10-01T09:22:56Z

mailman78
Oct 1, 2024
Author

User Message: connection error: desc = "transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden"] service/connect.go:100
2024-10-01T11:22:19+02:00 ERRO [PROC:1] Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:100656.1 service/connect.go:102
2024-10-01T11:22:19+02:00 INFO [PROC:1] Successfully obtained credentials to connect to the cluster. pid:100656.1 identity:Proxy service/connect.go:531
2024-10-01T11:22:23+02:00 INFO [PROC:1] Successfully obtained credentials to connect to the cluster. pid:100656.1 identity:Instance service/connect.go:531
2024-10-01T11:22:23+02:00 ERRO [PROC:1] Failed to establish connection to cluster. pid:100656.1 identity:Instance error:[
ERROR REPORT:
Original Error: trace.aggregate connection error: desc = "transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden"
Stack Trace:
github.com/gravitational/teleport/lib/service/connect.go:1407 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClientDirect
github.com/gravitational/teleport/lib/service/connect.go:1242 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient.func1
github.com/gravitational/teleport/lib/service/connect.go:1311 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient
github.com/gravitational/teleport/lib/service/connect.go:1133 github.com/gravitational/teleport/lib/service.(*TeleportProcess).getConnector
github.com/gravitational/teleport/lib/service/connect.go:539 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect
github.com/gravitational/teleport/lib/service/connect.go:218 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
github.com/gravitational/teleport/lib/service/connect.go:191 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
github.com/gravitational/teleport/lib/service/connect.go:84 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
github.com/gravitational/teleport/lib/service/service.go:3051 github.com/gravitational/teleport/lib/service.(*TeleportProcess).RegisterWithAuthServer.func1
github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
runtime/asm_amd64.s:1695 runtime.goexit
User Message: connection error: desc = "transport: Error while dialing: failed to dial: unable to proxy connection: 403 Forbidden"] service/connect.go:100
2024-10-01T11:22:23+02:00 ERRO [PROC:1] Check to see if the config has auth_server pointing to a Teleport Proxy. If it does, use proxy_server instead of auth_server. pid:100656.1 service/connect.go:102

0 replies

mailman78 · 2024-10-01T10:19:18Z

mailman78
Oct 1, 2024
Author

teleport.yaml is this:

version: v3
teleport:
nodename: teleport1.test.skidata.com
data_dir: /var/lib/teleport
log:
output: stderr
severity: INFO
format:
output: text
ca_pin: ""
diag_addr: ""
auth_service:
enabled: "yes"
listen_addr: 0.0.0.0:3025
cluster_name: teleport.sandbox.skidata.com
proxy_listener_mode: multiplex
ssh_service:
enabled: "yes"
proxy_service:
enabled: "yes"
web_listen_addr: 0.0.0.0:443
public_addr: teleport.sandbox.skidata.com:443
https_keypairs:

key_file: /var/lib/teleport/privkey.pem
cert_file: /var/lib/teleport/fullchain.pem
https_keypairs_reload_interval: 0s
acme: {}

0 replies

webvictim · 2024-10-02T20:16:02Z

webvictim
Oct 2, 2024
Collaborator

It looks like you have an HTTP proxy in your environment? Check env | grep -i proxy

At any rate, Teleport is attempting to use this configured HTTP proxy and is getting a 403 Forbidden error - likely a permissions/ACL error on the proxy itself.

You can also try export NO_PROXY=localhost,127.0.0.1,0.0.0.0 to avoid Teleport traffic going via the proxy at all.

0 replies

mailman78 · 2024-10-03T09:40:44Z

mailman78
Oct 3, 2024
Author

Hi webvictim,

It looks like that was an issue:

root@teleport1:~# env | grep -i proxy
root@teleport1:~# export NO_PROXY=localhost,127.0.0.1,0.0.0.0
root@teleport1:~# sudo teleport configure -o file \
    --cluster-name=teleport.sandbox.skidata.com \
    --public-addr=teleport.sandbox.skidata.com:443 \
    --cert-file=/var/lib/teleport/fullchain.pem \
    --key-file=/var/lib/teleport/privkey.pem
WARNING: The data directory /var/lib/teleport is not empty and may contain existing cluster state. Running this configuration is likely a mistake. To join a new cluster, specify an alternate --data-dir or clear the /var/lib/teleport directory.

A Teleport configuration file has been created at "/etc/teleport.yaml".
To start Teleport with this configuration file, run:

teleport start --config="/etc/teleport.yaml"

Happy Teleporting!

root@teleport1:/home/nfs/brdu/sterndl.sandbox.skidata.com/23092024# teleport start --config="/etc/teleport.yaml"
2024-10-03T11:52:11+02:00 INFO  Starting Teleport with a config file version:16.4.2 config_file:/etc/teleport.yaml common/teleport.go:756
2024-10-03T11:52:11+02:00 INFO [PROC:1]    Generating new host UUID pid:113971.1 host_uuid:b338fc2e-d913-4a58-a7c4-eb40979dc914 service/service.go:6295
2024-10-03T11:52:12+02:00 INFO [PROC:1]    Service is creating new listener. pid:113971.1 type:debug address:/var/lib/teleport/debug.sock service/signals.go:249
2024-10-03T11:52:13+02:00 INFO [AUTH]      Updating cluster configuration: StaticTokens([]). auth/init.go:467
2024-10-03T11:52:13+02:00 INFO [AUTH]      Creating namespace: "default". auth/init.go:474
2024-10-03T11:52:13+02:00 INFO [AUTH]      Creating cluster networking configuration: Kind:"cluster_networking_config" Version:"v2" Metadata:<Name:"cluster-networking-config" Namespace:"default" Labels:<key:"teleport.dev/origin" value:"config-file" > > Spec:<KeepAliveInterval:300000000000 KeepAliveCountMax:3 ProxyListenerMode:Multiplex TunnelStrategy:<AgentMesh:<> > > . auth/init.go:819
2024-10-03T11:52:13+02:00 INFO [AUTH]      Creating session recording config: Kind:"session_recording_config" Version:"v2" Metadata:<Name:"session-recording-config" Namespace:"default" Labels:<key:"teleport.dev/origin" value:"defaults" > > Spec:<Mode:"node" ProxyChecksHostKeys:"\010\001" > . auth/init.go:859
2024-10-03T11:52:13+02:00 INFO [AUTH]      Creating access graph settings: kind:"access_graph_settings"  version:"v1"  metadata:{name:"access-graph-settings"}  spec:{secrets_scan_config:ACCESS_GRAPH_SECRETS_SCAN_CONFIG_DISABLED}. auth/init.go:897
2024-10-03T11:52:13+02:00 INFO [AUTH]      Creating cluster auth preference: AuthPreference(Type="local",SecondFactor="otp"). auth/init.go:777
2024-10-03T11:52:13+02:00 INFO [AUTH]      Starting migration 1 create_db_cas migration/migration.go:123
2024-10-03T11:52:13+02:00 INFO [AUTH]      Completed migration 1 create_db_cas migration/migration.go:143
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating spiffe certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating host certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating oidc_idp certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating jwt certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating db_client certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating openssh certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating db certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating saml_idp certificate authority. auth/init.go:635
2024-10-03T11:52:13+02:00 INFO [AUTH]      First start: generating user certificate authority. auth/init.go:635
2024-10-03T11:52:14+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:terraform-provider time:2024-10-03T09:52:14.287Z uid:6063075c-d97c-4d9f-aafc-6861b4409fc9 user:system events/emitter.go:288
2024-10-03T11:52:14+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:editor time:2024-10-03T09:52:14.295Z uid:dcb091b6-d235-46bd-9294-56f366f3885c user:system events/emitter.go:288
2024-10-03T11:52:14+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:access time:2024-10-03T09:52:14.302Z uid:f9d425e6-a30f-45ed-813d-feb55913ea71 user:system events/emitter.go:288
2024-10-03T11:52:14+02:00 INFO [AUDIT]     role.created cluster_name:teleport.sandbox.skidata.com code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:auditor time:2024-10-03T09:52:14.31Z uid:b4446cb7-fa52-4839-886c-6bd48fd71aeb user:system events/emitter.go:288
2024-10-03T11:52:14+02:00 INFO [AUTH]      Auth server is running periodic operations. auth/init.go:574
2024-10-03T11:52:14+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:113971.1 identity:Admin service/connect.go:531
2024-10-03T11:52:14+02:00 INFO [PROC:1]    The process successfully wrote the credentials and state to the disk. pid:113971.1 identity:Admin service/connect.go:566
2024-10-03T11:52:14+02:00 INFO [AUTH:COMP] upload completer will run every 5m0s events/complete.go:167
2024-10-03T11:52:14+02:00 INFO [AUTH:1:CA] Cache "auth" first init succeeded. cache/cache.go:1035
2024-10-03T11:52:14+02:00 INFO [PROC:1]    Service is creating new listener. pid:113971.1 type:auth address:0.0.0.0:3025 service/signals.go:249
2024-10-03T11:52:14+02:00 WARN [AUTH:1]    'proxy_protocol' unspecified. Starting Auth service with external PROXY protocol support, but IP pinned connection affected by PROXY headers will not be allowed. Set 'proxy_protocol: on' in 'auth_service' config if Auth service runs behind L4 load balancer with enabled PROXY protocol, or set 'proxy_protocol: off' otherwise pid:113971.1 service/service.go:2187
2024-10-03T11:52:14+02:00 WARN [AUTH:1]    Configuration setting auth_service/advertise_ip is not set, using inferred address pid:113971.1 address:10.14.91.110:3025 service/service.go:2274
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  starting upload completer service pid:113971.1 service/service.go:3136
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log/upload service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log/upload/streaming service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log/upload/streaming/default service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log/upload service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log/upload/corrupted service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1]  Creating directory. pid:113971.1 directory:/var/lib/teleport/log/upload/corrupted/default service/service.go:3152
2024-10-03T11:52:14+02:00 INFO [AUTH:1]    Auth service is starting. pid:113971.1 version:16.4.2 git_ref:v16.4.2-0-ge414b8f listen_address:10.14.91.110:3025 service/service.go:2222
2024-10-03T11:52:14+02:00 INFO [AUTH:SPIF] Obtained lock, SPIFFEFederation syncer is starting pid:113971.1 machineidv1/spiffe_federation_syncer.go:209
2024-10-03T11:52:14+02:00 INFO [UPLOAD]    uploader will scan /var/lib/teleport/log/upload/streaming/default every 5s filesessions/fileasync.go:196
2024-10-03T11:52:14+02:00 INFO [UPLOAD:1:] upload completer will run every 5m0s events/complete.go:167
2024-10-03T11:52:14+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:113971.1 identity:Instance service/connect.go:531
2024-10-03T11:52:14+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:113971.1 identity:Proxy service/connect.go:531
2024-10-03T11:52:14+02:00 INFO [PROC:1]    features loaded from auth server pid:113971.1 identity:Instance features:Kubernetes:true App:true DB:true Desktop:true DeviceTrust:<> AccessRequests:<> AccessList:<> AccessMonitoring:<> Policy:<> SupportType:SUPPORT_TYPE_FREE JoinActiveSessions:true entitlements:<key:"AccessLists" value:<> > entitlements:<key:"AccessMonitoring" value:<> > entitlements:<key:"AccessRequests" value:<> > entitlements:<key:"App" value:<enabled:true > > entitlements:<key:"CloudAuditLogRetention" value:<> > entitlements:<key:"DB" value:<enabled:true > > entitlements:<key:"Desktop" value:<enabled:true > > entitlements:<key:"DeviceTrust" value:<> > entitlements:<key:"ExternalAuditStorage" value:<> > entitlements:<key:"FeatureHiding" value:<> > entitlements:<key:"HSM" value:<> > entitlements:<key:"Identity" value:<> > entitlements:<key:"JoinActiveSessions" value:<enabled:true > > entitlements:<key:"K8s" value:<enabled:true > > entitlements:<key:"MobileDeviceManagement" value:<> > entitlements:<key:"OIDC" value:<> > entitlements:<key:"OktaSCIM" value:<> > entitlements:<key:"OktaUserSync" value:<> > entitlements:<key:"Policy" value:<> > entitlements:<key:"SAML" value:<> > entitlements:<key:"SessionLocks" value:<> > entitlements:<key:"UpsellAlert" value:<> > entitlements:<key:"UsageReporting" value:<> >  service/connect.go:1150
2024-10-03T11:52:14+02:00 INFO [PROC:1]    The process successfully wrote the credentials and state to the disk. pid:113971.1 identity:Instance service/connect.go:566
2024-10-03T11:52:14+02:00 INFO [INSTANCE:] Successfully registered instance client. pid:113971.1 service/service.go:2695
2024-10-03T11:52:14+02:00 INFO [PROC:1]    Reusing Instance client. pid:113971.1 identity:Proxy additional_system_roles:[Auth Node Proxy] service/connect.go:1118
2024-10-03T11:52:15+02:00 INFO [PROC:1]    The process successfully wrote the credentials and state to the disk. pid:113971.1 identity:Proxy service/connect.go:566
2024-10-03T11:52:15+02:00 INFO [PROC:1]    Successfully obtained credentials to connect to the cluster. pid:113971.1 identity:Node service/connect.go:531
2024-10-03T11:52:15+02:00 INFO [PROC:1]    Reusing Instance client. pid:113971.1 identity:Node additional_system_roles:[Auth Node Proxy] service/connect.go:1118
2024-10-03T11:52:15+02:00 INFO [PROC:1]    The process successfully wrote the credentials and state to the disk. pid:113971.1 identity:Node service/connect.go:566
2024-10-03T11:52:15+02:00 INFO [PROXY:1:C] Cache "proxy" first init succeeded. cache/cache.go:1035
2024-10-03T11:52:15+02:00 INFO [PROC:1]    Service is creating new listener. pid:113971.1 type:proxy:web address:0.0.0.0:443 service/signals.go:249
2024-10-03T11:52:15+02:00 INFO [PROXY:SER] Starting reverse tunnel server pid:113971.1 version:16.4.2 git_ref:v16.4.2-0-ge414b8f listen_address: cache_policy:in-memory cache service/service.go:4266
2024-10-03T11:52:15+02:00 INFO [PROXY:SER] Starting web proxy service. pid:113971.1 version:16.4.2 git_ref:v16.4.2-0-ge414b8f listen_address:0.0.0.0:443 service/service.go:4541
2024-10-03T11:52:15+02:00 INFO [PROXY:SER]  Stating SSH proxy service pid:113971.1 version:16.4.2 git_ref:v16.4.2-0-ge414b8f listen_address:[::]:443 service/service.go:4750
2024-10-03T11:52:15+02:00 INFO [PROXY:AGE] Starting reverse tunnel agent pool. service/service.go:4804
2024-10-03T11:52:15+02:00 INFO [PROXY:PRO] Starting Kube proxy. pid:113971.1 listen_address:[::]:443 service/service.go:4899
2024-10-03T11:52:15+02:00 INFO [DB:SERVIC] Starting Database Postgres proxy server. pid:113971.1 listen_address:[::]:443 service/service.go:4999
2024-10-03T11:52:15+02:00 INFO [DB:SERVIC] Starting Database TLS proxy server. pid:113971.1 listen_address:0.0.0.0:443 service/service.go:5017
2024-10-03T11:52:15+02:00 INFO [PROC:1]    Starting proxy gRPC server. pid:113971.1 listen_address:[::]:443 service/service.go:6439
2024-10-03T11:52:15+02:00 INFO [PROC:1]    Starting proxy gRPC server. pid:113971.1 listen_address:[::]:443 service/service.go:6516
2024-10-03T11:52:15+02:00 INFO [PROXY:SER] Starting TLS ALPN SNI proxy server on. pid:113971.1 listen_address:[::]:443 service/service.go:5099
2024-10-03T11:52:15+02:00 INFO [NODE:1:CA] Cache "node" first init succeeded. cache/cache.go:1035
2024-10-03T11:52:15+02:00 WARN [NODE:1]    Login UID is set, but it shouldn't be. Incorrect login UID breaks session ID when using auditd. Please make sure that Teleport runs as a daemon and any parent process doesn't set the login UID. pid:113971.1 service/service.go:2788
2024-10-03T11:52:15+02:00 INFO [PROC:1]    Service is creating new listener. pid:113971.1 type:node address:0.0.0.0:3022 service/signals.go:249
2024-10-03T11:52:15+02:00 INFO [NODE:1]    SSH Service is starting. pid:113971.1 version:16.4.2 git_ref:v16.4.2-0-ge414b8f listen_address:0.0.0.0:3022 cache_policy:in-memory cache service/service.go:2931
2024-10-03T11:52:15+02:00 INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status. pid:113971.1 max_retry_period:1m30s service/connect.go:700

and it stay like this, if I do CTRL+C to continue to work, I get:

^C2024-10-03T11:56:54+02:00 INFO [PROC:1]    Got shutdown signal, exiting within timeout. pid:113971.1 signal:interrupt timeout:3 service/signals.go:103
2024-10-03T11:56:54+02:00 WARN [DEBUG:1]   Debug server exited with error. pid:113971.1 error:[accept unix /var/lib/teleport/debug.sock: use of closed network connection] service/service.go:3488
2024-10-03T11:56:54+02:00 INFO [AUTH:SPIF] SPIFFEFederation syncer has stopped pid:113971.1 machineidv1/spiffe_federation_syncer.go:214
2024-10-03T11:56:54+02:00 INFO [NODE:1]    Shutting down gracefully. pid:113971.1 service/service.go:3020
2024-10-03T11:56:54+02:00 INFO [NODE:1]    Exited. pid:113971.1 service/service.go:3030
2024-10-03T11:56:54+02:00 INFO [AUTH:1]    Shutting down immediately (auth service does not currently support graceful shutdown). pid:113971.1 service/service.go:2355
2024-10-03T11:56:54+02:00 INFO [UPLOAD:1]  File uploader is shutting down. pid:113971.1 service/service.go:3192
2024-10-03T11:56:54+02:00 INFO [UPLOAD:1]  File uploader has shut down. pid:113971.1 service/service.go:3194
2024-10-03T11:56:54+02:00 INFO [AUTH:1]    Exited. pid:113971.1 service/service.go:2370
2024-10-03T11:56:54+02:00 INFO [DEBUG:1]   Shutting down gracefully. pid:113971.1 service/service.go:3499
2024-10-03T11:56:54+02:00 INFO [DEBUG:1]   Exited. pid:113971.1 service/service.go:3503
2024-10-03T11:56:54+02:00 INFO [PROXY:SER] Shutting down gracefully. pid:113971.1 service/service.go:5175
2024-10-03T11:56:54+02:00 INFO [UPLOAD:1]  File upload completer is shutting down. pid:113971.1 error:<nil> service/service.go:3224
2024-10-03T11:56:54+02:00 INFO [UPLOAD:1]  File upload completer has shut down. pid:113971.1 service/service.go:3226
2024-10-03T11:56:54+02:00 WARN [AUTH:1]    Mux serve failed. error:[<nil>] auth/middleware.go:313
2024-10-03T11:56:54+02:00 WARN [PROXY:1:C] Re-init the cache on error: watcher is closed: error reading from server: EOF cache/cache.go:1158
2024-10-03T11:56:54+02:00 WARN  cert authority watch loop for client TLS config generator failed error:[
ERROR REPORT:
Original Error: *errors.errorString ca watcher exited with: event fanout system reset
Stack Trace:
        github.com/gravitational/teleport/lib/auth/client_tls_config_generator.go:241 github.com/gravitational/teleport/lib/auth.(*ClientTLSConfigGenerator).watchForCAChanges
        github.com/gravitational/teleport/lib/auth/client_tls_config_generator.go:189 github.com/gravitational/teleport/lib/auth.(*ClientTLSConfigGenerator).refreshClientTLSConfigs
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: ca watcher exited with: event fanout system reset] auth/client_tls_config_generator.go:195
2024-10-03T11:56:54+02:00 INFO [PROXY:SER] Exited. pid:113971.1 service/service.go:4547
2024-10-03T11:56:54+02:00 INFO [WEB]       Closing session cache. web/sessions.go:727
2024-10-03T11:56:54+02:00 WARN [PROC:1]    Teleport process has exited with error. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError listener is closed
Stack Trace:
        github.com/gravitational/teleport/lib/srv/alpnproxy/listener.go:88 github.com/gravitational/teleport/lib/srv/alpnproxy.(*ListenerMuxWrapper).Accept
        google.golang.org/[email protected]/server.go:885 google.golang.org/grpc.(*Server).Serve
        github.com/gravitational/teleport/lib/service/service.go:6440 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initPublicGRPCServer.func1
        github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: listener is closed] pid:113971.1 service:proxy.grpc.public service/supervisor.go:319
2024-10-03T11:56:54+02:00 WARN [NODE:1:CA] Re-init the cache on error: watcher is closed: error reading from server: EOF cache/cache.go:1158
2024-10-03T11:56:54+02:00 WARN [PROC:1]    Teleport process has exited with error. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError listener is closed
Stack Trace:
        github.com/gravitational/teleport/lib/srv/alpnproxy/listener.go:88 github.com/gravitational/teleport/lib/srv/alpnproxy.(*ListenerMuxWrapper).Accept
        google.golang.org/[email protected]/server.go:885 google.golang.org/grpc.(*Server).Serve
        github.com/gravitational/teleport/lib/service/service.go:6517 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initSecureGRPCServer.func1
        github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: listener is closed] pid:113971.1 service:proxy.grpc.secure service/supervisor.go:319
2024-10-03T11:56:54+02:00 INFO [WEB]       Closing session cache. web/sessions.go:727
2024-10-03T11:56:54+02:00 WARN [PROXY:SER] Failed to delete heartbeat. pid:113971.1 error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError dial tcp 0.0.0.0:3025: connect: connection refused
Stack Trace:
        github.com/gravitational/teleport/lib/httplib/httplib.go:206 github.com/gravitational/teleport/lib/httplib.ConvertResponse
        github.com/gravitational/teleport/lib/auth/authclient/http_client.go:321 github.com/gravitational/teleport/lib/auth/authclient.(*HTTPClient).Delete
        github.com/gravitational/teleport/lib/auth/authclient/http_client.go:590 github.com/gravitational/teleport/lib/auth/authclient.(*HTTPClient).DeleteProxy
        github.com/gravitational/teleport/lib/service/service.go:5218 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initProxyEndpoint.func21
        github.com/gravitational/teleport/lib/service/service.go:2389 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initProxyEndpoint.(*TeleportProcess).OnExit.func56
        github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        runtime/asm_amd64.s:1695 runtime.goexit
User Message: dial tcp 0.0.0.0:3025: connect: connection refused] service/service.go:5220
2024-10-03T11:56:54+02:00 INFO [PROXY:SER] Exited. pid:113971.1 service/service.go:5233
2024-10-03T11:56:54+02:00 INFO [PROC:1]    All services stopped or timeout passed, exiting immediately. pid:113971.1 service/signals.go:120

NOTE: I am allowed to do CTRL+C after teleport start --config="/etc/teleport.yaml"?
That should not be a problem or it is?

root@teleport1:~# systemctl enable teleport
Created symlink /etc/systemd/system/multi-user.target.wants/teleport.service → /lib/systemd/system/teleport.service.
root@teleport1:~# systemctl start teleport
root@teleport1:/home/nfs/brdu/sterndl.sandbox.skidata.com/23092024# systemctl status teleport
● teleport.service - Teleport Service
     Loaded: loaded (/lib/systemd/system/teleport.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-10-03 11:56:54 CEST; 40s ago
   Main PID: 114388 (teleport)
      Tasks: 12 (limit: 2204)
     Memory: 93.1M
        CPU: 7.647s
     CGroup: /system.slice/teleport.service
             └─114388 /usr/local/bin/teleport start --config /etc/teleport.yaml --pid-file=/run/teleport.pid

Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [PROXY:PRO] Starting Kube proxy. pid:114388.1 listen_address:[::]:443 s>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [DB:SERVIC] Starting Database Postgres proxy server. pid:114388.1 liste>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [DB:SERVIC] Starting Database TLS proxy server. pid:114388.1 listen_add>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [PROC:1]    Starting proxy gRPC server. pid:114388.1 listen_address:[::>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [PROC:1]    Starting proxy gRPC server. pid:114388.1 listen_address:[::>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [PROXY:SER] Starting TLS ALPN SNI proxy server on. pid:114388.1 listen_>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [NODE:1:CA] Cache "node" first init succeeded. cache/cache.go:1035
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [PROC:1]    Service is creating new listener. pid:114388.1 type:node ad>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [NODE:1]    SSH Service is starting. pid:114388.1 version:16.4.2 git_re>
Oct 03 11:56:56 teleport1.test.skidata.com teleport[114388]: 2024-10-03T11:56:56+02:00 INFO [PROC:1]    The new service has started successfully. Starting syncing >


root@teleport1:~# ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
161/udp                    ALLOW       Anywhere
9102/tcp                   ALLOW       Anywhere
5666/tcp                   ALLOW       Anywhere
443                        ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)
161/udp (v6)               ALLOW       Anywhere (v6)
9102/tcp (v6)              ALLOW       Anywhere (v6)
5666/tcp (v6)              ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

0 replies

mailman78 · 2024-10-03T09:59:49Z

mailman78
Oct 3, 2024
Author

So, I can not reach the page:

What is the problem now? Or what to check now?

0 replies

mailman78 · 2024-10-18T07:50:16Z

mailman78
Oct 18, 2024
Author

So, nobody can help me with this? Is there any contributors here or this is deprecated?

0 replies

mailman78 · 2024-10-18T08:05:23Z

mailman78
Oct 18, 2024
Author

0 replies

mailman78 · 2024-10-18T08:06:36Z

mailman78
Oct 18, 2024
Author

This is the config file:

root@teleport1:/var/lib/teleport# teleport configure -o file \
--cluster-name=teleport.sandbox.skidata.com \
--public-addr=teleport.sandbox.skidata.com:443 \
--cert-file=/var/lib/teleport/fullchain.pem \
--key-file=/var/lib/teleport/privkey.pem
WARNING: The data directory /var/lib/teleport is not empty and may contain existing cluster state. Running this configuration is likely a mistake. To join a new cluster, specify an alternate --data-dir or clear the /var/lib/teleport directory.

A Teleport configuration file has been created at "/etc/teleport.yaml".
To start Teleport with this configuration file, run:

teleport start --config="/etc/teleport.yaml"

Happy Teleporting!

0 replies

webvictim · 2024-10-21T14:00:07Z

webvictim
Oct 21, 2024
Collaborator

NOTE: I am allowed to do CTRL+C after teleport start --config="/etc/teleport.yaml"?

When you run teleport start it runs Teleport in the foreground. This means that if you press Ctrl+C, the process will exit.

To run Teleport in the background, you should use sudo systemctl start teleport as you've done.

You can check its status with sudo systemctl status teleport as you've noted.

You can view logs with sudo journalctl -fu teleport as you've seen.

The error with connecting to the web UI looks like a connection issue or timeout between the machine where you're running the browser and the host teleport.sandbox.skidata.com.

How are you connected to that machine? When I connect to it, I can get a ping response from Teleport just fine. The UI also loads fine in my browser.

If you're still having connectivity issues, try running this at the command line on your machine:

curl -v https://teleport.sandbox.skidata.com/webapi/ping

0 replies

mailman78 · 2024-10-22T09:14:33Z

mailman78
Oct 22, 2024
Author

thanks for the reply!

I have managed to do all, I have setup firewall differently and it is Ok now.
But when I try to add kubernetes cluster:

2024-10-22T09:04:31Z WARN [UPLOAD:1] The Instance connector is still not available, process-wide services such as session uploading will not function pid:6.1 service/service.go:3091
2024-10-22T09:04:32Z INFO [PROC:1] Joining the cluster with a secure token. pid:6.1 service/connect.go:471
2024-10-22T09:04:32Z INFO Attempting registration via proxy server. join/join.go:294
2024-10-22T09:04:32Z ERRO [PROC:1] Failed to establish connection to cluster. pid:6.1 identity:Kube error:[
ERROR REPORT:
Original Error: trace.aggregate Post "https://teleport.sandbox.skidata.com:443/v1/webapi/host/credentials": dial tcp: lookup teleport.sandbox.skidata.com on 10.96.0.10:53: no such host
Stack Trace:
github.com/gravitational/teleport/lib/auth/join/join.go:304 github.com/gravitational/teleport/lib/auth/join.Register
github.com/gravitational/teleport/lib/service/connect.go:515 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect
github.com/gravitational/teleport/lib/service/connect.go:218 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
github.com/gravitational/teleport/lib/service/connect.go:191 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
github.com/gravitational/teleport/lib/service/connect.go:84 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
github.com/gravitational/teleport/lib/service/service.go:3057 github.com/gravitational/teleport/lib/service.(*TeleportProcess).RegisterWithAuthServer.func1
github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
runtime/asm_amd64.s:1695 runtime.goexit
User Message: Post "https://teleport.sandbox.skidata.com:443/v1/webapi/host/credentials": dial tcp: lookup teleport.sandbox.skidata.com on 10.96.0.10:53: no such host] service/connect.go:100
2024-10-22T09:04:58Z INFO [PROC:1] Joining the cluster with a secure token. pid:6.1 service/connect.go:471
2024-10-22T09:04:58Z INFO Attempting registration via proxy server. join/join.go:294
2024-10-22T09:04:58Z ERRO [PROC:1] Failed to establish connection to cluster. pid:6.1 identity:Instance error:[
ERROR REPORT:
Original Error: trace.aggregate Post "https://teleport.sandbox.skidata.com:443/v1/webapi/host/credentials": dial tcp: lookup teleport.sandbox.skidata.com on 10.96.0.10:53: no such host
Stack Trace:
github.com/gravitational/teleport/lib/auth/join/join.go:304 github.com/gravitational/teleport/lib/auth/join.Register
github.com/gravitational/teleport/lib/service/connect.go:515 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect
github.com/gravitational/teleport/lib/service/connect.go:218 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
github.com/gravitational/teleport/lib/service/connect.go:191 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
github.com/gravitational/teleport/lib/service/connect.go:84 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
github.com/gravitational/teleport/lib/service/service.go:3057 github.com/gravitational/teleport/lib/service.(*TeleportProcess).RegisterWithAuthServer.func1
github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
runtime/asm_amd64.s:1695 runtime.goexit
User Message: Post "https://teleport.sandbox.skidata.com:443/v1/webapi/host/credentials": dial tcp: lookup teleport.sandbox.skidata.com on 10.96.0.10:53: no such host] service/connect.go:100
2024-10-22T09:05:01Z WARN [UPLOAD:1] The Instance connector is still not available, process-wide services such as session uploading will not function pid:6.1 service/service.go:3091

To be honest, I do not know what is IP 10.96.0.10:53.

root@ka1:~# curl -v https://teleport.sandbox.skidata.com/webapi/ping

Uses proxy env variable no_proxy == 'localhost,127.0.0.1,ka.test.skidata.com,test.skidata.com'
Uses proxy env variable https_proxy == 'http://proxy.test.skidata.com:3128'
Trying 10.14.15.10:3128...
Connected to (nil) (10.14.15.10) port 3128 (#0)
allocate connect buffer!
Establish HTTP proxy tunnel to teleport.sandbox.skidata.com:443

CONNECT teleport.sandbox.skidata.com:443 HTTP/1.1
Host: teleport.sandbox.skidata.com:443
User-Agent: curl/7.81.0
Proxy-Connection: Keep-Alive

< HTTP/1.1 500 Internal Server Error
< Server: squid/5.9
< Mime-Version: 1.0
< Date: Tue, 22 Oct 2024 09:07:29 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3891
< X-Squid-Error: ERR_CANNOT_FORWARD 0
< Vary: Accept-Language
< Content-Language: en
<

Received HTTP code 500 from proxy after CONNECT
CONNECT phase completed!
Closing connection 0
curl: (56) Received HTTP code 500 from proxy after CONNECT

Do you maybe see where is the issue here? Maybe on Cloudflare I need to do something extra?

3 replies

webvictim Oct 22, 2024
Collaborator

10.96.0.10 looks like the ClusterIP of the Kubernetes control plane - kubectl -n kube-system get svc will probably confirm that.

Port 53 is used for DNS requests, which makes sense. The error is to saying that your Kubernetes cluster's DNS forwarder (probably kube-dns - confirm with kubectl -n kube-system get pods) is not able to look up and return the IP for the host teleport.test.skidata.com. This would generally indicate a lack of working internet connectivity for the Kubernetes control plane.

The 500 ERR_CANNOT_FORWARD error you see when trying to use the Squid HTTP proxy appears to indicate that whatever proxy you're using is incapable of forwarding HTTP requests - perhaps because it also cannot look up the DNS hostname, or is misconfigured. You'd need to check Squid error logs for more detail.

mailman78 Oct 23, 2024
Author

Thanks, I have fixed it. Can you just let me know how to remove the old entry or why I see the old user and kubernetes which is not valid anymore:

brdu@WINAA5CG2363ZZD:~$ tsh login --proxy=teleport.sandbox.skidata.com:443 --user=teleport-admin
Enter password for Teleport user teleport-admin:
Enter your OTP token:

Profile URL: https://teleport.sandbox.skidata.com:443
Logged in as: teleport-admin
Cluster: teleport.sandbox.skidata.com
Roles: access, auditor, editor
Logins: root, ubuntu, ec2-user
Kubernetes: enabled
Kubernetes groups: system:masters
Valid until: 2024-10-24 01:33:07 +0200 CEST [valid for 12h0m0s]
Extensions: login-ip, permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy

Profile URL: https://teleport1.test.skidata.com:443
Logged in as: dev_user
Cluster: teleport1.test.skidata.com
Roles: dev-view, developer-read
Kubernetes: enabled
Kubernetes cluster: "qa-k8s-cluster"
Kubernetes groups: dev-view, developer-read
Valid until: 2024-09-18 00:26:07 +0200 CEST [EXPIRED]
Extensions: login-ip, permit-port-forwarding, permit-pty, private-key-policy

Can this part be removed somehow:
Profile URL: https://teleport1.test.skidata.com:443
Logged in as: dev_user
Cluster: teleport1.test.skidata.com
Roles: dev-view, developer-read
Kubernetes: enabled
Kubernetes cluster: "qa-k8s-cluster"
Kubernetes groups: dev-view, developer-read
Valid until: 2024-09-18 00:26:07 +0200 CEST [EXPIRED]
Extensions: login-ip, permit-port-forwarding, permit-pty, private-key-policy

I mean, that is not valid anymore (Expired)...

webvictim Oct 23, 2024
Collaborator

tsh logout --proxy teleport1.test.skidata.com --user dev_user ought to do that.

If not, a full tsh logout definitely will.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error while dialing: failed to dial: unable to proxy connection #47012

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 10 comments 3 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Error while dialing: failed to dial: unable to proxy connection #47012

mailman78 Oct 1, 2024

Replies: 10 comments · 3 replies

mailman78 Oct 1, 2024 Author

mailman78 Oct 1, 2024 Author

webvictim Oct 2, 2024 Collaborator

mailman78 Oct 3, 2024 Author

mailman78 Oct 3, 2024 Author

mailman78 Oct 18, 2024 Author

mailman78 Oct 18, 2024 Author

mailman78 Oct 18, 2024 Author

webvictim Oct 21, 2024 Collaborator

mailman78 Oct 22, 2024 Author

webvictim Oct 22, 2024 Collaborator

mailman78 Oct 23, 2024 Author

webvictim Oct 23, 2024 Collaborator

mailman78
Oct 1, 2024

Replies: 10 comments 3 replies

mailman78
Oct 1, 2024
Author

mailman78
Oct 1, 2024
Author

webvictim
Oct 2, 2024
Collaborator

mailman78
Oct 3, 2024
Author

mailman78
Oct 3, 2024
Author

mailman78
Oct 18, 2024
Author

mailman78
Oct 18, 2024
Author

mailman78
Oct 18, 2024
Author

webvictim
Oct 21, 2024
Collaborator

mailman78
Oct 22, 2024
Author

webvictim Oct 22, 2024
Collaborator

mailman78 Oct 23, 2024
Author

webvictim Oct 23, 2024
Collaborator