kubectl get --raw incorrectly truncates the server URL's path before appending the requested path

[b1zzu]

What happened:

Whether I'm running kubectl --context cluster-a get --raw "/api/v1/nodes" or kubectl --context cluster-b get --raw "/api/v1/nodes" it always returns the nodes from cluster-a.

Same if I switch context: kubectl config use-context cluster-b

What you expected to happen:

I expect to see the nodes form the selected context

How to reproduce it (as minimally and precisely as possible):

Have two clusters, and two contexts that shares the same authentication method.

Anything else we need to know?:

Environment:
Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.0

NAME="Fedora Linux"
VERSION="41 (Workstation Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-05-13
VARIANT="Workstation Edition"
VARIANT_ID=workstation

[xyz-li]

Can't reproduce it. Could you provide more log?

[b1zzu]

This is the best example I have

➜  ~ kubectl --v=9 --context=non-prod-eks-tooling get --raw "/api/v1/nodes"
I1106 11:41:20.520492   30805 loader.go:395] Config loaded from file:  /home/davide/.kube/config
I1106 11:41:20.520692   30805 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.31.2 (linux/amd64) kubernetes/5864a46" 'https://rancher-dev.willhaben.at/api/v1/nodes'
I1106 11:41:20.565680   30805 round_trippers.go:495] HTTP Trace: DNS Lookup for rancher-dev.willhaben.at resolved to [{172.31.20.2 }]
I1106 11:41:20.617801   30805 round_trippers.go:510] HTTP Trace: Dial to tcp:172.31.20.2:443 succeed
I1106 11:41:21.019694   30805 round_trippers.go:553] GET https://rancher-dev.willhaben.at/api/v1/nodes 200 OK in 498 milliseconds
I1106 11:41:21.019717   30805 round_trippers.go:570] HTTP Statistics: DNSLookup 34 ms Dial 52 ms TLSHandshake 326 ms ServerProcessing 74 ms Duration 498 ms
[...]

➜  ~ kubectl -v=9 --context non-prod-eks-tooling get nodes 
I1106 11:42:10.515162   31218 loader.go:395] Config loaded from file:  /home/davide/.kube/config
I1106 11:42:10.519156   31218 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.31.2 (linux/amd64) kubernetes/5864a46" 'https://rancher-dev.willhaben.at/k8s/clusters/c-m-knqq927b/api/v1/nodes?limit=500'
I1106 11:42:10.573768   31218 round_trippers.go:495] HTTP Trace: DNS Lookup for rancher-dev.willhaben.at resolved to [{172.31.20.2 }]
I1106 11:42:10.597673   31218 round_trippers.go:510] HTTP Trace: Dial to tcp:172.31.20.2:443 succeed
I1106 11:42:10.964421   31218 round_trippers.go:553] GET https://rancher-dev.willhaben.at/k8s/clusters/c-m-knqq927b/api/v1/nodes?limit=500 200 OK in 445 milliseconds
I1106 11:42:10.964466   31218 round_trippers.go:570] HTTP Statistics: DNSLookup 43 ms Dial 23 ms TLSHandshake 243 ms ServerProcessing 122 ms Duration 445 ms

And this is my cluster definition in the kubeconfig:

apiVersion: v1
clusters:
- cluster:
    server: https://rancher-dev.willhaben.at/k8s/clusters/c-m-knqq927b
  name: non-prod-eks-tooling
contexts:
- context:
    cluster: non-prod-eks-tooling
    user: rancher-dev
  name: non-prod-eks-tooling
kind: Config
preferences: {}
users:
- name: rancher-dev
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - token
      - --server=rancher-dev.willhaben.at
      - --user=rancher-dev
      - --auth-provider=oktaProvider
      command: rancher
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

As you can see when using the --raw option the full server url is not used

[brianpursley]

At first, I thought I couldn't reproduce it either. It worked fine when I created two kind clusters and specified the context, but kind server urls don't have a path like yours does.

I found in the code that when you perform a get with --raw, kubectl (well, client-go actually) replaces the path with the path you are requesting.
https://github.com/kubernetes/kubernetes/blob/072dfcb416fd4e1ddab0a89ac4faf519e268bc96/staging/src/k8s.io/client-go/rest/request.go#L378-L383

I think the intended behavior would be for it to append the requested path, preserving any existing path that is part of the server URL. So something like this:
https://rancher-dev.willhaben.at/k8s/clusters/c-m-knqq927b + /api/v1/nodes

Rancher must be doing something to handle when you don't specify the cluster so that it doesn't outright fail (choosing some default cluster? or the first one?). That's the only reason I can think why your request would not just fail, but instead run against a different cluster altogether.

I do think this is a bug in kubectl/client-go though, that should be fixed.
/triage accepted

---

@b1zzu Can you try this and see if it returns the result you expect?

kubectl --context=non-prod-eks-tooling get --raw "/k8s/clusters/c-m-knqq927b/api/v1/nodes"

I know it isn't ideal, but I think that might work.

[b1zzu]

Hi @brianpursley,

I've tried using

kubectl --context=non-prod-eks-tooling get --raw "/k8s/clusters/c-m-knqq927b/api/v1/nodes"

and it reaches the expected (non-prod-eks-tooling) cluster.

Personally, I expected the --raw to concat the cluster path with the request raw path, exactly like your example. But for now I can also use the full-path as workaround like you have suggested.

[dharmit]

I'd like to work on this. @brianpursley I hope it's OK to assign myself.

/assign

[feichashao]

I am hitting the same issue as we have a reverse-proxy-like mechanism to access our clusters.

server: https://example.com/proxy/cluster-id/

and kubectl get --raw /api will lead us to:

https://example.com/api

while we expect:

https://example.com/proxy/cluster-id/api

Looking at the logic around this:

kubectl/pkg/cmd/get/get.go

Line 449 in 285ed6c

return rawhttp.RawGet(restClient, o.IOStreams, o.Raw)

kubectl/pkg/rawhttp/raw.go

Line 72 in 285ed6c

request = restClient.Get().RequestURI(url)

https://github.com/kubernetes/kubernetes/blob/1b517e55013479f674f3fac7196f3af307d6e23f/staging/src/k8s.io/client-go/rest/request.go#L406

kubectl get --raw /api calls request = restClient.Get().RequestURI(url), but the RequestURI overrides the entire path, instead of appending /api to the existing pathPrefix.

This PR should solve the issue kubernetes/kubernetes#131165

Tested with the above PR in my environment, and the kubectl get --raw /api points to https://example.com/proxy/cluster-id/api as expected.