Error 503 when trying to login to private registry

[PKizzle]

Description

When using nerdctl login I am unable to successfully login to a private Gitlab container registry.

Steps to reproduce the issue

1. Configure an access_token with the correct permission on Gitlab
2. Try to login to the container registry as suggested on Gitlab's website but replace docker with nerdctl
3. Enter the username (short version with @ in front) and access token when requested

Describe the results you received and expected

I receive the following error message:

$ nerdctl login -u ${USERNAME} ${REGISTRY_DOMAIN}
Enter Password:
ERRO[0005] failed to call tryLoginWithRegHost            error="unexpected status code 503" i=0
FATA[0005] unexpected status code 503

Using docker login works perfectly fine as well as ctr to pull and push images directly. Also when using curl to retrieve https://${REGISTRY_DOMAIN}:443/v2 I never receive a 503 status code.

What version of nerdctl are you using?

2.0.4

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Namespace: default
Debug Mode: false

Server:
Server Version: v2.0.4
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file none syslog
Storage: native overlayfs
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-52-generic
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 251.2GiB
Name: IPA-AIMV-80908-WS1
ID: c0a19b0a-6f33-4389-bf05-912556ac9660

WARNING: IPv4 forwarding is disabled

[apostasie]

@PKizzle

Can you provide logs using --debug-full? (eg: nerdctl --debug-full login -u ${USERNAME} ${REGISTRY_DOMAIN})

Thanks

[apostasie]

nerdctl --debug-full login ghcr.io
Enter Username: apostasie
Enter Password:
DEBU[0023] loading host directory                        dir="/home/dmp.linux/.config/containerd/certs.d/ghcr.io:443"
DEBU[0023] len(regHosts)=2

WARNING! Your credentials are stored unencrypted in '/home/dmp.linux/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/

Login Succeeded

This is working fine with above ^, so, I assume something is different in your case - which is why I need logs.

Also, if you have any specific hosts.toml configuration, please share it as well.

[PKizzle]

Here is the output with the --debug-full flag:

$ sudo nerdctl --debug-full login -u ${USERNAME} ${REGISTRY_DOMAIN}
Enter Password:
DEBU[0004] len(regHosts)=1
ERRO[0004] failed to call tryLoginWithRegHost            error="unexpected status code 503" i=0
FATA[0004] unexpected status code 503

Sometimes I do not even get a chance to fill in the password and see that the 503 error was already logged.

I do not use any custom hosts.toml configuration.

[apostasie]

What is the value of $registry_domain?

[apostasie]

Also: do you use a proxy?

[apostasie]

ERRO[0004] failed to call tryLoginWithRegHost error="unexpected status code 503" i=0

Is this all you get in the logs?
No url? No scope information?

Given the symptoms you describe, intuition is that the registry you try to contact is not happy with a port in the url and/or Host header - which is why you get a 503.

To confirm that, I would need more information about that registry. Ideally the registry logs for that error. The Registry URL being used. What product is that (satisfactory, other). What version. Is it configured behind a reverse proxy. Etc.

Without these, it is going to be hard to give you a firm diagnosis.

[PKizzle]

I can’t share the exact registry domain but it’s the official Gitlab container registry offered in the Gitlab CE version.

Yes, that is the full output I get when using --debug-full.

I sadly can’t access the registry‘s logs. Is there maybe a specific header that is being sent when using docker login? I‘ll try to collect more information about the registry but that might take some time.

[apostasie]

I can’t share the exact registry domain

Understood.
Let's try with what we have then.

but it’s the official Gitlab container registry offered in the Gitlab CE version.

Unfortunately, we do not test against Gitlab yet, but I am not surprised that Gitlab Registry behavior is not fully ironed out.

Yes, that is the full output I get when using --debug-full.

I sadly can’t access the registry‘s logs. Is there maybe a specific header that is being sent when using docker login? I‘ll try to collect more information about the registry but that might take some time.

cat ~/.docker/config.json <- copy this here but be 100% SURE you do redact credentials before posting of course.

Then:

• reset credentials - either nerdctl logout $REGISTRY or mv ~/.docker/config.json ~/config.json_backup
• nerdctl --debug-full login $REGISTRY_WITHOUT_PORT <- copy the logs (be sure to redact domain name, but please keep PORT information from the logs if it is there)

[PKizzle]

Thank you for the quick reply. I do not have a ~/.docker/config.json as I fully use nerdctl and have not used docker login for that registry yet.

$ nerdctl logout $REGISTRY_WITHOUT_PORT
$ nerdctl login --debug-full=true $REGISTRY_WITHOUT_PORT
Enter Username: ${USERNAME}
Enter Password:
DEBU[0004] len(regHosts)=1
ERRO[0004] failed to call tryLoginWithRegHost            error="unexpected status code 503" i=0
FATA[0004] unexpected status code 503

[apostasie]

Ok...

REGISTRY_HOST="foo.com"
REGISTRY_PORT="443"
curl -iv -H "Host: $REGISTRY_HOST:$REGISTRY_PORT" -H "Accept-encoding: gzip" -H "User-Agent: Go-http-client/1.1"  https://$REGISTRY_HOST:$REGISTRY_PORT/v2/

^ copy the full output of this (redact your domain name of course).

[PKizzle]

Here you go:

*   Trying ${REGISTRY_IP}:${REGISTRY_PORT}...
* Connected to ${REGISTRY_HOST} (${REGISTRY_IP}) port ${REGISTRY_PORT} (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /usr/lib/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=XX; ST=XXXXX; O=XXXXXX; CN=XXXXXXXXXXXXXXX
*  start date: May 23 00:00:00 2024 GMT
*  expire date: May 23 23:59:59 2025 GMT
*  subjectAltName: host "${REGISTRY_HOST}" matched cert's "${REGISTRY_HOST}"
*  issuer: C=XXX; O=XXXXXXXXXXXXX; CN=XXXXXXXXXXXXXXXXXX
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /v2/ HTTP/1.1
> Host: ${REGISTRY_HOST}:${REGISTRY_PORT}
> Accept: */*
> Accept-encoding: gzip
> User-Agent: Go-http-client/1.1
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
< Content-Length: 62
Content-Length: 62
< Connection: close
Connection: close
< Cache-Control: no-cache,no-store
Cache-Control: no-cache,no-store
< Pragma: no-cache
Pragma: no-cache

<
* Closing connection 0
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.2 (IN), TLS alert, close notify (256):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.2 (OUT), TLS alert, close notify (256):

[PKizzle]

The issue is the host header. Without it I am able to access the registry correctly and also receive the expected 401 Unauthorized response.

Edit: It also works when editing the Host header to not include the port.