Agent gRPC listener silently stops accepting connections (TLS handshake fails)

[spdrman]

🔍 Check for existing issues

• Completed

How is Dozzle deployed?

Agents

📦 Dozzle version

v10.0.2

✅ Command used to run Dozzle

Dozzle Server (on Host A):

services:
  dozzle-server:
    image: amir20/dozzle:v10.0.2
    container_name: dozzle-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - dozzle_data:/data
      - /opt/dozzle/config/users.yml:/data/users.yml:ro
      - /opt/dozzle/certs:/certs:ro
    environment:
      - DOZZLE_ADDR=:8080
      - DOZZLE_LEVEL=info
      - DOZZLE_AUTH_PROVIDER=simple
      - DOZZLE_ENABLE_ACTIONS=true
      - DOZZLE_ENABLE_SHELL=true
      - DOZZLE_REMOTE_AGENT=<host-b-ip>:7007
      - DOZZLE_CERT=/certs/dozzle_cert.pem
      - DOZZLE_KEY=/certs/dozzle_key.pem
    restart: unless-stopped
    networks:
      - caddy-bridge

volumes:
  dozzle_data:
    external: true

networks:
  caddy-bridge:
    name: caddy-bridge
    external: true

Dozzle Agent (on Host B):

services:
  dozzle-agent:
    image: amir20/dozzle:v10.0.2
    container_name: dozzle-agent
    command: agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /opt/dozzle/certs:/certs:ro
    ports:
      - "7007:7007"
    environment:
      - DOZZLE_LEVEL=info
      - DOZZLE_CERT=/certs/dozzle_cert.pem
      - DOZZLE_KEY=/certs/dozzle_key.pem
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "/dozzle", "healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    networks:
      - caddy-bridge

networks:
  caddy-bridge:
    name: caddy-bridge
    external: true

🐛 Describe the bug / provide steps to reproduce it

Summary: The Dozzle agent's gRPC listener silently stops accepting new TLS connections after running for ~2 days. The agent process remains alive, the TCP port stays open, and the built-in healthcheck (/dozzle healthcheck) continues to report healthy. However, all new gRPC connections from the server fail with a TLS handshake error.

Setup:

• Two separate hosts (different physical servers on the same network)
• Server and agent use the same self-signed TLS certificate/key pair (SHA256 checksums verified matching on both hosts)
• Communication over port 7007
• Firewall rules verified — port 7007 is open between hosts

Timeline of failure:

1. Agent starts normally (Day 0):

   {"level":"info","message":"Loaded custom dozzle certificate and key"}
   {"level":"info","message":"Dozzle agent version v10.0.2"}
   {"level":"info","message":"Agent listening on [::]:7007"}
   

2. Agent encounters errors (~48 hours later):

   {"level":"error","error":"rpc error: code = Canceled desc = context canceled","message":"error while reading event"}
   {"level":"error","error":"read unix @->/run/docker.sock: use of closed network connection","message":"error while writing to ws"}
   {"level":"info","host":"host-b","message":"stopped container stats collector"}
   

   After these errors, the agent produces no further log output — the gRPC listener appears to enter a zombie state.

3. Server continuously fails to connect:
   Initially connection reset by peer, then transitions to context deadline exceeded:

   {"level":"warn","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: read tcp <server-container-ip>:<port>-><host-b-ip>:7007: read: connection reset by peer\"","host":"host-b","message":"error fetching host info for client"}
   

   {"level":"warn","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\"","host":"host-b","message":"error fetching host info for client"}
   

4. Agent healthcheck still reports healthy despite being unable to serve connections:

   docker inspect dozzle-agent --format '{{.State.Health.Status}}'
   healthy
   

Key observations:

• TLS certificates are identical on both hosts (verified via SHA256 checksum) — not a cert mismatch
• Agent container remains running with healthy health status throughout the failure
• Agent's TCP port 7007 remains bound (connections reach it but get reset during TLS handshake)
• Only restarting the agent container (docker restart dozzle-agent) resolves the issue
• After agent restart, the server immediately reconnects successfully ("clients":2,"message":"Connected to Docker")
• This is a recurring issue — it has happened multiple times

Steps to reproduce:

1. Deploy Dozzle server + agent on separate hosts with TLS certs
2. Verify connection works initially
3. Wait ~2 days (the exact trigger is unclear — may be related to Docker socket events like container restarts or network changes on the agent host)
4. Server loses connection to agent and cannot reconnect

Workaround: Restart the dozzle-agent container.

Suggested fix: The agent's gRPC goroutine appears to crash/hang after certain Docker socket errors (context canceled, use of closed network connection) without recovering or exiting. Either:

• The gRPC listener should recover and re-establish its ability to accept connections after Docker socket errors
• Or the agent should exit (allowing Docker's restart: unless-stopped to bring it back cleanly)
• The built-in healthcheck should also detect this state — currently it reports healthy even when the gRPC listener cannot accept new connections

💻 Environment

Agent host (Host B):

Client: Docker Engine - Community
 Version:    29.0.2
 Server Version: 29.0.2
 Storage Driver: overlay2
 Kernel Version: 6.1.0-32-cloud-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.581GiB

Server host (Host A):

Client: Docker Engine - Community
 Version:    28.5.1
 Server Version: 28.5.1
 Storage Driver: overlay2
 Kernel Version: 6.1.0-32-cloud-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.581GiB

📜 Dozzle logs are required for debugging purposes. You may need to enable debug mode. See https://dozzle.dev/guide/debugging.

Agent logs (full sequence from start to failure):

{"level":"info","version":"v10.0.2","cert":"/certs/dozzle_cert.pem","key":"/certs/dozzle_key.pem","time":"2026-02-20T10:18:36Z","message":"Loaded custom dozzle certificate and key"}
{"level":"info","version":"v10.0.2","time":"2026-02-20T10:18:36Z","message":"Dozzle agent version v10.0.2"}
{"level":"info","version":"v10.0.2","time":"2026-02-20T10:18:36Z","message":"Agent listening on [::]:7007"}
{"level":"error","version":"v10.0.2","error":"rpc error: code = Canceled desc = context canceled","time":"2026-02-22T01:28:28Z","message":"error while reading event"}
{"level":"error","version":"v10.0.2","error":"rpc error: code = Canceled desc = context canceled","time":"2026-02-22T01:28:28Z","message":"error while reading event"}
{"level":"error","version":"v10.0.2","error":"rpc error: code = Canceled desc = context canceled","time":"2026-02-22T01:28:28Z","message":"error while reading event"}
{"level":"error","version":"v10.0.2","error":"read unix @->/run/docker.sock: use of closed network connection","time":"2026-02-22T01:28:28Z","message":"error while writing to ws"}
{"level":"info","version":"v10.0.2","host":"host-b","time":"2026-02-22T07:26:57Z","message":"stopped container stats collector"}

Server logs (connection failures — representative sample):

{"level":"warn","version":"v10.0.2","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: read tcp 172.19.0.6:39036->10.0.0.2:7007: read: connection reset by peer\"","host":"host-b","time":"2026-02-22T18:32:53Z","message":"error fetching host info for client"}
{"level":"warn","version":"v10.0.2","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\"","host":"host-b","time":"2026-02-23T09:28:36Z","message":"error fetching host info for client"}
{"level":"warn","version":"v10.0.2","error":"host <agent-id> unavailable: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: context deadline exceeded\"","time":"2026-02-23T09:33:25Z","message":"error listing containers"}

Server logs after both containers restarted (connection restored):

{"level":"info","version":"v10.0.2","time":"2026-02-23T09:37:39Z","message":"Dozzle version v10.0.2"}
{"level":"info","version":"v10.0.2","cert":"/certs/dozzle_cert.pem","key":"/certs/dozzle_key.pem","time":"2026-02-23T09:37:39Z","message":"Loaded custom dozzle certificate and key"}
{"level":"info","version":"v10.0.2","clients":2,"time":"2026-02-23T09:37:39Z","message":"Connected to Docker"}
{"level":"info","version":"v10.0.2","time":"2026-02-23T09:37:39Z","message":"Accepting connections on :8080"}

📸 If applicable, add screenshots to help explain your bug

N/A

[amir20]

You do have a Dozzle that is two versions old. So I would try updating first. Meanwhile, I'll see if I can reproduce this. Over 10K people are using agent so I don't think this is related to Dozzle.

What is caddy-bridge? Are you running Agent on a caddy proxy?

[amir20]

I did more research.

{"level":"error","error":"rpc error: code = Canceled desc = context canceled","message":"error while reading event"}
{"level":"error","error":"read unix @->/run/docker.sock: use of closed network connection","message":"error while writing to ws"}

This is happening because of attach/exec shell session. Which is normal. You might have closed the websocket while it was writing.

{"level":"info","version":"v10.0.2","host":"host-b","time":"2026-02-22T07:26:57Z","message":"stopped container stats collector"}

This is also normal because Dozzle will listen on stats and stop the collector after a few hours of inactivity.

{"level":"warn","version":"v10.0.2","error":"rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: read tcp 172.19.0.6:39036->10.0.0.2:7007: read: connection reset by peer"","host":"host-b","time":"2026-02-22T18:32:53Z","message":"error fetching host info for client"}

This is the only real error that suggests something is wrong with TLS handshake. It could be related to Caddy or something else.

I would move its own network. I have also updated some of the flags at #4474. But I really doubt it would fix it.

[amir20]

The flag changes are in amir20/dozzle:master now. Doubt will fix anything.

[spdrman]

I'll try it. Caddy is running for handling https to dozzle server to connect to the server from WAN. Nothing to do with dozzle server to agent comms.

I'll keep testing. Dozzle agent is running in a docker container separately from the dozzle server running on a different server and docker container. TLS was generated on local machine and uploaded to both server and agent docker containers.

[amir20]

That's what I figured. Other people are using agents so I am not sure what is different about your system that is causing Dozzle to stop connecting.

When this happens, it would be good to help investigate. The healthcheck reuses the same certs to check for connection.

I think doing a thread dump would help. If it happens again, I would also bounce the UI server to see if it's actually related to the client for some reason.

The agent is pretty stable for me so it might something related to the custom certs that I haven't noticed.

[amir20]

Any update on this? No one else seems to have this problem so I am wondering what is going on.

[spdrman]

The problem was fixed once I moved it out of the caddy network and into its own for the docker containers running the client and server.