Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,31 @@ $ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard

See Environment Variables (see the `nemoclaw-user-reference` skill) for the full list of port overrides.

### Running multiple sandboxes simultaneously

Each sandbox requires its own dashboard port.
If you onboard a second sandbox without overriding the port, onboarding fails because port `18789` is already claimed by the first sandbox.

Assign a distinct port to each sandbox at onboard time:

```console
$ nemoclaw onboard # first sandbox — uses default 18789
$ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard # second sandbox — uses 19000
```

Each sandbox then has its own SSH tunnel and its own dashboard URL:

```text
http://localhost:18789 ← first sandbox
http://localhost:19000 ← second sandbox
```

You can verify which tunnel belongs to which sandbox with:

```console
$ openshell forward list
```

## Onboarding

### Cgroup v2 errors during onboard
Expand Down Expand Up @@ -434,6 +459,26 @@ $ openshell term
To permanently allow an endpoint, add it to the network policy.
Refer to Customize the Network Policy (see the `nemoclaw-user-manage-policy` skill) for details.

### Dashboard not reachable after setting `NEMOCLAW_DASHBOARD_PORT`

If you ran `NEMOCLAW_DASHBOARD_PORT=<port> nemoclaw onboard` and onboarding completed
but the dashboard URL is unreachable (browser shows connection refused or the page fails
to load), the sandbox was most likely created with an older NemoClaw version that had a
bug where `NEMOCLAW_DASHBOARD_PORT` was parsed on the host but not passed into the sandbox
at startup. The gateway inside the sandbox continued listening on the default port 18789
while the SSH tunnel forwarded the custom port — leaving nothing at the other end of the
tunnel.

Re-run onboarding on the current NemoClaw release with the desired port. This rebuilds
the sandbox image with the gateway bound to the configured port:

```console
$ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard
```

If you need to run multiple sandboxes at different ports at the same time, see
[Running multiple sandboxes simultaneously](#running-multiple-sandboxes-simultaneously).

### Blueprint run failed

View the error output for the failed blueprint run:
Expand Down
45 changes: 45 additions & 0 deletions docs/reference/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,31 @@ $ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard

See [Environment Variables](commands.md#environment-variables) for the full list of port overrides.

### Running multiple sandboxes simultaneously

Each sandbox requires its own dashboard port.
If you onboard a second sandbox without overriding the port, onboarding fails because port `18789` is already claimed by the first sandbox.

Assign a distinct port to each sandbox at onboard time:

```console
$ nemoclaw onboard # first sandbox — uses default 18789
$ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard # second sandbox — uses 19000
```

Each sandbox then has its own SSH tunnel and its own dashboard URL:

```text
http://localhost:18789 ← first sandbox
http://localhost:19000 ← second sandbox
```

You can verify which tunnel belongs to which sandbox with:

```console
$ openshell forward list
```

## Onboarding

### Cgroup v2 errors during onboard
Expand Down Expand Up @@ -464,6 +489,26 @@ $ openshell term
To permanently allow an endpoint, add it to the network policy.
Refer to [Customize the Network Policy](../network-policy/customize-network-policy.md) for details.

### Dashboard not reachable after setting `NEMOCLAW_DASHBOARD_PORT`

If you ran `NEMOCLAW_DASHBOARD_PORT=<port> nemoclaw onboard` and onboarding completed
but the dashboard URL is unreachable (browser shows connection refused or the page fails
to load), the sandbox was most likely created with an older NemoClaw version that had a
bug where `NEMOCLAW_DASHBOARD_PORT` was parsed on the host but not passed into the sandbox
at startup. The gateway inside the sandbox continued listening on the default port 18789
while the SSH tunnel forwarded the custom port — leaving nothing at the other end of the
tunnel.

Re-run onboarding on the current NemoClaw release with the desired port. This rebuilds
the sandbox image with the gateway bound to the configured port:

```console
$ NEMOCLAW_DASHBOARD_PORT=19000 nemoclaw onboard
```

If you need to run multiple sandboxes at different ports at the same time, see
[Running multiple sandboxes simultaneously](#running-multiple-sandboxes-simultaneously).

### Blueprint run failed

View the error output for the failed blueprint run:
Expand Down
15 changes: 12 additions & 3 deletions scripts/nemoclaw-start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,16 @@ else
exit 1
fi
fi
CHAT_UI_URL="${CHAT_UI_URL:-http://127.0.0.1:${_DASHBOARD_PORT}}"
# When NEMOCLAW_DASHBOARD_PORT is explicitly set (injected at sandbox create time
# via envArgs in onboard.ts), unconditionally override CHAT_UI_URL so the gateway
# starts on the configured port even if the Docker image has a different value
# baked in. Without this, the Docker ENV takes precedence and the gateway listens
# on the wrong port while the SSH tunnel forwards the custom port. (#1925)
if [ -n "${NEMOCLAW_DASHBOARD_PORT:-}" ]; then
CHAT_UI_URL="http://127.0.0.1:${_DASHBOARD_PORT}"
else
CHAT_UI_URL="${CHAT_UI_URL:-http://127.0.0.1:${_DASHBOARD_PORT}}"
fi
PUBLIC_PORT="$_DASHBOARD_PORT"
OPENCLAW="$(command -v openclaw)" # Resolve once, use absolute path everywhere
_SANDBOX_HOME="/sandbox" # Home dir for the sandbox user (useradd -d /sandbox in Dockerfile.base)
Expand Down Expand Up @@ -892,7 +901,7 @@ if [ "$(id -u)" -ne 0 ]; then
chmod 600 /tmp/auto-pair.log

# Start gateway in background, auto-pair, then wait
nohup "$OPENCLAW" gateway run >/tmp/gateway.log 2>&1 &
nohup "$OPENCLAW" gateway run --port "${_DASHBOARD_PORT}" >/tmp/gateway.log 2>&1 &
GATEWAY_PID=$!
echo "[gateway] openclaw gateway launched (pid $GATEWAY_PID)" >&2
trap cleanup SIGTERM SIGINT
Expand Down Expand Up @@ -951,7 +960,7 @@ harden_openclaw_symlinks
# SECURITY: The sandbox user cannot kill this process because it runs
# under a different UID. The fake-HOME attack no longer works because
# the agent cannot restart the gateway with a tampered config.
nohup gosu gateway "$OPENCLAW" gateway run >/tmp/gateway.log 2>&1 &
nohup gosu gateway "$OPENCLAW" gateway run --port "${_DASHBOARD_PORT}" >/tmp/gateway.log 2>&1 &
GATEWAY_PID=$!
echo "[gateway] openclaw gateway launched as 'gateway' user (pid $GATEWAY_PID)" >&2
trap cleanup SIGTERM SIGINT
Expand Down
44 changes: 44 additions & 0 deletions src/lib/agent-runtime.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { describe, it, expect } from "vitest";
// Import from compiled dist/ so coverage is attributed correctly.
import { buildRecoveryScript } from "../../dist/lib/agent-runtime";
import type { AgentDefinition } from "./agent-defs";

// Test fixture — only fields read by buildRecoveryScript are needed.
// Cast via unknown to avoid requiring the full AgentDefinition shape.
const minimalAgent = {
name: "test-agent",
displayName: "Test Agent",
binary_path: "/usr/local/bin/test-agent",
gateway_command: "test-agent gateway run",
healthProbe: { url: "http://127.0.0.1:19000/" },
} as unknown as AgentDefinition;

describe("buildRecoveryScript", () => {
it("returns null for null agent (OpenClaw inline script handles it)", () => {
expect(buildRecoveryScript(null, 18789)).toBeNull();
});

it("embeds the port in the gateway launch command (#1925)", () => {
const script = buildRecoveryScript(minimalAgent, 19000);
expect(script).toContain("--port 19000");
});

it("embeds the default port when called with default value", () => {
const script = buildRecoveryScript(minimalAgent, 18789);
expect(script).toContain("--port 18789");
});

it("uses the agent gateway_command, not a hardcoded openclaw", () => {
const script = buildRecoveryScript(minimalAgent, 19000);
expect(script).toContain("test-agent gateway run --port 19000");
});

it("falls back to openclaw gateway run when gateway_command is absent", () => {
const agent = { ...minimalAgent, gateway_command: undefined } as unknown as AgentDefinition;
const script = buildRecoveryScript(agent, 19000);
expect(script).toContain("openclaw gateway run --port 19000");
});
});
4 changes: 2 additions & 2 deletions src/lib/agent-runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ export function getHealthProbeUrl(agent: AgentDefinition | null): string {
* Returns the script string, or null if agent is null (use existing inline
* OpenClaw script instead).
*/
export function buildRecoveryScript(agent: AgentDefinition | null): string | null {
export function buildRecoveryScript(agent: AgentDefinition | null, port: number): string | null {
if (!agent) return null;

const probeUrl = getHealthProbeUrl(agent);
Expand All @@ -70,7 +70,7 @@ export function buildRecoveryScript(agent: AgentDefinition | null): string | nul
"touch /tmp/gateway.log; chmod 600 /tmp/gateway.log;",
`AGENT_BIN=${shellQuote(binaryPath as string)}; if [ ! -x "$AGENT_BIN" ]; then AGENT_BIN="$(command -v ${shellQuote((binaryPath as string).split("/").pop()!)})"; fi;`,
'if [ -z "$AGENT_BIN" ]; then echo AGENT_MISSING; exit 1; fi;',
`nohup ${gatewayCmd} > /tmp/gateway.log 2>&1 &`,
`nohup ${gatewayCmd} --port ${port} > /tmp/gateway.log 2>&1 &`,
"GPID=$!; sleep 2;",
'if kill -0 "$GPID" 2>/dev/null; then echo "GATEWAY_PID=$GPID"; else echo GATEWAY_FAILED; cat /tmp/gateway.log 2>/dev/null | tail -5; fi',
].join(" ");
Expand Down
7 changes: 7 additions & 0 deletions src/lib/dashboard.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,4 +113,11 @@ describe("buildControlUiUrls", () => {
const urls = buildControlUiUrls("tok");
expect(urls).toHaveLength(1);
});

it("uses the configured port in the displayed URL when NEMOCLAW_DASHBOARD_PORT overrides the default (#1925)", () => {
// getDashboardAccessInfo passes dashboardPort explicitly so the URL shown to
// the user reflects the custom port — not the default 18789.
const urls = buildControlUiUrls("my-token", 19000);
expect(urls).toEqual(["http://127.0.0.1:19000/#token=my-token"]);
});
});
21 changes: 20 additions & 1 deletion src/lib/onboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2787,6 +2787,13 @@ async function createSandbox(
// subprocesses (gateway start, openshell CLI) but the sandbox should
// never have access to the host's Kubernetes cluster or SSH agent.
const envArgs = [formatEnvAssignment("CHAT_UI_URL", chatUiUrl)];
// Pass the configured dashboard port into the sandbox so nemoclaw-start.sh
// can unconditionally override CHAT_UI_URL even when the Docker image was
// built with a different default. Without this, the baked-in Docker ENV
// value takes precedence and the gateway starts on the wrong port. (#1925)
if (process.env.NEMOCLAW_DASHBOARD_PORT) {
envArgs.push(formatEnvAssignment("NEMOCLAW_DASHBOARD_PORT", String(DASHBOARD_PORT)));
}
if (webSearchConfig?.fetchEnabled) {
const braveKey =
getCredential(webSearch.BRAVE_API_KEY_ENV) || process.env[webSearch.BRAVE_API_KEY_ENV];
Expand Down Expand Up @@ -4773,10 +4780,22 @@ function ensureDashboardForward(sandboxName, chatUiUrl = `http://127.0.0.1:${CON
// Use stdio "ignore" to prevent spawnSync from waiting on inherited pipe fds.
// The --background flag forks a child that inherits stdout/stderr; if those are
// pipes, spawnSync blocks until the background process exits (never).
runOpenshell(["forward", "start", "--background", forwardTarget, sandboxName], {
// Use stdio "ignore" to prevent spawnSync from waiting on inherited pipe fds.
// The --background flag forks a child that inherits stdout/stderr; if those are
// pipes, spawnSync blocks until the background process exits (never).
const fwdResult = runOpenshell(["forward", "start", "--background", forwardTarget, sandboxName], {
ignoreError: true,
stdio: ["ignore", "ignore", "ignore"],
});
// A non-zero exit from the parent means forward start rejected before forking —
// typically because the port is already bound by another process (e.g. a local
// Docker test container with -p PORT:PORT). The error is otherwise swallowed by
// ignoreError + stdio:ignore, leaving the dashboard URL silently unreachable (#1925).
if (fwdResult && fwdResult.status !== 0) {
console.warn(`! Port ${portToStop} forward did not start — port may be in use by another process.`);
console.warn(` Check: docker ps --format 'table {{.Names}}\\t{{.Ports}}' | grep ${portToStop}`);
console.warn(` Free the port, then reconnect: nemoclaw ${sandboxName} connect`);
}
}

function findOpenclawJsonPath(dir) {
Expand Down
12 changes: 10 additions & 2 deletions src/lib/ports.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,16 @@ export function parsePort(envVar: string, fallback: number): number {

/** OpenShell gateway port (default 8080, override via NEMOCLAW_GATEWAY_PORT). */
export const GATEWAY_PORT = parsePort("NEMOCLAW_GATEWAY_PORT", 8080);
/** Dashboard UI port (default 18789, override via NEMOCLAW_DASHBOARD_PORT). */
export const DASHBOARD_PORT = parsePort("NEMOCLAW_DASHBOARD_PORT", 18789);
/**
* The default port the OpenClaw dashboard listens on inside the sandbox.
* The sandbox image is built with CHAT_UI_URL=http://127.0.0.1:SANDBOX_DASHBOARD_PORT
* (patched by patchStagedDockerfile), so the gateway starts on whichever port was
* configured via NEMOCLAW_DASHBOARD_PORT at onboard time. This constant represents
* the hardcoded default when no override is set.
*/
export const SANDBOX_DASHBOARD_PORT = 18789;
/** Dashboard UI port (default SANDBOX_DASHBOARD_PORT, override via NEMOCLAW_DASHBOARD_PORT). This is the host-side port. */
export const DASHBOARD_PORT = parsePort("NEMOCLAW_DASHBOARD_PORT", SANDBOX_DASHBOARD_PORT);
/** vLLM / NIM inference port (default 8000, override via NEMOCLAW_VLLM_PORT). */
export const VLLM_PORT = parsePort("NEMOCLAW_VLLM_PORT", 8000);
/** Ollama inference port (default 11434, override via NEMOCLAW_OLLAMA_PORT). */
Expand Down
4 changes: 2 additions & 2 deletions src/nemoclaw.ts
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ function isSandboxGatewayRunning(sandboxName) {
*/
function recoverSandboxProcesses(sandboxName) {
const agent = agentRuntime.getSessionAgent(sandboxName);
const agentScript = agentRuntime.buildRecoveryScript(agent);
const agentScript = agentRuntime.buildRecoveryScript(agent, DASHBOARD_PORT);
// The recovery script runs as the sandbox user (non-root). This matches
// the non-root fallback path in nemoclaw-start.sh — no privilege
// separation, but the gateway runs and inference works.
Expand All @@ -258,7 +258,7 @@ function recoverSandboxProcesses(sandboxName) {
// Resolve and start gateway
'OPENCLAW="$(command -v openclaw)";',
'if [ -z "$OPENCLAW" ]; then echo OPENCLAW_MISSING; exit 1; fi;',
'nohup "$OPENCLAW" gateway run > /tmp/gateway.log 2>&1 &',
`nohup "$OPENCLAW" gateway run --port ${DASHBOARD_PORT} > /tmp/gateway.log 2>&1 &`,
"GPID=$!; sleep 2;",
// Verify the gateway actually started (didn't crash immediately)
'if kill -0 "$GPID" 2>/dev/null; then echo "GATEWAY_PID=$GPID"; else echo GATEWAY_FAILED; cat /tmp/gateway.log 2>/dev/null | tail -5; fi',
Expand Down
44 changes: 43 additions & 1 deletion test/nemoclaw-start.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ describe("nemoclaw-start non-root fallback", () => {

expect(src).toMatch(/if \[ "\$\(id -u\)" -ne 0 \]; then/);
expect(src).toMatch(/touch \/tmp\/gateway\.log/);
expect(src).toMatch(/nohup "\$OPENCLAW" gateway run >\/tmp\/gateway\.log 2>&1 &/);
expect(src).toMatch(/nohup "\$OPENCLAW" gateway run --port "\$\{_DASHBOARD_PORT\}" >\/tmp\/gateway\.log 2>&1 &/);
});

it("exits on config integrity failure in non-root mode", () => {
Expand Down Expand Up @@ -502,3 +502,45 @@ describe("nemoclaw-start signal handling", () => {
expect(src).toMatch(/AUTO_PAIR_PID=\$!/);
});
});

describe("nemoclaw-start CHAT_UI_URL override for configurable dashboard port (#1925)", () => {
const src = fs.readFileSync(START_SCRIPT, "utf-8");

it("unconditionally sets CHAT_UI_URL when NEMOCLAW_DASHBOARD_PORT is injected", () => {
// When the var is present (injected via envArgs in onboard.ts), the gateway
// must use the configured port even if the Docker image has a different
// CHAT_UI_URL baked in as a Docker ENV directive.
const overrideBlock = src.match(
/if \[ -n "\$\{NEMOCLAW_DASHBOARD_PORT:-\}" \]; then([\s\S]*?)else/,
);
expect(overrideBlock).toBeTruthy();
// Plain assignment — the Docker ENV value cannot take precedence
expect(overrideBlock[1]).toContain('CHAT_UI_URL="http://127.0.0.1:${_DASHBOARD_PORT}"');
// Must NOT use :- in this branch — that would let the baked-in Docker ENV win
// and restart the gateway on the wrong port (#1925)
expect(overrideBlock[1]).not.toMatch(/CHAT_UI_URL=.*:-/);
});

it("falls back to baked-in CHAT_UI_URL when NEMOCLAW_DASHBOARD_PORT is absent", () => {
// When no port override was injected (default install), honour whatever
// CHAT_UI_URL was baked into the Docker image at onboard time.
const ifElseBlock = src.match(
/if \[ -n "\$\{NEMOCLAW_DASHBOARD_PORT:-\}" \]; then[\s\S]*?else([\s\S]*?)fi/,
);
expect(ifElseBlock).toBeTruthy();
expect(ifElseBlock[1]).toContain(
'CHAT_UI_URL="${CHAT_UI_URL:-http://127.0.0.1:${_DASHBOARD_PORT}}"',
);
});

it("passes --port to openclaw gateway run in root path (gosu gateway) (#1925)", () => {
// The root path (run as root, then gosu'd to the gateway user) must also
// pass --port so the gateway binds to the configured port. Without this,
// a user with NEMOCLAW_DASHBOARD_PORT set would get a gateway on 18789
// even though the SSH tunnel forwards the custom port.
const rootBlock = src.split(/# ── Root path/)[1] || "";
expect(rootBlock).toMatch(
/nohup gosu gateway "\$OPENCLAW" gateway run --port "\$\{_DASHBOARD_PORT\}" >\/tmp\/gateway\.log 2>&1 &/,
);
});
});
Loading
Loading