Skip to main content

Gateway troubleshooting

This page is the deep runbook. Start at /help/troubleshooting if you want the fast triage flow first.

Command ladder

Run these first, in this order:
Expected healthy signals:
  • velaclaw gateway status shows Runtime: running and RPC probe: ok.
  • velaclaw doctor reports no blocking config/service issues.
  • velaclaw channels status --probe shows live per-account transport status and, where supported, probe/audit results such as works or audit ok.

Anthropic 429 extra usage required for long context

Use this when logs/errors include: HTTP 429: rate_limit_error: Extra usage is required for long context requests.
Look for:
  • Selected Anthropic Opus/Sonnet model has params.context1m: true.
  • Current Anthropic credential is not eligible for long-context usage.
  • Requests fail only on long sessions/model runs that need the 1M beta path.
Fix options:
  1. Disable context1m for that model to fall back to the normal context window.
  2. Use an Anthropic credential that is eligible for long-context requests, or switch to an Anthropic API key.
  3. Configure fallback models so runs continue when Anthropic long-context requests are rejected.
Related:

Local OpenAI-compatible backend passes direct probes but agent runs fail

Use this when:
  • curl ... /v1/models works
  • tiny direct /v1/chat/completions calls work
  • Velaclaw model runs fail only on normal agent turns
Look for:
  • direct tiny calls succeed, but Velaclaw runs fail only on larger prompts
  • backend errors about messages[].content expecting a string
  • backend crashes that appear only with larger prompt-token counts or full agent runtime prompts
Common signatures:
  • messages[...].content: invalid type: sequence, expected a string → backend rejects structured Chat Completions content parts. Fix: set models.providers.<provider>.models[].compat.requiresStringContent: true.
  • direct tiny requests succeed, but Velaclaw agent runs fail with backend/model crashes (for example Gemma on some inferrs builds) → Velaclaw transport is likely already correct; the backend is failing on the larger agent-runtime prompt shape.
  • failures shrink after disabling tools but do not disappear → tool schemas were part of the pressure, but the remaining issue is still upstream model/server capacity or a backend bug.
Fix options:
  1. Set compat.requiresStringContent: true for string-only Chat Completions backends.
  2. Set compat.supportsTools: false for models/backends that cannot handle Velaclaw’s tool schema surface reliably.
  3. Lower prompt pressure where possible: smaller workspace bootstrap, shorter session history, lighter local model, or a backend with stronger long-context support.
  4. If tiny direct requests keep passing while Velaclaw agent turns still crash inside the backend, treat it as an upstream server/model limitation and file a repro there with the accepted payload shape.
Related:

No replies

If channels are up but nothing answers, check routing and policy before reconnecting anything.
Look for:
  • Pairing pending for DM senders.
  • Group mention gating (requireMention, mentionPatterns).
  • Channel/group allowlist mismatches.
Common signatures:
  • drop guild message (mention required → group message ignored until mention.
  • pairing request → sender needs approval.
  • blocked / allowlist → sender/channel was filtered by policy.
Related:

Dashboard control ui connectivity

When dashboard/control UI will not connect, validate URL, auth mode, and secure context assumptions.
Look for:
  • Correct probe URL and dashboard URL.
  • Auth mode/token mismatch between client and gateway.
  • HTTP usage where device identity is required.
Common signatures:
  • device identity required → non-secure context or missing device auth.
  • origin not allowed → browser Origin is not in gateway.controlUi.allowedOrigins (or you are connecting from a non-loopback browser origin without an explicit allowlist).
  • device nonce required / device nonce mismatch → client is not completing the challenge-based device auth flow (connect.challenge + device.nonce).
  • device signature invalid / device signature expired → client signed the wrong payload (or stale timestamp) for the current handshake.
  • AUTH_TOKEN_MISMATCH with canRetryWithDeviceToken=true → client can do one trusted retry with cached device token.
  • That cached-token retry reuses the cached scope set stored with the paired device token. Explicit deviceToken / explicit scopes callers keep their requested scope set instead.
  • Outside that retry path, connect auth precedence is explicit shared token/password first, then explicit deviceToken, then stored device token, then bootstrap token.
  • On the async Tailscale Serve Control UI path, failed attempts for the same {scope, ip} are serialized before the limiter records the failure. Two bad concurrent retries from the same client can therefore surface retry later on the second attempt instead of two plain mismatches.
  • too many failed authentication attempts (retry later) from a browser-origin loopback client → repeated failures from that same normalized Origin are locked out temporarily; another localhost origin uses a separate bucket.
  • repeated unauthorized after that retry → shared token/device token drift; refresh token config and re-approve/rotate device token if needed.
  • gateway connect failed: → wrong host/port/url target.

Auth detail codes quick map

Use error.details.code from the failed connect response to pick the next action: Device auth v2 migration check:
If logs show nonce/signature errors, update the connecting client and verify it:
  1. waits for connect.challenge
  2. signs the challenge-bound payload
  3. sends connect.params.device.nonce with the same challenge nonce
If velaclaw devices rotate / revoke / remove is denied unexpectedly:
  • paired-device token sessions can manage only their own device unless the caller also has operator.admin
  • velaclaw devices rotate --scope ... can only request operator scopes that the caller session already holds
Related:

Gateway service not running

Use this when service is installed but process does not stay up.
Look for:
  • Runtime: stopped with exit hints.
  • Service config mismatch (Config (cli) vs Config (service)).
  • Port/listener conflicts.
  • Extra launchd/systemd/schtasks installs when --deep is used.
  • Other gateway-like services detected (best effort) cleanup hints.
Common signatures:
  • Gateway start blocked: set gateway.mode=local or existing config is missing gateway.mode → local gateway mode is not enabled, or the config file was clobbered and lost gateway.mode. Fix: set gateway.mode="local" in your config, or re-run velaclaw onboard --mode local / velaclaw setup to restamp the expected local-mode config. If you are running Velaclaw via Podman, the default config path is ~/.Zavianx/velaclaw-dev.json.
  • refusing to bind gateway ... without auth → non-loopback bind without a valid gateway auth path (token/password, or trusted-proxy where configured).
  • another gateway instance is already listening / EADDRINUSE → port conflict.
  • Other gateway-like services detected (best effort) → stale or parallel launchd/systemd/schtasks units exist. Most setups should keep one gateway per machine; if you do need more than one, isolate ports + config/state/workspace. See /gateway#multiple-gateways-same-host.
Related:

Gateway probe warnings

Use this when velaclaw gateway probe reaches something, but still prints a warning block.
Look for:
  • warnings[].code and primaryTargetId in JSON output.
  • Whether the warning is about SSH fallback, multiple gateways, missing scopes, or unresolved auth refs.
Common signatures:
  • SSH tunnel failed to start; falling back to direct probes. → SSH setup failed, but the command still tried direct configured/loopback targets.
  • multiple reachable gateways detected → more than one target answered. Usually this means an intentional multi-gateway setup or stale/duplicate listeners.
  • Probe diagnostics are limited by gateway scopes (missing operator.read) → connect worked, but detail RPC is scope-limited; pair device identity or use credentials with operator.read.
  • unresolved gateway.auth.* / gateway.remote.* SecretRef warning text → auth material was unavailable in this command path for the failed target.
Related:

Channel connected messages not flowing

If channel state is connected but message flow is dead, focus on policy, permissions, and channel specific delivery rules.
Look for:
  • DM policy (pairing, allowlist, open, disabled).
  • Group allowlist and mention requirements.
  • Missing channel API permissions/scopes.
Common signatures:
  • mention required → message ignored by group mention policy.
  • pairing / pending approval traces → sender is not approved.
  • missing_scope, not_in_channel, Forbidden, 401/403 → channel auth/permissions issue.
Related:

Cron and heartbeat delivery

If cron or heartbeat did not run or did not deliver, verify scheduler state first, then delivery target.
Look for:
  • Cron enabled and next wake present.
  • Job run history status (ok, skipped, error).
  • Heartbeat skip reasons (quiet-hours, requests-in-flight, alerts-disabled, empty-heartbeat-file, no-tasks-due).
Common signatures:
  • cron: scheduler disabled; jobs will not run automatically → cron disabled.
  • cron: timer tick failed → scheduler tick failed; check file/log/runtime errors.
  • heartbeat skipped with reason=quiet-hours → outside active hours window.
  • heartbeat skipped with reason=empty-heartbeat-fileHEARTBEAT.md exists but only contains blank lines / markdown headers, so Velaclaw skips the model call.
  • heartbeat skipped with reason=no-tasks-dueHEARTBEAT.md contains a tasks: block, but none of the tasks are due on this tick.
  • heartbeat: unknown accountId → invalid account id for heartbeat delivery target.
  • heartbeat skipped with reason=dm-blocked → heartbeat target resolved to a DM-style destination while agents.defaults.heartbeat.directPolicy (or per-agent override) is set to block.
Related:

Node paired tool fails

If a node is paired but tools fail, isolate foreground, permission, and approval state.
Look for:
  • Node online with expected capabilities.
  • OS permission grants for camera/mic/location/screen.
  • Exec approvals and allowlist state.
Common signatures:
  • NODE_BACKGROUND_UNAVAILABLE → node app must be in foreground.
  • *_PERMISSION_REQUIRED / LOCATION_PERMISSION_REQUIRED → missing OS permission.
  • SYSTEM_RUN_DENIED: approval required → exec approval pending.
  • SYSTEM_RUN_DENIED: allowlist miss → command blocked by allowlist.
Related:

Browser tool fails

Use this when browser tool actions fail even though the gateway itself is healthy.
Look for:
  • Whether plugins.allow is set and includes browser.
  • Valid browser executable path.
  • CDP profile reachability.
  • Local Chrome availability for existing-session / user profiles.
Common signatures:
  • unknown command "browser" or unknown command 'browser' → the bundled browser plugin is excluded by plugins.allow.
  • browser tool missing / unavailable while browser.enabled=trueplugins.allow excludes browser, so the plugin never loaded.
  • Failed to start Chrome CDP on port → browser process failed to launch.
  • browser.executablePath not found → configured path is invalid.
  • browser.cdpUrl must be http(s) or ws(s) → the configured CDP URL uses an unsupported scheme such as file: or ftp:.
  • browser.cdpUrl has invalid port → the configured CDP URL has a bad or out-of-range port.
  • No Chrome tabs found for profile="user" → the Chrome MCP attach profile has no open local Chrome tabs.
  • Remote CDP for profile "<name>" is not reachable → the configured remote CDP endpoint is not reachable from the gateway host.
  • Browser attachOnly is enabled ... not reachable or Browser attachOnly is enabled and CDP websocket ... is not reachable → attach-only profile has no reachable target, or the HTTP endpoint answered but the CDP WebSocket still could not be opened.
  • Playwright is not available in this gateway build; '<feature>' is unsupported. → the current gateway install lacks the full Playwright package; ARIA snapshots and basic page screenshots can still work, but navigation, AI snapshots, CSS-selector element screenshots, and PDF export stay unavailable.
  • fullPage is not supported for element screenshots → screenshot request mixed --full-page with --ref or --element.
  • element screenshots are not supported for existing-session profiles; use ref from snapshot. → Chrome MCP / existing-session screenshot calls must use page capture or a snapshot --ref, not CSS --element.
  • existing-session file uploads do not support element selectors; use ref/inputRef. → Chrome MCP upload hooks need snapshot refs, not CSS selectors.
  • existing-session file uploads currently support one file at a time. → send one upload per call on Chrome MCP profiles.
  • existing-session dialog handling does not support timeoutMs. → dialog hooks on Chrome MCP profiles do not support timeout overrides.
  • response body is not supported for existing-session profiles yet.responsebody still requires a managed browser or raw CDP profile.
  • stale viewport / dark-mode / locale / offline overrides on attach-only or remote CDP profiles → run velaclaw browser stop --browser-profile <name> to close the active control session and release Playwright/CDP emulation state without restarting the whole gateway.
Related:

If you upgraded and something suddenly broke

Most post-upgrade breakage is config drift or stricter defaults now being enforced.

1) Auth and URL override behavior changed

What to check:
  • If gateway.mode=remote, CLI calls may be targeting remote while your local service is fine.
  • Explicit --url calls do not fall back to stored credentials.
Common signatures:
  • gateway connect failed: → wrong URL target.
  • unauthorized → endpoint reachable but wrong auth.

2) Bind and auth guardrails are stricter

What to check:
  • Non-loopback binds (lan, tailnet, custom) need a valid gateway auth path: shared token/password auth, or a correctly configured non-loopback trusted-proxy deployment.
  • Old keys like gateway.token do not replace gateway.auth.token.
Common signatures:
  • refusing to bind gateway ... without auth → non-loopback bind without a valid gateway auth path.
  • RPC probe: failed while runtime is running → gateway alive but inaccessible with current auth/url.

3) Pairing and device identity state changed

What to check:
  • Pending device approvals for dashboard/nodes.
  • Pending DM pairing approvals after policy or identity changes.
Common signatures:
  • device identity required → device auth not satisfied.
  • pairing required → sender/device must be approved.
If the service config and runtime still disagree after checks, reinstall service metadata from the same profile/state directory:
Related: