TROUBLESHOOTING

When something does not connect

Start with the journal, then match the line or the error code you see against the tables below. Every entry says who can fix it: the gateway operator, the target's owner, or the user.

Where the logs are

The gateway logs to stderr, which systemd collects. The audit log is not a file: it lives in the database, hash-chained, and is read on the Events and Recordings pages of the admin console (the Audit portal, for an auditor-only account) or exported to your SIEM. Recordings are files under the data directory.

sudo journalctl -u zanskar -n 200 --no-pager     # the last 200 lines
sudo journalctl -u zanskar -f                    # follow live while you reproduce a problem
sudo journalctl -u zanskar --since "10 min ago" -p warning   # warnings and errors only
sudo journalctl -u zanskar -o cat | grep request_id=<id>     # one request, when a user reports an error id
sudo docker logs guacd --tail 100                # the desktop relay, if RDP/VNC misbehave
ls -la /var/lib/zanskar/recordings | tail        # recordings land here as sessions end

Compose: docker compose -f deploy/docker-compose.yml logs -f zanskar and ... logs guacd. For more detail set the log level to debug on the console's Settings page (applies at once, no restart; Reset afterwards), or ZANSKAR_LOG_LEVEL=debug in the env file and restart. The console's Logs page shows the last 2,000 lines from memory, filterable and downloadable, when you have no shell on the host.

Journal lines and what they mean

Line	Meaning	Fix
`listening without TLS; acceptable only behind a TLS-terminating proxy on loopback`	Behind-proxy mode: the gateway speaks plain HTTP on loopback and expects your proxy on 443.	Expected. If you meant the gateway to serve TLS itself, re-run `zanskar init -force` with `-tls-cert` and `-tls-key`.
no admin user exists; create one with `zanskar admin create`	Migrations ran but the first admin was never created.	Run the `admin create` line from the install page as the service user.
The console says Restart required, or Cannot check /etc/zanskar/env	A boot setting changed in the env file after the gateway started; boot settings apply only to a fresh process. The second message means the service user cannot read the file.	Use the banner's Restart the gateway (waits for live sessions, then systemd restarts it). For the second: `chgrp zanskar /etc/zanskar/env && chmod 0640 /etc/zanskar/env`.
`guacd dial failed ... connection refused`	The desktop relay is not running at `ZANSKAR_GUACD_ADDR`. The package does not install it. Users see the desktop service (guacd) is not running.	Start guacd, check the address in `/etc/zanskar/env`, restart the gateway.
`guacd dial failed ... expected ready` (or an `error` from guacd)	guacd answered but could not reach the machine: RDP or VNC disabled, port closed from the gateway, or the certificate pin no longer matches. Users see could not open a connection to the machine.	Open 3389 or 5900 from the gateway host, confirm the service on the target, probe the target again after a certificate change; read `docker logs guacd` for the underlying reason.
`ssh dial failed ... host key mismatch`	The target's SSH host key changed after it was trusted. An audit event `target.hostkey.changed` was written.	Verify the new fingerprint with the machine's owner, then probe and trust it again in the target drawer. Do not trust an unexpected change without investigating.
`ssh dial failed ... authentication`	The bound credential is wrong for this host, or the login user does not exist there.	Check the credential's username and material; for SSH-CA mode, that the host trusts the CA.
`key ring: keyring: cannot unwrap key version N with the configured KEK`	The service started with a master key that is not the one the data keys are wrapped under: the environment was not updated after `zanskar key rotate-master`, or a restored backup belongs to a different installation.	Put the key that `rotate-master` wrote (or the escrowed one for that backup) in `/etc/zanskar/env` and restart. Nothing is lost while the right key exists.
`open credential ... could not be opened`	A stored secret failed to decrypt: the master key in the environment is not the one that sealed it.	Restore the original `ZANSKAR_MASTER_KEY` from escrow. If it is gone, re-enter the credentials.
`instance connect push failed`	EC2 Instance Connect refused the one-time key: the role lacks `ec2-instance-connect:SendSSHPublicKey`, or the instance has no `ec2-instance-connect` package.	Fix the role's permissions policy (the IAM setup panel shows it) or the instance's packages.
`recording could not be started`	The recordings directory or bucket is not writable, or (with S3 storage) the local spool folder is not private; the session is refused rather than run unrecorded.	Check `ZANSKAR_RECORDINGS_DIR` ownership (the service user) or the S3 credentials. If the journal says `recording: spool … is owned by another user`, `… is writable by other users` or `… is not a directory`, someone else made the default spool under `/tmp`: set `ZANSKAR_RECORDINGS_SPOOL_DIR=/var/lib/zanskar/spool` in `/etc/zanskar/env` and restart.
`audit chain broken at event N`	An audit row was altered or removed after it was written, or written by a build whose hashing differed.	Treat as an incident: restore from a backup taken before N and compare, and review database access.

Error codes the browser or API returns

Code	Meaning	Fix
`wrong version number` (from curl or a browser, not the gateway)	You spoke TLS to a plain-HTTP listener: the gateway is in behind-proxy mode.	Use `http://127.0.0.1:8443/readyz` on the host, and your proxy's `https://` address from outside.
`csrf_invalid`	A state-changing request without the session's `X-CSRF-Token`, or whose `Origin` does not match the gateway's host, typically a reverse proxy rewriting `Host`.	Make the proxy pass the original `Host` (Caddy and nginx `proxy_set_header Host $host` do); reload the page to refresh the token.
`mfa_required`, `mfa_required_by_policy`	The sign-in still owes a second factor, or the policy demands an enrolled authenticator.	Finish enrolment; admins can reset a lost authenticator with `zanskar admin reset-mfa`.
`review_only`	The account holds only the auditor role, which cannot see or connect to targets.	Grant the user role as well if this person should connect; auditors review only.
`policy_denied`	No enabled policy grants this user this protocol on this target right now (selector, protocol list or time window).	Check the policy's targets or tags, protocols and time windows, and the user's group membership.
`approval_required`	The only policy covering this target requires approval and there is no active grant.	Request access from the Targets page; an administrator approves under Approvals.
`duplicate_request`, `already_granted`, `not_extendable`	One open request per target and protocol: one is pending, a grant is already active (extend it instead), or the extension names something that is not your active grant.	Wait for the decision, or use Extend on My access.
`host_key_untrusted`, `host_key_unverified`	The SSH host key has not been trusted yet (or, for an autoscaling instance, not yet captured).	Probe and trust in the target drawer; for instances, wait for the next sync.
`certificate_unpinned`	The RDP or WinRM certificate has never been captured.	Probe the target once; the fingerprint is pinned from then on.
`certificate_mismatch`	The target presented a different certificate from the pinned one.	Expected after a renewal: probe the host again. On an autoscaling instance the pin is never replaced, so the instance goes unhealthy until the group replaces it.
`not_authorized`	The WinRM credential authenticated and was then refused a shell.	Make the account a local administrator on the target. Remote Management Users does not cover the WinRS shell Zanskar opens. See the guide.
`tls_required`	WinRM was configured on 5985 (plain HTTP), which Zanskar never uses.	Create the HTTPS listener on 5986 (guide) and set the port to 5986.
`invalid or expired ticket`	The WebSocket came too late, was reused, or came from a different address than the connect request.	Reload and connect again. If it repeats behind a proxy, make sure the proxy forwards a consistent client address (`ZANSKAR_TRUSTED_PROXIES`).
`no_credential`, `credential_required`	No credential is bound to this protocol on the target, or the target uses your own account and you did not supply it.	Bind a credential in the target drawer, or enter your username and password when asked.
`engine_unavailable`	The target's database engine has no credential-holding sidecar in this build (a target enrolled by an older API client with an engine this version does not know).	Re-enrol the target with engine postgres, mysql or mariadb.
`in_use` (on delete)	A policy still names the target or group by id, or a session is open on it.	Remove it from those policies and end the sessions, then delete again. History is kept either way.
`address_forbidden`	The target address is loopback, link-local or the gateway's own, which the probe refuses.	Enrol the machine's real address.
`no_healthy_instances`	The autoscaling group has no instance that is in service, passing its checks and pinned.	Sync the group and read its instance list for the reason per instance.
`login_user_not_permitted`	The SSH certificate authority on this target only issues certificates for the login users on its allowlist, and you (or the credential's fixed username) are not one of them.	Add the login user to the authority's allowlist in its Setup panel, or clear the list.
`not_certificate_authority`, `no_rotation`, `rotation_pending`	A certificate login test on a target whose SSH slot is not an authority; a cut-over or retire with nothing prepared; a second prepare while one is pending.	Bind an ssh_ca credential first; follow the rotation steps in order.
`password_change_required`	The account was created or reset with a one-time password that must be replaced before anything else.	Sign in with the one-time password and set your own; the second factor comes after.
`boot_setting`	The setting is fixed at install and cannot be changed from the console.	Edit the named variable in `/etc/zanskar/env`; the console then shows Restart required.
`tls_proxy`	The TLS card was used in proxy mode, where the gateway serves no certificate.	Manage the certificate on the proxy, or set `ZANSKAR_TLS_MODE=managed` and restart.
`test_unavailable`, `assume` / `describe` / `group` stages	An autoscaling access test: no cloud provider is configured, or the failing stage of the test: the trust policy or ExternalId, the permissions policy, or no such group in that region.	Fix the named stage; the gateway's own role must also be allowed `sts:AssumeRole` on the customer role.

Reaching the gateway, certificates and the HTTP redirect

Plain HTTP answers with a short page instead of redirecting: the address you typed is not in the served certificate, and the listener will not send a browser to a name the certificate cannot serve. On a cloud instance this is the normal first experience, because the public address is translated upstream and is on no interface, so the generated certificate never covered it. Add it with ZANSKAR_TLS_HOSTS in the environment file and restart, or use Regenerate self-signed on the Settings page. The page itself names the address it was asked for.
The browser warns twice: unknown issuer and a name mismatch. The mismatch goes away once the address is in the certificate, as above. The issuer warning is expected of a self-signed certificate and goes away when you upload a real one on the Settings page, which applies to the next connection with no restart.
You set the names and nothing changed: an uploaded certificate takes precedence over the generated one and is never regenerated. Check which one is active on the Settings page.
Nothing answers on 80 or 443 at all: an unprivileged service cannot bind them without CAP_NET_BIND_SERVICE. The packaged unit grants it; a hand-written one must, and the startup error says so.

Install and startup

ZANSKAR_MASTER_KEY_FILE: … is mode 0640; only its owner may read it: the key file is readable by other accounts, so the gateway will not use it. sudo chown zanskar: /etc/zanskar/master.key && sudo chmod 0400 /etc/zanskar/master.key, then restart. set ZANSKAR_MASTER_KEY or ZANSKAR_MASTER_KEY_FILE, not both means the env file still has the key line; remove it.
cosign: command not found at the verification step: cosign is in none of the Ubuntu, Debian or RHEL repositories. Install it from Sigstore's release with the commands in step 1, then run the verification again.
N: Download is performed unsandboxed as root as file '…/zanskar_….deb' couldn't be accessed by user '_apt': harmless, and the package is installed. apt reads a local package as its unprivileged _apt user, which cannot enter a home directory (0750 on Ubuntu 24.04), so it read the file as root instead. Download into /tmp/zanskar, as the install page does, and the notice goes away.
error: /etc/zanskar/env already exists; re-run with --force to overwrite: something wrote the file before init ran, most often the guacd line appended too early. Look at which settings it holds (names only): sudo cut -d= -f1 /etc/zanskar/env. With a ZANSKAR_MASTER_KEY line, run sudo zanskar init -force, which keeps that key and rewrites the rest. Without one, nothing has been sealed under it: sudo systemctl stop zanskar, sudo rm /etc/zanskar/env, then run init with -guacd so the line is written for you.
The service starts and stops at once: journalctl -u zanskar -n 50. The usual causes are a missing ZANSKAR_MASTER_KEY, a database file owned by root (run migrate and admin create as the service user, or chown -R zanskar:zanskar /var/lib/zanskar), or plain HTTP on a non-loopback address (choose a TLS mode with init).
readyz fails: pick the line for your mode on the install page; curl -k https:// against behind-proxy mode gives wrong version number.
Browser says the site is unsafe: a self-signed certificate. Use Caddy with sslip.io or your own certificate; see TLS without a certificate.
Audit log shows the proxy's address for every user: set ZANSKAR_TRUSTED_PROXIES to the proxy's address so X-Forwarded-For is believed; init -behind-proxy does this for a same-host proxy.
The browser warns about the certificate on the first visit: a managed install serves a self-signed certificate until you upload yours on the Settings page. Compare the SHA-256 the gateway logged at start with the one the browser shows, then continue and upload the real certificate.
"bind: permission denied" on port 443 or 80: the service runs unprivileged; the packaged unit grants CAP_NET_BIND_SERVICE. A hand-written unit needs AmbientCapabilities=CAP_NET_BIND_SERVICE and CapabilityBoundingSet=CAP_NET_BIND_SERVICE, or use ports above 1024 behind a proxy.
"recording could not be started" after switching to a bucket: the Recording storage card's Test shows the exact error; while a Move runs, its progress and last error are on the same card.

RDP, VNC and guacd

"the desktop service (guacd) is not running": start it (install page) and check ZANSKAR_GUACD_ADDR.
"could not open a connection to the machine": guacd is up; the machine did not answer. Open 3389 (RDP) or 5900 (VNC) from the gateway host, confirm the service is enabled, and re-probe after any certificate change. docker logs guacd names the reason.
Black screen or immediate disconnect on Windows: NLA must stay on, and the account needs the Remote Desktop Users right; a domain credential needs the domain set on the credential.
Copy and paste does nothing: the policy's clipboard switch is off, or the browser refused clipboard access; the session bar's Clipboard panel works either way.

WinRM

No WinRM button in the user portal: the probe did not mark WinRM reachable (no HTTPS listener on 5986, or 5986 closed from the gateway), or no policy grants winrm. Create the listener, open the port, probe again, bind a credential.
Pin mismatch after a certificate renewal: probe again to pin the new certificate.
The credential works elsewhere but the session is refused: the account is not a local administrator on that host. WinRM grants the WinRS shell to BUILTIN\Administrators only, and the host logs event 192, "The authorization of the user failed with error 5", in Microsoft-Windows-WinRM/Operational. Add the account with net localgroup Administrators "CORP\svc-zanskar" /add.
Wrong password, expired account or a domain trust problem shows up as an authentication failure rather than the above; the host logs 401 responses.

Active Directory and LDAP sign-in

Test connection fails with a network error or a reset connection on port 636: the controller is listening but has not bound a certificate. A self-signed certificate does not chain to anything the controller trusts, so it resets every handshake, including openssl s_client. Import the certificate into the controller's own Trusted Root Certification Authorities store and restart NTDS, or issue the certificate from your internal CA.
Zanskar refuses to save the provider: a plain ldap:// URL needs StartTLS, and either a CA certificate or an explicit decision to skip verification. A bind password is never sent in clear.
The test passes but nobody can sign in: the user filter. Run the same filter against the same base DN with the same bind account using ldapsearch or dsquery; if it returns nothing there, it returns nothing here. On Active Directory the filter is usually (&(objectClass=user)(sAMAccountName=%s)).
People sign in but reach nothing: group mapping. It takes the directory group's plain name, not its full DN, on the left and a Zanskar group id on the right; the console lists the ids. Membership is recomputed at each sign-in, so ask the person to sign out and back in after a change.
Locked out after enabling a provider: local accounts are always tried first, so the admin you created at install still works. Keep one.

Certificate logins

"authentication to the target failed" with a certificate authority bound: the target does not trust the authority yet. Use Test certificate login in the target drawer: certificate rejected means the TrustedUserCAKeys line or the public key is missing on that host, or the login user does not exist there; unreachable and host key mismatch are the ordinary probe reasons.
After a rotation, one host refuses logins: it never received the new public key. Prepare and cut over are separate for exactly this reason; test every bound target with the next key before cutting over. Until the host is fixed, its sessions fail; nothing else is affected.
The target's auth log shows the login: look for ID zanskar:<user>:<login>:<time> on accepted lines and zanskar-probe: for tests.

Database sessions

Saving a database target says tls_mode verify-full (the default) needs tls_ca: since 1.2.1 the gateway checks the database's certificate by default, so it needs the CA bundle that signed it. For Amazon RDS paste your region's bundle from https://truststore.pki.rds.amazonaws.com/<region>/<region>-bundle.pem; the global bundle is too large. To connect without checking, choose Encrypted, not verified, knowing that whoever can intercept the connection can impersonate the database.
A verified session fails at start with a certificate error: the bundle does not sign the database's certificate, or the address you entered is not a name on it. Use the endpoint name the provider gives (not an IP) and the bundle for that endpoint's region.
"docker: command not found" or a permission error in the journal: database sessions run in containers; install Docker or Podman on the gateway host and let the service user reach it (ZANSKAR_DOCKER_PATH for a non-default binary).
First session is slow: the client and proxy images are pulled on first use; pull the client image (postgres:16-alpine, mysql:8 or mariadb:11) and the sidecar (edoburu/pgbouncer for PostgreSQL, the gateway's own ghcr.io/albatroxxx/zanskar image for MySQL and MariaDB) ahead of time.
Not available inside the Compose stack: use the package install for database targets.