Files
ZitPlane/docs/TROUBLESHOOTING.md
T

14 KiB

ZitPlane — Troubleshooting and safe diagnostics

Scope: Only describe issues the current source or recorded phase evidence supports. Run #86 verified an isolated disposable backend, but no developer-PC environment or production deployment has been verified.

Symptom Known possible cause Safe next action
docker command not found Docker CLI/Desktop was missing on the previously inspected Windows host during P02 Install supported Docker Engine/Desktop and run only in a local disposable environment; do not treat a source manifest as runtime acceptance
docker compose config / up fails Local machine Docker/config differences or occupied port may differ from validated CI stack Gitea Run #86 verified isolated portless CI Compose; check environment-specific logs with secrets redacted; do not confuse its portless override with P02 local loopback Compose
Admin Console login fails First-instance bootstrap, Login V2 container/PAT, auth proxy or credentials not validated Verify local health, first-init logs and policy in a secret-safe environment; do not reinitialize volumes casually
Account Portal shows no page App dependencies are not installed; no Account Portal image, runtime or domain is configured Complete reviewed lockfile/build/CI and explicit integration phase before troubleshooting live UI
Redirect/callback error after future login OIDC issuer/callback URL, client type/secret, proxy headers, PKCE/state or allowed redirect may mismatch Compare exact configured origins and registered callback; never weaken checks or wildcard redirects
Unauthorized on /api/account/me Designed result for absent/expired user identity Authenticate through ZITADEL; if unexpected in a future runtime, inspect token expiry and session trace securely
Profile Save button missing Portal source has read-only fields only Use permitted ZITADEL self-service (if configured) or wait for an approved custom edit feature
MFA/passkey settings not editable in portal Portal only shows informational cards and a link Use the identity provider's policy-supported controls; do not paste secrets into support messages
Sign-out seems to leave other applications signed in Portal signOut is portal-local, not federated logout Close respective sessions via app/identity provider; global logout requires a future dedicated implementation
auth.vib.tools / account.vib.tools unavailable They are planned hostnames, no production deployment configured Do not assume DNS/SSL/Coolify service exists; await P07
Login V2 custom source appears unchanged when starting P02 Compose P02 Compose uses prebuilt official Login UI, not P03 imported code Later P05 must build and wire custom image after CI and explicit approval

How to report a safe issue

Include environment (local disposable vs CI vs production), exact phase/commit, which page/action, a redacted error and reproduction conditions. Never publish .env, client secret, passwords, access tokens, cookies, private keys or user data. Link to relevant phase note; record the fix and updated user guidance together.

P05-CI-02 verification and error boundaries

  • Prior Account Portal TS2345 at src/auth.ts(14,61) in Gitea Run #81 was fixed by verified JWT subject narrowing; Gitea Run #83 Account Portal typecheck/build PASS.
  • Previous Alpine backend checkout missed Git; Run #83 verified the fix. Run #83 also exposed an occupied shared-runner host TCP 8080. Run #86 verified the portless CI Compose overlay: all four services ready, disposable Postgres restart persistence, ZITADEL ready/OIDC discovery/JWKS and complete CI-only cleanup PASS. This does not prove the separate P02 local PC Compose or real user login.
  • A green HIGH/CRITICAL advisory audit gate still allows Moderate findings. P05-SEC-01 updates only approved Next.js/OTel dependency manifests and lockfiles and adds an independent zero Moderate/High/Critical acceptance gate. Potential glob/minimatch transitive major-version issues remain open until a real registry CI audit proves otherwise; no suppressions or forced compatibility-breaking overrides.
  • CI negative access tests do not simulate real login; actual OIDC app registration/authorized login/MFA must receive separate approval.

Known limitations

Account Portal and Login V2 build CI passed in Run #83, disposable backend acceptance passed in Run #86, and dependency lockfiles/workflows exist. Real authenticated OIDC callback/positive session, MFA/passkey, Admin Console human login, production Coolify/TLS/backup and production support remain UNVERIFIED. Preserve the Scope Lock in notes/ and ZITADEL security boundaries.

P05-SEC-01 exact security-blocker troubleshooting

Gitea Run #87 produced 3 PASS and 1 FAIL. The FAILED security_login job is a deliberate fail-closed Moderate threshold: GHSA-3g3m-2fjr-6pw4 in glob (fix >=14.0.2) and GHSA-g3hv-75wg-m249 in minimatch (fix >=11.0.1). Account npm audit returned 0. Login's earlier Next.js/OTel advisories no longer reproduce, its frozen build and 1219 tests PASS, and High/Critical remain 0. Do not turn off auditing, ignore Moderate reports, modify protected auth code or force incompatible package majors. A separate P05-SEC-02 exact Scope Lock must allow controlled PNPM workspace overrides or verified compatible parent upgrades; only a new focused CI run can close security acceptance.

P05-SEC-02 — scoped major transitive override troubleshooting

If a frozen Login install, Nx build or unit suite fails after this patch, verify the seven exact parent@version>dependency selectors in auth/ui/v2/pnpm-workspace.yaml and the matching pnpm-lock.yaml snapshots; compare with notes/P05-SEC-02.md. glob@14 and minimatch@11 are semver-major transitive substitutions, so historical parents expecting v10 APIs may need a separately approved compatible parent upgrade. Do not apply global overrides, blanket pnpm audit --fix, disable the strict zero-Moderate gate, or edit protected Login authentication code. Full security acceptance requires current CI frozen install/build/Vitest and registry audit, not just a clean lockfile.

P05-SEC-02 Run #88: false-positive override validation (not a runtime failure)

Run #88 Login build job #190 stopped before pnpm install --frozen-lockfile when the new transitive_override_contract.py classified old axios@>=... / brace-expansion@>=... / esbuild@>=... selectors as extra parent overrides. The regex was narrowed after the run to accept only direct >glob or >minimatch overrides. This fix has not been tested by CI. Independent Login production security audit job #192 passed with 0 Moderate, 0 High, 0 Critical; do not confuse that security result with an authenticated login, a production deployment, or a successful frozen build. A separate, explicitly approved Gitea CI run is required for complete Login dependency compatibility acceptance.

P05-SEC-02-A01 — Run #88 validator false-positive closed in Run #89

The earlier Run #88 CI helper misread axios@>= and similar version-range keys as parent overrides. The already-committed suffix-only >glob / >minimatch correction passed in Gitea Run #89, followed by successful frozen PNPM install, original Login Nx production build and 1219/1219 unit tests. The production registry vulnerability audit also passed with 0 Moderate/High/Critical. The prior false-positive is now CI-verified resolved; no repeat local test or production deployment. If future login failures occur, distinguish CI build/unit success from still-untested authenticated OIDC/callback/MFA integration.

P05-OIDC-01 — isolated real OIDC browser acceptance troubleshooting

The new manual Gitea p05-oidc-acceptance.yml is not a production deployment. If runner Docker ownership preflight fails, do not delete an existing network/volume or reuse secrets; use a clean runner. If IAM_OWNER PAT or API client/user creation fails, do not use P02's restricted IAM_LOGIN_CLIENT PAT as a substitute. If the original Login V2 browser or Auth.js callback fails, report only the safe blocker marker and do not mock tokens, disable PKCE/state/nonce or alter production auth source. If cleanup cannot prove empty run-owned Docker resources, the job must remain FAILED. The HTTP localhost issuer works only in the private Gitea CI development environment; production HTTPS requires a separate Scope Lock. Exact current evidence: notes/P05-OIDC-01.md.

P05-OIDC-01 Run #90 — private loopback forwarder NOT ready

Job #197 in Run #90 failed after four healthy backend services. The CI-only forwarding sidecar never made localhost:8080 reachable within the 70-second provisioner wait. The sidecar suppresses its apk add socat startup diagnostics, so its exact failure cause is not confirmed. Do not disable issuer checks, expose port 8080, reuse an IAM_LOGIN_CLIENT token as admin or modify protected Auth.js/Login V2 source. The proper next step is a separately approved narrow CI-only forwarder diagnostics/pinning fix, followed by one isolated run. always() cleanup PASS; no CI-owned Docker resources remained. No real client/user/browser OIDC assertions ran.

P05-OIDC-02 diagnostic separation: If private Traefik:80 returns a non-200 status with Host: localhost:8080, the proxy/Host route gate fails before launching the sidecar. If the pinned prebuilt socat container exits or is not running, the CI job records a nonsecret container state and exit code. If the process stays running but localhost:8080 HTTP 200 fails, only the HTTP status or fixed socket marker is printed. None of these is a real login/MFA issue until client/user/browser stages execute. Do not enable host ports, disable PKCE or print secret URLs; require successful owned cleanup on every run. Actual evidence: notes/P05-OIDC-02.md.

P05-OIDC-02 Run #91 verified distinction: a private Host: localhost:8080 request to Traefik:80 is healthy, but the version-pinned prebuilt socat sidecar immediately exited code 1. Image entrypoint/digest check passed; this is a forwarder-process startup blocker, not evidence of an Auth.js/password/PKCE problem. No full 8080 forwarder response or browser flow was attempted after exit. Its exact stderr cause is unknown. The only approved next action is narrowly scoped safe startup diagnosis/forwarder fix in P05-OIDC-03; no open host port or production change. Owned cleanup PASS.

P05-OIDC-03 CI-only follow-up: Run #91's socat process exited code 1 despite healthy Traefik Host route; exact stderr root cause was unknown. New run.sh captures only a fixed pre-credential startup error class (never raw logs), then can switch once within the same run to the official, SHA256-digest-selected Python stdlib forwarder. If private HTTP 200 still fails, stop before OIDC client/user provisioning and keep the failure marked BLOCKED. No host ports, no PKCE/nonce/state bypass, no local app tests, and mandatory existing cleanup remains unchanged. See docs/P05-OIDC-03.md for actual terminal result.

P05-OIDC-03-A01 — Gitea Run #93 queued, job not started

Existing Gitea Run #93/job #206 has not been assigned a running job at the inspected snapshot: no steps, no actual start timestamp and job log API returns 404 job not started. Workflow p05-oidc-acceptance.yml is active and requests ubuntu-latest. This does not prove the runner is offline, that labels mismatch, or that the Python/socat forwarder failed. To investigate, have a suitably authorized Gitea administrator inspect runner online status, matching labels and capacity without changing settings. Do not manually dispatch another run or open production ports. No Run #93 teardown occurred because no resources were created. Details: notes/P05-OIDC-03-A01.md.

P05-OIDC-03-A02 — Run #93 real terminal blocker, not a runner-queue outage

Previously queued Run #93 eventually ran: job #206 started and failed after healthy backend/proxy. Socat and Python fallback both could not bind private 127.0.0.1:8080. Socat classified SOCKET_BIND_ERROR; Python explicitly ADDRESS_IN_USE. That is strong evidence of a port collision, but there is no verified socket inode/process owner yet; do not attribute it to Traefik, ZITADEL or another process by guess. Do not disable issuer/PKCE checks, expose host 8080, reuse production secrets, or edit original backend/Account code. Safe next step requires separate approved CI-only private listener inspection before forwarding. Existing always() cleanup PASS; no residual run-owned containers/network/secrets. Full evidence: docs/P05-OIDC-03-A02.md.

P05-OIDC-04 — Failed private readiness is not an authentication regression

Gitea Run #95 passed isolated Docker startup and four healthy backend services, then the separate no-secret private Traefik GET /debug/ready with Host: localhost:8080 FAILED with OIDC_CI_PRIVATE_PROXY_SOCKET_OR_HTTP_BLOCKED. This marker does not distinguish refused, timeout or unexpected HTTP response. The planned pre-bind 8080 socket/PID collector and both forwarders did not execute; therefore an earlier ADDRESS_IN_USE owner was not determined. Do not blame a service or Auth.js, open a host port, disable OIDC checks or repeat CI without separate approval. Always-run cleanup PASS. Next: P05-OIDC-04-A01, narrowly classify direct private socket/HTTP readiness while redacting all payloads. Exact evidence.