Troubleshooting

Common issues and their solutions when running Patrimoine locally or in production.

---

Frontend loads but API calls fail

Cause: nginx is not proxying requests correctly to the backend.

Solutions:

• Check that the backend is running and healthy: podman compose ps — the status must be healthy.
• Check BACKEND_URL in frontend/container/.env (or the compose default http://backend:8000): it must be an origin only, http(s)://host[:port], with no path and no trailing slash. Inspect the rendered config with podman compose exec frontend cat /etc/nginx/conf.d/default.conf. A wrong host answers 502 until the name resolves; see FRONTEND_BACKEND_RESOLVER.
• Check backend logs: podman compose logs backend.

---

502 Bad Gateway

Cause: The backend is not ready yet or has crashed.

Solutions:

• Wait a few seconds — the backend may still be starting up (Alembic migration
  • uvicorn).
• Check logs: podman compose logs backend.
• Check the healthcheck: podman compose ps — the backend must reach healthy status.

---

502 Bad Gateway on the OIDC callback

Cause: The backend completes the sign-in, but its /auth/callback response carries three token cookies (access, refresh and ID tokens) that do not fit in nginx’s default header buffer. The nginx log shows upstream sent too big header while reading response header from upstream. Larger tokens (extra claims or groups, offline_access) make this more likely.

Solutions:

• Use a frontend image whose nginx configuration sets proxy_buffer_size 16k; and proxy_busy_buffers_size 16k;.
• If you run your own nginx in front of the app, add proxy_buffer_size 16k; and proxy_busy_buffers_size 16k; to the /auth/ location — see Reverse Proxy.
• Check the nginx log: podman compose logs frontend.

---

OIDC: “Missing PKCE verifier cookie”

Cause: The pkce_verifier and oauth_state cookies set by /auth/oidc/start are not being sent back by the browser on the OIDC callback.

Solutions (check each in order):

1. SameSite=Lax, not None — Firefox Total Cookie Protection partitions SameSite=None cookies, preventing them from being sent on the callback The app uses SameSite=Lax by default.
2. nginx /auth/ location — only relevant if you run your own nginx: it must use proxy_http_version 1.1; and proxy_buffering off;. Without these, nginx uses HTTP/1.0 and enables buffering, which can strip Set-Cookie headers on 302 responses.
3. Uvicorn --proxy-headers — the backend must run with --proxy-headers so Uvicorn processes X-Forwarded-Proto and X-Forwarded-Host from the reverse proxy. This is the default in backend/container/Containerfile.
4. TRUSTED_PROXY=true (default) — enables reading X-Real-IP for rate limiting and X-Forwarded-Proto for HTTPS detection.
5. X-Forwarded-Proto: https — the reverse proxy must send this header so the backend sets the Secure flag on cookies.

---

OIDC: “State mismatch”

Cause: Cookies are set by the browser but not sent on the callback request.

Solutions:

• Check the SameSite and Secure attributes — if the callback is over HTTPS, cookies must be Secure. If over HTTP, they must not be Secure.
• Check that Path=/ is set on the cookies.
• Inspect the Firefox Network tab for the Cookie header presence on the callback request.

---

OIDC: Sessions expire quickly (refresh_token empty)

Cause: The OIDC provider does not return a refresh_token in the token exchange response, or returns it empty. When the access token expires (typically 5–15 minutes for OIDC providers), the backend cannot refresh it, and the user is logged out.

Symptom: The refresh_token cookie may appear empty or contain """" in the browser’s cookie inspector. This happens when the OIDC provider returns an empty string as the refresh_token value — the app now skips setting the cookie entirely when the value is empty, preventing confusion.

Diagnosis:

• Check backend logs for the warning: OIDC provider did not return refresh_token — sessions will expire when access_token expires. Ensure the provider grants 'offline_access' scope.
• This warning is logged in the /auth/callback handler when the token exchange response lacks a refresh_token field or returns it empty.
• Enable DEBUG=true in backend/container/.env to see additional diagnostic logging: OIDC token response keys: [...] — refresh_token present: True/False, length: N

Solutions:

1. Grant offline_access scope in Authentik — Go to Admin → Applications → Providers → [your provider] → Edit. In Advanced protocol settings, ensure offline_access is listed in the selected scopes. Alternatively, create a Scope Mapping for offline_access and assign it to the application.
2. Verify the scope is requested — The app requests openid profile email offline_access in /auth/oidc/start (auth.py). This is hardcoded and correct — the issue is always provider-side.
3. Check provider logs — Authentik logs the token exchange in the Authorisation flow logs. Verify the offline_access scope is present in the granted scopes.

Note

Note: Even with a valid refresh token, sessions can still expire if the OIDC provider rotates refresh tokens (as Authentik does by default). The backend handles rotation by updating the refresh_token cookie when the provider returns a new one during the refresh flow.

---

Container won’t start

Cause: Permissions on the ./data directory.

Solutions:

• Verify the directory exists: ls -la ./data. Create it if missing: mkdir -p ./data.
• In rootless mode, Podman maps the container user to your host user. The ./data directory must be writable by your user.
• Check logs: podman compose logs backend for the exact error.

---

Backend exits at startup with a settings error

Cause: a setting failed validation; the log line names the variable (for example JWT_SECRET is too short, METRICS_TOKEN is too short, TRUSTED_PROXIES must be a comma-separated list of IP addresses or CIDRs). E2E_TEST_MODE=true without DEBUG=true also stops the backend.

Solutions: fix that variable in backend/container/.env using Environment Variables and restart. Generate secrets with python -c "import secrets; print(secrets.token_urlsafe(48))".

With podman.sh, a PostgreSQL or Redis on a non-default port, or a PostgreSQL cluster of an older major version, is also refused at start.

---

Scheduled jobs never run

Cause: REDIS_URL is set but no arq worker runs; while it is set the API leaves the scheduled jobs to the worker.

Solutions: run the worker (Background Worker), or clear REDIS_URL.

---

/metrics answers 404 or 401

404 means METRICS_TOKEN is empty; 401 means a missing or wrong bearer token. The frontend’s nginx does not proxy /metrics: scrape the backend on the container network (backend:8000). See Metrics.

---

Migration failures

Cause: Corrupt database or inconsistent schema version.

Solutions:

• Check current migration state:

  podman compose exec backend alembic current
  podman compose exec backend alembic history

• If the database is empty or corrupt, you can force a fresh creation: stop the containers, delete ./data/patrimoine.db, and restart.

  WARNING: This deletes all data.

---

Database is locked

Cause: SQLite does not handle concurrent access from multiple processes. This typically happens when sqlite3 commands are run directly on ./data/patrimoine.db while the backend container is running.

Diagnosis:

• Backend logs show: sqlite3.OperationalError: database is locked on INSERT/UPDATE statements.
• The error may persist after restarting the container because the WAL/SHM files were created by the external sqlite3 process with different permissions.

Solutions:

1. Restart the backend — podman compose restart backend. This forces SQLAlchemy to create fresh connections.
2. Fix file permissions — if the container user (UID 10001) can’t read the database files after a direct sqlite3 access:

   chmod 666 data/patrimoine.db data/patrimoine.db-shm data/patrimoine.db-wal
   podman compose restart backend

3. Disable automatic backup — the internal backup task can also cause lock contention. Set AUTO_BACKUP=false in backend/container/.env and restart.
4. Prevent recurrence — never run sqlite3 directly on the database while the container is running. Use podman compose exec backend sqlite3 instead, or stop the backend first:

   podman compose stop backend
   sqlite3 data/patrimoine.db "..."
   podman compose start backend

---

ADMIN_EMAILS has no effect

Cause: ADMIN_EMAILS and ADMIN_SUBS only promote accounts that signed in through your OIDC provider, which has verified their email. A local-auth account (email and password registered in Patrimoine) is never promoted, whatever these variables contain — otherwise anyone could claim admin by registering with the configured address before its owner did. For the same reason, a local:<email> value in ADMIN_SUBS is ignored. Email matching is case-insensitive; ADMIN_SUBS must match the sub claim exactly.

Solutions (local-auth instance):

1. Register the first account — when no account exists yet (deleted accounts do not count), the first account registered becomes admin automatically. Registration must be open (REGISTRATION_ENABLED=true); close it again afterwards.

2. Ask an existing admin — open the Administration entry in the profile menu, go to the Users page and tick the Admin checkbox on the user’s row. See User management.

3. Last resort: edit the database — when accounts already exist but none is an admin (for example after updating from a release without the admin role). Back up the database first, then set is_admin on that account’s row in the users table:

   • SQLite (the booleans are stored as 0/1):

     podman compose exec backend sqlite3 /data/patrimoine.db \
       "UPDATE users SET is_admin = 1 WHERE email = 'you@example.com' AND auth_provider = 'local' AND deleted_at IS NULL;"

   • PostgreSQL:

     docker compose -f docker-compose.yml -f docker-compose.postgres.yml exec postgres \
       psql -U patrimoine patrimoine -c "UPDATE users SET is_admin = true WHERE email = 'you@example.com' AND auth_provider = 'local' AND deleted_at IS NULL;"

   Use the account’s email in lowercase. The role is read from the database on every request and is not stored in the session, so no restart is needed: sign out and back in (or reload the page) and the Administration entry appears.

OIDC instance: check that the email in ADMIN_EMAILS (or the sub in ADMIN_SUBS) matches what your provider sends, then restart the backend or sign in again.

---

Cookie not sent on callback

Cause: Browser is rejecting or partitioning the cookie.

Solutions:

• Open Firefox DevTools → Network tab. Check the callback request for a Cookie header. If absent, the browser did not send the cookies.
• Open Firefox DevTools → Storage tab. Inspect the cookie attributes:
  • Path=/
  • SameSite=Lax
  • Secure=true (HTTPS only)
  • HttpOnly=true

---

Log inspection

Command	Purpose
podman compose logs backend	Backend (FastAPI/uvicorn) logs
podman compose logs frontend	Frontend (nginx) logs
./podman.sh logs [backend|frontend|docs|postgres|redis|worker]	Logs of a podman.sh deployment

---

See Also

• Cookies & CSRF
• Reverse Proxy
• Backup & Restore — Internal automatic backup, AUTO_BACKUP env var