Background Worker (arq + Redis)

Patrimoine supports an optional background worker for distributed task processing via arq and Redis. When Redis is not configured, the backend falls back to an asyncio-based scheduler.

When to Use the Worker

The worker enables a distributed task queue for background jobs that benefit from running outside the request-response cycle:

Database backups (SQLite .backup)
Recurring snapshot creation
Foreign exchange rate refresh
Audit log cleanup
Delta cache refresh (heatmap data)

For single-user deployments, the asyncio fallback is sufficient. The worker is recommended for multi-user or high-traffic deployments where background jobs should not compete with API requests.

Setup

1. Uncomment the worker service

In docker-compose.yml, uncomment the worker block:

worker:
  build:
    context: ./backend
    args:
      UV_EXTRA_ARGS: "--extra redis"
  command: arq app.worker.WorkerSettings
  env_file: backend/container/.env
  depends_on:
    backend:
      condition: service_healthy
  restart: unless-stopped
  mem_limit: 256m
  cpus: 0.5

The default backend image already includes the redis extra (arq) and the postgres extra, so the worker can reuse it. The UV_EXTRA_ARGS build arg in the example only matters if you build the worker image separately; setting it replaces the default, so it installs redis only. If you use PostgreSQL, set UV_EXTRA_ARGS: "--extra postgres --extra redis", otherwise the worker cannot connect to the database.

2. Add Redis and set REDIS_URL

docker-compose.yml does not ship a Redis service. Add one (for example redis: { image: docker.io/library/redis:7-alpine }) or point at an existing Redis. Both the backend and worker services must reach it.

In backend/container/.env, use the Redis service name, not localhost (each compose service has its own network namespace):

REDIS_URL=redis://redis:6379

3. Start the stack

podman compose --env-file backend/container/.env up --build

With podman.sh, none of the steps above are needed: set REDIS_URL=redis://127.0.0.1:6379 and ./podman.sh up starts Redis and the worker in the pod. See Optional Redis and worker containers.

Asyncio Fallback

If REDIS_URL is empty or not set, the backend uses an asyncio fallback (asyncio.create_task in the main.py lifespan). This runs the same background jobs in-process without Redis or a separate worker container.

Mode	REDIS_URL	Worker container	Behavior
Distributed	`redis://...`	Running (arq)	Jobs run in a separate process via arq cron schedule
Fallback	empty	Not needed	Jobs run in-process via asyncio scheduler

In both modes every job run is counted as a success or an error on the metrics endpoint.

The fallback is suitable for single-user deployments. It does not provide a distributed queue — if the backend restarts, in-flight tasks are lost.

Background Jobs

The asyncio fallback (backend/app/main.py’s lifespan) runs thirteen recurring jobs:

#	Job	Interval (asyncio)	Description
1	Daily backup	86400s (24h)	SQLite online backup via `sqlite3.connect().backup()`. Only runs when `AUTO_BACKUP=true`. The asyncio fallback runs it at startup and then every 24h; the worker runs it at 03:00
2	Recurring snapshots	3600s (1h)	`run_due_snapshots` — reschedules due reminder rules (sets their last and next run; it creates no snapshot and sends no email, the alert check raises the reminder)
3	Planned transaction sweep	3600s (1h)	`run_due_planned_transactions` — a unified sweep that auto-posts due planned transactions and sends reminders for ones coming due soon (replaces the former separate recurring-transaction and bill-reminder checks)
4	FX rate refresh	86400s (24h)	`refresh_fx_rates` for all users with non-EUR accounts
5	Audit log cleanup	86400s (24h)	Deletes audit logs older than `AUDIT_LOG_RETENTION_DAYS`
6	Delta cache refresh	3600s (1h)	`refresh_delta_cache` — recomputes daily delta cache for the heatmap visualization
7	Price refresh	86400s (24h)	Refreshes investment holding prices from the market-data provider
8	Scheduled reports	3600s (1h)	Sends each user’s periodic net-worth/CAGR report email on its configured cadence (the arq cron below runs it once daily instead)
9	Webhook delivery retry	60s	Retries failed webhook deliveries — the shortest backoff is 1 minute, so this runs far more often than the others; it always runs in the API process, with or without the worker
10	Webhook delivery cleanup	86400s (24h)	Deletes webhook deliveries older than `WEBHOOK_DELIVERY_RETENTION_DAYS`
11	Alert check	1800s (30 min)	`run_alert_checks` — evaluates each user’s alert rules, net worth thresholds and due snapshot reminders, stores one alert per occurrence and posts `alert.fired` to the user’s webhooks, and emails each new notification of a type the user set to email or both when SMTP is configured (never for a user’s first check)
12	Session purge	86400s (24h)	Deletes user session rows that expired more than 30 days ago. Always runs in the API process
13	Revoked-token purge	86400s (24h)	Deletes expired revoked refresh tokens (also once at startup). Always runs in the API process

The first time alerts are evaluated for a user, by this check or when the user saves their alert rules, the alerts found are stored without sending alert.fired, so webhook consumers do not get a burst of events for every account and bill already stale or due (for example right after upgrading to the version that added alerts, or once a new user has imported their history). That first evaluation is recorded per user and happens only once: every alert raised after it is posted, a few at a time, including the very first alert of a user who had nothing stale or due at that point.

arq Cron Schedule

When running with arq (Redis mode), backend/app/worker.py’s WorkerSettings.cron_jobs covers 10 of the 13 jobs above:

Job	Schedule	Meaning
Daily backup	`hour=3`	Runs at 03:00 every day
Recurring snapshots	`minute={0, 30}`	Runs at :00 and :30 every hour
Planned transaction sweep	`minute={0, 30}`	Runs at :00 and :30 every hour
FX rate refresh	`hour=4`	Runs at 04:00 every day
Audit log cleanup	`hour=5`	Runs at 05:00 every day
Webhook delivery cleanup	`hour=5, minute=30`	Runs at 05:30 every day
Delta cache refresh	`hour={6, 12, 18}`	Runs at 06:00, 12:00, and 18:00
Price refresh	`hour=4, minute=30`	Runs at 04:30 every day
Scheduled reports	`hour=7`	Runs at 07:00 every day
Alert check	`minute={0, 30}`	Runs at :00 and :30 every hour