Troubleshooting

This page lists failures that operators of the self-hosted Compose stack meet. Each row gives the symptom, the cause, and the fix. Run the commands from /opt/hephaestus/docker/self-host.

Startup and sign-in

Symptom	Cause / fix
application-server exits: ProxyTrustGuard	HEPHAESTUS_TRUSTED_PROXIES is blank. The Compose default is non-blank. Your override therefore set it to empty. Unset the override. If you changed the network, you can instead set it to match reverse-proxy's IP.
Exits: encryption-key length error	HEPHAESTUS_SECURITY_ENCRYPTION_KEY must be exactly 32 printable, non-space ASCII characters. Restore the original key for an existing database. On a new installation, clear the invalid value. Then rerun ./setup.sh.
Login page has no sign-in button	No login provider configured: both GH_OAUTH_CLIENT_ID and _SECRET must be non-empty (a half-filled pair is skipped and logged at ERROR).
OAuth redirect loop / cookie never sticks	A proxy in front of Traefik is injecting a Domain= attribute on cookies, which browsers reject for __Host- cookies. Serve the stack directly on ports 80/443.
Signed in but not admin	HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS did not match. On the account's first login, the server logs (INFO) the exact provider subject username it saw. Copy that into the allowlist. Restart. (For an account that already existed, check that log line from its first sign-in, or use the numeric id.)
down or logs aborts with "variable is missing"	Pass both --env-file .env --env-file release-lock.env. Every subcommand validates required variables. Restore the matching configuration if either file is missing.
docker compose up -d stops with dependency failed to start, because application-server is unhealthy. The worker and the webhook server stay stopped.	The server reports healthy only after its database migrations finish. Its health check allows 10 minutes for startup, and a large database can take longer. Follow docker compose --env-file .env --env-file release-lock.env logs -f application-server until Liquibase finishes. Then run up -d again. Do not stop the server during a migration. Upgrade § If migration fails owns recovery.
A changed GH_OAUTH_* or GITLAB_OAUTH_* value in .env has no effect after a restart.	The environment creates a login provider only when that provider does not exist yet. Change an existing provider under Instance admin → Login providers.
The browser shows a certificate warning, or Traefik serves its default certificate.	Let's Encrypt could not complete the challenge for a hostname. Check that its DNS record points at this host, that port 80 is reachable from the internet, and that APP_HOST_MATCH names it. Serving more than one hostname explains the matcher.

Provider events

Symptom	Cause / fix
GitHub webhook deliveries show 401	WEBHOOK_SECRET on GitHub differs from .env — must be byte-identical.
Data appears only ~hourly	No GitHub webhook configured. See First steps § 4.
A GitLab workspace receives no live events and shows Insufficient permissions (HTTP …). You need the Owner role on GitLab group '…' with the Premium tier.	Hephaestus registers a group hook, which needs the Owner role on a GitLab Premium group. Use an Owner token on such a group. Without the hook, only scheduled sync (MONITORING_SYNC_CRON, hourly by default) brings in changes. When Hephaestus finds a GitLab hook that GitLab disabled after failed deliveries, it registers the hook again and logs Webhook auto-disabled (alert_status=disabled), re-registering.

Practice reviews

Symptom	Cause / fix
Reviews never start. The item events show No AI model is assigned to practice review.	Set AGENT_ENABLED and GIT_CHECKOUT_ENABLED on the server and the worker. Then assign a model to practice reviews, as Connect an AI Provider describes.
Feedback is withheld with The work was already merged, so a note on it would arrive too late. or The review did not start before its deadline.	Reviews wait longer than the work stays open. Read agent.queue.depth and agent.queue.oldest_age_seconds (Queue health). Add memory, then raise SANDBOX_MAX_CONCURRENT, as Sizing describes.
The worker restarts, and docker inspect -f '{{.State.OOMKilled}}' <container> prints true with exit code 137.	More reviews prepare at the same time than the worker memory limit holds. Raise APPLICATION_WORKER_MEM_LIMIT before you raise concurrency, and set BPL_JVM_HEAD_ROOM (Memory headroom).
Sandboxes do not start. The worker logs Sandbox execution failed: errorType=…, or its Docker health reports Docker daemon not reachable.	Compare DOCKER_GROUP_ID with stat -c %g /var/run/docker.sock. Check the Docker Engine version against Tested versions. If you set SANDBOX_DOCKER_CONTAINER_RUNTIME=runsc, install gVisor first.
Script runs end with Practice script stopped without a result: it ran out of time.	The model provider is slow or limits the rate of calls. Practice review operations lists the reasons. Raise the provider limits or use a faster model before you raise concurrency.

Disk and storage

Symptom	Cause / fix
The host runs out of disk. Reviews fail in preparation, or JetStream reports no space left on device.	Four things grow on disk: repository mirrors on the worker-git-repos volume, images of earlier releases, the PostgreSQL write-ahead log, and the NATS store. Find the largest with df -h and docker system df. Remove unused images with docker image prune. On a host with pull-based deployment whose reconciler predates the image cleanup, follow Recover a host that is already full instead. A newer reconciler removes its own images. The next row covers its refusal. Size the broker store as Webhook ingestion operations describes, or add disk.
A host with pull-based deployment does not move. Its journal says Only … GiB is free for Docker's data.	The reconciler refuses a pull below HEPHAESTUS_MIN_FREE_GIB after its image cleanup. Free space on the host, or lower the threshold.
Providers receive 503 from time to time. The webhook-server log has records with event.name=webhook.publish.failed, and webhook.publish counts outcome=failure.	Each record names the cause in webhook.publish.reason (timeout, no_responders, connection, connection_closed, stream_full, rejected or interrupted) and the attempts in webhook.publish.attempts. Reading log records shows how to filter them. For timeout, slow disk I/O delays the broker, because it calls fsync for each message. Read /proc/pressure/io and vmstat 1 for I/O wait. Give the JetStream volume storage that a database or container runtime does not saturate. Do not restart the containers in a loop. Hephaestus redelivers failed GitHub deliveries. See Webhook ingestion operations.
webhook-server does not start and logs that the webhook stream bounds are over the broker storage budget.	The byte bounds of the streams add up to more than the broker store. Webhook ingestion operations gives the budget rule.