# Troubleshooting This page lists failures that operators of the self-hosted Compose stack meet. Each row gives the symptom, the cause, and the fix. Run the commands from `/opt/hephaestus/docker/self-host`. ## Startup and sign-in | Symptom | Cause / fix | | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `application-server` exits: *ProxyTrustGuard* | `HEPHAESTUS_TRUSTED_PROXIES` is blank. The Compose default is non-blank. Your override therefore set it to empty. Unset the override. If you changed the network, you can instead set it to match `reverse-proxy`'s IP. | | Exits: encryption-key length error | `HEPHAESTUS_SECURITY_ENCRYPTION_KEY` must be exactly 32 printable, non-space ASCII characters. Restore the original key for an existing database. On a new installation, clear the invalid value. Then rerun `./setup.sh`. | | Login page has no sign-in button | No login provider configured: both `GH_OAUTH_CLIENT_ID` **and** `_SECRET` must be non-empty (a half-filled pair is skipped and logged at ERROR). | | OAuth redirect loop / cookie never sticks | A proxy in front of Traefik is injecting a `Domain=` attribute on cookies, which browsers reject for `__Host-` cookies. Serve the stack directly on ports 80/443. | | Signed in but not admin | `HEPHAESTUS_AUTH_BOOTSTRAP_ADMINS` did not match. On the account's **first** login, the server logs (INFO) the exact `provider subject username` it saw. Copy that into the allowlist. Restart. (For an account that already existed, check that log line from its first sign-in, or use the numeric id.) | | `down` or `logs` aborts with "variable is missing" | Pass both `--env-file .env --env-file release-lock.env`. Every subcommand validates required variables. Restore the matching configuration if either file is missing. | | `docker compose up -d` stops with `dependency failed to start`, because `application-server` is `unhealthy`. The worker and the webhook server stay stopped. | The server reports healthy only after its database migrations finish. Its health check allows 10 minutes for startup, and a large database can take longer. Follow `docker compose --env-file .env --env-file release-lock.env logs -f application-server` until Liquibase finishes. Then run `up -d` again. Do not stop the server during a migration. [Upgrade § If migration fails](/admin/upgrade.md#if-migration-fails) owns recovery. | | A changed `GH_OAUTH_*` or `GITLAB_OAUTH_*` value in `.env` has no effect after a restart. | The environment creates a login provider only when that provider does not exist yet. Change an existing provider under **Instance admin → Login providers**. | | The browser shows a certificate warning, or Traefik serves its default certificate. | `Let's Encrypt` could not complete the challenge for a hostname. Check that its DNS record points at this host, that port 80 is reachable from the internet, and that `APP_HOST_MATCH` names it. [Serving more than one hostname](/admin/install.md#serving-more-than-one-hostname) explains the matcher. | ## Provider events | Symptom | Cause / fix | | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GitHub webhook deliveries show `401` | `WEBHOOK_SECRET` on GitHub differs from `.env` — must be byte-identical. | | Data appears only \~hourly | No GitHub webhook configured. See [First steps § 4](/admin/first-steps.md#4-connect-repository-data). | | A GitLab workspace receives no live events and shows `Insufficient permissions (HTTP …). You need the Owner role on GitLab group '…' with the Premium tier.` | Hephaestus registers a group hook, which needs the Owner role on a GitLab Premium group. Use an Owner token on such a group. Without the hook, only scheduled sync (`MONITORING_SYNC_CRON`, hourly by default) brings in changes. When Hephaestus finds a GitLab hook that GitLab disabled after failed deliveries, it registers the hook again and logs `Webhook auto-disabled (alert_status=disabled), re-registering`. | ## Practice reviews | Symptom | Cause / fix | | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Reviews never start. The item events show `No AI model is assigned to practice review.` | Set `AGENT_ENABLED` and `GIT_CHECKOUT_ENABLED` on the server and the worker. Then assign a model to practice reviews, as [Connect an AI Provider](/admin/ai-providers.md) describes. | | Feedback is withheld with `The work was already merged, so a note on it would arrive too late.` or `The review did not start before its deadline.` | Reviews wait longer than the work stays open. Read `agent.queue.depth` and `agent.queue.oldest_age_seconds` ([Queue health](/admin/practice-review-operations.md#queue-health)). Add memory, then raise `SANDBOX_MAX_CONCURRENT`, as [Sizing](/admin/sizing.md) describes. | | The worker restarts, and `docker inspect -f '{{.State.OOMKilled}}' ` prints `true` with exit code 137. | More reviews prepare at the same time than the worker memory limit holds. Raise `APPLICATION_WORKER_MEM_LIMIT` before you raise concurrency, and set `BPL_JVM_HEAD_ROOM` ([Memory headroom](/admin/sizing.md#memory-headroom)). | | Sandboxes do not start. The worker logs `Sandbox execution failed: errorType=…`, or its Docker health reports `Docker daemon not reachable`. | Compare `DOCKER_GROUP_ID` with `stat -c %g /var/run/docker.sock`. Check the Docker Engine version against [Tested versions](/admin/install.md#tested-versions). If you set `SANDBOX_DOCKER_CONTAINER_RUNTIME=runsc`, install gVisor first. | | Script runs end with `Practice script stopped without a result: it ran out of time.` | The model provider is slow or limits the rate of calls. [Practice review operations](/admin/practice-review-operations.md) lists the reasons. Raise the provider limits or use a faster model before you raise concurrency. | ## Disk and storage | Symptom | Cause / fix | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The host runs out of disk. Reviews fail in preparation, or JetStream reports `no space left on device`. | Four things grow on disk: repository mirrors on the `worker-git-repos` volume, images of earlier releases, the PostgreSQL write-ahead log, and the NATS store. Find the largest with `df -h` and `docker system df`. Remove unused images with `docker image prune`. On a host with pull-based deployment whose reconciler predates the image cleanup, follow [Recover a host that is already full](/admin/pull-based-deployment.md#recover-a-host-that-is-already-full) instead. A newer reconciler removes its own images. The next row covers its refusal. Size the broker store as [Webhook ingestion operations](/admin/webhook-ingestion-operations.md) describes, or add disk. | | A host with [pull-based deployment](/admin/pull-based-deployment.md) does not move. Its journal says `Only … GiB is free for Docker's data`. | The reconciler refuses a pull below `HEPHAESTUS_MIN_FREE_GIB` after its [image cleanup](/admin/pull-based-deployment.md#images-and-free-space). Free space on the host, or lower the threshold. | | Providers receive `503` from time to time. The `webhook-server` log has records with `event.name=webhook.publish.failed`, and `webhook.publish` counts `outcome=failure`. | Each record names the cause in `webhook.publish.reason` (`timeout`, `no_responders`, `connection`, `connection_closed`, `stream_full`, `rejected` or `interrupted`) and the attempts in `webhook.publish.attempts`. [Reading log records](/admin/observability.md#reading-log-records) shows how to filter them. For `timeout`, slow disk I/O delays the broker, because it calls `fsync` for each message. Read `/proc/pressure/io` and `vmstat 1` for I/O wait. Give the JetStream volume storage that a database or container runtime does not saturate. Do not restart the containers in a loop. Hephaestus redelivers failed GitHub deliveries. See [Webhook ingestion operations](/admin/webhook-ingestion-operations.md#while-the-broker-is-slow). | | `webhook-server` does not start and logs that the webhook stream bounds are over the broker storage budget. | The byte bounds of the streams add up to more than the broker store. [Webhook ingestion operations](/admin/webhook-ingestion-operations.md) gives the budget rule. |