# Troubleshooting

This page covers a self-hosted deployment on the Helm chart. Before you change anything, collect the release version, the Helm history of the release, and the events and logs of the affected Pod. Do not print Secret objects or credentials while you collect diagnostics. [Kubernetes troubleshooting](/next/admin/kubernetes.md#troubleshooting) covers a failed first install and other symptoms of the chart.

## Startup fails: `server.tomcat.remoteip.internal-proxies is unset while forward-headers-strategy=native in prod`

**Cause:** `trustedProxies` holds only whitespace. The chart schema refuses an empty value. The server does not start in production with a blank value.

**Fix:**

1. Find the addresses of the Gateway Pods, as [Prepare the Gateway](/next/admin/kubernetes.md#prepare-the-gateway) describes.
2. Set `trustedProxies` to a Tomcat regular expression for the Pod network that holds them.
3. Run `helm upgrade` with your site values.

**Check:** The server Pod becomes ready.

## Startup fails: `An encryption key is required in production`

**Cause:** The release Secret has no `HEPHAESTUS_SECURITY_ENCRYPTION_KEY`. A server with the `prod` profile does not start without it.

**Fix:**

1. Restore the exact value of the key from your off-host copy, as [System key](/next/admin/secret-custody.md#system-key) describes. Do not make a new key for an existing database. The server cannot read the values that the old key sealed.
2. Restart every role, as [Restart a role](/next/admin/secret-custody.md#restart-a-role) describes.

**Check:** The server Pod becomes ready, and you can sign in.

## The login page has no sign-in button

**Cause:** No complete GitHub or GitLab login provider is configured.

**Fix:**

1. On a new instance, configure the OAuth App, as [First steps](/next/admin/first-steps.md#1-configure-a-login-provider) describes.
2. To change an existing provider, open **Instance administration → Login providers**.

**Check:** The login page offers **Continue with GitHub** or the button of your GitLab provider.

## Sign-in succeeds, but the account has no instance-admin navigation

**Cause:** `config.auth.bootstrapAdmins` does not name the account by the provider's stable account identifier. Hephaestus promotes an account only when it signs in.

**Fix:**

1. Compare each entry of `config.auth.bootstrapAdmins` with the account, as [First steps](/next/admin/first-steps.md#2-name-the-first-admin) describes.
2. Correct the entry, and run `helm upgrade` with your site values.
3. Sign in again.
4. If no admin exists and the entry is correct, follow [Bootstrap and recovery](/next/admin/instance-admin.md#bootstrap-and-recovery).

**Check:** After sign-in, the sidebar shows **Instance administration**.

## Helm reports `failed`, or a Pod waits for a pending change set

**Cause:** The migration Job has not finished the schema change. A Helm failure does not prove that the Job stopped.

**Fix:**

1. Read the log of the migration Job:

   ```
   kubectl --namespace hephaestus logs job/<migration-job>
   ```

   Replace `<migration-job>` with the Job name of the Helm revision.

2. Follow [Recover a stopped cutover](/next/admin/pull-based-deployment.md#recover-a-stopped-cutover).

**Check:** The migration Job completes, and the Pod of each role becomes ready.

## Every path returns `404`

**Cause:** The listener of the Gateway does not allow routes from the release namespace.

**Fix:**

1. Read the route conditions:

   ```
   kubectl --namespace hephaestus describe httproute
   ```

2. If the conditions show `Accepted=False NotAllowedByListeners`, set `allowedRoutes.namespaces` on the listener, as [Prepare the Gateway](/next/admin/kubernetes.md#prepare-the-gateway) shows.

**Check:** The route conditions show `Accepted=True` and `ResolvedRefs=True`.

## The browser reports a certificate error

**Cause:** The Gateway does not serve a valid certificate for the host name.

**Fix:**

1. Check that the DNS record of the host name points to the Gateway.
2. Check the Gateway certificate and the status of its cert-manager Certificate.
3. On one host, follow [Certificate renewal](/next/admin/k3s.md#certificate-renewal).

**Check:** The browser opens `https://<your hostname>` without a certificate error.

## GitHub deliveries return `401`

**Cause:** The secret of the GitHub webhook does not match `WEBHOOK_SECRET` in the release Secret, or a proxy changes the body or the signature header.

**Fix:**

1. Compare the configured webhook secret through a protected process. Do not put either value in logs.
2. Set the secret of the App or of the manual webhook to the value of `WEBHOOK_SECRET`.
3. Make sure that each proxy keeps the body and the `X-Hub-Signature-256` header.
4. Deliver the rejected event again, as [Failed deliveries and redelivery](/next/admin/github-integration.md#failed-deliveries-and-redelivery) describes.

**Check:** **Recent deliveries** on GitHub shows `2xx` for a real event. The initial `ping` returns `200` even with a wrong secret, so it does not prove the fix.

## Data appears only after a sync

**Cause:** No provider webhook reaches the server's `/webhooks` path. Without it, updates wait for the scheduled sync.

**Fix:**

1. Check that the provider webhook exists. [Connect GitHub](/next/admin/github-integration.md) and [Manage workspace integrations](/next/admin/integrations.md) give the setup of each provider.
2. Check that its deliveries reach the server's `/webhooks` path.

**Check:** A change to a pull request appears in the workspace without a wait for the scheduled sync.

## Deliveries return `503`

**Cause:** PostgreSQL did not commit the delivery in time. Each `503` has a log record with `event.name=webhook.publish.failed`.

**Fix:**

1. Read `webhook.publish.reason` in the log records of the server.
2. Check the PostgreSQL connection, locks, and volume space, as [Webhook ingestion operations](/next/admin/webhook-ingestion-operations.md) describes.
3. After the repair, recover the failed deliveries, as [Which providers deliver again](/next/admin/webhook-ingestion-operations.md#which-providers-deliver-again) describes.

**Check:** The `webhook.publish` meter records no new `outcome=failure`.

## Reviews record `No AI model is assigned to practice review.`

**Cause:** No enabled model is assigned to **Practice reviews** in the workspace.

**Fix:**

1. Assign an enabled model to practice reviews, as [Connect an AI provider](/next/admin/ai-providers.md) describes.

**Check:** On the workspace's **AI models** page, the **Practice reviews** row shows the model and does not read **Not ready**.

## A server stays unready

**Cause:** The server cannot watch its sandbox namespace, or its isolation probe failed.

**Fix:**

1. Read the probe result and the isolation tier in the server log:

   ```
   kubectl --namespace <release namespace> logs deployment/<release>-server
   ```

2. Find the row of [Other symptoms](/next/admin/kubernetes.md#other-symptoms) that matches the log, and do its fix.

3. Check the chart service account, the namespace, the quota, and NetworkPolicy, as [Sandbox prerequisites](/next/admin/sandboxes.md#sandbox-prerequisites) describes.

**Check:** The server Pod becomes ready.

## A sandbox Pod is refused or cannot start

**Cause:** Admission policy, a quota, an image pull, or a missing [sandbox prerequisite](/next/admin/sandboxes.md#sandbox-prerequisites) stops the Pod.

**Fix:**

1. Read the events of the sandbox Pod.
2. Read the server log.
3. Correct the admission policy, the quota, the image pull, or the prerequisite that the events name.

**Check:** New sandbox Pods start, and reviews leave the queue.

## A server or sandbox ends with `OOMKilled`

**Cause:** Its memory limit is lower than its peak memory under the workload.

**Fix:**

1. Compare its peak memory with its limit under the qualified workload.
2. Change the resource budget, as [Sizing and scaling](/next/admin/sizing.md) describes.

**Check:** The Pod runs the same workload without an `OOMKilled` status.

## Reviews wait past their deadline

**Cause:** The reviews have no capacity: the model is slow, or the sandbox quota holds too few reviews.

**Fix:**

1. Read queue depth and oldest age, as [Queue health](/next/admin/practice-review-operations.md#queue-health) describes.
2. Check the model provider.
3. Check `sandbox.reviewMemory` against [Sizing and scaling](/next/admin/sizing.md).

**Check:** Queue depth and oldest age go down.

## A practice script stopped without a candidate observation: `it ran out of time`

**Cause:** The model provider is slow or limits the rate of calls.

**Fix:**

1. Read the stop reasons in [Practice review operations](/next/admin/practice-review-operations.md).
2. Raise the provider limits, or use a faster model.
3. Raise concurrency only after steps 1 and 2.

**Check:** New script runs end with a result.

## A volume or a node disk is full

**Cause:** PostgreSQL data and WAL, repository caches, or the container runtime use the space.

**Fix:**

1. Measure each of these stores separately.
2. Do not prune volumes to recover space. The kubelet owns image garbage collection.
3. If the PostgreSQL volume is full, read the log of the latest backup Job. A failed WAL archive keeps the WAL on the volume.
4. If data or a volume is lost, use [Backup and restore](/next/admin/backup-restore.md).

**Check:** Each Pod of the release is ready:

```
kubectl --namespace hephaestus get pods
```