Skip to main content

Troubleshooting

This page covers a self-hosted deployment on the Helm chart. Before you change anything, collect the release version, the Helm history of the release, and the events and logs of the affected Pod. Do not print Secret objects or credentials while you collect diagnostics. Kubernetes troubleshooting covers a failed first install and other symptoms of the chart.

Startup fails: server.tomcat.remoteip.internal-proxies is unset while forward-headers-strategy=native in prod​

Cause: trustedProxies holds only whitespace. The chart schema refuses an empty value. The server does not start in production with a blank value.

Fix:

  1. Find the addresses of the Gateway Pods, as Prepare the Gateway describes.
  2. Set trustedProxies to a Tomcat regular expression for the Pod network that holds them.
  3. Run helm upgrade with your site values.

Check: The server Pod becomes ready.

Startup fails: An encryption key is required in production​

Cause: The release Secret has no HEPHAESTUS_SECURITY_ENCRYPTION_KEY. A server with the prod profile does not start without it.

Fix:

  1. Restore the exact value of the key from your off-host copy, as System key describes. Do not make a new key for an existing database. The server cannot read the values that the old key sealed.
  2. Restart every role, as Restart a role describes.

Check: The server Pod becomes ready, and you can sign in.

The login page has no sign-in button​

Cause: No complete GitHub or GitLab login provider is configured.

Fix:

  1. On a new instance, configure the OAuth App, as First steps describes.
  2. To change an existing provider, open Instance administration → Login providers.

Check: The login page offers Continue with GitHub or the button of your GitLab provider.

Sign-in succeeds, but the account has no instance-admin navigation​

Cause: config.auth.bootstrapAdmins does not name the account by the provider's stable account identifier. Hephaestus promotes an account only when it signs in.

Fix:

  1. Compare each entry of config.auth.bootstrapAdmins with the account, as First steps describes.
  2. Correct the entry, and run helm upgrade with your site values.
  3. Sign in again.
  4. If no admin exists and the entry is correct, follow Bootstrap and recovery.

Check: After sign-in, the sidebar shows Instance administration.

Helm reports failed, or a Pod waits for a pending change set​

Cause: The migration Job has not finished the schema change. A Helm failure does not prove that the Job stopped.

Fix:

  1. Read the log of the migration Job:

    kubectl --namespace hephaestus logs job/<migration-job>

    Replace <migration-job> with the Job name of the Helm revision.

  2. Follow Recover a stopped cutover.

Check: The migration Job completes, and the Pod of each role becomes ready.

Every path returns 404​

Cause: The listener of the Gateway does not allow routes from the release namespace.

Fix:

  1. Read the route conditions:

    kubectl --namespace hephaestus describe httproute

  2. If the conditions show Accepted=False NotAllowedByListeners, set allowedRoutes.namespaces on the listener, as Prepare the Gateway shows.

Check: The route conditions show Accepted=True and ResolvedRefs=True.

The browser reports a certificate error​

Cause: The Gateway does not serve a valid certificate for the host name.

Fix:

  1. Check that the DNS record of the host name points to the Gateway.
  2. Check the Gateway certificate and the status of its cert-manager Certificate.
  3. On one host, follow Certificate renewal.

Check: The browser opens https://<your hostname> without a certificate error.

GitHub deliveries return 401​

Cause: The secret of the GitHub webhook does not match WEBHOOK_SECRET in the release Secret, or a proxy changes the body or the signature header.

Fix:

  1. Compare the configured webhook secret through a protected process. Do not put either value in logs.
  2. Set the secret of the App or of the manual webhook to the value of WEBHOOK_SECRET.
  3. Make sure that each proxy keeps the body and the X-Hub-Signature-256 header.
  4. Deliver the rejected event again, as Failed deliveries and redelivery describes.

Check: Recent deliveries on GitHub shows 2xx for a real event. The initial ping returns 200 even with a wrong secret, so it does not prove the fix.

Data appears only after a sync​

Cause: No provider webhook reaches the server's /webhooks path. Without it, updates wait for the scheduled sync.

Fix:

  1. Check that the provider webhook exists. Connect GitHub and Manage workspace integrations give the setup of each provider.
  2. Check that its deliveries reach the server's /webhooks path.

Check: A change to a pull request appears in the workspace without a wait for the scheduled sync.

Deliveries return 503​

Cause: PostgreSQL did not commit the delivery in time. Each 503 has a log record with event.name=webhook.publish.failed.

Fix:

  1. Read webhook.publish.reason in the log records of the server.
  2. Check the PostgreSQL connection, locks, and volume space, as Webhook ingestion operations describes.
  3. After the repair, recover the failed deliveries, as Which providers deliver again describes.

Check: The webhook.publish meter records no new outcome=failure.

Reviews record No AI model is assigned to practice review.​

Cause: No enabled model is assigned to Practice reviews in the workspace.

Fix:

  1. Assign an enabled model to practice reviews, as Connect an AI provider describes.

Check: On the workspace's AI models page, the Practice reviews row shows the model and does not read Not ready.

A server stays unready​

Cause: The server cannot watch its sandbox namespace, or its isolation probe failed.

Fix:

  1. Read the probe result and the isolation tier in the server log:

    kubectl --namespace <release namespace> logs deployment/<release>-server

  2. Find the row of Other symptoms that matches the log, and do its fix.

  3. Check the chart service account, the namespace, the quota, and NetworkPolicy, as Sandbox prerequisites describes.

Check: The server Pod becomes ready.

A sandbox Pod is refused or cannot start​

Cause: Admission policy, a quota, an image pull, or a missing sandbox prerequisite stops the Pod.

Fix:

  1. Read the events of the sandbox Pod.
  2. Read the server log.
  3. Correct the admission policy, the quota, the image pull, or the prerequisite that the events name.

Check: New sandbox Pods start, and reviews leave the queue.

A server or sandbox ends with OOMKilled​

Cause: Its memory limit is lower than its peak memory under the workload.

Fix:

  1. Compare its peak memory with its limit under the qualified workload.
  2. Change the resource budget, as Sizing and scaling describes.

Check: The Pod runs the same workload without an OOMKilled status.

Reviews wait past their deadline​

Cause: The reviews have no capacity: the model is slow, or the sandbox quota holds too few reviews.

Fix:

  1. Read queue depth and oldest age, as Queue health describes.
  2. Check the model provider.
  3. Check sandbox.reviewMemory against Sizing and scaling.

Check: Queue depth and oldest age go down.

A practice script stopped without a candidate observation: it ran out of time​

Cause: The model provider is slow or limits the rate of calls.

Fix:

  1. Read the stop reasons in Practice review operations.
  2. Raise the provider limits, or use a faster model.
  3. Raise concurrency only after steps 1 and 2.

Check: New script runs end with a result.

A volume or a node disk is full​

Cause: PostgreSQL data and WAL, repository caches, or the container runtime use the space.

Fix:

  1. Measure each of these stores separately.
  2. Do not prune volumes to recover space. The kubelet owns image garbage collection.
  3. If the PostgreSQL volume is full, read the log of the latest backup Job. A failed WAL archive keeps the WAL on the volume.
  4. If data or a volume is lost, use Backup and restore.

Check: Each Pod of the release is ready:

kubectl --namespace hephaestus get pods