# Server runtime Hephaestus ships one image. It boots as the server, or it runs the database migration and exits ([ADR 0005](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0005-two-role-runtime-via-conditional-on-property.md)). The [configuration reference](/next/admin/configuration-reference.md) lists every setting. This page gives the facts that an operator acts on. ## Server role The server serves the API, receives provider webhooks, syncs provider work, and delivers feedback. With `sandbox.enabled`, it also runs the practice reviews and conversation turns in sandboxes, their gateway, and the LLM proxy. | Port | Listener | Who reaches it | | ---- | -------------------------------------------------- | ----------------------------------- | | 8080 | The API and `/webhooks/**`, plain HTTP | The Gateway data plane | | 8081 | The sandbox gateway, with `sandbox.enabled` | The sandboxes of the release | | 9090 | The management listener: health probes and metrics | The kubelet and the metrics scraper | At startup, the server logs one record with `event.name=runtime.roles.configured`. Use it to find the start of each server in a log store. ## Reception during a restart While no server is ready, providers cannot deliver webhooks. The chart keeps a ready server during a schema-compatible update ([What the chart runs](/next/admin/helm-chart.md#what-the-chart-runs)). A stopped upgrade, or the loss of all server replicas, stops reception. After reception resumes: * GitHub redelivers the deliveries that failed. * Slack retries a failed event. * The routine sync repairs the lifecycle facts of GitLab and Outline. ## Reviews and conversation turns `sandbox.enabled` turns on practice reviews and conversation turns. It is `false` by default, and the `small` and `cluster` profiles turn it on. Without it, the server submits and claims no practice reviews, runs no conversation turn, and starts no sandbox. It still takes back expired review leases, closes finished reviews, and records the spend of attempts that already ran. The server needs no LLM environment variable. Register providers under **Instance administration → AI models**, or on the AI models page of a workspace ([AI providers](/next/admin/ai-providers.md)). Two chart values set the capacity: | Chart value | Effect | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `sandbox.reviewMemory` | The review sandboxes that the installation runs at once ([Sizing and scaling](/next/admin/sizing.md#raise-capacity)). | | `sandbox.conversationTurns` | The conversation turns that the installation runs at once ([Conversation turn capacity](#conversation-turn-capacity)). | A review's sandbox holds a lease of 5 minutes on the database clock and renews it every 25 seconds. When a lease expires, any server queues the review again, or fails it after 5 lost leases. A cancel ends a review at once. Its sandbox stops at the next lease renewal. The server runs sandboxes only on Kubernetes, with the settings that the chart gives it. The server removes orphaned sandbox resources at startup and then at a fixed interval. Each server generates the key that signs its sandbox tokens at start. The key needs no configuration. ## Sandbox gateway With `sandbox.enabled`, the server opens the sandbox gateway on port 8081. The API port serves no gateway path, and the gateway port serves no API path. Other paths return an empty `404`. The Service `-sandbox-gateway` selects every server, so any server can serve a sandbox's call. No route of the Gateway reaches this port. | Limit | Value | Response past the limit | | -------------------------------------------------- | ----------------- | ------------------------------------------------------------------------------- | | Request body | 4 MiB | `413`. A request without a declared body length gets `411`. | | Requests from one connection address | 12,000 per minute | `429` with `Retry-After`, before the gateway reads a credential | | Requests of one sandbox | 680 per minute | `429` with `Retry-After` | | Open connections, `sandbox.gateway.maxConnections` | 4096 by default | Up to 100 more connections wait in the operating system's queue. The rest fail. | The gateway also answers `429` when the request bodies in flight would pass a quarter of the server's heap. If the gateway cannot read its rate-limit counts, it serves the call without a limit and counts `sandbox.gateway.limiter.errors`. A caller that stays silent for 30 seconds loses its connection. ## Database migrations The `migrate` run mode applies the database migration of a release, then exits. The chart's migration Job starts the image with `SPRING_PROFILES_ACTIVE=prod,migrate` once for each revision. The run mode starts no web server. It exits with status 0 when the schema is current, and with a non-zero status when the migration fails. The run mode needs `DATABASE_URL`, `DATABASE_USERNAME`, and `DATABASE_PASSWORD`, with the login of `hephaestus_owner`. The server uses the login of `hephaestus_app` ([Database roles](/next/admin/helm-chart.md#database-roles)). Under the `prod` profile, the server does not migrate at start. It waits until the database holds every change set of its release, and writes nothing. A schema that is ahead of the server is correct: during a rollout, the previous release runs against the new schema. After 10 minutes, the server stops. Its error names the first change set that has not run. A migration that takes longer needs the [database maintenance](/next/admin/maintenance.md#database-maintenance) procedure. ### One migration at a time Every migration holds a PostgreSQL advisory lock while it runs. The lock lives in the database session, so the migration needs a direct or session-pooled connection. A proxy that pools by transaction can leave the lock on a server connection. A second migration waits until the first ends, and logs `database.migration.lock_awaited`. After 10 minutes, the waiting process stops. PostgreSQL releases the lock when the process stops, also when the process is killed. The run mode also releases the Liquibase lock that a killed run left, and logs `database.migration.lock_released` with the holder and the time of that lock. ### Rollouts and rollback Between the migration and the end of the rollout, both releases use the new schema. The [expand/contract rule](/next/contributor/database-migration.md#expand-and-contract) keeps this safe. The schema is never rolled back. If the new release fails after the migration, return the server to the previous release, which runs on the new schema. Then roll forward to a release that fixes the problem. A failed migration stops at a change set boundary. The previous release can run on that partly migrated schema, unless the release has a **Stop every role** entry in its migration guide. Save the log of the failed Job, because the chart deletes a finished Job after 24 hours: ``` kubectl -n hephaestus logs job/ -c migration > migration.log ``` A failed Helm status does not mean that the Job has stopped. Correct the cause, then follow [If migration fails](/next/admin/upgrade.md#if-migration-fails). A release with a **Stop every role** entry needs a [stopped schema upgrade](/next/admin/upgrade.md#stopped-schema-upgrades). ## Conversation turns Each question in the app or in a Slack direct message starts a turn in the conversation's sandbox. A conversation runs one turn at a time. Each thread is a separate conversation. [Conversation runtime](/next/contributor/unified-pi-runtime.md#turn-lifecycle) describes dispatch, leases and failures. ### Conversation turn capacity `sandbox.conversationTurns` sets the turns that the whole installation runs at once. More servers do not add turns. A server starts no turn while the installation already holds that many running turns. A queued turn does not spend its time limit. [Sandbox capacity budget](/next/admin/helm-chart.md#sandbox-capacity-budget) gives `auto` and the checks of an explicit count. ### Live answers The server serves the live answer stream at `GET /api/workspaces/{slug}/mentor/events`. It sends a `ping` comment every 20 seconds. Every proxy between the browser and the server needs an idle timeout of at least 60 seconds ([Gateway routes](/next/admin/helm-chart.md#gateway-routes)). A developer can have at most 10 open streams. When a server stops, it ends its live streams first. The browser connects to another server and gets the whole conversation again. Slack receives one completed answer after the turn ends, and no live text. ### Turn limits The chart has no setting for these limits: | Limit | Value | | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | The lease of a running turn | 60 seconds. The sandbox renews it every 25 seconds. A turn that loses its lease runs once more, then fails. | | The wait of one model call of a turn, or one streamed model call of a review, for the provider's response headers | 5 minutes | | The time that the stream of one model call can send nothing | 2 minutes | | The questions that can wait in one conversation or thread | 3 | | The longest question in the web app | 100,000 characters | | The questions that one account can ask | 20 in 10 minutes | The timeout of the workspace's conversation model binding is the time limit of each turn, at most 900 seconds. Set it above the time of a few of your model's slowest calls, because one answer takes a few calls. A turn also stops after 40 model calls. A gateway in front of the model that buffers whole answers looks like a silent provider ([Gateways that buffer answers](/next/admin/ai-providers.md#gateways-that-buffer-answers)). ## Model calls ### Call pools Each model is in a call pool: one provider quota boundary across servers, models and workspaces. The pool learns its limit of calls in flight. A new pool starts at 4. The limit rises while the provider serves every call. It falls after a refusal, and when the first tokens of calls arrive late. Conversation calls have an interactive reserve of a quarter of the limit. [AI providers](/next/admin/ai-providers.md#6-sharing-provider-capacity) tells how to share a pool and set a ceiling. [Call pools](/next/contributor/agent-runs.md#call-pools) gives the learned limit, the grant order and the leases. ### Provider breaker Each server keeps a circuit breaker for each provider connection. A call that stalls, that cannot connect, or that ends with a server error counts as a failure. A rate limit, a request error, or a client that went away does not count. When at least 3 calls in 5 minutes ended and half of them failed, the breaker opens for 60 seconds. While it is open, the server starts no conversation turn on that connection. Each such turn fails at once with **Hephaestus's AI model is not responding right now, so Hephaestus did not start this reply. Try again in a few minutes.** Model calls of running turns and of practice reviews get an error at once. After 60 seconds, one call tries the provider. If it succeeds, the breaker closes. If it fails, the breaker opens again. [Conversation turn metrics](/next/admin/metrics.md#conversation-turn-metrics) names the breaker's metrics. ## Graceful drain On `SIGTERM`, the server stops taking review and conversation claims before the web server closes. It keeps the sandbox gateway and the database open while leased runs finish. The gateway Service still includes the stopping server, so its sandboxes can renew their leases. A model stream that loses its server connection is tried again by the sandbox runtime, within the run's time limit. `server.drainTimeoutSeconds` bounds the wait: 1800 seconds by default, 600 in the `small` profile. The chart gives the Pod that interval plus two minutes for the other shutdown phases. Every other shutdown phase has 20 seconds. If the drain time ends, the server logs `agent.review.drain.expired` with the count of runs that remain, and closes. Any server takes back those runs when their leases expire ([Review drain deadline](/next/admin/log-records.md#review-drain-deadline)).