Server runtime

Hephaestus ships one image. It boots as the server, or it runs the database migration and exits (ADR 0005). The configuration reference lists every setting. This page gives the facts that an operator acts on.

Server role

The server serves the API, receives provider webhooks, syncs provider work, and delivers feedback. With sandbox.enabled, it also runs the practice reviews and conversation turns in sandboxes, their gateway, and the LLM proxy.

Port	Listener	Who reaches it
8080	The API and /webhooks/**, plain HTTP	The Gateway data plane
8081	The sandbox gateway, with sandbox.enabled	The sandboxes of the release
9090	The management listener: health probes and metrics	The kubelet and the metrics scraper

At startup, the server logs one record with event.name=runtime.roles.configured. Use it to find the start of each server in a log store.

Reception during a restart

While no server is ready, providers cannot deliver webhooks. The chart keeps a ready server during a schema-compatible update (What the chart runs). A stopped upgrade, or the loss of all server replicas, stops reception. After reception resumes:

• GitHub redelivers the deliveries that failed.
• Slack retries a failed event.
• The routine sync repairs the lifecycle facts of GitLab and Outline.

Reviews and conversation turns

sandbox.enabled turns on practice reviews and conversation turns. It is false by default, and the small and cluster profiles turn it on. Without it, the server submits and claims no practice reviews, runs no conversation turn, and starts no sandbox. It still takes back expired review leases, closes finished reviews, and records the spend of attempts that already ran.

The server needs no LLM environment variable. Register providers under Instance administration → AI models, or on the AI models page of a workspace (AI providers).

Two chart values set the capacity:

Chart value	Effect
sandbox.reviewMemory	The review sandboxes that the installation runs at once (Sizing and scaling).
sandbox.conversationTurns	The conversation turns that the installation runs at once (Conversation turn capacity).

A review's sandbox holds a lease of 5 minutes on the database clock and renews it every 25 seconds. When a lease expires, any server queues the review again, or fails it after 5 lost leases. A cancel ends a review at once. Its sandbox stops at the next lease renewal.

The server runs sandboxes only on Kubernetes, with the settings that the chart gives it. The server removes orphaned sandbox resources at startup and then at a fixed interval. Each server generates the key that signs its sandbox tokens at start. The key needs no configuration.

Sandbox gateway

With sandbox.enabled, the server opens the sandbox gateway on port 8081. The API port serves no gateway path, and the gateway port serves no API path. Other paths return an empty 404. The Service <release>-sandbox-gateway selects every server, so any server can serve a sandbox's call. No route of the Gateway reaches this port.

Limit	Value	Response past the limit
Request body	4 MiB	413. A request without a declared body length gets 411.
Requests from one connection address	12,000 per minute	429 with Retry-After, before the gateway reads a credential
Requests of one sandbox	680 per minute	429 with Retry-After
Open connections, sandbox.gateway.maxConnections	4096 by default	Up to 100 more connections wait in the operating system's queue. The rest fail.

The gateway also answers 429 when the request bodies in flight would pass a quarter of the server's heap. If the gateway cannot read its rate-limit counts, it serves the call without a limit and counts sandbox.gateway.limiter.errors. A caller that stays silent for 30 seconds loses its connection.

Database migrations

The migrate run mode applies the database migration of a release, then exits. The chart's migration Job starts the image with SPRING_PROFILES_ACTIVE=prod,migrate once for each revision. The run mode starts no web server. It exits with status 0 when the schema is current, and with a non-zero status when the migration fails.

The run mode needs DATABASE_URL, DATABASE_USERNAME, and DATABASE_PASSWORD, with the login of hephaestus_owner. The server uses the login of hephaestus_app (Database roles).

Under the prod profile, the server does not migrate at start. It waits until the database holds every change set of its release, and writes nothing. A schema that is ahead of the server is correct: during a rollout, the previous release runs against the new schema. After 10 minutes, the server stops. Its error names the first change set that has not run. A migration that takes longer needs the database maintenance procedure.

One migration at a time

Every migration holds a PostgreSQL advisory lock while it runs. The lock lives in the database session, so the migration needs a direct or session-pooled connection. A proxy that pools by transaction can leave the lock on a server connection. A second migration waits until the first ends, and logs database.migration.lock_awaited. After 10 minutes, the waiting process stops.

PostgreSQL releases the lock when the process stops, also when the process is killed. The run mode also releases the Liquibase lock that a killed run left, and logs database.migration.lock_released with the holder and the time of that lock.

Rollouts and rollback

Between the migration and the end of the rollout, both releases use the new schema. The expand/contract rule keeps this safe. The schema is never rolled back. If the new release fails after the migration, return the server to the previous release, which runs on the new schema. Then roll forward to a release that fixes the problem.

A failed migration stops at a change set boundary. The previous release can run on that partly migrated schema, unless the release has a Stop every role entry in its migration guide. Save the log of the failed Job, because the chart deletes a finished Job after 24 hours:

kubectl -n hephaestus logs job/<migration-job> -c migration > migration.log

A failed Helm status does not mean that the Job has stopped. Correct the cause, then follow If migration fails. A release with a Stop every role entry needs a stopped schema upgrade.

Conversation turns

Each question in the app or in a Slack direct message starts a turn in the conversation's sandbox. A conversation runs one turn at a time. Each thread is a separate conversation. Conversation runtime describes dispatch, leases and failures.

Conversation turn capacity

sandbox.conversationTurns sets the turns that the whole installation runs at once. More servers do not add turns. A server starts no turn while the installation already holds that many running turns. A queued turn does not spend its time limit. Sandbox capacity budget gives auto and the checks of an explicit count.

Live answers

The server serves the live answer stream at GET /api/workspaces/{slug}/mentor/events. It sends a ping comment every 20 seconds. Every proxy between the browser and the server needs an idle timeout of at least 60 seconds (Gateway routes). A developer can have at most 10 open streams. When a server stops, it ends its live streams first. The browser connects to another server and gets the whole conversation again.

Slack receives one completed answer after the turn ends, and no live text.

Turn limits

The chart has no setting for these limits:

Limit	Value
The lease of a running turn	60 seconds. The sandbox renews it every 25 seconds. A turn that loses its lease runs once more, then fails.
The wait of one model call of a turn, or one streamed model call of a review, for the provider's response headers	5 minutes
The time that the stream of one model call can send nothing	2 minutes
The questions that can wait in one conversation or thread	3
The longest question in the web app	100,000 characters
The questions that one account can ask	20 in 10 minutes

The timeout of the workspace's conversation model binding is the time limit of each turn, at most 900 seconds. Set it above the time of a few of your model's slowest calls, because one answer takes a few calls. A turn also stops after 40 model calls. A gateway in front of the model that buffers whole answers looks like a silent provider (Gateways that buffer answers).

Model calls

Call pools

Each model is in a call pool: one provider quota boundary across servers, models and workspaces. The pool learns its limit of calls in flight. A new pool starts at 4. The limit rises while the provider serves every call. It falls after a refusal, and when the first tokens of calls arrive late. Conversation calls have an interactive reserve of a quarter of the limit.

AI providers tells how to share a pool and set a ceiling. Call pools gives the learned limit, the grant order and the leases.

Provider breaker

Each server keeps a circuit breaker for each provider connection. A call that stalls, that cannot connect, or that ends with a server error counts as a failure. A rate limit, a request error, or a client that went away does not count. When at least 3 calls in 5 minutes ended and half of them failed, the breaker opens for 60 seconds.

While it is open, the server starts no conversation turn on that connection. Each such turn fails at once with Hephaestus's AI model is not responding right now, so Hephaestus did not start this reply. Try again in a few minutes. Model calls of running turns and of practice reviews get an error at once.

After 60 seconds, one call tries the provider. If it succeeds, the breaker closes. If it fails, the breaker opens again.

Conversation turn metrics names the breaker's metrics.

Graceful drain

On SIGTERM, the server stops taking review and conversation claims before the web server closes. It keeps the sandbox gateway and the database open while leased runs finish. The gateway Service still includes the stopping server, so its sandboxes can renew their leases. A model stream that loses its server connection is tried again by the sandbox runtime, within the run's time limit.

server.drainTimeoutSeconds bounds the wait: 1800 seconds by default, 600 in the small profile. The chart gives the Pod that interval plus two minutes for the other shutdown phases. Every other shutdown phase has 20 seconds.

If the drain time ends, the server logs agent.review.drain.expired with the count of runs that remain, and closes. Any server takes back those runs when their leases expire (Review drain deadline).