Runtime Roles

Hephaestus ships one JAR that boots in any of three runtime roles. The same artifact can run as an application server, worker pod, or webhook receiver. It can also run all three together as a monolith. Three boolean properties control the roles:

Role	Property	Default
Server	hephaestus.runtime.server.enabled	true
Worker	hephaestus.runtime.worker.enabled	true
Webhook	hephaestus.runtime.webhook.enabled	true

All three properties default to on, which starts the monolith role set. Conversation turns need at least one process with the worker role, even when the server and worker roles share that JVM. The server role never runs a conversation turn itself.

Boot log

On startup, the JVM logs the enabled roles:

INFO  RuntimeRoleStartupLogger : Runtime roles enabled: [SERVER, WORKER, WEBHOOK]

Under prod, disabling all roles prevents startup. Disabling NATS for a server or webhook role also prevents startup. A worker-only process must disable NATS. The warning for a JVM with no roles is a non-production diagnostic, not a production recovery mode. The startup event also records event.name=runtime.roles.configured and the three runtime.*.enabled booleans.

Monolith vs split-pod

Use the monolith when:

• You are a single contributor who runs locally.
• You have one app server and no horizontal scale yet.
• LLM capacity limits your worker, not Docker capacity.

Split the worker out when:

• LLM reviews and HTTP traffic compete for the same JVM heap / sandbox quota.
• You want to scale practice-review throughput independently of webhook ingestion.

The shipped Compose stacks use separate containers in both the multi-file stack (docker/compose.{proxy,core,app}.yaml) and the single-host install. application-server turns off the worker role and has no Docker socket. application-worker runs all sandboxes and is the only container with the socket. The gate:env check fails a container that mounts the socket without the worker role.

Split the webhook receiver out when:

• You want to deploy app-server changes without dropping inbound GitHub/GitLab events. The webhook pod runs the JetStream publisher in isolation. Redeploys of the app pod do not touch the inbound HTTP surface.

Per-role overlays

Each role has a Spring profile YAML that turns off the other two flags and removes their beans:

• application-worker.yml: runtime.server=false and runtime.webhook=false. It excludes OAuth2 autoconfigure but keeps JPA enabled. Its HTTP connector also serves management endpoints. It binds to loopback for in-container health checks. A separate sandbox gateway listens on SANDBOX_API_PORT (default 8081). If you set it to 0, the gateway binds a free port at start, and sandboxes dial that port.
• application-webhook.yml: runtime.server=false and runtime.worker=false. It enables sync NATS and disables the agent job queue. Webhook pods never claim jobs. The webhook role has no database. It loads only the receiver. It needs no database settings and no encryption keys. It starts and accepts deliveries while PostgreSQL is down. It reaches only NATS. Outline deliveries verify against a secret derived from WEBHOOK_ROUTING_SECRET. Slack consent comes from a NATS key-value bucket that the server writes (ADR 0066).
• No application-server.yml exists. The monolith default is the server profile.

The worker gateway's connector-specific security chain accepts only job-token-authenticated sandbox capabilities. Other paths on that port return an empty 404. Capability calls receive these responses:

• Over SANDBOX_API_MAX_REQUEST_BYTES: 413.
• No declared body length: 411.
• Past SANDBOX_API_REQUESTS_PER_MINUTE: 429 with a Retry-After.

If the gateway cannot reach the store for rate-limit counts, it serves the call without a limit. It counts that failure under sandbox.gateway.limiter.errors. Thus, a limiter outage never stops a review. Traefik never routes to this port.

To deploy a worker pod:

1. Set SPRING_PROFILES_ACTIVE=prod,worker.
2. Give it the app pod's database. Workers and the app pod coordinate only through PostgreSQL.

docker/compose.app.yaml contains a functional Compose template.

Build-only profiles

specs and cds-training generate artifacts. They are not deployment modes. They load the application contract without runtime bootstrap, container-image validation or image pulls. They supply no shared placeholder secrets. If an authentication component needs a key to initialize, the build uses the existing non-production ephemeral-key behavior, without a runtime warning.

Never combine either profile with prod, including through the worker-node or webhook-server profile groups. Startup rejects that combination even when production secrets are configured. Production encryption requirements apply to Spring's expanded profiles, so selecting a production profile group requires the same keys as selecting prod directly.

Database migrations

The migrate run mode applies the database migration of a release, then exits. Start the application image with SPRING_PROFILES_ACTIVE=prod,migrate. It needs only DATABASE_URL, DATABASE_USERNAME and DATABASE_PASSWORD. It starts no web server and no runtime role. It exits with status 0 when the schema is current, and with a non-zero status when the migration fails.

Under the prod profile, no role migrates at start. The server and worker roles wait until the database holds every change set of their release. The webhook role reads no database and does not wait. The wait reads the migration history and writes nothing. A schema that is ahead of the role is correct: during a rollout, the previous release runs against the new schema.

After hephaestus.database.migration-wait (HEPHAESTUS_DATABASE_MIGRATION_WAIT, default 10m), the role stops. The bundled Compose stacks do not pass this variable, so the default applies there. Its error names the first change set that has not run.

Run the migration once for each release, before the new roles start:

• Compose. The shipped stacks contain a one-shot database-migration service. The server and the worker start only after it completes successfully. Run docker compose run --rm -T --no-deps database-migration first. --no-deps keeps Compose from restarting PostgreSQL under the running roles. The command ends with the exit status of the migration, and the previous roles keep serving until then. If it fails, Compose replaces no role. Follow Rollouts and rollback. Then run docker compose up -d to replace the roles. It stops the previous roles and runs the migration service again, which finds nothing to apply. Then it starts the new roles. A plain docker compose up -d alone stops the previous roles for the whole migration. Pull-based deployment runs the migration first.
• Kubernetes. The Helm chart runs the migration as a Job of each revision, beside the new roles. The server and worker roles wait for the Job.
• Other deployments. Run the image once with the migrate profile, then start the new roles.

One migration at a time

Every migration holds a PostgreSQL advisory lock while it runs. The lock lives in the database session, so the migration needs a direct or session-pooled connection. A proxy that pools by transaction can leave the lock on a server connection. This applies to the run mode and to a process that migrates at start, such as the development server or a role with SPRING_LIQUIBASE_ENABLED=true. A second migration waits until the first ends, and logs database.migration.lock_awaited. After hephaestus.database.migration-wait, the waiting process stops.

PostgreSQL releases this lock when the process stops, also when the process is killed. Thus, a killed run cannot block the next one. The run mode releases the Liquibase lock that a killed run left. It logs database.migration.lock_released with the holder and the time of that lock.

A release from before the run mode takes no advisory lock, and its server migrates at start. During the first upgrade to this release, do not restart the previous server while the migration starts. Otherwise, the run mode can release the Liquibase lock of that server while it still migrates. Release a Liquibase lock by hand only when such a process left it, and only after you prove that it no longer runs.

Rollouts and rollback

Between the migration and the end of the rollout, both releases use the new schema. The expand/contract rule keeps this safe. The schema is never rolled back. If the new release fails after the migration, you can return the roles to the previous release. The previous release runs on the new schema. Then roll forward to a release that fixes the problem.

A failed migration stops at a change set boundary. The previous release can run on that partly migrated schema, but only if the release has no Stop every role entry in its migration guide. With the bundled Compose stacks, this is the default case:

1. If docker compose run --rm -T --no-deps database-migration fails, do not run docker compose up -d. The previous roles continue to serve.
2. Correct the cause that the migration error names.
3. Run the migration command again. It continues from that boundary.
4. Run docker compose up -d to replace the roles.

A release whose migration guide entry starts with Stop every role is the exception. The previous release cannot run on its schema, and also not on a partly migrated one. For that release, do these steps:

1. Stop every role, including remote workers.
2. Make a database backup with the matching signed release lock, as the silent deployment checklist describes.
3. Run the migration. Do not let a deployment tool return the roles to the previous release when the migration fails.
4. Start the new release.
5. If the migration or the new release fails, follow Upgrade § If migration fails. It restores the backup and the previous signed release lock together.

Upgrade § Rollback explains why images alone cannot undo a schema change.

Webapp container

The webapp image runs nginx as an unprivileged user and needs no capabilities.

Property	Value
User and group	UID 101, GID 101
Port	8080, plain HTTP. Your proxy handles TLS.
Health check	GET / on port 8080 returns 200
Writable path	/tmp/hephaestus-webapp only. It holds the page shell, the runtime configuration and the nginx files.
Root file system	Can be read-only

The Compose stacks run the container with a read-only root file system, all capabilities dropped and no-new-privileges. They mount a 16 MiB tmpfs at /tmp/hephaestus-webapp, owned by UID 101 with mode 0700. The maintenance page uses the same image and the same settings. The Helm chart sets the same constraints. If you run the image on another platform, mount an empty directory at /tmp/hephaestus-webapp that UID 101 can write, for example an emptyDir with fsGroup: 101. To mount legal overrides, see Legal Pages.

Configuration cheatsheet

Each cell shows the value a pod uses when the operator sets no environment variable for it. It uses the profile overlay's value if the overlay specifies one. Otherwise, it uses the base default.

Property	Server	Worker	Webhook	Set by
hephaestus.runtime.server.enabled	true	false	false	profile overlay
hephaestus.runtime.worker.enabled	true	true	false	profile overlay
hephaestus.runtime.webhook.enabled	true	false	true	profile overlay
hephaestus.workspace.init-default	false	false	false	prod profile
hephaestus.git.enabled	false	false	false	env default
hephaestus.sync.nats.enabled	false	false	true	env default / profile overlay
hephaestus.agent.enabled	false	false	false	env default
hephaestus.integration.outline.allowed-origins	empty	empty	empty	env default
hephaestus.sandbox.gateway.port	8081	8081	—	env default (SANDBOX_API_PORT)
hephaestus.sandbox.gateway.requests-per-minute	120	120	—	env default (SANDBOX_API_REQUESTS_PER_MINUTE)
hephaestus.sandbox.gateway.max-request-bytes	4194304	4194304	—	env default (SANDBOX_API_MAX_REQUEST_BYTES)
Needs user-auth wiring	yes	no	no	role gating

Bold cells show values that the role's profile overlay specifies, not the base defaults. The three runtime.* rows in a column show the pod's role set:

• The worker overlay turns off the server and webhook roles.
• The webhook overlay turns off the server and worker roles.
• The monolith, with no overlay, runs all three.

The server and the worker enforce Outline origin approval. The server controls connections, sync and the change events it consumes. The worker projects evidence. The webhook role reads no connection, so it does not need the allowlist.

1. Supply the same allowlist to the server and the worker.
2. After a change, restart both roles.

The reference Compose files supply the allowlist automatically.

Identical rows across the columns are not role-specific. They are instance-wide defaults, shown per role because operators ask per role. init-default is false in the prod profile for all roles. These properties do not change with the active overlay:

• git.enabled (GIT_CHECKOUT_ENABLED).
• agent.enabled (AGENT_ENABLED).
• integration.outline.allowed-origins.

The two environment-variable flags default to false on all roles until the operator sets them. No profile turns on work automatically.

Needs user-auth wiring is not in that group. Its cells show yes | no | no because it differs by role. User authentication exists only with the server role. hephaestus.runtime.server.enabled=false removes it. The worker overlay also excludes Spring's OAuth2 resource-server autoconfiguration. A worker-only pod therefore never depends on user-auth wiring at boot.

The webhook overlay has no such exclusion. It uses only the role gate.

No hephaestus.sandbox.enabled exists, and an architecture test forbids one. The Docker sandbox is the worker capability, controlled by hephaestus.runtime.worker.enabled. A server pod that keeps the worker role, such as a monolith, opens the sandbox gateway on the same port. It needs the same firewall controls as a separate worker. The webhook role turns off the worker role and opens no gateway. Its three sandbox.gateway cells therefore show —.

Git needs no such row. It runs in-process through JGit in the role that does the work. It uses that container's mirrors under hephaestus.fabric.root. The server role handles commit ingestion. The worker role handles review preparation. A server pod with the worker role enabled uses one root for both.

The webhook role has no Git.

The worker role also writes each review's feedback after its sandbox ends. These model calls take the same proxy path and rules as the sandbox's calls, so they need no new egress.

Sandbox maintenance follows the worker role in both split-out and single-container deployments. The worker removes orphaned sandbox resources at startup and then at a fixed interval. These resources include a conversation host whose worker is gone. They also include the sandbox of a review attempt after its review moved to a later attempt. This maintenance does not enable server-wide scheduled jobs on the worker. API-contract generation and CDS training do not run sandbox maintenance, server scheduled jobs or startup sync-job recovery.

The worker application HTTP endpoint binds to loopback. Every role has a separate private management listener on port 9090, including a single-JVM topology. Management does not run on SERVER_PORT. See Observability for network restrictions.

One row reads differently from what a bare docker compose up produces, because the shipped compose file sets the env var the base default leaves off:

• sync.nats.enabled is ${NATS_ENABLED:false} in application-prod.yml, but docker/compose.app.yaml passes NATS_ENABLED: ${NATS_ENABLED:-true} to application-server — so the compose server pod runs with NATS on. The webhook overlay pins it true regardless (it needs the shared Connection bean). The worker overlay pins it false (the job queue is PostgreSQL, ADR 0025).

hephaestus.agent.enabled (AGENT_ENABLED) controls submission and execution. Server maintenance runs independently of this flag:

• Submission (turning pull request, issue and bot-command events into queued jobs) needs this flag and the server role. A worker-only container executes jobs. It does not submit these reviews.
• Orphan recovery reclaims the jobs of a worker that stopped. A worker holds a lease of 5 minutes on each job that it runs, on the database clock. It renews the lease with each heartbeat (hephaestus.agent.heartbeat-interval). The server checks the leases every 20 seconds. When a lease expires, the server requeues the job, or fails it at the retry limit.
  • After the server starts, it requeues nothing for one heartbeat interval. Every live worker renews its leases within that interval.
  • A stall of more than 40 seconds between two checks also starts such an interval. A slow check alone does not delay recovery.
  • A job from before the upgrade to 1.0 has no lease. The server recovers it as an expired lease.
  • Orphan recovery needs the server role, including when AGENT_ENABLED=false. It therefore does not run twice on a dedicated worker pod.
  • If recovery cannot finish, the timeout backstop ends the attempt. ADR 0070 defines the fresh timeout for a cutover claim.
• Completion closes the parent review of a terminal job and records its stop reason. This server maintenance also runs when AGENT_ENABLED=false.
• Execution (the poll loop that claims and runs jobs) needs this flag and the worker role. Worker admission and server preparation read the same practice-page feedback support, so that read exists in every role. A review attempt that was cancelled, requeued or moved to another worker writes nothing more. The gateway refuses each of its calls, and the practice review stops at its next refused call. A worker that polls the requeued job while that attempt still runs on it stops that attempt first. Otherwise, the attempt holds its capacity until its next call or its deadline.

AGENT_ENABLED is false on all roles until you set it, regardless of how the pod starts.

1. Set it to true on each pod that needs submission or execution.

In a split-pod topology, both pods need it:

• application-server: submission. Recovery runs independently of the flag.
• application-worker: execution.

A worker with this flag set to false claims no practice reviews. Server maintenance still records the spend of attempts that already ran.

Conversation turns

Each question to Hephaestus, in the webapp or in a Slack DM, starts one turn. Turns run in the conversation host, one long-lived container on each worker. The glossary defines both terms. The application server keeps these functions:

• The web API, the live answer stream, and Slack delivery.
• A first admission check when a question arrives. The worker admits the turn again, with the budget check, when it claims it.
• The conversation store in PostgreSQL.

The server role needs no Docker client and never connects to a conversation host. AGENT_ENABLED controls queued practice reviews, not Hephaestus.

Dispatch

When a developer asks a question, the server stores it in PostgreSQL with a queued turn. Each worker listens for turn changes through PostgreSQL LISTEN/NOTIFY. It also checks the queue every 30 seconds as a backstop. A worker with free capacity claims the next turn and holds a lease on it. The worker runs the turn in its conversation host and writes the answer to PostgreSQL while Hephaestus writes it. The server then sends the answer to the webapp or posts it to Slack.

A turn that finds no free capacity waits in the queue until a worker can claim it. Each conversation runs one turn at a time. A thread is a separate conversation, so a thread turn can run at the same time as a turn in the main conversation. Questions that arrive during a turn wait in their conversation. When hephaestus.mentor.turn.max-follow-ups questions wait, Hephaestus refuses the next one.

Conversation turn capacity

hephaestus.worker.capacity.mentor-max sets how many conversation turns one worker runs at once. The default is 8. The model provider usually sets the real limit, not the worker:

• A turn uses little of the conversation host. In a load test, one host ran 90 turns at once in 364 MiB of memory.
• Each turn that streams writes its partial answer up to 4 times each second, one short statement each time.
• A provider that queues calls past its concurrency limit makes each call slower. In the same load test, one provider key answered a single call in 4 seconds. With 8 calls at once, each call took 34 seconds.

A turn that waits in the Hephaestus queue does not count against its execution deadline. A call that waits at the provider counts against the deadline and against the model idle timeout. Thus, find how many calls your provider serves at once at full speed. Keep the total capacity of all workers at or below that number.

A queued turn has a separate wait of three minutes to start, counted from its stored queue time. Server maintenance ends an overdue queued question with a message that asks the developer to try again.

A follow-up stays queued while an answer holds its conversation. The wait survives worker restarts. A lease retry gets a new wait.

Within that limit, size the capacity from the expected load. Concurrent turns are about the questions each second multiplied by the seconds for each answer. For example, 1.8 questions each second with 4.5-second answers need 8 concurrent turns. mentor_turn_queue_wait_seconds and mentor_turn_queued show how long and how many questions wait (Observability).

The server must also know the capacity of all workers together: hephaestus.mentor.turn.capacity (HEPHAESTUS_MENTOR_TURN_CAPACITY). While that many turns run, a question that a worker can start next says that Hephaestus is answering other questions. The Helm chart sets the value to worker.replicas multiplied by worker.conversationTurns. If you run workers without the chart, set it on the server to the sum of their mentor-max values. If it is not set, no question says why it waits.

During a rollout, the old workers finish their turns beside the new workers, and those turns count too. Thus, for a short time, a question can say that Hephaestus is busy while a new worker has a free slot. The question starts at the next claim.

Lease, retry, and deadline

The worker renews the lease of each running turn at one third of hephaestus.mentor.turn.lease. If the worker or its conversation host stops, the lease expires. The server's lease reaper then returns the turn to the queue once, and any worker can claim it. If the lease of the second attempt also expires, the turn fails with Hephaestus could not finish this reply. Try again.

The lease end and its expiry both use the database clock. Thus, a clock difference between the server and the worker does not take back a running turn.

The timeout of the workspace's conversation model binding is the deadline of each attempt, at most 300 seconds. A binding accepts no longer Hephaestus timeout, and an older binding with a longer one acts as 300 seconds. A turn also stops after 40 model calls, retries included.

At the deadline or the call limit, the worker stops the turn. The turn fails with Hephaestus took too long to reply and stopped. Try again. The worker ends a stopped turn itself if the conversation host does not end it within 30 seconds.

The worker's LLM proxy cuts a conversation model call whose provider stays silent too long. Two bounds apply:

• hephaestus.mentor.turn.model-first-byte-timeout (default 5 minutes) bounds the wait for the response headers. A provider can queue a request before it answers, so this bound is the longer one.
• hephaestus.mentor.turn.model-idle-timeout (default 2 minutes) bounds each gap in the stream after the headers.

Any byte counts, also an SSE comment that a gateway sends to keep the connection alive. The turn's deadline also bounds both, because the stop at the deadline cancels the call. The turn fails with Hephaestus's AI model stopped responding, so Hephaestus could not finish this reply. Try again in a moment. The worker logs the cut call and the failure with the reason STALLED and the bound that was passed.

In a load test, the slowest 5 % of single calls took 21 seconds, and calls under load took 34 seconds on average. Both defaults are above these times. A gateway that buffers the whole answer and sends nothing until then shows a slow answer as one long silence. Configure it to stream or to send keep-alive comments, for example with proxy_buffering off in nginx. Otherwise, set the timeout above your longest answer.

Provider breaker

Each worker keeps a circuit breaker for each provider connection. A call that stalls, that cannot connect, or that ends with a server error counts as a failure. When at least 3 calls in 5 minutes ended, and half of them failed, the breaker opens for 60 seconds.

While it is open, the worker starts no conversation turn on that connection. Each such turn fails at once with Hephaestus's AI model is not responding right now, so Hephaestus did not start this reply. Try again in a few minutes. Model calls of turns that already run, and of practice reviews, get an error at once.

After 60 seconds, one call probes the provider. If it succeeds, the breaker closes. If it fails, the breaker opens again.

A rate limit, a request error, or a client that went away does not count. Observability names the breaker's metrics.

Model calls in flight for each attempt

The worker limits the model calls that one attempt has in flight at the same time, on every model:

• A conversation turn has at most 2: one call, and one that a stop left open.
• A practice review has at most 2 × (practice concurrency + 1) + 4. With the default practice concurrency of 3, the limit is 12.

The review limit holds two calls for each practice session, 2 more, and the 4 calls of the practice scripts. The proxy notices that a client went away only at its next write to that client. Thus, a call that a stalled session left open still counts for a time. A model call ends when its provider sends nothing for 5 minutes, the time after which a review's runner gives up on the session. The 2 extra calls are for calls that wait for that end.

The proxy refuses a call past the limit with HTTP 429 and Retry-After: 1. It refuses the call before the call reaches the provider. Thus, no attempt takes more than its share of the provider's capacity. sandbox.gateway.calls_in_flight.refused counts these refusals.

A refused call is not lost:

• The review's sessions and Hephaestus ask again with their own retries.
• A practice script's direct call waits as the refusal asks, and then asks again. It stops at the review's work deadline. Thus, a full limit delays a script's call but does not leave it unrated.
• The feedback composition of a review counts against the same limit, because it is a call of the same attempt. It starts after the review's sandbox ends. Then, only calls that a stalled session left open can fill the limit. The composition treats a refusal as a provider pause and tries again.

PRACTICE_REVIEW_PRACTICE_CONCURRENCY sets the practice concurrency, and the limit follows it.

Call pools

A call pool represents one provider quota boundary across workers, models, and workspaces. The instance admin assigns model membership explicitly. A shared host name alone does not establish a shared quota. AI providers owns the setup procedure.

Every call takes a database grant before it reaches the provider. Review calls, composition, and direct script calls share the background capacity. Conversation calls can also use the interactive reserve. The reserve is never lent to background work. Both the total limit and the background limit apply independently. A finite pool must leave background capacity above zero.

A fair queue gives workspaces, composition, and individual attempts turns before repeated grants to a busy peer. Retry and backfill origins receive no extra priority. A review claim reserves its first grant before capture or Pod creation. A saturated pool leaves the review queued. The first call consumes only its exact attempt and worker reservation.

A script can call another pool before its primary model. The worker first releases its unused reservation. That release commits before another pool lock or capacity wait. Active calls retain their grants until their transports close.

Unused reservations expire when their owner stops renewing them for a full permit lease. Any worker can reclaim these reservations under the pool lock, with an atomic state fence against activation. Active grants never use that expiry path. The fair queue counts grants even when unused provider debits are refunded.

A review that cannot claim records its demand in the same fair queue as background calls. It stays queued and creates no Pod. After a capacity refusal, the same poll scans other endpoint pools before it sleeps. The job retains its original readiness time.

If another claim owns the next turn, the poll dispatches that claim. A budget hold or workspace limit withdraws ineligible claim demand. A paused workspace or future readiness time also withdraws that demand. Conversation and background grants have separate fairness history, so conversations do not defer their workspace's composition.

The pool also accounts for request and token rates across workers. Each physical request, including a protocol retry, consumes a rate debit. A configured per-call token bound reserves the provider's maximum input-plus-output charge. The reservation remains conservative when usage is absent or the call is cancelled. Closed calls retain their debit for a full minute after closure. The rate account also leaves an interactive share when the pool has an interactive reserve.

A live worker cancels a call after three quarters of the permit lease without renewal. The proxy releases capacity only after the response completes or the cancelled transport closes. Cancellation also closes a JSON response body after its headers arrive. The request UUID fences renewal and release. A late release cannot affect another call. An ended attempt cannot start a provider call after a capacity wait.

Admission and administration reclaim grants that the worker has not renewed within hephaestus.llm.call-pools.permit-lease (one minute by default). When a worker process ends, its sockets close. A suspended worker can keep a provider connection open after its grant expires. A database connection failure or a cancellation that cannot close the transport can also leave such a connection open. The pool limit can then be exceeded until those connections close.

The expired grant cannot renew or debit a retry. Its provider rate debit remains for one minute after lease expiry. Pool edits, deletion, and membership changes wait for open requests, retained rate debits, and cooldowns to end.

An instance-controlled connection can impose a shared cooldown after HTTP 429. A workspace-controlled connection cannot impose that cooldown on other tenants. The response still reaches its caller. The cooldown follows the retry headers and never shortens an existing pause.

The worker loads at most four compositions at once by default. It acquires this local capacity before it loads or restores composition inputs. This bound is separate from the existing sixteen-local-attempt ceiling. It does not reduce the resident review limit to four.

The conversation host

Each worker starts its conversation host when it claims its first turn. A worker that never claims a turn starts no host. The host runs in a container of the agent image. Its script comes from the worker, not from the image. The host has these fixed limits:

• 2 GiB of memory on Docker. On Kubernetes, 1 GiB (Kubernetes § Sandbox sizes).
• 1 CPU. On Kubernetes, it requests 50m.
• 256 processes.

These limits do not come from the SANDBOX_* variables, and the host does not count against SANDBOX_MAX_CONCURRENT. The host is a separate container beside the worker container. Its 2 GiB do not count against APPLICATION_WORKER_MEM_LIMIT or the worker's memory headroom. The machine must hold them in addition to the worker container. The host has its own internal Docker network and volumes. Its container has the label hephaestus.kind=heph-host.

Two cases change the network:

• A worker that runs on the machine itself, outside any container, cannot join a Docker network. It gives its host an ordinary bridge network with internet access (see the threat model, SB-3). The host reaches the worker through host.docker.internal. Local development runs this way.
• A worker in a container that it cannot identify starts no host. Each conversation turn on it fails with Hephaestus could not start. Try again in a moment. The worker logs why. Docker configuration gives the hostname rule.

The host is ready when it opens its command channel to the worker's sandbox gateway. If the host cannot start, or the channel does not open within 3 minutes, the turn fails. The developer then sees Hephaestus could not start. Try again in a moment.

If the command channel closes, the worker removes the host. The worker pings the channel every 5 seconds. If the host leaves 6 pings in a row without an answer, about 30 seconds, the worker closes the channel and removes the host.

The worker logs the removal with the host's id at INFO. It does not copy the host's output into its own log, because that output can hold the conversation, a provider's answer, or a credential. When a sandbox ends abnormally, the worker logs why from the Pod's status: the container, its exit code and the Kubernetes reason (sandbox.exited). Only for a practice review does that line also carry the review runtime's own last error lines. When a host cannot start, the worker logs the cause that it knows. For example, it logs the hostname rule or the exit code of the host's workspace setup.

A turn that ran in a removed host runs again when its lease expires, if it was on its first attempt. A turn on its second attempt fails. The next turn that the worker claims starts a new host. On a graceful drain, the worker stops claiming turns and waits for its running turns up to hephaestus.worker.drain.timeout, the same timeout as for reviews. Then it removes the host. If a worker stops without a drain, sandbox maintenance removes its host (see Configuration readiness).

The worker gives each attempt a credential that is valid only for that turn. The worker resolves provider keys and never sends them to the host. PostgreSQL remains the billing record.

Turn sandboxes

Hephaestus reads its context documents, code, and Git history in a turn sandbox on the worker, not in the conversation host. The first read of a turn attempt takes one turn sandbox. The worker removes it when the attempt ends. Each worker also keeps a pool of 2 empty turn sandboxes, so that a turn does not wait for a container start.

Plan for these containers on each worker, beside the review sandboxes and the conversation host:

• One turn sandbox for each running conversation turn that reads, at most hephaestus.worker.capacity.mentor-max.
• 2 empty turn sandboxes in the pool.

Each turn sandbox has these fixed limits:

• 1 CPU.
• 1 GiB of memory on Docker, which includes its files. On Kubernetes, 512 MiB, and its files are on the node's disk (Kubernetes § Sandbox sizes).
• 256 processes.
• No network.

The limits are bounds, not reservations. An idle sandbox runs only sleep. SANDBOX_MAX_CONCURRENT and the review capacity of the worker do not count turn sandboxes. Their containers have the label hephaestus.kind=heph-sandbox.

Turn sandboxes need Docker on the worker and the agent image of the release. If the worker cannot start a turn sandbox, Hephaestus cannot read the developer's work and says so. The worker logs Could not fill the turn sandbox pool at WARN and tries again every 60 seconds. Hephaestus reads repositories only where the worker has Git mirrors (GIT_CHECKOUT_ENABLED). Without Git mirrors, Hephaestus tells the developer that it cannot read the code. No setting turns turn sandboxes on or off.

The worker removes leftover turn sandboxes at startup and every 60 seconds. It never removes a turn sandbox of another worker that still runs. A worker that runs outside a container, as in local development, cannot find its leftovers after a crash. Remove them with docker rm -f $(docker ps -aq --filter label=hephaestus.kind=heph-sandbox).

The Conversation runtime gives the other limits and the measured times.

Live answers

The server role serves the live answer stream at GET /api/workspaces/{slug}/mentor/events. It sends a ping comment every 20 seconds. The reverse proxy must keep an idle connection open for longer than 20 seconds. The bundled Traefik keeps it open for 300 seconds. A developer can have at most 10 open streams.

When the server stops, it ends every live stream before it drains its other requests. The browser connects again to the next server, which first sends the whole conversation. The workspace's sync stream works the same way: on the next connect, the page reads its data again.

The mentor-turns rate limit applies to POST /api/workspaces/{slug}/mentor/turns. HEPHAESTUS_RATE_LIMIT_MENTOR_TURNS_CAPACITY (default 20) and HEPHAESTUS_RATE_LIMIT_MENTOR_TURNS_PERIOD (default 10m) set it. The limit counts the questions of one account. A load test that asks more questions as one account must raise the capacity.

Tunables

Property	Default	Effect
hephaestus.worker.capacity.mentor-max	auto (8)	The number of conversation turns that one worker's conversation host runs at once (Conversation turn capacity)
hephaestus.mentor.turn.lease	PT1M	The lease of a running turn. The worker renews it at one third of this.
hephaestus.mentor.turn.model-first-byte-timeout	PT5M	How long one model call of a turn may wait for the provider's response headers
hephaestus.mentor.turn.model-idle-timeout	PT2M	How long the stream of one model call of a turn may send nothing
hephaestus.mentor.turn.lease-reaper-delay	PT15S	The interval at which the server looks for expired leases
hephaestus.mentor.turn.max-follow-ups	3	The number of questions that can wait in one conversation or thread
hephaestus.mentor.turn.capacity	Not set	The conversation turns that all workers together run at once. Only the server reads it (Conversation turn capacity)
hephaestus.mentor.agent.max-prompt-chars	100000	The maximum length of a question sent in the webapp, in characters

Give all roles the same values. The Compose files forward only the capacity, as HEPHAESTUS_WORKER_MENTOR_MAX. To change another key, add its environment variable to the services with a Compose override. An example is HEPHAESTUS_MENTOR_TURN_LEASE.

Observability lists the conversation turn metrics.

Worker pod env vars

The bare minimum to boot a worker, beside the database settings:

SPRING_PROFILES_ACTIVE=prod,worker
HEPHAESTUS_WORKER_DRAIN_TIMEOUT=5m
AGENT_ENABLED=true

Without the last line, the worker boots but claims no practice-review jobs. It can still run conversation turns: AGENT_ENABLED does not gate them.

No LLM environment variables are necessary. The LLM proxy runs in-process on each pod with the worker role, whatever AGENT_ENABLED says. It serves queued practice reviews and conversation turns. It reads providers from the same instance/workspace catalog as the app pod.

1. Register providers at runtime under Instance admin → AI models, or under the workspace's AI models page.
2. Never register providers through an environment variable.

Both capacities default to auto. Review capacity then sizes from Runtime.availableProcessors(). Tunables gives the Conversation turn capacity. Override them only if the host is noisy.

The JVM binds HEPHAESTUS_WORKER_CAPACITY_REVIEW_MAX and HEPHAESTUS_WORKER_CAPACITY_MENTOR_MAX. The shorter HEPHAESTUS_WORKER_REVIEW_MAX and HEPHAESTUS_WORKER_MENTOR_MAX are Compose-file aliases only. docker/compose.app.yaml translates them. A Kubernetes or systemd deployment ignores them without a warning.

SANDBOX_MAX_CONCURRENT also caps review capacity. A worker never claims a review when it already runs as many review sandboxes as the host permits. A review that only writes its feedback holds no sandbox, so it does not count. The single-host self-hosted Compose file sets this limit to 1. A higher review capacity there has no effect until you also raise the sandbox limit.

1. Raise the sandbox limit only with sufficient RAM, as Sizing describes.

Each worker signs the attempt tokens of its sandboxes with a key that it generates at start and keeps in memory. A sandbox calls only the gateway of the worker that runs its attempt, so the key needs no configuration.

Sizing owns the connection pool budget of each role and the JVM memory headroom.

Sandbox runtime

HEPHAESTUS_SANDBOX_RUNTIME selects where a worker runs its sandboxes: docker, the default, or kubernetes. A worker runs exactly one runtime, and a runtime that cannot start never hands its work to the other one. A value that names no runtime stops the worker's start.

The Kubernetes runtime reads the HEPHAESTUS_SANDBOX_KUBERNETES_* variables of the configuration reference. The Helm chart sets each of them.

Every worker beats in worker_registry, also with AGENT_ENABLED=false. The other workers keep a worker's conversation host and turn sandboxes while it beats.

Graceful drain

On SIGTERM, the worker does these actions in order:

1. It reports itself not ready and stops claiming new jobs.
2. It waits up to hephaestus.worker.drain.timeout (default 5 min) for active jobs to finish.
3. It settles each remaining job under its ownership fence before it waits for any container stop.
4. It stops the settled jobs' containers in parallel, with one 30s cleanup deadline for the batch.

The drain phase may take hephaestus.worker.drain.timeout plus 55 seconds for the poll thread, step 3 and step 4. Running conversation turns drain in the same phase, under the same timeout. Every other shutdown phase keeps spring.lifecycle.timeout-per-shutdown-phase (SHUTDOWN_TIMEOUT, 20 seconds). Give the worker a stop grace period of at least the drain timeout plus 2 minutes. The chart sets drainTimeoutSeconds plus 2 minutes.

The worker's HTTP connectors, including the sandbox gateway, stay open during step 2. Thus, an active review can still send model requests while the worker waits for it. The HTTP graceful shutdown starts only after the drain.

Unadmitted reviews return to the queue for another worker to claim. The drain is not the review's fault, so it spends none of AGENT_MAX_RETRIES. The next attempt still gets its own number, so the stopped sandbox cannot write into it. After 3 drains, the next drain cancels the review with its drain reason.

A concurrent cancellation leaves the job terminal instead. Admitted results are never measured again. An interrupted review without saved completed feedback fails with an explicit reason and retains its results. Completed reviews continue through normal delivery and delivery recovery. timeout=0 skips step 2 and immediately applies the same recovery decisions.

Container cleanup does not delay settlement of the other jobs. A failed settlement does not stop the other jobs from being settled. Only a job whose settlement won its ownership fence has its container stopped.

If a stop fails or exceeds the cleanup deadline, the worker records incomplete cleanup. A Docker call that does not respond to interruption can finish after the deadline. The worker does not wait indefinitely for that call. Leftover containers remain subject to normal sandbox reconciliation.

The Helm chart sets the drain timeout from worker.drainTimeoutSeconds, 3600 by default, and the grace period to that value plus two minutes. The cleanup deadline does not extend this grace period or guarantee that all containers stop.

Cancellation

A cancel ends a review on its row at once. The sandbox gateway then refuses the review's next model call. The worker that runs the review stops its sandbox at its next heartbeat (hephaestus.agent.heartbeat-interval). A requeue or a reap of the review ends it in the same way.

See also

The full design rationale lives in the architecture decision records, which are kept in the repository rather than on this site:

• ADR 0005 — original two-role design.
• ADR 0008 — webhook role rationale.
• ADR 0009 — worker substrate. Its WSS control channel is superseded by ADR 0068.
• ADR 0025 — agent job queue moved off NATS onto PostgreSQL.