Conversation runtime
Use this page to change how a conversation turn runs. ADR 0057 records the decisions. Runtime roles gives the operator settings.
The ledger
Practice feedback language § Conversation defines the conversation, the thread, the message and the weekly digest. Practice review glossary § Runtime terms defines the turn, the conversation host and the context document.
PostgreSQL holds all Hephaestus state in three workspace-scoped tables:
| Table | Holds |
|---|---|
mentor_conversation | One main conversation for each account and workspace, and one row for each thread. account_id names the owner. A main conversation can carry the Slack team, the Slack user, and the Slack channel id of the direct message. |
mentor_entry | Each question, answer, tool result, system entry, summary, and weekly digest. The column seq orders the entries of one conversation. surface tells whether a message was written in the app or in Slack. |
mentor_turn | One question's run: status, lease, attempt, partial answer, usage, and error. |
A thread is a mentor_conversation row with parent_conversation_id and parent_entry_id.
It forks the main conversation at that entry.
The mentor module owns these tables, their services, and the path builder.
The agent.mentor packages own admission, dispatch, the host, and the web API.
An entry that Pi wrote keeps the Pi session entry verbatim in payload.
The server derives the kind from the Pi entry (MentorPiEntries.kind).
text holds only the developer's words or Hephaestus's text.
Thus, export, search, and erasure read one column.
Owner and tenancy
The account owns the conversation.
mentor_conversation.account_id is a foreign key to account with ON DELETE RESTRICT.
The server finds the conversation by account.
In the app, it uses the account of the request (CurrentDeveloperLookup).
In Slack, it uses the account of the Slack identity (SlackMentorIdentityResolver.resolveActiveAccountId).
The first write of an account in a workspace creates its main conversation (MentorConversationService.findOrCreateMain).
A lock on the account and the workspace makes sure that two first writes create one main conversation.
Each of the three tables has a unique key on (id, workspace_id).
Each reference inside a workspace also names the workspace.
These references are the parent of a thread, the conversation and the turn of an entry, and the conversation of a turn.
feedback_copy.mentor_entry_id and feedback_copy.mentor_turn_id also name the workspace.
Thus, no row can refer to a row of another workspace.
Roster sync can create an unclaimed account, and a conversation can belong to that account.
When a person claims the account, MentorAccountAbsorptionContributor moves the conversations of the unclaimed account to the claiming account.
If both accounts have a main conversation in a workspace, it merges them:
- The entries of the absorbed main conversation follow the entries of the claiming main conversation, in their own order. They get new
seqvalues and keep their ids, so threads, turns, and feedback copies still find them. - Its turns and threads move to the claiming main conversation. Before they move, a running attempt of the absorbed conversation goes back to the queue (
MentorTurnService.requeueRunning), as for an expired lease. Its host loses the lease, so the ledger refuses its later writes. The attempt is billed, and its feedback claims getUNCONFIRMEDreceipts (MentorLedgerEvents.AttemptTakenBack). - Its summaries are erased, so the next turn summarizes the merged conversation again.
- Its Slack identity moves only if the claiming main conversation has none.
- The absorbed conversation row is deleted.
Turn lifecycle
- The server writes the question entry and a
QUEUEDturn in one transaction (MentorTurnService.submit). The clientrequestIdmakes a repeat return the same rows. Personal data tells when the server refuses the write. - A worker claims the first queued turn of a conversation that has no running turn (
MentorTurnStarter.startNext). The order in a conversation is theseqof the questions, which the server gives under the lock of the conversation. The clock of a server does not decide it. The turn becomesRUNNINGwith a lease. - The worker admits the turn (
MentorAdmission.admit) and freezes the model and price inmentor_turn.admission. A refusal ends the turnFAILEDwith the refusal text. - The worker sends
startto its conversation host. The host reads the context and runs Pi. While Pi works, the host reports the partial answer. - The host ends the turn as
DONE,STOPPED, orFAILED(MentorTurnCompletion.end). The end carries all Pi entries of the turn, and the worker stores them in one transaction. A model call that Pi retried or recovered stays out, with thecontext_editthat leaves it out of the model's view (heph-host.tswithoutOmitted). The worker bills the attempt from the proxy's row counters. - In the transaction of the end, the worker records the receipts that the attempt's claims are due. After the end commits, it stores them (Delivery of shown feedback).
A conversation has at most one running turn.
The partial unique index ux_mentor_turn_running enforces this.
The main conversation and its threads are separate conversations, so they run at the same time.
A question for a busy conversation waits as QUEUED.
A conversation holds at most hephaestus.mentor.turn.max-follow-ups queued turns.
The next question gets 409 with the reason FOLLOW_UPS_FULL.
A queued turn tells its clients when it waits for a free Hephaestus (MentorTurnDTO.waitingFor, MentorTurnWaits).
Two facts decide the reason, and both are rows:
- A worker may start the turn. The SQL rule is the same as the claim (
MentorTurnRepository.CLAIMABLE), so the two cannot disagree. - The turns that hold a Hephaestus slot are at least
hephaestus.mentor.turn.capacity. A turn holds a slot while it runs, and while it is stopped and its host still commits the answer (MentorTurnRepository.countHeld).
The capacity is the number of turns that all workers together run at once.
The Helm chart sets it on the server to worker.replicas multiplied by worker.conversationTurns.
The chart helper hephaestus.conversationTurns resolves worker.conversationTurns for the server and the workers, so the two numbers agree.
FREE_HEPH: a worker may start the turn, and every Hephaestus slot of the deployment is taken.- No value: the server knows no reason. The turn starts at once, waits behind an earlier question of its conversation, or the server has no capacity.
A turn whose worker stopped holds its slot until the lease reaper takes it back.
During a rollout, the turns of the draining workers count too.
Thus, for a short time, a question can show FREE_HEPH while a new worker has a free slot.
It starts at the next claim.
All workspaces share the workers' Hephaestus slots.
Thus, FREE_HEPH depends on turns of other workspaces.
It tells a member only that every slot answers some question, never whose question it is.
A start, an end, a stop, and a requeue can each change the held slots.
They can also let the next question of a conversation start.
Each of these changes publishes MentorTurnWaits.SlotsChanged.
After the change commits, MentorTurnWaits.publishWaits publishes a turn change for each turn that a worker may start.
Only these turns can wait for a free Hephaestus.
Because the publication comes after the commit, two changes that commit together each read the other. Clients read the new reason when they receive the change.
The publication covers all workspaces, newest turn first, and at most 100 turns. A turn past that bound shows its new reason at its next read. The process that commits the change publishes it, also a worker without the capacity, because the reader computes the reason.
Leases and retries
The worker renews the lease every third of hephaestus.mentor.turn.lease, while the turn runs on a live host.
The renewal tick has a thread of its own (HephHostConfiguration.hephTurnTicks).
No sandbox call runs on that thread, so a stalled Kubernetes API cannot expire the lease of a turn that runs well.
Each host call checks that the attempt runs on this worker (MentorLeaseLostException).
The worker knows that without a read: a lost lease, the end, and a lost host remove the attempt.
Each write of the attempt is also conditional on the lease, the attempt, and the worker.
The end is the only write of the turn's entries. An attempt that crashes or loses its lease stores no entries. Thus, its retry starts from the same path as the first attempt.
The worker can fail to answer a context read or an end, with a network error or a 5xx.
Then the host sends the call again, after 1, 2, and 4 seconds.
The end releases the lease, so the worker answers a repeated end with 409 and stores nothing twice.
MentorTurnLeaseReaper (server role) takes back an expired lease:
- A running turn on its first attempt returns to
QUEUEDwith the same key. Its partial answer is cleared. - A running turn that loses its second lease ends
FAILED. - A stopped turn whose host did not end it stays
STOPPED, without an answer.
The reaper bills the lost attempt from the proxy's record.
The reaper can return a turn to the queue.
Its QUEUED notification names the attempt that lost the lease.
A worker acts on that notification only when it runs that attempt.
Then the worker sends stop with the reason lease for that attempt, before it starts the retry.
Each command to the host names the attempt.
A host that gets start for a later attempt of a turn that it runs stops the earlier attempt first.
The earlier attempt then ends without a call to the worker.
Stop and deadline
A developer stops a turn with PATCH …/mentor/turns/{turnId}/status.
- A queued turn becomes
STOPPEDand never runs. The question stays without an answer. - A running turn becomes
STOPPEDat once, but it keeps its lease. The host aborts Pi and ends the turn with the cut-off answer.
The next queued turn of that conversation starts after the host ends the stopped turn, or after the lease expires.
An abort can leave a tool call without its result.
The host stores no result for it.
In the stopped turn, Pi gives each open tool call a result when it builds a request (transformMessages).
For later turns, the path builder removes the tool calls from the cut-off answer (MentorContextPathBuilder.keepCutOffAnswer).
The deadline of a turn is the LLM timeout of the conversation model binding, at most five minutes (AgentBindingLimits.MAX_HEPH_TURN_SECONDS).
A worker runs few turns at once, so a longer turn would keep every question queued behind it waiting.
At the deadline, the worker sends stop with the reason deadline.
If the host does not end the turn within HephTurnDispatcher.STOP_GRACE after a developer or deadline stop, the worker ends it.
A turn that the developer stopped then stays STOPPED.
A lease stop has no such wait: the worker no longer counts the attempt, and the reaper decides about the turn.
A turn also ends after 40 model calls, retries included (MAX_TURN_MODEL_CALLS in heph-host.ts).
A turn that keeps calling loops on a tool or on a failing call.
The worker logs one line for each model call and each tool call of a turn.
The line names the tool, the outcome and the duration, never the words.
A turn gets a context folder only while the worker can start turn sandboxes.
Without one, the turn has no read, bash or load_context tool, and its situation says so (HephTurnContexts.NO_CONTEXT_FOLDER).
How a turn fails
A failed turn stores no entries. Its question stays in the path with a marker (The path), and the developer reads the text in this table.
| Cause | Who ends the turn | Text for the developer |
|---|---|---|
| Admission refuses the turn. The causes are no model, no AI choice, budget, unpriced usage, and a model below Hephaestus's smallest model (Limits). A requested erasure of the developer's personal data is also a cause. | The worker, before the host runs (MentorTurnStarter). | The refusal text. |
After a claim, the host does not start, or its channel does not open within HephTurnDispatcher.HOST_READY_TIMEOUT, on two hosts in a row. | The worker. | "Hephaestus could not start. Try again in a moment." |
| The deadline passes. | The host, with the reason DEADLINE. If the host is silent for STOP_GRACE, the worker. | "Hephaestus took too long to reply and stopped. Try again." |
| The turn reaches its model call limit. | The host, with the reason DEADLINE and the detail call_limit. | "Hephaestus took too long to reply and stopped. Try again." |
The provider of a model call sends no response headers within hephaestus.mentor.turn.model-first-byte-timeout, or its stream sends no byte for model-idle-timeout, or its breaker refuses the call. The proxy answers with the code upstream_stalled or upstream_paused. | The host, with the reason STALLED. | "Hephaestus's AI model stopped responding, so Hephaestus could not finish this reply. Try again in a moment." |
| The provider breaker of the admitted model's connection is open (Runtime roles). | The worker, before the host runs (MentorTurnStarter). | "Hephaestus's AI model is not responding right now, so Hephaestus did not start this reply. Try again in a few minutes." |
| Pi writes no answer, or the model call fails. | The host, with the reason ERROR and a detail. | "Hephaestus could not finish this reply. Try again." |
| The conversation stays too long after compaction. | The host, with the reason ERROR and a detail. | "Hephaestus could not finish this reply. Try again." |
The worker refuses the end with 400, for example for too many entries. | The host, with the reason ERROR and a detail. | "Hephaestus could not finish this reply. Try again." |
The worker refuses the end with 413 again, after the host replaced its documents with read notes. | The host, with the reason ERROR and a detail. | "Hephaestus could not finish this reply. Try again." |
| The second attempt loses its lease. | MentorTurnLeaseReaper. | "Hephaestus could not finish this reply. Try again." |
An attempt that loses its lease does not fail by itself. The host stops it without a call to the worker, and the reaper decides (see Leases and retries).
The worker logs each attempt that fails on it at WARN, with the reason and the host's detail.
The detail is a kind of failure from a closed set, for example model_error, context_overflow or worker_refused (FAILURE_DETAILS in heph-host-protocol.ts and in HephTurnController).
It is never the text of the model provider, a tool, or the worker.
The worker logs a detail outside the set as unrecognized.
The host writes the same kind to its own output (failureSummary), with the worker's HTTP status when the worker answered.
It never writes a failure's message, name, or stack, because any of them can carry the words of the worker's answer.
The worker does not read the host's output (The conversation host).
The worker logs an admission refusal at INFO.
The reaper on the server role logs a lost second attempt at INFO.
Observability names the log lines.
Delivery of shown feedback
Practice feedback language § Destinations owns the delivery rule for feedback in conversation. This section tells how a turn applies it.
Three tools put Hephaestus's words about practice feedback into the answer.
Each takes an id and Hephaestus's own text.
The text appears in the answer where Hephaestus calls the tool.
| Tool | Endpoint | The worker admits | Effect on delivery |
|---|---|---|---|
link_result | POST link-result | A result of the developer that Hephaestus may read for conversational mentoring (ResultHistoryQuery). | None. |
link_problem | POST link-problem | A problem of such a result. The result decides for the problem. | None. |
show_feedback | POST show-feedback | Prepared conversation feedback of the developer. The call claims its copy. | The worker settles the copy when the attempt ends. |
Each endpoint answers 200 with the id, or 422 with a detail.
After a 422, the host fails the tool call, and the answer shows nothing for it.
A link alone never delivers feedback.
Prepared conversation feedback is READY feedback with the destination CONVERSATION.
prepared_conversation_feedback.json lists at most three pieces for the conversation of the turn.
The system prompt tells Hephaestus to raise at most one piece in each turn.
The code sets no limit for each turn.
The claim
show-feedback calls FeedbackStore.claimConversationCopy while the tool call runs, in its own transaction.
The claim holds the person-data write fence of the developer (Personal data).
It succeeds only when all of these are true:
- The feedback is
READY, has the destinationCONVERSATION, is not withdrawn, and passes the delivery policy. - Its only copy is a
PLANNEDcopy of the kindCHAT_TURN. - The turn runs this attempt, in a conversation of the developer.
The claim sets the copy to WRITE_STARTED and records the turn in feedback_copy.mentor_turn_id.
The feedback becomes UNCONFIRMED while the claim is open.
Thus, an attempt that stops after the claim never leaves feedback that looks unsent.
A second show_feedback call for the same feedback in the same attempt keeps the claim that the attempt holds.
An attempt cannot use a claim of an earlier attempt.
The receipt
The transaction that records how an attempt or its Slack reply ended also publishes ConversationReceiptsDue.
The event publication registry keeps the event until MentorTurnCompletion.settle stored one receipt for each copy that the attempt claimed (FeedbackStore.recordConversationReceipt).
A crash after the commit or a failed receipt write leaves the publication incomplete, and NotificationRedeliveryJob resubmits it.
A receipt is stored once, so a repeat settles only what is left.
A Hephaestus message shows a piece of feedback when it stores a show_feedback call for it whose tool result succeeded.
The call names the feedback by a well-formed id in any letter case.
| How the attempt ended | Feedback |
|---|---|
A turn asked in the app ended DONE, and a stored Hephaestus message showed the feedback. | DELIVERED |
A turn asked in Slack ended DONE. | What Slack shows decides (Slack replies). |
Any other end, and the attempt's entries or its last partial answer have a show_feedback call for the feedback. For example, the turn ended STOPPED or FAILED after the call. | UNCONFIRMED |
Any other end, and no show_feedback call for the feedback. | FAILED |
The attempt lost its lease (MentorTurnCompletion.billLostAttempt). | UNCONFIRMED |
A delivered copy records the Hephaestus message that showed the feedback in feedback_copy.mentor_entry_id.
It also stores the words that the answer showed: the text of the successful show_feedback calls of that message for that feedback.
The database function conversation_receipt_words holds this rule.
The receipt query and the trigger that guards feedback_copy both use it.
A receipt that the store refuses for good leaves the copy claimed, so the feedback stays UNCONFIRMED.
An unconfirmed or failed piece is no longer READY.
Thus, no later turn or later attempt lists it or shows it again.
This applies the rule that unknown visibility never permits an automatic repeat write.
Expiry and the weekly digest
ConversationFeedbackTtlSweeper (server role, every hour) expires prepared conversation feedback that is older than 14 days and whose copy has no write.
The feedback becomes WITHHELD with the stop reason CONVERSATION_EXPIRED.
A later show_feedback call for it gets 422.
The weekly digest links each piece of conversation feedback that became DELIVERED in its week to the message in feedback_copy.mentor_entry_id.
The digest creates no copy and changes no delivery.
In a thread on a digest, Hephaestus discusses a piece through link_result or link_problem.
A show_feedback call for a piece of the digest gets 422, because that feedback is not READY.
ADR 0057 § Weekly digest owns what the digest lists.
Slack replies
SlackMentorProjector (server role) sends the reply to each turn that the developer asked in Slack.
A turn asked in the app never reaches Slack.
The reply is what the app shows for the turn (SlackMentorPosts.reply).
It has each message that Hephaestus wrote, followed by the quoted words of each result, problem, and piece of feedback that it showed.
One server projects a turn: the server whose MentorSlackDelivery.claim sets the question's slack_delivery from empty to PENDING.
The question's slack_delivery is the state of the whole reply: PENDING, DELIVERED, UNCERTAIN, or FAILED.
Hephaestus's answer entry carries only slack_ts, the message that shows it, so that a Slack thread on that message finds it.
Every post carries Slack metadata with the entry id (event type heph_entry).
A notice to the developer, such as a refusal, carries the event type heph_notice.
Thus, only a stream that no server ended is a Hephaestus message without metadata.
| Where the developer asked | While the turn runs | When the turn ends |
|---|---|---|
| The top level of the direct message | Hephaestus reacts with 👀. | One post with the reply, then Hephaestus removes 👀. Slack shows at most 12,000 characters. A longer reply is cut and ends with "… The full answer is in Hephaestus." |
| A Slack thread | Hephaestus reacts with 👀 and sets the thread's agent session to processing (agents.sessions.setStatus), so Slack shows its Stop. The partial answer streams (chat.startStream, then chat.appendStream). The question's slack_stream_ts holds the stream's ts. | The rest of the answer, then chat.stopStream with the Helpful and Not helpful buttons, then the session's status, then Hephaestus removes 👀. |
A question whose turn has not run 10 seconds after it was asked gets one notice where it was asked (SlackMentorProjector.WAITING_NOTICE).
Earlier turns hold the worker's turn slots, and without the notice Slack shows nothing while it waits.
The server deletes the notice when the turn runs or ends.
A turn that ran and waits again gets no notice.
If the turn waits for a free Hephaestus (MentorTurnWaits), the notice says that Hephaestus answers other questions (WAITING_FOR_HEPH_NOTICE).
The notice does not claim the reply. A claim before the turn runs would lose the answer if the server restarted during the wait. Thus, the notice relies on the one server that the chart runs. That server keeps the notice in memory, so after a restart a question that still waits can get it again.
A failed turn gets its reason as the reply. A stopped turn's reply is what Hephaestus wrote before the stop. If Hephaestus wrote nothing, nothing is posted.
A developer can stop any active turn in the app, also a turn asked in Slack.
When Slack rejects a running turn's stream append with stopped_by_user, the projector calls the same workspace-scoped turn stop.
The stream belongs to one claimed turn, so the stop changes that turn only. Queued follow-ups stay queued.
Other stream failures do not mean the developer asked to stop the turn.
Stop in a Slack thread
In a Slack thread, the developer can also press Slack's Stop, also before the first words and during a tool wait.
Slack sends agent_session_stopped, and SlackAgentSessionStoppedMessageHandler hands it to SlackMentorService.handleSessionStop.
- The event must come from the developer's own DM. The team's current Slack connection must match. The DM of the main conversation, the existing parent entry and its existing thread must also match. A thread is never opened, and no turn id is taken from the event.
- An event that lacks a field, or has a field of another type, is acknowledged and changes nothing.
- The event stops the thread's running Slack turn only, under the turn's lock (
MentorTurnService.stopRunningwith a predicate).event_tsmust be after the start of the running attempt. Whenstreaming_message_tsnames streams, one of them must also be the question'sslack_stream_ts. Queued follow-ups and the main conversation stay as they are, and a turn that already ended stays unchanged. - The stop writes only to the ledger. It calls no Slack API. The host aborts the turn on its own time, as for a stop in the app. The turn's end then ends the stream and the session's status.
The time check has no margin.
It compares Slack's clock (event_ts) with the database's clock (the attempt's started_at).
A difference between the two clocks can make a stop just after a start do nothing.
In a narrow window, it can also let a delayed stop of an earlier attempt stop the next attempt.
The recorded stream is more evidence, but it names no attempt.
After a lost lease, slack_stream_ts still names the lost attempt's stream until the projector starts the answer again.
The session's status
The projector sets the status of a thread's agent session.
processing means that a Slack turn of the thread runs now. A queued question alone shows active.
- When it takes a running thread turn, the projector sets
processing, if that attempt still runs. - It opens a stream only while the stream's attempt still runs.
- When a stream ends,
chat.stopStreamsends the status of the thread's running Slack work assession_status. Slack's default isactive. - At the end of the turn,
agents.sessions.setStatussends that status again, also for a reply that is posted instead of streamed.
Slack keeps the status that it got last.
Thus, each of these calls runs in MentorSlackDelivery.callInThreadSession.
First it holds the developer's person-data write fence (PersonDataWriteFence.holdForAccountWrite).
This precedes every ledger lock, as for other writes about the developer (Personal data).
If the fence refuses the developer, for example after an erasure request, no Slack call is made.
Then it shares the main conversation's row and locks the thread's row, until Slack answers.
An erasure that holds the fence makes a session call wait before any row lock. An erasure also waits at the fence for an active session call. On every server, the session calls of one thread run one at a time. Each reads the thread's running Slack work after the preceding call ends.
A worker's claim takes neither row, so the next turn can start while an earlier stream ends. Its own status call then comes after. While such a call waits for Slack, appends to that thread and writes that lock the main conversation's row wait. No turn, entry or receipt is written in the call. The projector records Slack's answer after it. A top-level question gets no session status.
Retries, rate limits and stream failures
A retried attempt starts its answer again, in a new message. If Slack still streams the lost attempt's stream, it ends with "… Hephaestus had to start again. The answer follows below."
A stream appends at most once each second.
chat.appendStream is Tier 4: Slack allows 100 calls each minute for each workspace.
On one server, the appends of running turns in one workspace are at least 600 ms apart (SlackMentorProjector.WORKSPACE_APPEND_SPACING).
Server replicas do not share this spacing, and the rest of an answer at the end of its turn is not spaced.
For these appends, Slack's rate limit applies: the server waits for the Retry-After time, then tries again.
Each HTTP call of the Slack gateway's own client (SlackMessageService) has a deadline of 10 seconds.
One SDK call can make two HTTP calls: auth.test for a token that the SDK has not seen, then the requested call.
A send has a budget of 30 seconds and a history read 150 seconds, counted on the monotonic clock with the time of the calls.
After a 429, a retry waits for the whole Retry-After.
The wait must be within its cap.
It and two more HTTP calls must also fit in the remaining budget.
Otherwise it gives up with the 429, and never tries earlier than Slack asked.
The deadline does not end a call that already runs when the budget ends.
A call that runs out of time is a transport_failure: Slack may still have applied the write, so the reply's outcome stays unknown.
The table gives what the end of the turn does with the stream.
It applies to a refused append and to an answer that no longer starts with what the stream shows.
The same rules apply to the append and chat.stopStream at the end of the turn.
A stream that takes no more text gets no more text until the turn ends.
Slack documents no way to keep a stream open while it gets no text.
| Cause | The stream | When the turn ends |
|---|---|---|
stopped_by_user: the developer stopped the stream in Slack. | Takes no more text. A refused append while the turn runs also stops that turn. | The stream ends where it is. |
message_not_in_streaming_state: Slack ended the stream, for example after a timeout that it does not document. | Takes no more text. | chat.update replaces the message with the whole reply. |
| A transport failure or a Slack server error. Slack can show the text already, so the append is not sent again. | Takes no more text. | chat.stopStream, then chat.update with the whole reply. If this error answered chat.startStream, the server does not know the stream's ts. It reads the thread for a message with the app's bot_id (from auth.test) and without metadata, from within 2 seconds of the refused start. It replaces that message only if exactly one matches. Otherwise, it posts the whole reply. |
| A rate limit, or any other error. | Takes the text again a second later. After 8 refused appends in a row, it takes no more text. | As for a transport failure. |
| No error, but Hephaestus's answer no longer starts with what the stream shows, for example after a retried model call. | Takes no more text. | chat.stopStream, then chat.update with the whole reply. |
An error in SlackMentorProjector.STREAM_GONE, for example a deleted message. | Takes no more text. | The server clears slack_stream_ts and posts the whole reply as a new message. |
For a stopped turn, the stream ends where it is, unless it shows text that the stored answer does not start with. For a failed turn, the reason replaces the message instead of the whole reply.
A Slack error decides the state of the reply (SlackMentorPoster):
- A transport failure or a Slack server error leaves
UNCERTAIN. The server cannot tell whether Slack stored the post. - A rate limit after the retry budget also leaves
UNCERTAIN. Every try got a 429, so Slack stored nothing. The conservative state is stillUNCERTAIN. - Any other error that Slack answered is final:
FAILED. - Silent Mode is
FAILED.
A reply that can no longer reach Slack is FAILED, for example after the Slack connection or the direct message is gone (MentorSlackDelivery.giveUp).
Feedback in a Slack reply
The end of a turn that ended DONE does not settle the feedback of a turn asked in Slack.
What Slack shows settles it (MentorTurnCompletion.recordDelivery).
Slack shows a piece of feedback when the shown part of the reply holds each of its quotes whole.
A piece that the reply quotes more than once, with only some quotes whole, is partly shown.
The shown part is what the stream sent, or the part of a post or a chat.update that Slack renders.
| What happened in Slack | Feedback |
|---|---|
The post reached Slack, the stream ended, or chat.update replaced the message, and Slack shows the feedback. | DELIVERED |
| The same, but Slack shows the feedback only partly. | UNCONFIRMED |
| The same, but Slack does not show the feedback. | FAILED |
| Slack refused the reply. | DELIVERED if a stream already showed it, otherwise FAILED |
Silent Mode kept the reply from Slack (MentorTurnCompletion.recordSilenced). | DELIVERED if a stream already showed it, otherwise FAILED |
The reply is UNCERTAIN on the projecting server. | It stays UNCONFIRMED until the reconciler reads the history. |
| The reply can no longer reach Slack. | It stays UNCONFIRMED. |
A turn asked in Slack that ended STOPPED or FAILED settles its feedback at the end, as an app turn does (Delivery of shown feedback).
Only a DONE turn's Slack projection publishes feedback receipts. A later stream, post, Silent Mode refusal, or disconnected-reply reconciliation cannot replace the end's cutoff receipt.
Reconciler
SlackMentorReconciler (server role, every minute) finishes each reply that no server settled.
The projecting server died, missed the end of the turn, or left the reply UNCERTAIN.
The reconciler takes a reply 2 minutes after the turn ended, and up to 1 day after.
One run reads at most 50 due posts, and at most 5 of each workspace.
A run stops taking posts after 5 minutes.
The reconciler takes each unsettled post once (SlackMentorReconciler.MAX_ATTEMPTS is 1).
It never posts again what Slack can hold:
- A reply that no server took is claimed as the projecting server claims it, and sent now. If a server claimed it after the reconciler selected it, the claim fails and the reconciler skips the reply.
- A post that a server took gets one read of the history of the direct message, by its metadata (
SlackMentorPoster.reconcile).- The post is there: the reply is
DELIVERED, and its feedback settles as for a post that reached Slack. - The post is not there, and Slack has no more pages:
slack_deliveryisFAILED, and the feedback that the reply claimed isFAILED. The reply is never posted again. The app shows "Not fully delivered to Slack. The full answer is here." under the answer. - The history cannot be read: the reply stays
UNCERTAIN, and its feedback isUNCONFIRMED(MentorTurnCompletion.recordUncertain). - The post is not on the page that the reconciler reads, and Slack has more pages: the reply stays
UNCERTAIN.
- The post is there: the reply is
- A stream that another server left is never posted again, because its shown part is unknown.
For a turn that ended
DONE, the reconciler ends the stream with "… The full answer is in Hephaestus.", and the feedback of the reply isUNCONFIRMED. If Slack already ended that stream, the reconciler replaces the message with the whole reply (chat.update). The feedback then settles as for a replaced message. For a stopped turn, the reconciler ends the stream where it is. For a failed turn, it ends the stream with the reason. If Slack already ended the stream, it replaces the message with the reason.
The reconciler does not take a post a second time.
The Slack note of a weekly digest gets the same single history read.
A note whose direct message Hephaestus never learned stays UNCERTAIN.
Slack disconnect
A Slack disconnect, an uninstall of the app, or a revoked token erases every Hephaestus entry of the workspace that was written in Slack.
It also stops their running turns (SlackWorkspaceContentEraser).
A disconnect takes the lifecycle lock of the Slack connection before it erases.
A Slack question is stored in one transaction that takes the same lock and checks that the connection is still active for the Slack team (SlackWorkspaceResolver.whileConnected).
Thus, a question that arrives during a disconnect is either not stored or erased by it.
The conversation host
Each worker runs one conversation host container (HephHost, HephHostSpec).
A worker with Hephaestus capacity starts its host before it claims a turn, also when no turn is queued.
The host is ready when it opens its command channel.
The worker's readiness includes hephTurnDispatcher, so a rolling update keeps the old worker until the replacement host is ready.
A channel that does not open is tried again after 1, 2, 4 and 8 seconds (runCommands), then the host exits.
Until a host is ready, turns stay queued without spending an attempt.
Server maintenance ends overdue queued questions without claiming an attempt.
The queue startup budget uses the stored mentor_turn.queued_at time, so a restart cannot extend it.
A lease retry records a new queue time. A worker restart does not. The reaper locks the turn and checks its eligibility again, so a question that a ready worker claimed stays with that worker.
The developer gets "Hephaestus could not start. Try again in a moment."
The turn records start_wait_expired and spends no new attempt or model call.
If the host is lost after a claim, the worker tries one fresh host before the claimed turn fails.
| Property | Value |
|---|---|
| Image | The agent image of the release, runtime contract 9 (SandboxLayout.RUNTIME_CONTRACT_VERSION). It provides Node and the Pi SDK. |
| Host script | heph-host.ts, its modules, and the system prompt come from the worker JAR. The worker stages them into the workspace (HephHostSpec). The image does not hold them. |
| Process | node --max-old-space-size=1024 --permission, with no child-process grant. |
| Readable paths | The workspace, /opt/pi-sdk, the empty home, and the practice-script library (/opt/practice-scripts). The provider helper imports that library through the workspace link that the practice runners also use (PiRunnerProfile.PRACTICE_SCRIPT_LIBRARY_SETUP). |
| Writable paths | Pi's agent directory and the temporary directory only. |
| Limits | HephHostSpec.LIMITS. Runtime roles gives the values. |
| Network | Its own internal network. It reaches only the worker's Sandbox Gateway. Runtime roles gives the exception for a worker that runs outside a container. |
| Docker label | Kind heph-host. SESSION_ID is the host id. |
| Tools | link_result, link_problem, show_feedback, search_conversation, and read, bash and load_context when the worker can serve them (Context folder). The three run in a turn sandbox, not in the host. |
The sandbox SPI agent.sandbox.spi.HostSandbox launches the host.
SandboxHosts implements it on the sandbox backend that runs practice reviews (DockerSandboxBackend on Docker).
A host that does not open its channel in time fails the claimed turn (How a turn fails). Runtime roles gives the timeouts and the ping interval of the host.
A closed channel means that the host is gone.
The worker pings the channel on each dispatcher tick, and the WebSocket client of the host answers each ping.
A host that leaves HephHost.UNANSWERED_PINGS pings in a row without an answer is also gone, and the worker closes its channel (HephHost.ping).
The worker removes a gone host and leaves its turns to their leases.
The next turn starts a new host.
SandboxReconciler removes a host whose worker is gone.
The worker never reads the host's output, and it does not copy it into its own log (SandboxHosts.remove).
Pi, Node, and the host itself can all write to that output, so it can hold the conversation, a provider's answer, or a credential.
The worker logs the removal with the host's id, and the class of a removal step that fails.
When a host cannot start, the worker logs the cause that it knows itself.
For example, it logs the hostname rule or the exit code of the workspace setup.
On drain, the worker stops claiming turns.
It waits up to HephTurnDispatcher.DRAIN_TIMEOUT for running turns, then removes the host.
Command channel
The host opens one WebSocket to its worker: GET /internal/llm/runtime/{hostId}/commands on the gateway port.
It authenticates with the gateway-session token of the host.
The worker accepts that token once, and only for the current host id.
The worker sends WebSocket pings and two JSON frames.
heph-host-protocol.ts parses them and ignores any other frame.
{"type":"start","turnId","attempt","token"}. The token is the attempt'sAttemptJwtof kindTURN.{"type":"stop","turnId","attempt","reason"}. The reason isdeveloper,deadline, orlease.
Turn endpoints
The host calls the worker over HTTP with the turn's attempt token (HephTurnController, hidden from OpenAPI).
All paths start with /internal/llm/turn/{turnId}/.
| Method and path | Purpose |
|---|---|
GET context | The session id, the stored path, the prompt, the model, and the situation. |
POST link-result | Admits one result for the answer ({resultId}). A refused result gets 422. |
POST link-problem | Admits one problem for the answer ({problemId}). A refused problem gets 422. |
POST show-feedback | Claims the copy of one piece of prepared conversation feedback for the attempt ({feedbackId}). A refused claim gets 422 (Delivery of shown feedback). |
POST search | Full-text search in the conversation tree. |
POST sandbox/read, sandbox/exec, sandbox/load | The read, bash and load_context calls in the attempt's turn sandbox (Context folder). |
PUT partial | The partial answer and the stage. The partial answer has the text and the link_result, link_problem, and show_feedback calls of each assistant message of the turn so far. The last message can be unfinished. The gateway rate limit does not apply. A lost snapshot is not sent again. |
POST end | Ends the turn with DONE, STOPPED, or FAILED, with the Pi entries that the turn appended (Limits). A failed end carries a detail, a kind of failure for the worker log (How a turn fails). The end first waits up to 10 seconds for the proxy to record the attempt's model calls (ProxyCallsInFlight). An end without an answer cancels them first. |
An end larger than the gateway's request bound gets 413.
The host then sends the end again, with each read, bash and load_context result replaced by its note (see Tool results).
The next turn reads the notes instead of this turn's results, and reads again what it needs.
The worker builds the fixed context documents of an attempt once, when the host reads the context. The turn sandbox stages that build, so one turn sees one set of documents. It stages an item document, for example the detail of one issue, when Hephaestus reads it (Staging). The end drops the build. A build that no call reads for 15 minutes expires, and a later read builds it again for the same question.
The proxy authentication of the worker (JobTokenAuthenticationProvider) knows two credentials: an attempt token (AttemptJwt) and the gateway-session token of a conversation host.
An attempt token names only its kind (REVIEW or TURN), the attempt's id, its workspace and its attempt number.
A review token also carries its work deadline.
The kind decides which row the id names: a review job or a conversation turn.
A turn's conversation and account come from the turn that this worker claimed, never from the token.
Each credential becomes a ProxyIdentity.
A turn token reaches the model and its own turn paths only.
A turn path that names another turn gets 401.
On a model call, the turn row must be leased by this worker with the same attempt and workspace.
Otherwise the call gets 409.
On a turn path, the provider reads no row.
The endpoint checks that the attempt runs on this worker, and takes the turn's conversation and account from it.
A lost lease gets 409 there too.
A host's gateway-session token reaches only GET /internal/llm/runtime/{hostId}/commands of its own host.
ProxyRoutingService bills each model call of a turn to that turn (BilledAttempt with MENTOR_TURN).
The LLM proxy cuts a conversation model call whose provider stays silent (LlmProxyService.Bounds).
Before the response headers, the first-byte bound applies, and the proxy answers 504.
In a stream, the idle bound applies, and the proxy ends the stream with an error event.
The proxy relays only whole server-sent events, so this error event never joins half an event that the provider left unfinished.
Lines can end with LF, CRLF or CR.
An unfinished event longer than 1 MiB fails the stream.
Both carry an error whose message starts with upstream_stalled.
A call that the connection's breaker refuses gets 503 with upstream_paused (LlmConnectionHealth).
The host fails the turn on either code and does not let Pi retry the call.
An end without an answer cancels the attempt's calls in flight (ProxyCallsInFlight.cancel), so a silent provider cannot hold it.
A partial answer is one conditional UPDATE … RETURNING and its NOTIFY, in one short transaction (MentorTurnService.updatePartial).
A snapshot equal to the one that the attempt stored last is not written.
The proxy records a model call's usage after it relayed the whole response.
The host can read the end of the answer earlier and end the turn.
Thus, the end waits for the attempt's calls, or the fence on the turn row would drop the last call as late.
Context folder
Hephaestus reads its context folder with the tools read, bash and load_context.
The folder is in a turn sandbox on the worker, not in the host.
ADR 0057 § Context folder and turn sandbox gives the decision.
The folder has this layout:
/workspace/
├── INDEX.md # What each area holds, what Hephaestus can load, and the rules.
├── FIELDS.md # What each document's fields mean and do not show.
├── inputs/context/<doc>.json # The context documents of the attempt.
└── repos/<host>/<path>/ # A repository with its Git history, after load_context.
ContextFolderIndex writes the core part of INDEX.md.
Each context loader adds its own section.
INDEX.md lists the documents for the model.
FIELDS.md (agent/mentor/fields.md) explains each document's fields, with one section for each document.
Hephaestus reads a section only when it needs it, so the system prompt that each model call sends stays short.
ContextFolderIndexTest keeps one section for each fixed document.
The host holds no list of document keys.
The host offers the three tools only when GET context returns contextFolder: true (TurnContextFolder.available).
The value is false after the worker failed to start a turn sandbox, for example without Docker or without the agent image.
No setting turns the tools on or off.
Endpoints
HephTurnSandboxController serves three endpoints under /internal/llm/turn/{turnId}/sandbox/, with the turn's attempt token.
Each endpoint checks that the attempt holds the lease, as the other turn endpoints do.
A refusal that the model must read is 200 with {"ok": false, "error", "detail"}.
The host returns detail as the error text of the tool.
| Endpoint | Tool | Does |
|---|---|---|
POST read | read {path, offset?, limit?} | Returns one page of a file, with the notices of Pi 1.0 read. The worker pages in the sandbox, so the host never holds more than one page. |
POST exec | bash {command, timeout?} | Runs bash -c <command> in /workspace as the unprivileged user of the image. Returns the end of the output. |
POST load | load_context {path} | Stages one item through the loader that owns the path prefix. Returns the staged path, the resolved ref, the SHA, and how to cite it. |
These rules apply to the endpoints:
- A path that leaves
/workspacegets the sameNOT_FOUNDtext as an absent file.readalso reads/tmp/heph-out/<n>.log. - A page is at most 2,000 lines or 50 KB (
PiText). A longer line gets Pi's notice with asedcommand. - A command runs as
timeout --kill-after=2s <t> bash -c …. The boundtis the smallest of the asked timeout, 60 seconds, and the time left in the turn. - The worker cancels sandbox commands that exceed their bound by 5 seconds or are killed by their timeout (
ExecSandboxes.cancelCommands). The cancellation ends every process of the sandbox user, also a child that left the command's process group. The sandbox stays in use for the next calls. On Docker, the worker runskill -KILL -1as the sandbox user, because the image has nopkill. execwrites the full output to/tmp/heph-out/<n>.logand returns the last 2,000 lines or 50 KB, with Pi's notice.- The command gets no environment from the host or the worker.
GIT_CONFIG_*setssafe.directory=*, because root owns the folder. - A load path is
repos/<host>/<full path>[@ref]. A ref is a branch, a SHA,pull/Non GitHub, ormerge-request/Non GitLab. - A load without a ref takes the default branch. A second load of the same repository replaces the first.
| Error | Endpoint | Cause |
|---|---|---|
NOT_FOUND, IS_DIRECTORY, OFFSET_BEYOND_END | read | The file is absent or not visible, the path is a directory, or the page starts after the end. |
NOT_VISIBLE | load | The repository is unknown, unmonitored, hidden, or in another workspace. All cases get one text. |
REF_NOT_FOUND | load | No commit matches the ref. |
TIMEOUT | load | The turn ended or its deadline passed before the load finished. |
BUDGET_EXHAUSTED | load | The attempt already loaded 5 items. |
INVALID_PATH | load | No loader owns the path. |
UNAVAILABLE | all | The worker cannot start a turn sandbox now, or the mirror cannot load the repository. |
CONTEXT_CHANGED | all | The workspace's data changed and the worker removed the sandbox. The next call gets a new one. |
The request fixtures sandbox-read, sandbox-exec and sandbox-load in agent/heph-turn-contract/ bind the host and the controller.
heph-turn-api.spec.ts and HephTurnGatewayIntegrationTest both read them.
Turn sandboxes
TurnSandboxes owns the turn sandboxes of one worker.
DockerExecSandboxes starts each one with its own Docker client, so a long command never takes a connection that a review waits on.
| Property | Value |
|---|---|
| Image and policy | The agent image of the release, under ContainerSecurityPolicy |
| Process | sleep infinity under Docker --init, as root. Commands run as user 1000. |
| Folder | /workspace, 512 MiB, read-only for the user: a tmpfs on Docker, a volume on the node's disk on Kubernetes |
| Scratch | /tmp: the tmpfs of the policy on Docker, a volume on the node's disk on Kubernetes |
| Network | none |
| Limits | 1 CPU, 1 GiB of memory, 256 processes (TurnSandboxes.LIMITS) |
| Labels | hephaestus.kind=heph-sandbox, hephaestus.heph-sandbox=<id>, hephaestus.heph-worker=<worker id>, and the owner and creator labels |
A sandbox has three states: POOLED, ASSIGNED and REMOVED.
The worker keeps 2 sandboxes in the pool (HephHostConfiguration.SANDBOX_POOL).
The first read, bash or load_context call of an attempt takes one from the pool, or starts one if the pool is empty.
The pool then fills again in the background.
An assigned sandbox never goes back to the pool.
The assignment exists only in the memory of the worker.
The worker removes the sandbox of an attempt with docker rm -f -v on each path that seals the attempt's model calls:
- The end of the turn from the host (
finish). - A lost lease.
- A lost host.
- The claim of a retry, for the earlier attempt.
A command that runs at that time ends with the container. A load that runs at that time is abandoned.
The reconciler (TurnSandboxes.reconcile) runs at startup and every 60 seconds on the worker.
It removes the sandboxes of this worker that it does not know, and the assigned sandboxes of attempts that no longer run here.
It also removes a sandbox whose creator worker container is gone.
It never removes a sandbox of a live other worker.
SandboxReconciler ignores turn sandboxes.
A worker that runs outside a container labels its sandboxes with an id for each process.
After a crash of such a worker, its sandboxes stay until you remove them with docker rm.
A change to what a workspace's turns may read sends MentorLedgerEvents.ContextChanged over the ledger NOTIFY channel.
The listener of each worker then removes every turn sandbox of that workspace (releaseWorkspace).
These changes send it:
- The erasure of a person (
MentorToolResultCopyPersonDataStore). - A repository that a team hides or shows (
AgentContextChangedEventfromWorkspaceTeamSettingsService). - A change of monitored repositories, and the erasure of a mirror (
MentorContextChanges). - A changed AI choice of an account, for each workspace of the account (
AiChoiceChangedEvent).
The next call of the running attempt gets CONTEXT_CHANGED once.
The call after it gets a new sandbox.
Staging
TurnContextFolder stages the folder on the first call of an attempt (ExecSandboxes.stage).
An Add sends INDEX.md, FIELDS.md and the attempt's documents as one tar into /workspace.
A load sends a Replace: the tar replaces one directory, for example repos/<host>/<path>, with all its contents.
The worker owns the tar stream, and the sandbox reads exactly its size.
A staged path is a plain path in /workspace.
After a staging, the sandbox user cannot write the folder or what the tar wrote.
exec runs commands only as the sandbox user, so no command can stage.
A failed staging can leave part of the tar, so the worker removes that sandbox.
On Docker, a staging is a script that runs through docker exec --user 0.
The script extracts the tar and then runs chmod -R a-w.
The script reads the stream with head -c, because docker-java does not close the standard input of an exec.
HephTurnContexts builds the fixed documents once for each attempt, as JSON with times in the developer's time zone.
The folder stages them with indentation, so that read can page them by line.
An item document is lazy.
Examples are issue/<id>.json, merge_readiness/<id>.json, results_history/<id>.json, delivered_feedback/<id>.json and review_attempts/<kind>/<id>.json under inputs/context/.
A read of an item path that is not staged asks the owning ContentSource.read.
The worker stages the document and returns its first page.
A path that no source owns, or that the turn may not see, gets NOT_FOUND.
Context loaders
A context loader (integration.core.spi.ContextLoader) stages a heavy item on request:
prefix()is the folder path that the loader owns, for examplerepos/.index(workspaceId, accountId)returns the loader's section of INDEX.md.load(workspaceId, accountId, path, abandoned)prepares the item and returns its directory and resolved version.
Core routes a path to a loader by its prefix only. A loader applies the same visibility rules as the context documents of its integration. An integration without heavy items contributes documents only.
The SCM integration implements the loader on RepositoryCheckouts (ScmContextLoader).
It lists up to 40 repositories that the workspace monitors on its active connection and that no team hides.
It lists each with its default branch and up to 10 open pull requests or merge requests.
A load resolves the ref, fetches a missing head into the mirror, and builds a checkout as for a review.
The checkout has the Git history that the resolved commit reaches.
Without Git mirrors (GIT_CHECKOUT_ENABLED), INDEX.md has no repos/ section.
| Limit | Value |
|---|---|
| Loads in one attempt | 5 (TurnContextFolder.MAX_LOADS_PER_ATTEMPT) |
| Loads at once on one worker | 2 (TurnContextFolder.LOADS_AT_ONCE) |
| Time of a load | The time left in the turn |
| Size of a checkout | The checkout bound of a review (GIT_MAX_SNAPSHOT_BYTES) |
Measured times
TurnSandboxLiveTest measured these times with real Docker and the agent image:
| Step | Time |
|---|---|
| Cold start of one pool sandbox | 0.45 to 0.8 seconds |
| First call: take a sandbox from the pool, stage INDEX.md and 2 documents, and read | 129 to 195 ms |
One read page | 51 to 54 ms at the median, 70 to 82 ms at p95 |
One load of a small repository with .git, without a fetch | 73 to 115 ms |
bash with git log | 56 to 67 ms at the median, 66 to 103 ms at p95 |
| End of a command after the attempt stopped | 28 to 64 ms |
A load that must fetch, and a full turn with a model, were not measured.
The output of each tool is untrusted data from the workspace. After the turn, the results follow the rules of Tool results.
Dispatch
HephTurnDispatcher runs on the worker role.
It claims turns while fewer than hephaestus.worker.capacity.mentor-max turns run on this worker.
A mentor_ledger notification from PostgreSQL wakes it, with a timed backstop.
MentorLedgerNotifications sends NOTIFY before commit, so PostgreSQL delivers it only on commit.
Live updates
The server role serves GET /workspaces/{slug}/mentor/events as server-sent events (MentorLiveHub).
The browser authenticates the stream with its cookie, and a native client with Authorization: Bearer.
- On each connect, the stream sends a
snapshotof the conversation. - Then it sends
turnandentryevents. Each event carries the full DTO that the server reads after the notification. - The end of a turn sends one
entrynotification with the ids of the messages that it appended. Tool results, summaries, and system entries are never shown, so the notification does not list them. - An erasure sends one notification for each conversation that shows a changed message.
- A large end or erasure splits its notification, so that each payload stays below the 8000-byte limit of PostgreSQL. The app does not show tool results and summaries, so their erasure sends no notification.
- A comment
pingkeeps a quiet stream open through the proxy (Runtime roles). - A stream belongs to the account that opened it.
Before each snapshot, each event, and each
ping, the server checks the membership of the account (AccountMembershipQuery.isActiveVisibleMember). The account must still be an active, visible member of the workspace. If it is not, the server closes the stream without the content. Each server checks its own streams, so a removed member gets no more content on any server.
The stream has no replay log and ignores Last-Event-ID.
The snapshot on each connect is the resume.
After each snapshot, the client reads every loaded page of the conversation and of its threads again.
An edit, a deletion, or the end of a turn does not change latestSeq, so the client cannot use it to find what changed.
The client applies the events and the answers to its own writes that arrive during this read after the read completes.
Thus, an older response cannot replace them.
The snapshot carries lastAnswerAt and lastReadAt.
The conversation has an unread answer when lastAnswerAt is later than lastReadAt.
PUT /workspaces/{slug}/mentor/conversation/read moves lastReadAt of the main conversation forward.
An earlier time changes nothing, so each device of the developer can send its own time.
When the marker moves, the developer's other open streams get a read event with the new lastReadAt (MentorRead).
One LISTEN connection serves each server replica.
MentorLiveHub limits the streams of each account in a workspace (Runtime roles).
A stream whose queue of 256 events overflows gets a new snapshot.
Context of a turn
The host rebuilds the Pi session of a turn from rows. Nothing about a conversation stays in the host after the turn. The request has four parts, in this order:
- The system prompt
agent/mentor/system.mdand the shared feedback writing guide. They are the same for each turn.HephHostSpecTestkeeps the two together at 15 KiB or less, because each model call sends them again. - The tools. They are the same for each turn of one worker.
- The path from the server (
MentorContextPathBuilder). It is the same bytes for each turn of one conversation, until it changes. - The situation (
MentorSituation). The host adds it as the last message and never stores it.
The first three parts form a stable prefix for the prompt cache. The Anthropic cache breakpoint sits on the last stored message, before the situation. The Pi session id is the conversation id, so each thread has its own cache key.
The path
The server groups entries by turn and orders the groups by their first seq.
A thread's path is the main conversation up to and including its parent entry, then the thread's own entries.
The path starts at the newest summary at or before the leaf. For a thread, that is the newest summary at or before the parent. The path leaves out erased entries. An erased answer or tool result keeps a Pi skeleton, so no tool call loses its result. An erased question keeps its Pi entry id, so a summary that keeps the conversation from that question still finds its start.
The server renders each question and weekly digest as a Pi user message with a prefix, for example [Thu 2026-10-08 09:12 Europe/Berlin · Slack].
The time zone is the developer's account.time_zone, or UTC.
A stopped answer ends with [stopped by the developer].
A failed turn adds [no answer: the turn failed] to its question.
forTurn returns the prompt that the host must pass to session.prompt() unchanged.
The situation
The situation message gives these facts:
- The time now, and whether the question was written in the app or in Slack.
- The time since the last message.
- For a thread, its parent and the main entries since the thread started.
- For the main conversation, up to 10 open threads.
- The page of the floating Hephaestus panel, if the client sent one.
- Who the developer is: their name and their handle in each connected system (
HephSituation). Context files mark the developer's own words with these handles, for examplebyYouon a Slack message. - The days of today, yesterday, this week and last week in the developer's time zone (
HephSituation). Hephaestus uses these days and does not calculate dates. - The prepared feedback that waits in this conversation (
MentorTurnEvidence.receipt): the count, and up to 3 feedback ids with their practices. It never contains the words of the feedback. It names the document that holds the feedback. - The line "Start with
read INDEX.md: it lists what this turn can read." The situation contains no content of the work, so Hephaestus must read the folder before it answers about the work.
Context documents
A context document is one file under inputs/context/ in the context folder.
An evidence.capture.ContentSource builds each document from a MentorContextRequest: the workspace, the account, the conversation, and the question.
MentorContextKeys lists the fixed keys.
INDEX.md gives Hephaestus one line about each staged document (ContextFolderIndex).
That index is the only description of the documents.
An item path, for example the detail of one issue, comes only from a resource field in a document.
The source of each document is in the module that owns its data:
Document under inputs/context/ | Source |
|---|---|
user.json, workspace.json, practice_catalog.json, practice_profile.json, delivered_feedback.json with the item path delivered_feedback/<id>, prepared_conversation_feedback.json, weekly_digest.json | agent/context/providers/mentor/ |
results_history.json and review_attempts.json, and the item paths results_history/<id> and review_attempts/<kind>/<id> | ResultHistoryContentSource and ReviewAttemptsContentSource in agent/context/providers/mentor/ |
recent_authored_work.json, activity.json, merge_readiness.json, and the item paths merge_readiness/<id> and issue/<id> | integration/scm/context/ |
slack_conversations.json | integration/slack/capture/ConversationCaptureSource |
outline_docs.json | integration/outline/capture/DocumentCaptureSource |
An admitted entry in delivered_feedback.json can include developerResponse.resolution from the developer's recorded response.
An omitted response is unknown.
It does not prove that the developer never responded.
This metadata does not change the recorded review result.
Each document obeys these rules:
- It covers the developer's own or assigned work and the developer's activity on it. Where the work needs them, it also has the names and words of other people on that work. Examples are review comments on a pull request and comments on an issue.
- Slack messages come only from threads that the developer took part in, in monitored channels whose consent is active (
ConversationThreadProjection). Other members' messages in those threads are included and marked as untrusted content. - It reads only repositories that the workspace monitors on its connected instance.
- It reads no repository that a team hides from contributions.
- It reads a related item, such as the parent of an issue, under the scope of that issue.
- It dates each item and names its repository.
- The host writes each time in the developer's time zone with its weekday and offset, for example
Mon 2026-10-12 14:41 +02:00(ContextTimes). Hephaestus does not calculate a weekday. - Each item whose web page Hephaestus stores carries that whole
url: work, reviews, comments, commits and documents. Hephaestus copies a link and never builds one. Slack messages and delivered feedback have no stored page. - It fits its own size limit. When it leaves something out, it says so (
truncated,omittedThreads,omittedPractices,omittedForSize). - It keeps Hephaestus practice reviews apart from reviews by people.
- It lists practice standings by practice group and name, never by standing.
- It never contains the diff of a pull request. It gives the files, their line counts, and
filesCoveragewhen file changes were not captured. Hephaestus reads the code and the diff in a repository that it loads into the context folder.
The stage
The host sets the stage of each partial answer from the tool call in progress.
turnStage in heph-host.ts is the only place that holds the labels.
A read or bash call that names a context document gets the label of that document, for example "Reading your practice feedback…".
A call that names repos/ or runs git gets "Reading the repository…".
A load_context call gets "Loading the repository…".
A search_conversation call gets "Looking back through this conversation…".
At all other times the stage is "Thinking…". A tool's stage holds for the partial-answer window in which the tool started, so a fast read is shown at least once. A stage never contains a tool name, a path, a command, or a repository name, because the model selects them.
The app shows the stage of a running turn (MentorTurnPartialDTO.stage).
Slack shows 👀 and the streamed text, not the stage.
Tool results
When a turn settles, the host appends Pi context_edit entries.
They replace each older read, bash and load_context result with a note: the tool, its path or command, the time, and a hint to read again.
Only the latest turn's results of these tools stay verbatim.
A load_context note says that each turn has a new context folder, so Hephaestus must load the repository again.
A fetch_context result that a development build stored before this change is not replaced.
link_result, link_problem, show_feedback, and search_conversation results stay, because they record what Hephaestus showed and found.
Compaction
Hephaestus uses Pi's own compaction, with Pi's default summary prompt.
- Before the prompt. Above the working trigger, the host compacts the stored conversation before it sends the question. The trigger is 49,152 tokens, or less for a small window. The status line shows Thinking… while the host compacts, and the summary call counts against the turn. If the conversation stays too long, the turn fails.
- Thinking off. Every summary call runs with thinking off. A reasoning model would otherwise spend Pi's summary output limit on thinking and leave the summary unfinished. The host sets the level on the agent state for the summary only, so the conversation stores no level change. Pi maps the level to each API's standard reasoning control and sends no provider-specific field. A model server that thinks by default keeps thinking (AI providers).
- A failed summary. If Pi cannot write the summary, the history stays whole. The question goes out while the conversation fits the model's window, and the next turn compacts again. Thus, one failed summary never fails every later question of the conversation.
- The trigger.
mentorCompactionstarts with a reserve of min(16,384, a quarter of the window). The trigger is the smaller of 49,152 tokens and the window less that reserve. Pi's reserve is then the window less the trigger. For a window of 128,000 tokens, it is 78,848 tokens. The reserve also caps the summary's output. - Recent messages. Pi keeps recent messages up to a target. The target is the smallest of 20,000 tokens, half the trigger and a sixth of the window. It is not a bound. Pi keeps a message whole and a tool call with its results. Thus, one long message can exceed the target.
- Room to answer. After a summary, the host estimates the messages and the fixed prompt. The answer needs min(output limit, an eighth of the window). If that does not fit the window, the turn fails with "still too long after shortening it".
- A repeat. A conversation that stays past the trigger after its summary is shortened again on the next turn. Nothing guarantees that one summary is enough.
- A kept summary. A turn that fails after its summary still stores that summary. The next turn starts from it. It summarises again only if the conversation is still past the trigger.
- During the turn. Pi's own check between tool batches is the fallback.
Pi's own check before a prompt reads only the usage of its last answer.
The host also counts what the ledger holds after that answer, for example the question of a failed turn.
The summary is one of the turn's entries, so the end of the turn stores it.
The server stores it as a SUMMARY entry, with fromSeq and toSeq of the entries that it covers.
Erasure of an entry erases every summary that covers it.
An edit of a Slack message keeps the message, but erases every summary that covers it and every search_conversation result that quotes it.
A turn that ran during an erasure or an edit erases the same copies when it ends (MentorRedaction.eraseStaleCopies).
The next turn then pays for one new compaction.
An erasure locks the main conversation row of each tree that it changes before it changes an entry. The context read shares that row, and the end of a turn locks it before it stores entries. Thus, a context read either omits an erased entry, or it ran before the erasure started. In the second case, the end of the turn erases each summary and search result that it stores from that read.
Search
search_conversation finds entries with PostgreSQL full-text search (simple configuration).
It searches the words that the app shows: the text of a message and the words of each link that Hephaestus showed.
The worker stores these words once, when it stores the message, in mentor_entry.search_text.
The GIN index ix_mentor_entry_text_search covers that column.
An edit replaces the words, and an erasure clears them.
The personal-data export reads the same column.
It searches the main conversation, this thread, or the whole tree. It returns questions, answers, and weekly digests only, never tool results, summaries, or erased entries. It returns a bounded number of hits (Limits), each with an entry id that Hephaestus can cite.
Personal data
Each write to the ledger first holds the person-data write fence of the conversation's owner (PersonDataWriteFence.holdForAccountWrite), before it locks a ledger row.
The writes are a question, the end of a turn, a feedback claim, a vote, the edit of a Slack message, and a weekly digest.
An erasure of the account waits for each write that holds the fence.
After the erasure is requested, the fence refuses each later write:
- A question, a vote, or the edit of a Slack message gets
PERSON_ERASED. - The end of a turn stores no entries.
- A
show_feedbackclaim gets422. - The weekly digest is not written.
The erasure of a person (MentorPersonDataCatalog) exports and deletes the person's own conversations.
In the conversations of other people, it redacts the Hephaestus messages that showed feedback about the person (MentorFeedbackCopyPersonDataStore).
As its last step, it redacts the tool results that can copy the person's data (MentorToolResultCopyPersonDataStore).
A turn that ran during that step erases the same copies when it ends (MentorRedaction.eraseStaleCopies).
A show_feedback result copies no one's data, so it stays.
The data map owns the inventory of person data.
Limits
This table gives the fixed limits that only contributors change.
Runtime roles gives the operator settings and their defaults, the limits and timeouts of the host, and the stream limits.
Runtime roles gives the request bound of the gateway (SANDBOX_API_MAX_REQUEST_BYTES).
| Limit | Value | Where |
|---|---|---|
| Attempts per turn | 2 | MentorTurnService.MAX_ATTEMPTS |
One read page, or the output that bash returns | 2,000 lines or 50 KB | PiText |
Time of one bash command | 60 seconds | TurnContextFolder.MAX_COMMAND_TIME |
| Loads in one attempt | 5 | TurnContextFolder.MAX_LOADS_PER_ATTEMPT |
| Empty turn sandboxes on one worker | 2 | HephHostConfiguration.SANDBOX_POOL |
| Partial answer interval | 250 ms | heph-host.ts |
| Wait of an end for the attempt's model calls | 10 seconds | ProxyCallsInFlight.SETTLE_WAIT |
| Smallest conversation model | 32,768 tokens of context, 4,096 tokens of output | AgentBindingLimits |
| Entries per end | 500 | HephTurnController |
| Search hits | 20 | MentorEntryService.MAX_SEARCH_HITS |
Words of a delivered piece in delivered_feedback.json, and in its item delivered_feedback/<id>.json | 600 and 2,000 characters. bodyTruncated marks a cut. | DeliveredFeedbackContentSource |
| Prepared conversation feedback in one turn's context | 3 pieces | PreparedConversationFeedbackContentSource |
| Age of prepared conversation feedback | 14 days | ConversationFeedbackTtlSweeper.TTL_DAYS |
Billing
The host reports no usage.
The worker bills each attempt from the LLM proxy's counters on the mentor_turn row, priced at the frozen admission rates.
The ledger row in llm_usage_event has the turn id and the attempt number.
A summary call during the turn counts against the turn's budget.