Product feedback architecture

Product feedback is a report about Hephaestus, separate from practice feedback. The instance stores the report first, drafts an issue, and files it in an operator-named GitHub repository. Without a destination, it retains the report for instance admins, and sends the admin email if the instance can send one.

The constraint is that each reporter’s effort ends at Send. Code must decide what can become public before GitHub receives it. The model supplies structured data but has no tools and no authority to publish. ADR 0073 records the decision and rejected alternatives.

Modules and runtime roles

The productfeedback module owns the lifecycle and trust decisions. It owns input stripping, redaction, the prompt, output validation, routing, rendering, and the durable filing state.

The closed agent module supplies the draft adapter in agent.productfeedback. It is the outbox adapter of DRAFT_REPORT, and it runs on a server with the agent runtime. That adapter and the LLM proxy call the named productfeedback.drafting API. The dependency is agent → productfeedback. Product feedback never imports agent internals.

The drafting API and the other product feedback services load on every server. Only the model adapter needs the agent runtime. The server refreshes issue snapshots, files issues, polls follow-up, processes /publish, transfers issues, and performs external cleanup. It also makes the one model call of each draft, through the LLM proxy in its own JVM. No sandbox runs for a draft.

The productfeedback.drafting contract has these operations:

• waiting(feedbackId, generation) returns the DraftCall while the report waits for that generation: the opaque llmModelId, the two messages, and the token cap.
• waits(feedbackId, generation) tells the proxy whether the report still waits for that generation.
• answered(attempt, feedbackId, generation, rawText) keeps the model output while the attempt holds its action.
• claimed and deferred apply filing lifecycle events while the action holds its fence.
• count(feedbackId, inputTokens, outputTokens) adds the tokens of one measured call to the report.
• ended(ended) moves the report on when its action ends.

Each write checks the generation before it changes the row.

The GitHub adapter (integration.scm.github.app.productfeedback) implements integration.scm.spi.ProductFeedbackIssues for repository lookup, snapshots, filing and reconciliation. It implements integration.scm.spi.ProductFeedbackIssueUpdates for transfers, edits, comments, permissions and follow-up. It files as the instance's GitHub App, so it carries an EgressExempt(PRODUCT_FEEDBACK) declaration and consults Silent Mode before each write. Each issue operation uses the numeric repository and installation IDs frozen during routing. GitLab, Slack, and Outline do not implement product feedback destinations.

The draft action

A report enters DRAFT_QUEUED in four cases:

• At submission.
• After the reporter edits a held report.
• When an admin selects Retry for a report without a draft.
• When an admin files a saved report whose author was told about the model.

In the same transaction, FileIssueQueue.draft makes three checks. The author was told about the model, the destination names a model, and the daily limits allow a draft. A report that fails a check goes to ANSWERED without a draft. Otherwise the report's generation rises by one, and one DRAFT_REPORT action is enqueued (The outbox).

The model call offers the model no tools. This is a security property, not a limit of the current drafter (OWASP LLM06). Before each call, the proxy checks that the destination still names the call's model. It also checks that the report still waits for this generation. It sends the call through its permits and its egress guard, as it does for a review call.

After capacity is granted, admission locks the destination settings and reserves the daily allowance before provider work. A queued action reads the current limits. A reservation covers its retries. A limit of zero stops them.

The action's end moves the report to ANSWERED. When the action is DONE, the report keeps the model's answer. When the action failed, expired or was cancelled, the report has no draft.

State machine and leases

The filing row belongs to one saved report. The outbox owns the claims, leases, retries, and expiry of model calls and GitHub writes. Network calls run outside the transaction. Fenced action results apply filing lifecycle events in the transaction that records the result.

A capacity refusal applies DRAFT_DEFERRED and enters DRAFT_WAITING without spending an attempt. The next claim applies DRAFT_CLAIMED and enters DRAFTING. The terminal event DRAFT_ENDED moves any draft state to ANSWERED.

FilingLifecycle is the one definition of the machine. It names every state, every event, and every allowed move. ProductFeedbackFiling.apply is the only way to change a state, and it refuses a move that the table does not hold. The status that a reporter sees derives from the state and the route. Thus a status cannot claim something that did not happen. A new state does not compile until it has a status, because reporterState is an exhaustive switch.

The diagram shows the main moves. The table in FilingLifecycle holds all of them. Erasure is not a state: it deletes the row from any state, and a cleanup record continues the work. Withdrawal and promotion are moves of a filed issue, so the state stays FILED and the route changes.

FilingLifecycleTest checks every pair of a state and an event against a second, hand-written table. FilingLifecycleModelTest searches every reachable state of one report and its world. That world includes Pause, a repository that changes, a lost write answer, and erasure. It asserts the invariants: no private report is public, no report has two issues, erasure is always open, and every report can come to rest.

The model test fails if the search leaves any move of the table untaken. When you add a state or an event, change both tests on purpose. A refused move keeps the row as it was. The sweepers count it, log the report ID, and raise HephaestusProductFeedbackMoveRefused.

An expired lease returns work to its previous waiting state with backoff. An expired or ambiguous GitHub attempt requires reconciliation before a new write or cleanup step. The server allows up to three automatic retries before Needs action requires Retry. Every report not on GitHub stays saved in the console.

The admin email announces it only if email delivery is set up, so no state says that an email was sent. MODEL_UNAVAILABLE, LIMIT_REACHED, and DRAFT_INVALID select the no-draft route instead of stopping the report. DESTINATION_UNAVAILABLE and GITHUB_REFUSED stop filing.

Person actions use guarded transitions on the same version. An edit restarts the five-minute public hold and checks the outgoing revision again. Keep private cannot cancel a write that has already started. The saved response describes the intended route. The receipt toast confirms storage and links to Your reports, which follows the row until it reaches its actual destination.

The header opens the same composer directly on all screen sizes. Report history belongs to the account menu. The form asks for a message and any required sharing choice, with diagnostics behind an optional disclosure. The draft retains its original page context across navigation within the same workspace.

Routing gate and consent

RoutingGate decides whether a draft is eligible for public filing. FilingService checks current consent and the destination before dispatch. Maintainer promotion calls RoutingGate.refusePromotion and then checks repository permissions. The ordinary public route requires all these conditions:

• The reporter allows publication for this report under the current consent event.
• The draft describes an actionable issue, regardless of the submitted report kind.
• A public destination and a valid draft exist for the exact outgoing revision.
• The disposition is ISSUE, rather than SECURITY or NOT_AN_ISSUE.
• The report has no risk latch or blocking redaction finding.
• Safety checks find no abuse or named-person subject and find the report coherent.
• Confidence is HIGH.

Otherwise, the report goes to the private repository, or is saved for instance admins when no private destination exists. The prompt asks the model to classify praise and support questions as NOT_AN_ISSUE. Code does not independently identify those semantics. A misclassification can therefore become public when permission and the other checks allow it. Private issues record the first applicable reason: opted out, security, redaction, about a person, low confidence, abuse, or no draft. No-draft private issues use the code-redacted original text.

A security disposition, blocking redaction finding, or aboutNamedPerson result on any revision sets the risk latch. No edit or retry clears it. Each secret, credential, email address, or known person’s name blocks public filing even after redaction removes it.

The append-only product_feedback_consent events record the account, choice or withdrawal, notice version, and time. Events determine the current choice. Each report records the event used for its route.

The first choice has no selected answer. An account withdrawal sends held public reports to the private route immediately. A new notice version requires a new choice. The hold catches mistakes. It does not supply consent through inactivity.

The route freezes repository and installation IDs. Destination changes keep queued reports for the previous destination saved for instance admins with DESTINATION_CHANGED. The server checks private visibility before each private write and stops on an unknown result. The two destinations have distinct IDs and the same owner.

Redaction and the renderer

Code strips raw HTML, HTML comments, invisible Unicode, and bidirectional Unicode before inference. Deterministic redaction removes detected email addresses, secrets, and known identities before the provider receives report text. Known identities include the reporter and the names, slugs, and members of their workspaces. Redaction can miss personal data. It does not prove anonymity.

The model receives the report as delimited untrusted data and a bounded snapshot of public issue titles. The snapshot contains at most 500 recently updated open issues, excludes pull requests, and refreshes every 15 minutes. Titles have a 120-character cap and share a fixed token budget. The model never receives private issue titles. The snapshot is a bounded set of titles, not a complete duplicate search or source inspection.

The model returns a disposition, issue type, title, sections, optional match, safety fields, and confidence. It leaves absent facts empty rather than invent reproduction steps. Code validates types, lengths, and that a proposed match exists and is open in the snapshot. An invalid response selects the no-draft route.

The constrained renderer escapes model fields as plain text under fixed bug or feature headings. It rejects model-supplied HTML, images, links, headings, and comments. Bare issue references and issue URLs become code, including references within the same repository. Only code adds a validated “Possibly related” link. Each report creates its own issue. A model match never causes a comment or edit on another issue.

Redaction runs again on every rendered body, including private bodies and edited drafts. It removes detected secrets and identities, restricts URLs, and neutralizes mentions and cross-repository references. Only code can add the reporter’s opted-in GitHub mention to the public footer. The footer discloses that a model drafted the issue.

Idempotent filing and pace

Each report has a stable marker across retries:

<!-- hephaestus-product-feedback:<uuid> -->

Before a write, the attempt freezes its target, payload hash, and transport deadline. The transport deadline is shorter than the lease. A lease expiry cannot stop an in-flight GitHub request. Reconciliation waits until its deadline passes before it concludes that no matching issue exists.

The lookup reads every page of the frozen repository’s issues with the app as creator, state=all, and the first-attempt time. It checks the exact app identity, marker, and frozen payload hash. An incomplete lookup, conflict, failed lookup, or live attempt is unknown, never absent. An unknown result blocks another write. A matching issue is adopted instead of created again.

An atomic pace reservation allows one filing at a time per instance, at least ten seconds apart and at most 30 per hour. Public filing also has a five-per-account daily cap. Before queueing a report, submission allows ordinary accounts five accepted reports per UTC day across all workspaces and the instance endpoint. An account-scoped transaction advisory lock serializes the count and insert across server replicas. Instance admins bypass this submission cap. The existing per-minute, drafting, and filing limits still apply.

Exhausting a model budget cannot bypass the submission cap through private fallback filing.

The server honors Retry-After and reset headers. It waits when the installation has less than 20 percent of its rate limit left. These bounds leave headroom for practice feedback but cannot guarantee GitHub’s undisclosed limits.

Model admission

Drafts use the instance model as background calls, without tools or interactive reserve. A draft's call is the only call without a workspace. It receives a grant only when no other call of any band has a live waiting request, and it does not rise with its wait. At most one product feedback call runs per pool. It cannot pass a queued practice-review call.

A later request can wait for that one bounded transport. Calls are not preempted. Provider quota and cooldown remain shared.

The proxy acquires the permit with Caller(null, feedbackId, generation, instanceModelRef), as for every call it serves. The generation identifies the report’s draft, not a review attempt. Only product feedback omits the workspace. A model without a configured pool is unavailable. The permit remains held until the response body is consumed or the cancelled transport closes. The proxy allows one call in flight for each draft.

The proxy opens each call in the shared per-call ledger as UsageSourceType.PRODUCT_FEEDBACK_DRAFT before provider work. The row has no workspace or run. It keeps its action ID and transport deadline for recovery. It freezes the current instance price under the model catalogue lock.

Completion measures that row and adds its tokens to the report once, in one transaction. A call without reported usage stays unverifiable and retains its cost bound. The action end settles its open calls. A server pass also settles calls after their deadline or claim loss.

Person export includes these calls. Erasure settles an open call and detaches its report identifier while it retains accounting facts. A late completion measures the same detached row and cannot restore the identifier.

Workspace budget caps do not apply to instance report drafts. The daily draft limits still apply.

Promotion, follow-up, and cleanup

The server reads private issue comments for /publish every five minutes, through every page after the stored cursor. It records each command ID durably before acting. At dispatch, the commenter needs current write or admin permission on both repositories. An unknown permission denies promotion. authorAssociation does not establish authority.

Promotion requires reporter permission and a valid redacted draft. It overrides confidence only. The draft must remain an actionable, coherent ISSUE. It never overrides consent, security, the risk latch, or abuse.

It creates a public issue with its own durable attempt marker, then closes the private issue with the public link. It adds no private backlink to the public issue. The no-draft fallback text cannot be promoted.

Every 30 minutes, the server polls followed issues in GraphQL batches of 100. Cursors and notification keys prevent repeated emails. Public comments from another actor, state changes, and duplicate closure can produce reporter emails. Private comments do not. Duplicate closure moves follow-up to the original issue.

Pause keeps new and unfiled reports saved for instance admins and holds publication commands until resume. It leaves follow-up, withdrawal, and erasure cleanup active. An in-flight write finishes or reconciles under the normal rules.

Erasure integrates with PersonDataCatalog, AccountErasureContributor, and WorkspacePurgeContributor. Before source rows are deleted, the same transaction copies the reconciliation envelope to product_feedback_cleanup. It retains repository and installation IDs, marker, payload hash, deadline, known issue number, and required actions, without report text or account identity. External cleanup runs asynchronously after reconciliation.

It cancels held reports, moves public issues to private, replaces issue text, and closes private issues. Issue deletion and revision removal require maintainer action. After 30 days, incomplete cleanup becomes Needs action, with an admin email if email delivery is set up. Its locator stays until the steps finish or an admin confirms manual completion.

Routing evaluation

The routing evaluation in scripts/product-feedback-eval/ uses about 30 synthetic cases. Its fixtures cover bugs, ideas, general feedback, security, identities, secrets, email addresses, abuse, duplicates, German text, and prompt injection. Each case records the expected route and reason, plus strings that must not reach the output.

The unit tier runs real routing, redaction, and rendering code without provider credentials on every build. It supplies the most permissive model answer to test that code still blocks privacy violations. Poisoned snapshot titles must cause no write to the named issue and must not enter the rendered body.

The live tier uses the real drafter and provider under @Tag("live"):

cd server
./gradlew :application:liveTest

Any expected-private case that routes public fails the evaluation. Any forbidden string in output also fails it. Expected-public reports routed private count toward a separate rate. Each run records the model ID, prompt digest, and case-set hash under the evaluation provenance rules. The routing evaluation reuses provenance conventions, not the practice feedback claim scorer.

Design trade-offs

Automatic filing removes a separate publish task for the reporter. The public gate, persistent risk latch, and hold carry that responsibility instead. Every possible duplicate becomes a new issue because automatic comments create a harassment path and complicate withdrawal. Polling keeps issue updates out of workspace ingestion. GitHub issue types classify reports without labels that could start automation. The ADR records the full alternatives and conditions for revisiting them.

The admin guide owns setup, recovery, and manual cleanup. The user guide owns the reporting flow and choices. The personal-data map owns export and erasure coverage.

Source investigation boundary

The current drafter has no source checkout or execution tools. It receives the release name stored with the report, the attached route, and a public issue-title snapshot. It must not claim a reproduction, root cause, relevant file, or fixed release from those inputs alone. The composer submits FEEDBACK for every report and leaves issue classification to the drafter. The API still accepts the stored report kinds. They do not determine eligibility for public filing.

The model classification limits are described under Routing gate and consent.

A source-aware investigation needs the same capture boundary as practice reviews:

• Resolve the reported release to an immutable commit in an operator-selected source repository. The issue destination need not contain the code.
• Capture a bounded read-only checkout and relevant open and closed issue excerpts. Record their revisions and completeness. Record unavailable releases or incomplete searches as unavailable evidence.
• Run investigation in the existing isolated runtime with a separate product feedback job contract. Give it a deadline, token budget and lowest-priority model admission. Do not create a practice review or charge a workspace for instance reports.
• Separate user statements from observations supported by source evidence. Require commit and file references for technical claims. Retain uncertainty where reproduction is unavailable.
• Return a structured candidate to the existing validation, redaction, consent, hold and filing stages. Give investigation no GitHub write credentials. It never decides to publish, close or comment on another issue.

This is an extension boundary, not a capability of the current single-call drafter. It deliberately requires captured evidence and an explicit runtime contract before source inspection can be enabled. GitHub’s task guidance supports well-scoped problems and acceptance criteria. Anthropic’s workflow guidance supports starting with a bounded workflow and measuring quality before adding autonomous execution. GitHub’s API guidance supports paced writes and honoring backoff signals.