Observability
Every runtime role writes one JSON record per event to standard output.
Spring Boot writes the records in its built-in Elastic Common Schema (ECS) format.
job.id and workspace.id are fields, not metric labels.
The application exports no logs.
The container runtime keeps them on the host until an operator collects them (Collecting logs).
Questions that the signals answer
Each row is a question that an operator asks to run an instance or to make it better. The signal answers the question. The alert or the objective tells the operator when the answer needs work.
| Question | Signal | Alert or objective |
|---|---|---|
| Can provider activity be recorded? | activity_events_failed_total and activity_events_recorded_total | Investigate sustained failure increases. A write failure stays queued and retries within minutes. Inspect its exception in server logs. A vanished or unmonitored source ends its pending fact without a write and also increments the failure counter. No workspace or person labels are emitted |
| Do item reviews end complete? | practice_review_ended_total | HephaestusReviewsIncomplete |
| Which items got no result, and why? | practice_review_items_ended_total, by state and reason | HephaestusReviewsIncomplete |
| Does feedback reach each destination, and why was it withheld? | practice_feedback_settled_total, by destination, state and reason | HephaestusFeedbackDeliveriesFailing |
| How long after a push or a request does the developer get feedback? | practice_review_first_delivery_seconds | HephaestusFeedbackDeliverySlow |
| Do feedback sends repeatedly retry? | practice_feedback_send_attempts and practice_feedback_send_readback_limit | HephaestusFeedbackSendRetrying |
| Does feedback wait and never arrive? | practice_feedback_oldest_waiting_seconds | HephaestusFeedbackDeliveryStalled |
| Does the outbox carry out its actions, and do any end uncertain? | action_backlog, action_oldest_due_seconds, action_ended_total, by kind | HephaestusOutboxStalled, HephaestusOutboxUncertain |
| Do review attempts fail, and where? | agent_job_total, agent_job_infra_retry_requeued_total, agent_run_reaped_total | HephaestusAgentJobsFailing and the other review alerts |
| How long does each stage of a review take? | agent_job_stage_duration_seconds, by stage and outcome. Admission: http_server_requests_seconds on the two admission routes | None |
| Which statements take the most database time, and do tables collect dead rows? | pg_stat_statements_seconds_total by queryid, pg_stat_user_tables_n_dead_tup, pg_long_running_transactions_oldest_timestamp_seconds, with the database metrics on | HephaestusDatabaseDeadlocks, HephaestusDatabaseTransactionOpenLong, HephaestusDatabaseDeadRowsHigh, HephaestusDatabaseTransactionIdAgeHigh |
| How long does a provider event wait for its handling, and how long does the handling take? | integration_message_lag_seconds, integration_message_process_duration_seconds, by kind, with retries in integration_consumer_retry_total and dead letters in integration_consumer_poison_total | HephaestusEventsDelayed |
| Does feedback earn trust? | practice_result_recorded_total, practice_result_marked_total, practice_feedback_response_total, and the query by practice | HephaestusResultsMarkedIncorrect |
| Does work wait for a server? | agent_queue_oldest_age_seconds, agent_job_duration_seconds with phase="queue" | HephaestusAgentQueueStalled |
| Do model calls wait for their pool? | llm_pool_permit_wait_seconds, llm_pool_upstream_throttled_total | HephaestusConversationModelWaitLong, HephaestusReviewModelWaitLong |
| Did a server lose its model calls in flight? | llm_pool_permits_expired_total, llm_pool_lease_renewal_failures_total | HephaestusModelPermitsExpired, HephaestusModelPermitRenewalFailing |
| Do erasure and retention run, and do they finish? | privacy_job_completed_total, by exported_job and outcome | HephaestusPrivacyJobFailing, HephaestusPrivacyJobNotRunning |
| Did the kernel kill a container for its memory? | kube-state-metrics | HephaestusContainerOomKilled |
| How long does a model call take, how many tokens does it use, and how does it end? | gen_ai_client_operation_duration_seconds, gen_ai_client_tokens_total, gen_ai_client_calls_total by outcome | None |
| Which calls does the proxy refuse before a provider sees them? | llm_proxy_refused_total, by refusal | None |
| Does the API answer without errors, and fast enough? | http_server_requests_seconds | The API objectives |
| Which route is slow at the 95th percentile? | http_server_requests_seconds_bucket, by uri | The Activity objective |
| Does the database pool make requests wait? | hikaricp_connections_acquire_seconds, hikaricp_connections_timeout_total | HephaestusDatabasePoolSlow, HephaestusDatabaseConnectionTimeouts |
| Do pages show their content fast enough? | web_vital_duration_seconds | HephaestusPageExperiencePoor, HephaestusPageExperiencePoorWeekly |
| Does a conversation answer, and how fast does it start? | mentor_turn_completed_total, agent_run_first_token_seconds | HephaestusConversationTurnsFailing and the conversation objective |
| Do provider events arrive and get processed? | webhook_publish_total, integration_consumer_* | The webhook and provider event alerts |
| Does a workspace sync wait for a new credential? | workspace_sync_paused_total, by kind and reason | None. Workspace admins receive the connection alert |
| Does the GitLab sweep verify each note deletion? | gitlab_note_reconciliation_unverified_total counts issues and merge requests whose notes a sweep could not check | None. The next reconciliation checks them again, and the sync job of that workspace ends with a warning |
| Is the data safe? | The WAL archive, backup and volume metrics | The critical alerts |
| Which release runs in each Pod? | application_build_info | None |
| Did feedback whose write was in doubt reach the work? | practice_feedback_readback_total | HephaestusFeedbackReadbackUnknown |
| Is the monitoring itself working? | up | HephaestusWatchdog, at an outside dead-man check |
| Does the cluster evaluate the rules of the release that runs? | The monitoring stack, or the ConfigMap or the PrometheusRule of observability.alertRules | None |
Questions without a signal
These questions have no signal yet. The most important is first.
- What does a reader wait for on a page? When does the browser show the first word of an answer? No signal exists.
Objectives for 1.0
Each objective has a measured baseline from the project's test instance on 2026-11-03. The samples are small.
| Objective | Baseline | Signal |
|---|---|---|
| 95 % of item reviews end complete | 4 of 15 reviews in seven days | HephaestusReviewsIncomplete |
| 90 % of reviews post their first feedback within 30 minutes of their push or request | From review creation to the first posted copy: median 21 minutes, 95th percentile 2.9 hours, 83 reviews in 30 days | HephaestusFeedbackDeliverySlow, HephaestusFeedbackDeliveryStalled |
| 95 % of Activity reads finish within 1 second | The slowest Activity read in seven days took 0.19 seconds | HephaestusActivityLatencyBudgetBurnFast, HephaestusActivityLatencyBudgetBurnSlow |
| 95 % of conversation turns get their first token within 10 seconds | All 7 first tokens of seven days within 10 seconds, the slowest 5.9 seconds | HephaestusConversationFirstTokenBudgetBurnFast, HephaestusConversationFirstTokenBudgetBurnSlow |
Service level objectives
The group hephaestus-slos holds seven objectives.
The two API objectives, the Activity objective and the conversation objective each have an error budget for 30 days, and burn-rate alerts on that budget.
A fast burn spends 2 % of that budget in one hour, and a slow burn spends 5 % in six hours.
Each of these alerts checks a long window and a short window, so it ends soon after the cause ends (Google SRE workbook).
The two API objectives also have a ticket window of three days, with six hours as the short window. At a burn rate of 1, it spends 10 % of the budget. It finds a slow leak that the fast and slow windows do not see, and it needs work on the next working day. These ticket alerts need 1,000 requests in three days.
The review and feedback objectives each have one threshold over six hours, not a budget. The page load objective has a threshold over six hours and a second threshold over seven days. Each alert needs a minimum number of requests, page loads or reviews in its long window. Thus, one failure on an idle instance does not page.
| Objective | Signal | Why this objective |
|---|---|---|
99.5 % of API requests get no server error (HephaestusApiErrorBudgetBurnFast, HephaestusApiErrorBudgetBurnSlow, HephaestusApiErrorBudgetBurnTicket) | http_server_requests_seconds_count of the server role, without the /internal/ routes | A reader sees every server error. The /internal/ routes serve review sandboxes and the model proxy, which have their own alerts |
99.5 % of API reads finish within 2 seconds (HephaestusApiLatencyBudgetBurnFast, HephaestusApiLatencyBudgetBurnSlow, HephaestusApiLatencyBudgetBurnTicket) | http_server_requests_seconds_bucket with le="2.0" against le="+Inf", for GET without server errors, event streams, downloads and exports | A page waits for its reads. An event stream stays open on purpose, and a download or an export takes as long as its size. A server from before the 2-second bucket counts on neither side |
95 % of Activity reads finish within 1 second (HephaestusActivityLatencyBudgetBurnFast, HephaestusActivityLatencyBudgetBurnSlow) | http_server_requests_seconds_bucket with le="1.0" against le="+Inf", for GET without server errors on the workspace and public Activity routes | Activity is the page that a developer opens most. The alerts use the same windows as the API objectives. They need 20 reads in one hour or 60 in six hours, so that one slow read on a quiet instance does not fire |
95 % of conversation turns get their first token within 10 seconds (HephaestusConversationFirstTokenBudgetBurnFast, HephaestusConversationFirstTokenBudgetBurnSlow) | agent_run_first_token_seconds_bucket with le="10.0" against the count, for web and Slack turns | The first part of an answer is what a developer waits for. A turn that ends before any answer records no first token, so HephaestusConversationTurnsFailing covers it. An instance has few turns, so the alerts need 5 turns in one hour or 10 in six hours |
75 % of page loads show their main content within 2.5 seconds (HephaestusPageExperiencePoor, HephaestusPageExperiencePoorWeekly) | web_vital_duration_seconds_bucket for lcp with le="2.5", over six hours, and over seven days | The Core Web Vitals threshold for a good Largest Contentful Paint (web.dev). It needs page experience metrics on. The six-hour alert needs 50 loads in six hours, which a small instance seldom has. The weekly alert needs 50 loads in seven days |
95 % of item reviews end complete (HephaestusReviewsIncomplete) | practice_review_ended_total with kind="item", over six hours | A review that is not complete gives a developer no feedback on part of its practices. The alert fires when more than 25 % end incomplete, which spends the 5 % budget five times as fast. Readiness waits and budget stops are policy decisions with their own signals, so they count on neither side. A cancellation counts only when no sandbox room was left. The alert needs four reviews in six hours |
90 % of reviews post their first feedback within 30 minutes of their push or request (HephaestusFeedbackDeliverySlow) | practice_review_first_delivery_seconds with trigger event or request and approval="none", over six hours | A developer waits for this feedback. The alert fires when more than 30 % come later than 30 minutes, which spends the 10 % budget three times as fast. Feedback that a person approves first, and scheduled or backfill reviews, count on neither side. The alert needs four reviews in six hours. Feedback that never arrives has no observation, so HephaestusFeedbackDeliveryStalled covers it: feedback on the work that waited more than two hours for its first copy |
Health probes
Each probe group answers one question. The probes read only state inside the process. A slow or missing dependency therefore never restarts a Pod, and never takes every Pod out of rotation at once.
| Group | Members | Question |
|---|---|---|
Liveness (/actuator/health/liveness, /livez) | livenessState | Does the process run? |
Readiness (/actuator/health/readiness, /readyz) | readinessState | Can this Pod serve now? |
integrationConsumer, practiceReview and kubernetesSandbox are in neither group.
A server serves its API whether or not it may start sandboxes, and its readiness does not qualify sandbox isolation. Run the platform isolation gate before enabling work.
They stay in /actuator/health and the metrics, which the alerts read.
One state inside the process breaks liveness: a provider event handler that ran past hephaestus.integration.consumer.handler-timeout and did not stop within 30 seconds of its interrupt.
Kubernetes then restarts the server Pod, so the stuck handler ends before another server handles its event.
Next steps
- Set up monitoring scrapes the metrics, loads the alert rules and collects the logs.
- Alert rules gives the reason for each alert.
- Metrics and Log records list each signal.