# Observability Every runtime role writes one JSON record per event to standard output. Spring Boot writes the records in its built-in [Elastic Common Schema (ECS) format](https://docs.spring.io/spring-boot/reference/features/logging.html#features.logging.structured). `job.id` and `workspace.id` are fields, not metric labels. The application exports no logs. The container runtime keeps them on the host until an operator collects them ([Collecting logs](/next/admin/monitoring.md#collect-logs)). ## Questions that the signals answer Each row is a question that an operator asks to run an instance or to make it better. The signal answers the question. The alert or the objective tells the operator when the answer needs work. | Question | Signal | Alert or objective | | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Can provider activity be recorded? | `activity_events_failed_total` and `activity_events_recorded_total` | Investigate sustained failure increases. A write failure stays queued and retries within minutes. Inspect its exception in server logs. A vanished or unmonitored source ends its pending fact without a write and also increments the failure counter. No workspace or person labels are emitted | | Do item reviews end complete? | `practice_review_ended_total` | `HephaestusReviewsIncomplete` | | Which items got no result, and why? | `practice_review_items_ended_total`, by `state` and `reason` | `HephaestusReviewsIncomplete` | | Does feedback reach each destination, and why was it withheld? | `practice_feedback_settled_total`, by `destination`, `state` and `reason` | `HephaestusFeedbackDeliveriesFailing` | | How long after a push or a request does the developer get feedback? | `practice_review_first_delivery_seconds` | `HephaestusFeedbackDeliverySlow` | | Do feedback sends repeatedly retry? | `practice_feedback_send_attempts` and `practice_feedback_send_readback_limit` | `HephaestusFeedbackSendRetrying` | | Does feedback wait and never arrive? | `practice_feedback_oldest_waiting_seconds` | `HephaestusFeedbackDeliveryStalled` | | Does the outbox carry out its actions, and do any end uncertain? | `action_backlog`, `action_oldest_due_seconds`, `action_ended_total`, by `kind` | `HephaestusOutboxStalled`, `HephaestusOutboxUncertain` | | Do review attempts fail, and where? | `agent_job_total`, `agent_job_infra_retry_requeued_total`, `agent_run_reaped_total` | `HephaestusAgentJobsFailing` and the other review alerts | | How long does each stage of a review take? | `agent_job_stage_duration_seconds`, by `stage` and `outcome`. Admission: `http_server_requests_seconds` on the two [admission routes](/next/admin/metrics.md#agent-job-metrics) | None | | Which statements take the most database time, and do tables collect dead rows? | `pg_stat_statements_seconds_total` by `queryid`, `pg_stat_user_tables_n_dead_tup`, `pg_long_running_transactions_oldest_timestamp_seconds`, with the [database metrics](/next/admin/monitoring.md#turn-on-database-metrics) on | `HephaestusDatabaseDeadlocks`, `HephaestusDatabaseTransactionOpenLong`, `HephaestusDatabaseDeadRowsHigh`, `HephaestusDatabaseTransactionIdAgeHigh` | | How long does a provider event wait for its handling, and how long does the handling take? | `integration_message_lag_seconds`, `integration_message_process_duration_seconds`, by `kind`, with retries in `integration_consumer_retry_total` and dead letters in `integration_consumer_poison_total` | `HephaestusEventsDelayed` | | Does feedback earn trust? | `practice_result_recorded_total`, `practice_result_marked_total`, `practice_feedback_response_total`, and the [query by practice](/next/admin/metrics.md#feedback-trust) | `HephaestusResultsMarkedIncorrect` | | Does work wait for a server? | `agent_queue_oldest_age_seconds`, `agent_job_duration_seconds` with `phase="queue"` | `HephaestusAgentQueueStalled` | | Do model calls wait for their pool? | `llm_pool_permit_wait_seconds`, `llm_pool_upstream_throttled_total` | `HephaestusConversationModelWaitLong`, `HephaestusReviewModelWaitLong` | | Did a server lose its model calls in flight? | `llm_pool_permits_expired_total`, `llm_pool_lease_renewal_failures_total` | `HephaestusModelPermitsExpired`, `HephaestusModelPermitRenewalFailing` | | Do erasure and retention run, and do they finish? | `privacy_job_completed_total`, by `exported_job` and `outcome` | `HephaestusPrivacyJobFailing`, `HephaestusPrivacyJobNotRunning` | | Did the kernel kill a container for its memory? | kube-state-metrics | `HephaestusContainerOomKilled` | | How long does a model call take, how many tokens does it use, and how does it end? | `gen_ai_client_operation_duration_seconds`, `gen_ai_client_tokens_total`, `gen_ai_client_calls_total` by `outcome` | None | | Which calls does the proxy refuse before a provider sees them? | `llm_proxy_refused_total`, by `refusal` | None | | Does the API answer without errors, and fast enough? | `http_server_requests_seconds` | The API objectives | | Which route is slow at the 95th percentile? | `http_server_requests_seconds_bucket`, by `uri` | The Activity objective | | Does the database pool make requests wait? | `hikaricp_connections_acquire_seconds`, `hikaricp_connections_timeout_total` | `HephaestusDatabasePoolSlow`, `HephaestusDatabaseConnectionTimeouts` | | Do pages show their content fast enough? | `web_vital_duration_seconds` | `HephaestusPageExperiencePoor`, `HephaestusPageExperiencePoorWeekly` | | Does a conversation answer, and how fast does it start? | `mentor_turn_completed_total`, `agent_run_first_token_seconds` | `HephaestusConversationTurnsFailing` and the conversation objective | | Do provider events arrive and get processed? | `webhook_publish_total`, `integration_consumer_*` | The webhook and provider event alerts | | Does a workspace sync wait for a new credential? | `workspace_sync_paused_total`, by `kind` and `reason` | None. Workspace admins receive the connection alert | | Does the GitLab sweep verify each note deletion? | `gitlab_note_reconciliation_unverified_total` counts issues and merge requests whose notes a sweep could not check | None. The next reconciliation checks them again, and the sync job of that workspace ends with a warning | | Is the data safe? | The WAL archive, backup and volume metrics | The critical alerts | | Which release runs in each Pod? | `application_build_info` | None | | Did feedback whose write was in doubt reach the work? | `practice_feedback_readback_total` | `HephaestusFeedbackReadbackUnknown` | | Is the monitoring itself working? | `up` | `HephaestusWatchdog`, at an outside dead-man check | | Does the cluster evaluate the rules of the release that runs? | The [monitoring stack](/next/admin/monitoring.md#run-the-monitoring-stack), or the ConfigMap or the PrometheusRule of `observability.alertRules` | None | ### Questions without a signal These questions have no signal yet. The most important is first. 1. What does a reader wait for on a page? When does the browser show the first word of an answer? No signal exists. ### Objectives for 1.0 Each objective has a measured baseline from the project's test instance on 2026-11-03. The samples are small. | Objective | Baseline | Signal | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | | 95 % of item reviews end complete | 4 of 15 reviews in seven days | `HephaestusReviewsIncomplete` | | 90 % of reviews post their first feedback within 30 minutes of their push or request | From review creation to the first posted copy: median 21 minutes, 95th percentile 2.9 hours, 83 reviews in 30 days | `HephaestusFeedbackDeliverySlow`, `HephaestusFeedbackDeliveryStalled` | | 95 % of Activity reads finish within 1 second | The slowest Activity read in seven days took 0.19 seconds | `HephaestusActivityLatencyBudgetBurnFast`, `HephaestusActivityLatencyBudgetBurnSlow` | | 95 % of conversation turns get their first token within 10 seconds | All 7 first tokens of seven days within 10 seconds, the slowest 5.9 seconds | `HephaestusConversationFirstTokenBudgetBurnFast`, `HephaestusConversationFirstTokenBudgetBurnSlow` | ## Service level objectives The group `hephaestus-slos` holds seven objectives. The two API objectives, the Activity objective and the conversation objective each have an error budget for 30 days, and burn-rate alerts on that budget. A fast burn spends 2 % of that budget in one hour, and a slow burn spends 5 % in six hours. Each of these alerts checks a long window and a short window, so it ends soon after the cause ends ([Google SRE workbook](https://sre.google/workbook/alerting-on-slos/)). The two API objectives also have a ticket window of three days, with six hours as the short window. At a burn rate of 1, it spends 10 % of the budget. It finds a slow leak that the fast and slow windows do not see, and it needs work on the next working day. These ticket alerts need 1,000 requests in three days. The review and feedback objectives each have one threshold over six hours, not a budget. The page load objective has a threshold over six hours and a second threshold over seven days. Each alert needs a minimum number of requests, page loads or reviews in its long window. Thus, one failure on an idle instance does not page. | Objective | Signal | Why this objective | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 99.5 % of API requests get no server error (`HephaestusApiErrorBudgetBurnFast`, `HephaestusApiErrorBudgetBurnSlow`, `HephaestusApiErrorBudgetBurnTicket`) | `http_server_requests_seconds_count` of the server role, without the `/internal/` routes | A reader sees every server error. The `/internal/` routes serve review sandboxes and the model proxy, which have their own alerts | | 99.5 % of API reads finish within 2 seconds (`HephaestusApiLatencyBudgetBurnFast`, `HephaestusApiLatencyBudgetBurnSlow`, `HephaestusApiLatencyBudgetBurnTicket`) | `http_server_requests_seconds_bucket` with `le="2.0"` against `le="+Inf"`, for `GET` without server errors, event streams, downloads and exports | A page waits for its reads. An event stream stays open on purpose, and a download or an export takes as long as its size. A server from before the 2-second bucket counts on neither side | | 95 % of Activity reads finish within 1 second (`HephaestusActivityLatencyBudgetBurnFast`, `HephaestusActivityLatencyBudgetBurnSlow`) | `http_server_requests_seconds_bucket` with `le="1.0"` against `le="+Inf"`, for `GET` without server errors on the workspace and public Activity routes | Activity is the page that a developer opens most. The alerts use the same windows as the API objectives. They need 20 reads in one hour or 60 in six hours, so that one slow read on a quiet instance does not fire | | 95 % of conversation turns get their first token within 10 seconds (`HephaestusConversationFirstTokenBudgetBurnFast`, `HephaestusConversationFirstTokenBudgetBurnSlow`) | `agent_run_first_token_seconds_bucket` with `le="10.0"` against the count, for web and Slack turns | The first part of an answer is what a developer waits for. A turn that ends before any answer records no first token, so `HephaestusConversationTurnsFailing` covers it. An instance has few turns, so the alerts need 5 turns in one hour or 10 in six hours | | 75 % of page loads show their main content within 2.5 seconds (`HephaestusPageExperiencePoor`, `HephaestusPageExperiencePoorWeekly`) | `web_vital_duration_seconds_bucket` for `lcp` with `le="2.5"`, over six hours, and over seven days | The Core Web Vitals threshold for a good Largest Contentful Paint ([web.dev](https://web.dev/articles/lcp)). It needs [page experience metrics](/next/admin/metrics.md#page-experience-metrics) on. The six-hour alert needs 50 loads in six hours, which a small instance seldom has. The weekly alert needs 50 loads in seven days | | 95 % of item reviews end complete (`HephaestusReviewsIncomplete`) | `practice_review_ended_total` with `kind="item"`, over six hours | A review that is not complete gives a developer no feedback on part of its practices. The alert fires when more than 25 % end incomplete, which spends the 5 % budget five times as fast. Readiness waits and budget stops are policy decisions with their own signals, so they count on neither side. A cancellation counts only when no sandbox room was left. The alert needs four reviews in six hours | | 90 % of reviews post their first feedback within 30 minutes of their push or request (`HephaestusFeedbackDeliverySlow`) | `practice_review_first_delivery_seconds` with `trigger` `event` or `request` and `approval="none"`, over six hours | A developer waits for this feedback. The alert fires when more than 30 % come later than 30 minutes, which spends the 10 % budget three times as fast. Feedback that a person approves first, and scheduled or backfill reviews, count on neither side. The alert needs four reviews in six hours. Feedback that never arrives has no observation, so `HephaestusFeedbackDeliveryStalled` covers it: feedback on the work that waited more than two hours for its first copy | ## Health probes Each probe group answers one question. The probes read only state inside the process. A slow or missing dependency therefore never restarts a Pod, and never takes every Pod out of rotation at once. | Group | Members | Question | | --------------------------------------------------- | ---------------- | ----------------------- | | Liveness (`/actuator/health/liveness`, `/livez`) | `livenessState` | Does the process run? | | Readiness (`/actuator/health/readiness`, `/readyz`) | `readinessState` | Can this Pod serve now? | `integrationConsumer`, `practiceReview` and `kubernetesSandbox` are in neither group. A server serves its API whether or not it may start sandboxes, and its readiness does not qualify sandbox isolation. Run the platform isolation gate before enabling work. They stay in `/actuator/health` and the metrics, which the alerts read. One state inside the process breaks liveness: a provider event handler that ran past `hephaestus.integration.consumer.handler-timeout` and did not stop within 30 seconds of its interrupt. Kubernetes then restarts the server Pod, so the stuck handler ends before another server handles its event. ## Next steps * [Set up monitoring](/next/admin/monitoring.md) scrapes the metrics, loads the alert rules and collects the logs. * [Alert rules](/next/admin/alert-rules.md) gives the reason for each alert. * [Metrics](/next/admin/metrics.md) and [Log records](/next/admin/log-records.md) list each signal.