# Observability contract Every runtime role writes one JSON record per event to standard output. Spring Boot writes the records in its built-in [Elastic Common Schema (ECS) format](https://docs.spring.io/spring-boot/reference/features/logging.html#features.logging.structured). `job.id` and `workspace.id` are fields, not metric labels. The application exports no logs. The container runtime keeps them on the host until an operator collects them ([Collecting logs](#collecting-logs)). ## Scrape metrics Each runtime role serves `GET /actuator/prometheus` without a user token on a separate management listener, port **9090** by default (`MANAGEMENT_PORT` outside the supported Compose stacks). The application listener refuses this metrics path, including requests with a user token. Port 8081 is the sandbox gateway, not a metrics endpoint. Liveness and readiness also remain available on the application listener at `/livez` and `/readyz`. Container probes use these application-listener paths. The multi-file and single-host Compose stacks bind management to the container's own `shared-network` address through its network-qualified hostname. They expose 9090 **only on that network**. This excludes sandbox interfaces, including after a restart with other networks attached. Docker's [network-scoped DNS resolution](https://github.com/moby/moby/blob/v28.2.2/libnetwork/sandbox.go#L401-L425) selects the management network without a fixed container IP. The stacks do not publish the management port to the host or configure a proxy route. Docker `expose` is not a firewall. Any container on that network can read metrics. The management listener's default address is loopback (`127.0.0.1`). 1. Attach your own Prometheus to that network. 2. Do not add a `ports` mapping, Traefik router, or internet ingress for management. 3. Restrict network membership to trusted services. 4. For non-Compose deployments, restrict the management listener with private networks and firewall policy before startup. 5. For a remote scraper, set `MANAGEMENT_SERVER_ADDRESS` to a specific trusted private interface. 6. Do not use the application or sandbox gateway port for management. See [Spring Boot management listeners](https://docs.spring.io/spring-boot/reference/actuator/monitoring.html) and [Prometheus security](https://prometheus.io/docs/operating/security/). No monitoring service or outbound metric exporter is enabled by this configuration. Operators who do not run Prometheus need no monitoring setup. Traces remain opt-in and separate from scraping. Example `prometheus.yml` for the multi-file Compose stack: ``` scrape_configs: - job_name: hephaestus scrape_interval: 30s scrape_timeout: 10s metrics_path: /actuator/prometheus static_configs: - targets: [application-server:9090] labels: runtime_role: server - targets: [application-worker:9090] labels: runtime_role: worker - targets: [webhook-server:9090] labels: runtime_role: webhook rule_files: - hephaestus-alerts.yml ``` For multiple replicas: 1. Discover and scrape **all containers**, not a load-balanced service address that can switch replicas between scrapes. 2. Keep each target's `instance` label distinct. 3. Apply `increase` or `rate` to each counter before you sum them. 4. Never add `workspace.id` or `job.id` as metric labels. This handles resets per process. `runtime_role` above is an operator-supplied target label, not an application meter label. `workspace.id` and `job.id` remain log fields. Example `prometheus.yml` for the [Helm chart](/admin/kubernetes.md), with the release in the namespace `hephaestus`: ``` scrape_configs: - job_name: hephaestus scrape_interval: 30s scrape_timeout: 10s metrics_path: /actuator/prometheus kubernetes_sd_configs: - role: pod namespaces: names: [hephaestus] relabel_configs: - source_labels: - __meta_kubernetes_pod_label_app_kubernetes_io_name - __meta_kubernetes_pod_container_port_name regex: hephaestus;management action: keep - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component] target_label: runtime_role - source_labels: [__meta_kubernetes_namespace] target_label: namespace - source_labels: [__meta_kubernetes_pod_name] target_label: pod rule_files: - hephaestus-alerts.yml ``` Each pod of the server, the webhook role and the worker is then one target. The `app.kubernetes.io/component` label of the pod gives `runtime_role`. Set `networkPolicy.metrics` to the namespace and the pod labels of this Prometheus ([Network policy](/admin/kubernetes.md#network-policy)). With the Prometheus Operator, a PodMonitor must give the same labels. Set `job` to `hephaestus` with a relabeling. ### Alert rules The release checkout contains the alert rules in `deploy/observability/hephaestus-alerts.yml`. Load that file as it is, from the release that you run: `https://github.com/hephaestus-build/Hephaestus/blob/v/deploy/observability/hephaestus-alerts.yml`. With the Prometheus Operator, put the `groups` list of the file into the `spec.groups` of a PrometheusRule. The chart does not render a PrometheusRule. The rules of the group `hephaestus` read the `job` and `runtime_role` labels that your scrape configuration sets, not labels that the chart controls. Helm reads files only from the chart directory, so a template would need a second copy of the tested file. The plain file has one home, and it works with any Prometheus. [Alert response](/admin/production-operations-runbook.md#alert-response) gives the response to each alert. The rule file is the home of each threshold, window, and summary. This section gives the reason for each threshold. `hephaestus-alerts.test.yml` beside the rules contains a `promtool` unit test for each alert. Each test gives series that fire the alert and series that do not. CI runs these tests for each change to the rules. Each alert has one of two `severity` labels: * `critical`: data is at risk now. Send these alerts to a person at once. The critical alerts are a full PostgreSQL volume, a failing WAL archive, a stale backup, and no ready webhook receiver. * `warning`: a fault needs work during working hours. All other alerts are warnings. The rules read the exported Prometheus names, not the dotted Micrometer names. The rules of the group `hephaestus` read the application's metrics. Budget admission and committed agent-job outcomes can occur in different roles, so their rules aggregate all scraped roles. They need these target labels from the scrape configuration: * `job="hephaestus"` on each target of the runtime roles. * `runtime_role`, with the value `server`, `worker` or `webhook`. * A different `instance` on each target. `HephaestusMetricsUnavailable` fires only for a target that Prometheus still discovers. When service discovery loses all targets of a role, no `up` series remains. To detect that, add a rule for each role that you run, for example `absent(up{job="hephaestus",runtime_role="server"})`. The rules of the group `hephaestus-kubernetes` read cluster metrics. Each rule needs its exporter, scraped by the same Prometheus: | Exporter | Metrics | Alerts | | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/cronjob-metrics.md) | `kube_cronjob_status_last_successful_time`, `kube_cronjob_created`, `kube_deployment_status_replicas_available`, `kube_deployment_status_replicas_unavailable`, `kube_deployment_spec_replicas`, `kube_deployment_labels` | `HephaestusBackupStale`, `HephaestusWebhookReceiverDown`, `HephaestusWebhookReceiverUnready` | | The kubelet's `/metrics` endpoint, with a storage driver that reports volume statistics | `kubelet_volume_stats_available_bytes`, `kubelet_volume_stats_capacity_bytes` | `HephaestusPostgresVolumeFilling`, `HephaestusJetStreamVolumeFilling` | | [node-exporter](https://github.com/prometheus/node_exporter) on each node | `node_filesystem_avail_bytes`, `node_filesystem_size_bytes`, `node_filesystem_readonly` | `HephaestusNodeDiskFilling` | | [cert-manager](https://cert-manager.io/docs/devops-tips/prometheus-metrics/) | `certmanager_certificate_expiration_timestamp_seconds` | `HephaestusCertificateExpiring` | The group `hephaestus-pull-deploy` reads the metrics of the [pull-based deployment](/admin/pull-based-deployment.md#watching-it) reconciler. node-exporter's textfile collector on each host that runs the reconciler exports them. Load this group only when Prometheus scrapes such a host: `HephaestusDeployMetricsMissing` fires when no host reports. A rule whose exporter is not scraped has no series and never fires. Before you rely on a rule, query its metrics in Prometheus. The kube-state-metrics, kubelet and cert-manager rules have no series outside Kubernetes. The webhook rules find the release's webhook Deployment, `-webhook`, by the chart's labels `app.kubernetes.io/name=hephaestus` and `app.kubernetes.io/component=webhook`. kube-state-metrics exports these labels only when you allow them: * Start kube-state-metrics with `--metric-labels-allowlist=deployments=[app.kubernetes.io/name,app.kubernetes.io/component]`. * With kube-prometheus-stack, set the value `kube-state-metrics.metricLabelsAllowlist` to `["deployments=[app.kubernetes.io/name,app.kubernetes.io/component]"]`. Without these labels, the two webhook rules find no Deployment and never fire. The other rules find the release's objects by the names that the chart gives them: `data--postgresql-0`, `data--nats-0`, and the CronJobs `-postgresql-backup-full` and `-postgresql-backup-diff`. Another application in the cluster can use the same names, for example another PostgreSQL chart. To limit these rules to the release, add a matcher for its namespace, for example `namespace="hephaestus"`. Do the same for `HephaestusCertificateExpiring`, which covers every certificate that cert-manager manages in the cluster. The chart shortens a name above 52 characters, and then the name loses its suffix. For a release name longer than 34 characters, change the name patterns of these rules. | Alert | Signal | Why this threshold | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `HephaestusMetricsUnavailable` | `up` | Several failed scrapes in a row, not one missed scrape | | `HephaestusConsumerPoison` | `integration_consumer_poison_total` | Each poison event is a delivery that the application stopped retrying | | `HephaestusStreamPollStale` | `webhook_stream_poll_age_seconds` | Three times the default 60-second monitor interval. `NaN` means a stream was never polled successfully, not an age of zero | | `HephaestusStreamPollMissing` | No `webhook_stream_poll_age_seconds` from a scraped receiver | The stream monitor is gone, which a failed scrape does not show | | `HephaestusStreamBacklog` | `webhook_stream_consumer_pending`: events that the broker did not deliver to a consumer | Under steady load, a few events wait between two fetches. A backlog above that level at each evaluation for 15 minutes shows consumers that fall behind or do not fetch. At its bound, the stream removes its oldest message, also when a consumer still needs it. The gauge counts an event once for each consumer that it matches | | `HephaestusWebhookPublishFailing` | `webhook_publish_total{outcome="failure"}` | GitHub redelivery covers a short stall of the broker, not a long one | | `HephaestusWebhookRedeliveryNotRunning` | `github_webhook_redelivery_total` | Redelivery requests that GitHub refused or that could not read the App's deliveries. One refusal can be a race. Three in 30 minutes are not | | `HephaestusWebhookRedeliveriesRequested` | `github_webhook_redelivery_total{outcome="requested"}` divided by `webhook_publish_total{stream="github",outcome="success"}`, over one hour | The server asks for a redelivery when GitHub recorded no successful delivery of an event. This includes a delivery that the receiver never saw, for example at an ingress timeout or while the receiver restarted. One event can be asked for up to five times. A ratio scales with traffic: more than 1 request for each 20 received deliveries is more than occasional network faults. A minimum of three requests keeps a single redelivery in a quiet hour quiet. A client error from the receiver, such as from a wrong webhook secret, gives no request and no count. The alert stays up for up to one hour after the requests stop. Only GitHub reports its delivery outcomes: GitLab, Slack and Outline have no such metric | | `HephaestusWebhookEventsUnrecovered` | `github_webhook_unrecovered` | Each counted event failed all six delivery attempts | | `HephaestusBudgetExhausted` | `llm_budget_exhausted_total` | A workspace crossed the instance-funded monthly cap | | `HephaestusBudgetBlocked` | `llm_budget_blocked_total` | Admission refused work at a cap. A single refusal is expected at a cap. Refusals that continue for 15 minutes need a decision on the budget | | `HephaestusAgentJobsFailing` | `agent_job_total` with `failed` or `timed_out` | All terminal failures, whatever their cause. Three in one hour point to the platform also on a small instance that runs a few reviews each day. Cancellations do not count. The alert stays up for up to one hour after the last failure | | `HephaestusReviewSandboxFailing` | `agent_job_infra_retry_requeued_total{cause="sandbox"}` | The sandbox could not start, initialize or finish, and the attempt runs again. Three in one hour point to the cluster: the sandbox namespace, its quota, the agent image or the nodes. The alert stays up for up to one hour after the last failure | | `HephaestusReviewUnreachable` | `agent_job_infra_retry_requeued_total` with `cause` `server_unreachable` or `provider_unreachable` | The sandbox reached neither the server's gateway nor the model provider. The `cause` label tells which. Three in one hour point to the network policy or to the provider. The alert stays up for up to one hour after the last failure | | `HephaestusReviewPreparationFailing` | `agent_job_infra_retry_requeued_total{cause="preparation"}` | An attempt failed before its sandbox started, for example because the reviewed source was not captured yet. A few retries are normal after a push. Retries that continue point to provider sync or the database | | `HephaestusReviewWorkerLost` | `agent_job_orphan_failed_total` | Each count is a review that failed because a worker stopped during each of its attempts. Workers that stop that often are evicted, out of memory, or on lost nodes. The alert stays up for one hour after the count | | `HephaestusAgentQueueStalled` | `agent_queue_oldest_age_seconds` | A practice review usually runs for 10 to 16 minutes. A wait of several review durations shows that no worker claims work | | `HephaestusSandboxIsolationFailed` | `sandbox_isolation_passed` of a worker | A later probe failed, and the worker starts no sandbox until a probe passes. The wait covers the first probe at the start | | `HephaestusWalArchiveFailing` | `database_wal_archive_failing`, and `database_wal_archive_sample_failures_total` | PostgreSQL retries a failed segment about once each minute, so a short fault of the bucket clears itself. A longer failure stops point-in-time recovery after the last archived segment. A server that cannot read the archiver for the same time cannot show that the archive works, so the alert also fires then | | `HephaestusWebhookReceiverDown` | `kube_deployment_status_replicas_available` of the webhook Deployment, while its desired replicas are above 0 | No available receiver means that every provider delivery fails. The wait covers the start of a replacement pod. A Deployment scaled to 0 for maintenance does not fire. A receiver that is unavailable for a shorter time than the wait, for example for 16 seconds to 2 minutes, does not fire. `HephaestusWebhookRedeliveriesRequested` can then fire for the GitHub deliveries that it lost, when they reach its threshold. Deliveries of other providers have no such alert | | `HephaestusWebhookReceiverUnready` | `kube_deployment_status_replicas_unavailable` of the webhook Deployment, while another replica is available | The other replicas serve, so this is a warning. The wait is longer than the 10 minutes that a new pod waits for the schema | | `HephaestusBackupStale` | The newest successful full or differential backup of a release, or the creation time of CronJobs with no successful backup | A differential backup runs each day. The hours above one day let a full backup run | | `HephaestusPostgresVolumeFilling` | Free space of the PostgreSQL volume | A failed WAL archive fills the volume. The warning gives time to repair the archive. The critical level comes shortly before PostgreSQL stops. The warning covers only the band above the critical level, so the two never fire together | | `HephaestusJetStreamVolumeFilling` | Free space of the JetStream volume | The volume is `max_file` plus 25 %, so a broker at its storage budget leaves 20 % free. Below the threshold, something outside the budget fills the volume. A full file system wedges the broker | | `HephaestusNodeDiskFilling` | Free space of a writable node file system, not the volumes that the kubelet mounts for pods | The kubelet's image garbage collection starts at 85 % use and frees space to 80 %. When the space stays low, eviction comes next. The volume rules cover pod volumes | | `HephaestusDeployReconcileFailing` | `hephaestus_deploy_reconcile_success` | The timer runs the reconciler each minute. A run that fails for 15 minutes does not clear itself. For example, a chart that refuses the host's values fails each run in the same way. The host then keeps its old release | | `HephaestusDeployReconcileStale` | `hephaestus_deploy_reconcile_timestamp_seconds` | The reconciler writes the timestamp when a run ends, and one apply may take up to the unit's 30-minute start timeout. No finished run for 40 minutes means that the timer or the unit stopped | | `HephaestusDeployMetricsMissing` | No `hephaestus_deploy_reconcile_timestamp_seconds` | The textfile collector, the metrics file or the reconciler is gone, which no other rule of the group shows | | `HephaestusDeployToolingPending` | `hephaestus_deploy_tooling_pending` | The host runs tooling that no apply recorded. The next promotion of the release that the host runs ends it. An hour leaves time for that promotion | | `HephaestusCertificateExpiring` | `certmanager_certificate_expiration_timestamp_seconds` | cert-manager renews a certificate when one third of its lifetime remains: 15 days of a 45-day certificate. For a certificate with a lifetime of less than 21 days, lower the threshold. cert-manager reports 0 for a certificate that it did not issue yet. Therefore, the alert also fires when a first issuance takes longer than its wait | The WAL archive rule reads the server's [WAL archive metrics](#wal-archive-metrics). It also applies to an external PostgreSQL that archives its WAL. No rule reads the age of the last archived segment. PostgreSQL switches a segment only after WAL activity, so an idle database archives nothing, and an age rule would fire falsely. An archive command that hangs without a failure stops the next backup, because pgBackRest waits for the backup's WAL in the archive. `HephaestusBackupStale` and `HephaestusPostgresVolumeFilling` then fire. To adapt the rules: 1. Adjust windows, thresholds, and routing to your traffic and recovery objectives. 2. If you change the monitor interval, increase the poll-age threshold. 3. If you change `postgresql.backup.schedule`, change the age of `HephaestusBackupStale`. 4. Enable stream rules only where you expect the webhook stream monitor. 5. To detect an individually missing stream, add an `absent_over_time` rule for each expected `stream` and receiver target. The missing-series rule detects loss of the whole monitor. Counters created on their first event can be absent before that event. An `increase` rule needs at least two samples. It cannot recover an event that occurred before its first sample. Scraping does not replace warning and error logs. Poison counts mean the application stopped retrying a delivery. They are not a durable dead-letter queue. See [Webhook ingestion operations](/admin/webhook-ingestion-operations.md#what-to-alert-on). 6. Validate the config with: ``` promtool check config prometheus.yml ``` 7. Validate the rules and run their tests with: ``` promtool check rules hephaestus-alerts.yml promtool test rules hephaestus-alerts.test.yml ``` A changed threshold also needs a changed test. 8. Before you rely on the rules, test the notification route. Prometheus evaluates rules. Your Alertmanager or equivalent controls notification delivery, grouping, and silences. See the [Prometheus alerting guidance](https://prometheus.io/docs/practices/alerting/). ## Health probes Each probe group answers one question. The probes read only state inside the process. A slow or missing dependency therefore never restarts a Pod, and never takes every Pod out of rotation at once. | Group | Members | Question | | --------------------------------------------------- | -------------------------------------------------------------------------- | ----------------------- | | Liveness (`/actuator/health/liveness`, `/livez`) | `livenessState` | Does the process run? | | Readiness (`/actuator/health/readiness`, `/readyz`) | `readinessState`, and on a worker `practiceReview` and `kubernetesSandbox` | Can this Pod serve now? | NATS (`integrationConsumer`) and JetStream (`webhook`) are in neither group. They stay in `/actuator/health` and the metrics, which the alerts read. While the broker is slow, the receiver stays ready and answers deliveries with `503`. ## Database statement statistics The bundled PostgreSQL image loads [`pg_stat_statements`](https://www.postgresql.org/docs/current/pgstatstatements.html). It records the run count and time of each statement for all databases. To read the statistics, create the extension once in the `postgres` database. Then the application schema stays unchanged. From `/opt/hephaestus/docker/self-host`, run: ``` docker compose --env-file .env --env-file release-lock.env exec postgres \ psql -U root -d postgres -c 'CREATE EXTENSION IF NOT EXISTS pg_stat_statements' docker compose --env-file .env --env-file release-lock.env exec postgres \ psql -U root -d postgres -c 'SELECT calls, round(mean_exec_time) AS mean_ms, round(total_exec_time) AS total_ms, left(query, 120) AS query FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20' ``` For most statements, the statistics replace constants with placeholders. A clean restart keeps them, and `SELECT pg_stat_statements_reset()` clears them. ## Reading log records | Field | Meaning | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `@timestamp` | UTC event time, in ISO-8601 format | | `log.level`, `log.logger`, `message` | Severity, originating logger, and human-readable event description | | `service.name`, `service.version` | `Hephaestus` and the release that the deployment names in `APP_VERSION`, the same values that traces carry | | `service.environment` | `DEPLOYMENT_ENVIRONMENT` | | `service.node.name` | The container ID, the same value as the `X-Hephaestus-Replica` response header | | `runtime.role` | `server`, `worker` or `webhook`, or `all` for a local process that runs every role | | `context.event.name` | Stable name on structured application events. Do not parse the message to identify these events | | `context.traceId`, `context.spanId` | The active trace scope, when there is one | | `context.job.id`, `context.workspace.id` | Thread-scoped context, when available | | `context.installation_id` | The GitHub App installation of a workspace-scoped request, when it has one | | `context.integration.route`, `context.integration.delivery` | On `integration.message.failed`: provider and event (`github.push`), and the delivery count | | `context.*` | Every other event field and thread-scoped value | | `error.type`, `error.stack_trace` | Exception class and frames when a call site includes the exception. Not present on every failure | Spring Boot writes a dotted name such as `log.level` as nested JSON objects. Event fields and thread-scoped values sit in the `context` object, so they cannot replace a field of the record. When an event names a field that the thread-scoped context also holds, for example `job.id`, the record carries the event's value once. This document names them without the `context.` prefix below. Read records with a JSON tool, not by text position. Event-specific numbers and booleans remain JSON numbers and booleans. A JSON string can contain escaped newlines. Read logs as JSON, not as extra records from their rendered text. A stack trace is at most 8192 characters and at most 32 frames for each exception. Docker splits a log line at 16 KiB, and the pieces are not JSON, so one record must stay below that. The trace starts with the root cause, and each exception that wraps it follows on a `Wrapped by:` line. When a long chain of wrappers reaches the limit, the outermost frames are cut, not the root cause. `error.type` is still the class of the outermost exception. JSON escaping makes the written trace slightly longer than its limit. A message that a call site writes has no limit of its own. Each exception line carries a hash, so you can count repeated failures of one kind. A record names each exception in a stack trace, including causes and suppressed exceptions, by its class only. It contains no `error.message`. A database driver quotes rejected row values in its messages, an HTTP client the request URL, and a parser the input that it rejected. These boundaries have their own guard: * The PostgreSQL driver runs with `logServerErrorDetail=false`, so its messages omit the `DETAIL` line that quotes rejected row values. The primary message, the SQLSTATE and the constraint name remain. A primary message can still quote a value for some kinds of error. * Hibernate's own log of the driver message for a failed statement, the `org.hibernate.orm.jdbc.error` category, is off. The application still logs the exception class and frames, and conflicts still record the SQLSTATE. * Tomcat's request parser, `org.apache.coyote.http11.Http11Processor`, logs at `WARN` and above only. At `INFO` and `DEBUG` it writes the raw target of a request that it cannot parse, query string included. When the [database migration](/admin/runtime-roles.md#database-migrations) stops because the changelog fails validation or a changeset precondition fails, its `APPLICATION FAILED TO START` record gives the reason in its message. An example is a database from before v0.77.4 that needs the [baseline synchronization runbook](/admin/liquibase-baseline-runbook.md). A changeset precondition that cannot run, or a failed migration statement, quotes the driver message, so its record names the exception class only. The application writes an exception's class, not its message, into its own log messages. Sync, provider, webhook and NATS failures log the class with the operation, the status where known, and opaque identifiers. A failed NATS delivery logs its route and delivery count, and attaches the stack trace on the first delivery only. The subject, which names an owner and a repository, stays out of the record. Workspaces are identified by `workspace.id`. The workspace slug is not a log field. Run production at `INFO`. `DEBUG` on framework, client or driver packages can write request content, provider responses and container settings that no guard above covers. The startup event `runtime.roles.configured` records `runtime.server.enabled`, `runtime.worker.enabled`, and `runtime.webhook.enabled` as booleans. It is informational when any role is enabled and warns when all are disabled. A role configuration log is not a readiness probe. At the shared HTTP boundary, `integration.request.failed` identifies the request method, upstream host, and exception class without recording the full URL, headers, or exception message. `database.constraint.conflict` identifies the exception class and a valid five-character SQLSTATE (`db.response.status_code`) without recording SQL or rejected row values. Both remain warnings. The response still reports service unavailability or a conflict. GitLab webhook registration failures expose the workspace and HTTP status in `integration.webhook.registration.failed`, never the provider's response body. Outline request diagnostics use `integration.request.failed` with the API operation, host, exception type, and HTTP status when available. They stay at debug level and do not include raw transport/decoding messages. When a review sandbox fails, the log records the size of its transcript as `sandbox.transcript.withheld`, not its content. The transcript holds what the run was asked, what it read and the provider errors that it saw. No log level writes it. For a next-day investigation, use the worker's bounded debug session transcripts ([retention decision](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0041-compose-1x-kubernetes-2.md#update--2026-10-09-debug-transcripts)). The formatter masks secret field names (for example `token`, `authorization`, `PRIVATE-TOKEN`, `X-Gitlab-Token` and `privateKey`) and known credential formats. These formats are bearer and basic credentials, GitHub, GitLab and Slack tokens (including `xoxe-` and `xapp-`), JWTs, credentials in a URL, and signed or token query parameters. This masking is defense in depth, not redaction. It does not detect names, email addresses or private text that a call site writes into a message. Keep raw requests, provider responses, signed URLs, and private runtime text out of logs. Restrict log access and retention as part of the instance's data-protection policy. Identifiers such as `workspace.id`, `job.id`, a trace ID or a container ID are not anonymous. They connect records to a workspace and its people, so they are personal data under the same policy. ## Agent-job lifecycle Lifecycle records contain `event.name`, `job.id`, `workspace.id`, `job.type`, `job.phase`, `job.outcome`, and `duration.ms`. | `event.name` | Emitted when | `job.phase` | `job.outcome` | `duration.ms` | | -------------------- | ----------------------------- | ----------- | ----------------------------------------------- | ----------------------------- | | `agent.job.queued` | A new job transaction commits | `queue` | `queued` | `0` | | `agent.job.started` | A worker starts a claimed job | `queue` | `started` | Creation to start | | `agent.job.terminal` | A terminal transition commits | `total` | `completed`, `failed`, `timed_out`, `cancelled` | Creation to terminal emission | An attempt that fails before its sandbox starts logs `agent.job.preparation_failed` at `WARN`. It has no phase, outcome or duration. `job.preparation.stage` names the stage that failed. `job.preparation.kind` and `job.preparation.reason` name the kind and the reason of the failure. `exception.type` and `exception.cause.types` name the exception classes. `job.result` names what the job does next. ## Agent-job metrics | Prometheus name | Type | Labels | Meaning | | ---------------------------- | --------- | --------- | --------------------------------------------- | | `agent_job_total` | Counter | `outcome` | Committed terminal transitions | | `agent_job_duration_seconds` | Histogram | `phase` | Queue, execution-attempt, and total durations | `outcome` is one of `completed`, `failed`, `timed_out`, or `cancelled`. `phase` is one of `queue`, `execution`, or `total`. Identifiers, model and provider names, worker IDs, and exception text are not labels. See [Practice Review Operations](/admin/practice-review-operations.md) for queue health and the existing execution metric. ## Conversation turn metrics Each claim of a conversation turn by a worker is one attempt. A turn has a second attempt only after a lease expiry ([Runtime Roles](/admin/runtime-roles.md#conversation-turns)). | Prometheus name | Type | Labels | Meaning | | ------------------------------------------------------- | --------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `mentor_turn_started_total` | Counter | none | Attempts that a worker claimed | | `mentor_turn_completed_total` | Counter | `outcome` | Attempts that ended | | `mentor_turn_retried_total` | Counter | none | Turns that went back to the queue after a lease expiry | | `mentor_turn_queued` | Gauge | none | Turns that wait for a worker, sampled by the server every 30 seconds | | `mentor_turn_queue_wait_seconds` | Histogram | none | From a question to the start of its first attempt, also while an earlier question of its conversation runs | | `llm_proxy_upstream_stalled_total` | Counter | `apiProtocol` | conversation model calls that the worker's LLM proxy cut after the first-byte or the idle bound | | `resilience4j_circuitbreaker_state` | Gauge | `name`, `state` | The provider breaker of each connection, named `llm-connection--`. `state="open"` is 1 while it refuses calls | | `resilience4j_circuitbreaker_not_permitted_calls_total` | Counter | `name`, `kind` | Model calls that a provider breaker refused | | `mentor_turn_lease_reaper_failure_total` | Counter | none | Expired leases that the lease reaper could not take back on one pass | | `mentor_turn_duration_seconds` | Histogram | none | An attempt, from its claim to its end | | `mentor_turn_first_token_seconds` | Histogram | none | From the start of an attempt to the first part of Hephaestus's answer | | `mentor_turn_cost_usd` | Summary | none | The cost of one billed attempt in US dollars | | `mentor_host_ready` | Gauge | none | `1` while the conversation host of this worker has its command channel open | | `mentor_host_turns_active` | Gauge | none | Turns that run in the conversation host of this worker now | | `mentor_turn_claim_failure_total` | Counter | none | Claims and starts of a turn that failed with an error | | `mentor_slack_reply_total` | Counter | `state` | Replies to questions asked in Slack, by the state they were left in | | `mentor_live_streams` | Gauge | none | Live Hephaestus streams that the app holds open on this server | `outcome` is one of `success`, `stopped`, `failed`, or `lease_expired`. `state` is one of `delivered`, `failed`, or `uncertain`. `failed` includes a reply that Silent Mode kept from Slack. A turn with nothing to post, such as a turn that the developer stopped before Hephaestus wrote, is not counted. The Slack reconciler takes each unsettled reply once and counts it again in its new state. It reads the Slack history once for a reply that a server tried to post. A reply that it finds is `delivered`. A reply that it does not find is `failed`, and Hephaestus never posts it again. A reply whose history it cannot read stays `uncertain`. [Conversation runtime](/contributor/unified-pi-runtime.md#reconciler) gives its limits. Web and Slack turns both count. An attempt that ends before any part of an answer records no first-token time. `mentor_turn_cost_usd` has `_count`, `_sum` and `_max` series. An attempt records a cost only when it made a model call and its model has a price. Workers record most of these series. The server role records `lease_expired` outcomes, the two lease-reaper counters, the cost of an attempt that lost its lease, the Slack replies and the live streams. Thus, sum each series across all roles. `mentor_host_ready` is `0` until the first turn on a worker starts its host. It is also `0` while a host starts and after a host is lost. Thus, a `0` alone does not show a fault. The share of attempts that did not succeed shows problems that developers see. This query gives that share for the last hour: ``` sum(increase(mentor_turn_completed_total{outcome=~"failed|lease_expired"}[1h])) / sum(increase(mentor_turn_completed_total[1h])) ``` A high `lease_expired` share shows workers or conversation hosts that stop during turns. A high `failed` share with no lease expiries shows one of these causes: * Admission declines turns. * Turns reach their deadline. * Turns cannot start a host. * Turns get an error from the model. The worker logs each failed attempt at `WARN` as `conversation turn attempt failed: ; last failed model call: `, or `none`. The reason tells which of these happened, for example `its conversation host reported ERROR: model_error`. The conversation host reports a kind of failure from a closed set, for example `model_error`, `context_overflow` or `upstream_stalled`. It does not report the error text of the model provider, a tool or the worker, or the messages of the conversation. A worker that ends an attempt itself names the class of the failure, for example `starting it failed: IllegalStateException`. The worker logs each failed model call of a turn at `WARN` as `conversation model call failed`. The line has the turn id, the attempt, the kind of failure and the status that the provider or the proxy sent. It does not have the request or the response. It is the only `WARN` line for a failed model call of a conversation turn. A call that the end of its turn cancelled is not a failure. The turn row keeps the cause, so the cause stays after the worker stops: | Column of `mentor_turn` | Holds | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `failure_cause` | The reason of the log line | | `failed_call` | The kind of the last failed model call of the attempt. From the provider: an exception class, `upstream_status`, `upstream_stalled`, `upstream_paused` or a stream outcome such as `upstream_idle`. From the proxy: a refusal such as `refused_budget`, or `gateway_error`. | | `failed_call_status` | The HTTP status that the provider or the proxy sent for that call, if one was sent | | `llm_calls` | The model calls of the attempt that completed | The conversation host's end, the worker's own end and a lease that expires write the first three columns. A lease that expires writes no `failed_call`, because the server role, not the worker, ends the turn. The columns are empty for a turn that did not fail, and for a turn that admission declines. A turn that admission declines, for example because the budget is spent, is logged at `INFO` as `was declined at its start`. A turn whose second attempt loses its lease fails on the server role, which logs it at `INFO` as `Mentor turn lease expired`. Its `failure_cause` is `its lease expired on its last attempt`. An increase in `mentor_turn_claim_failure_total` shows a worker that claims turns but cannot start them. The worker fails each such turn and continues to claim. [Runtime roles](/admin/runtime-roles.md#the-conversation-host) tells when a worker replaces its conversation host, what it logs, and what happens to the turns of that host. ## Call pool metrics Each worker reports the [call pools](/admin/runtime-roles.md#call-pools) whose models it calls. A worker registers every series of a pool at start, so a rate starts from zero. The labels are `pool` (the pool's slug), `class` and `reason`, each from a closed set. `class` is `turn`, `composition`, `request`, `event`, `schedule`, or `backfill`. It identifies conversation, composition, or the origin of a review call. No series names a workspace, a model, a job or a turn. | Prometheus name | Type | Labels | Meaning | | --------------------------------------- | --------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `llm_pool_permits_active` | Gauge | `pool`, `class` | Permits that this worker holds. Sum over the workers for the pool's calls in flight | | `llm_pool_permit_wait_seconds` | Histogram | `pool`, `class`, `reason` | How long a queued call waited for its permit. `reason=capacity` includes concurrency, rate, and cooldown admission | | `llm_pool_permit_abandoned_total` | Counter | `pool`, `class`, `reason` | Calls whose attempt ended, or whose work deadline passed, while they waited | | `llm_pool_upstream_throttled_total` | Counter | `pool` | HTTP 429 answers from the pool's provider | | `llm_pool_cooldown_active` | Gauge | `pool` | 1 while this worker knows the pool to be paused after a 429. Take the maximum over the workers | | `llm_pool_self_fence_cancelled_total` | Counter | `pool`, `reason` | Calls that a worker cancelled because it no longer held their permit. `reason` is `no_renewal`, `lease_lost`, or `attempt_ended` | | `llm_pool_lease_renewal_failures_total` | Counter | none | Failed renewals of a worker's permits | Claim reservations do not add a second wait observation when the model call uses them. ## Privacy-job metrics Account erasure, export generation, export expiry and LLM usage retention each report one terminal outcome per run, published by the server role. | Prometheus name | Type | Labels | Meaning | | ----------------------------- | ------- | ---------------- | -------------------------------------------- | | `privacy_job_completed_total` | Counter | `job`, `outcome` | Terminal privacy-job executions | | `privacy_job_affected_total` | Counter | `job` | Rows or subjects the run removed or produced | `job` is one of `account_erasure`, `export_generation`, `export_retention`, or `llm_usage_retention`. `outcome` is `success`, `incomplete`, or `failure`. An empty backlog is a `success`. No increase in the success counter therefore also warrants an alert. `incomplete` means the run ended without error but exhausted its time budget while eligible work remained. The LLM usage retention sweep deletes in batches for at most five minutes. It reports `incomplete` when expired rows remain. A deployment that reports this each day never finishes deletion. Account, workspace, export, and source identifiers are not labels. The [personal-data map](/admin/dsms/personal-data-map.md) lists the erasure and retention paths these metrics cover. For export generation, `success` and the affected count are published only after the export commits. A failed attempt reports `failure` even if the database is unavailable and cannot save the failed status. Consult the server log when an export remains queued or processing after a reported failure. ## WAL archive metrics The server role reads `pg_stat_archiver` and `archive_mode` from the database every 30 seconds. A scrape does not query the database. | Prometheus name | Type | Meaning | | -------------------------------------------- | ------- | -------------------------------------------------------------------------------- | | `database_wal_archive_failing` | Gauge | `1` when a failed archive attempt came after the last archived segment, else `0` | | `database_wal_archive_sample_failures_total` | Counter | Reads of the archiver statistics that failed | The gauge makes no claim that the server did not check: * Before the first read, the gauge is `NaN`. * While `archive_mode` is `off`, the gauge is `NaN`. A database without archiving is not reported as healthy or as failing. * After a read that failed, the gauge is `NaN` until the next read, and the sample counter increases. The last value is not kept, because it could report a healthy archive that the server did not check. `HephaestusWalArchiveFailing` also fires on failed reads that continue, so a read outage does not resolve it. The bundled PostgreSQL archives its WAL only when `postgresql.backup.enabled` is `true`. The Compose stacks archive with the backup overlay. ## Correlating a request When tracing is enabled, HTTP responses carry an `X-Request-Id` header containing the request's W3C trace ID. It is generated by tracing or extracted from valid incoming trace context, not derived from user attributes. While a trace scope is active, JSON logs carry its IDs as `context.traceId` and `context.spanId`. With tracing disabled, the correlation filter passes through without inventing an ID. Access logs stay off at every layer. A queued review records its submission correlation, but an execution attempt starts its own trace. Join the request and subsequent lifecycle by `job.id`. Do not assume every log for a job shares the request's trace ID. Sandbox proxy calls propagate the execution trace. The [trace export contract](#exporting-opentelemetry-traces) describes the separate submission attribute. No collector or log aggregator is necessary. From the multi-file Compose stack: 1. Take the ID a user reports, or the `X-Request-Id` from a failed response. 2. Search the container logs: ``` docker compose logs --no-color | grep '"traceId":""' ``` That command finds records in the matching trace scope. To follow a queued job into later execution attempts: 3. Take its `job.id` from the queued event. 4. Search that identifier too. Scheduler work can have execution traces without an original HTTP response. A trace ID proves that correlation context was present, not that a collector received the trace. ## HTTP route metrics `http.server.requests` uses route templates, not individual workspace slugs, message IDs, or query strings. Its URI-tag budget is 320: enough for the supported route inventory and the framework's bounded unmatched-request and management tags. A new URI beyond the budget is refused rather than creating unbounded time series. A limit warning therefore means metrics are incomplete and needs investigation, not a logging-level change. Concrete IDs or URLs in `uri` tags indicate a tagging defect. 1. When the limit is reached, first inspect the actual `uri` tag values. 2. Fix a tagging defect before considering a larger budget. 3. Do not turn off the bound. ## Collecting logs Hephaestus sends no logs and no error reports to any service. The browser app reports nothing either. ### What the host keeps by default The supported Compose stacks use Docker's `json-file` driver with a size limit for each container: | Containers | Limit | | ------------------------------------------------------------------------------- | ---------------- | | `application-server`, `application-worker`, `webapp`, `postgres`, `volume-init` | 5 files of 50 MB | | `webhook-server`, `nats-server` | 3 files of 10 MB | Docker writes these files in its default blocking mode. An application thread waits for the write, so no line is lost while the disk accepts writes. When a container reaches its limit, Docker deletes its oldest file. A line that no collector read before that deletion is lost. Sandbox containers use the host's default Docker logging configuration, which Hephaestus does not change. ### Ship logs to Loki This is optional. [Grafana Alloy](https://grafana.com/docs/alloy/latest/) reads the host journal and sends the lines to a Loki that you operate. Grafana then queries that Loki. The repository ships an example only. It does not start Alloy, and it names no destination. 1. Upgrade to a release with this log format first. Lines written by an earlier release are not redacted after the fact, and nothing in this setup collects them. 2. Add [`docker/observability/compose.journald.yaml`](https://github.com/hephaestus-build/Hephaestus/blob/main/docker/observability/compose.journald.yaml) after the single-host stack's compose files. It needs Compose 2.24.4 or later. It moves the three Java runtime roles, `application-server`, `application-worker` and `webhook-server`, to the `journald` driver and tags their entries `hephaestus-runtime`. The tag is new, so journal entries written before the overlay never match it. The webapp keeps its local `json-file` log: nginx error lines can name a client address and a request with its query. PostgreSQL keeps `json-file`, because its error lines can quote rejected row values. Sandbox containers, NATS and the proxy are not tagged. 3. The multi-file Compose stack renders each stack as its own Compose project. Give each of the three Java services the same `logging` block in that stack's override. 4. Run Alloy with [`docker/observability/config.alloy`](https://github.com/hephaestus-build/Hephaestus/blob/main/docker/observability/config.alloy). Set `LOKI_PUSH_URL` to your Loki push endpoint and `DEPLOYMENT_ENVIRONMENT` to the instance's name. The label reads `unset` when the variable is empty. Give Alloy read access to the journal through the `systemd-journal` group. That group can read the whole host journal, including `sshd` and `sudo` entries. The `matches` filter limits what Alloy ships, not what it can read, so treat the Alloy host account as privileged. Do not give it the Docker socket. The socket gives full control of the Docker daemon, also when you mount it read-only. 5. Set Loki's retention to the period in your record of processing. The example reads only entries with the `hephaestus-runtime` tag. It sets two index labels, `service_name` and `deployment_environment`. Both come from a small fixed set. The trace ID goes to [structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/), so `| trace_id=""` finds a request without a label per request. Workspace, job, user and repository identifiers stay in the line. Read them with `| json`. Do not make them labels: each value becomes its own stream ([Loki label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/)). Each stage has a bound, and each bound can lose lines: | Stage | Bound | When it is reached | | -------------------------- | ---------------------------------------------- | --------------------------------------------- | | Docker to journald | 4 MiB buffer, non-blocking | Docker drops new lines. The app does not wait | | journald | The host's `RateLimitBurst` and `SystemMaxUse` | journald drops or deletes old entries | | Alloy start or long outage | `max_age = "12h"` | Older entries are not sent | | Alloy to Loki | 10 retries, at most 5 minutes apart | Alloy drops the batch | Hephaestus does not change `journald.conf`. Check its limits on the host before you rely on the journal as a buffer. ## Exporting OpenTelemetry traces Hephaestus uses Spring Boot's OpenTelemetry starter and the OTLP HTTP exporter. Export is disabled by default. Configure these settings on the runtime roles whose spans you want to collect: | Variable | Default | Purpose | | ------------------------------ | --------------------------------- | ------------------------------------------------------------------------------ | | `TRACING_OTLP_ENABLED` | `false` | Enable the OTLP trace exporter. | | `TRACING_OTLP_ENDPOINT` | `http://localhost:4318/v1/traces` | Full OTLP HTTP receiver URL, reachable from the application container. | | `TRACING_SAMPLING_PROBABILITY` | `0.0` | Fraction of new traces sampled. Use `1.0` for a dedicated evaluation instance. | The supported Compose stack forwards these settings to server, worker and webhook containers. Each span carries the running version as `service.version` and the deployment environment as `deployment.environment.name`. For authenticated collectors, configure Spring Boot's `management.opentelemetry.tracing.export.otlp.headers` through protected deployment configuration. Do not put collector credentials in a URL. OTLP log and metric export stay disabled: enabling traces does not start exporting application logs, private transcripts or a second metrics stream. Prometheus metrics are available through the private listener described in [Scrape metrics](#scrape-metrics). Each practice-review attempt starts a `practice_review.execute` span. Its trace ID can differ from the job's submission correlation ID, which is carried as the `hephaestus.job.submission_trace_id` attribute rather than as a parent span. Sandbox proxy requests propagate the execution trace's sampling decision. A `gen_ai.chat`, `gen_ai.decision`, `gen_ai.embeddings` or `gen_ai.rerank` span covers each authenticated model request, streamed or not. It records the model, the HTTP outcome and the token usage that the provider reported, as `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. A reranking call that a provider billed in search units records `hephaestus.usage.search_units` instead. Framework HTTP spans describe the transport. Prompts, tool results, request bodies, headers and tokens used for authentication are not span attributes. For streaming requests, `hephaestus.stream.outcome` records transport completion separately from the HTTP status. A client disconnect or stream failure marks the span as an error with a bounded `error.type`, even if HTTP 200 headers were already sent. No replacement response body is sent after the stream has been committed. A trace ID is not proof of export. Sampling, exporter errors, collector retention and abrupt process termination can all leave partial coverage. A trace describes when calls happened and how they ended. It is not a record of what a review read or said. Durable review records retain admitted results, their problems and citation checks, not raw inputs, model requests or session transcripts. The [payload retention](/admin/data-retention.md#postgresql) strips diagnostic job output. The worker can separately retain bounded debug session transcripts. The [retention decision](https://github.com/hephaestus-build/Hephaestus/blob/main/docs/decisions/0041-compose-1x-kubernetes-2.md#update--2026-10-09-debug-transcripts) defines their ownership checks, metadata fallback, erasure and expiry.