Observability contract

Every runtime role writes one JSON record per event to standard output. Spring Boot writes the records in its built-in Elastic Common Schema (ECS) format. job.id and workspace.id are fields, not metric labels. The application exports no logs. The container runtime keeps them on the host until an operator collects them (Collecting logs).

Scrape metrics

Each runtime role serves GET /actuator/prometheus without a user token on a separate management listener, port 9090 by default (MANAGEMENT_PORT outside the supported Compose stacks). The application listener refuses this metrics path, including requests with a user token. Port 8081 is the sandbox gateway, not a metrics endpoint. Liveness and readiness also remain available on the application listener at /livez and /readyz. Container probes use these application-listener paths.

The multi-file and single-host Compose stacks bind management to the container's own shared-network address through its network-qualified hostname. They expose 9090 only on that network. This excludes sandbox interfaces, including after a restart with other networks attached. Docker's network-scoped DNS resolution selects the management network without a fixed container IP. The stacks do not publish the management port to the host or configure a proxy route.

Docker expose is not a firewall. Any container on that network can read metrics. The management listener's default address is loopback (127.0.0.1).

1. Attach your own Prometheus to that network.
2. Do not add a ports mapping, Traefik router, or internet ingress for management.
3. Restrict network membership to trusted services.
4. For non-Compose deployments, restrict the management listener with private networks and firewall policy before startup.
5. For a remote scraper, set MANAGEMENT_SERVER_ADDRESS to a specific trusted private interface.
6. Do not use the application or sandbox gateway port for management.

See Spring Boot management listeners and Prometheus security.

No monitoring service or outbound metric exporter is enabled by this configuration. Operators who do not run Prometheus need no monitoring setup. Traces remain opt-in and separate from scraping.

Example prometheus.yml for the multi-file Compose stack:

scrape_configs:
  - job_name: hephaestus
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: [application-server:9090]
        labels:
          runtime_role: server
      - targets: [application-worker:9090]
        labels:
          runtime_role: worker
      - targets: [webhook-server:9090]
        labels:
          runtime_role: webhook
rule_files:
  - hephaestus-alerts.yml

For multiple replicas:

1. Discover and scrape all containers, not a load-balanced service address that can switch replicas between scrapes.
2. Keep each target's instance label distinct.
3. Apply increase or rate to each counter before you sum them.
4. Never add workspace.id or job.id as metric labels.

This handles resets per process. runtime_role above is an operator-supplied target label, not an application meter label. workspace.id and job.id remain log fields.

Example prometheus.yml for the Helm chart, with the release in the namespace hephaestus:

scrape_configs:
  - job_name: hephaestus
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /actuator/prometheus
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [hephaestus]
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_container_port_name
        regex: hephaestus;management
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
        target_label: runtime_role
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
rule_files:
  - hephaestus-alerts.yml

Each pod of the server, the webhook role and the worker is then one target. The app.kubernetes.io/component label of the pod gives runtime_role. Set networkPolicy.metrics to the namespace and the pod labels of this Prometheus (Network policy). With the Prometheus Operator, a PodMonitor must give the same labels. Set job to hephaestus with a relabeling.

Alert rules

The release checkout contains the alert rules in deploy/observability/hephaestus-alerts.yml. Load that file as it is, from the release that you run: https://github.com/hephaestus-build/Hephaestus/blob/v<version>/deploy/observability/hephaestus-alerts.yml. With the Prometheus Operator, put the groups list of the file into the spec.groups of a PrometheusRule.

The chart does not render a PrometheusRule. The rules of the group hephaestus read the job and runtime_role labels that your scrape configuration sets, not labels that the chart controls. Helm reads files only from the chart directory, so a template would need a second copy of the tested file. The plain file has one home, and it works with any Prometheus. Alert response gives the response to each alert.

The rule file is the home of each threshold, window, and summary. This section gives the reason for each threshold. hephaestus-alerts.test.yml beside the rules contains a promtool unit test for each alert. Each test gives series that fire the alert and series that do not. CI runs these tests for each change to the rules.

Each alert has one of two severity labels:

• critical: data is at risk now. Send these alerts to a person at once. The critical alerts are a full PostgreSQL volume, a failing WAL archive, a stale backup, and no ready webhook receiver.
• warning: a fault needs work during working hours. All other alerts are warnings.

The rules read the exported Prometheus names, not the dotted Micrometer names. The rules of the group hephaestus read the application's metrics. Budget admission and committed agent-job outcomes can occur in different roles, so their rules aggregate all scraped roles. They need these target labels from the scrape configuration:

• job="hephaestus" on each target of the runtime roles.
• runtime_role, with the value server, worker or webhook.
• A different instance on each target.

HephaestusMetricsUnavailable fires only for a target that Prometheus still discovers. When service discovery loses all targets of a role, no up series remains. To detect that, add a rule for each role that you run, for example absent(up{job="hephaestus",runtime_role="server"}).

The rules of the group hephaestus-kubernetes read cluster metrics. Each rule needs its exporter, scraped by the same Prometheus:

Exporter	Metrics	Alerts
kube-state-metrics	kube_cronjob_status_last_successful_time, kube_cronjob_created, kube_deployment_status_replicas_available, kube_deployment_status_replicas_unavailable, kube_deployment_spec_replicas, kube_deployment_labels	HephaestusBackupStale, HephaestusWebhookReceiverDown, HephaestusWebhookReceiverUnready
The kubelet's /metrics endpoint, with a storage driver that reports volume statistics	kubelet_volume_stats_available_bytes, kubelet_volume_stats_capacity_bytes	HephaestusPostgresVolumeFilling, HephaestusJetStreamVolumeFilling
node-exporter on each node	node_filesystem_avail_bytes, node_filesystem_size_bytes, node_filesystem_readonly	HephaestusNodeDiskFilling
cert-manager	certmanager_certificate_expiration_timestamp_seconds	HephaestusCertificateExpiring

The group hephaestus-pull-deploy reads the metrics of the pull-based deployment reconciler. node-exporter's textfile collector on each host that runs the reconciler exports them. Load this group only when Prometheus scrapes such a host: HephaestusDeployMetricsMissing fires when no host reports.

A rule whose exporter is not scraped has no series and never fires. Before you rely on a rule, query its metrics in Prometheus. The kube-state-metrics, kubelet and cert-manager rules have no series outside Kubernetes.

The webhook rules find the release's webhook Deployment, <release>-webhook, by the chart's labels app.kubernetes.io/name=hephaestus and app.kubernetes.io/component=webhook. kube-state-metrics exports these labels only when you allow them:

• Start kube-state-metrics with --metric-labels-allowlist=deployments=[app.kubernetes.io/name,app.kubernetes.io/component].
• With kube-prometheus-stack, set the value kube-state-metrics.metricLabelsAllowlist to ["deployments=[app.kubernetes.io/name,app.kubernetes.io/component]"].

Without these labels, the two webhook rules find no Deployment and never fire.

The other rules find the release's objects by the names that the chart gives them: data-<release>-postgresql-0, data-<release>-nats-0, and the CronJobs <release>-postgresql-backup-full and <release>-postgresql-backup-diff. Another application in the cluster can use the same names, for example another PostgreSQL chart. To limit these rules to the release, add a matcher for its namespace, for example namespace="hephaestus". Do the same for HephaestusCertificateExpiring, which covers every certificate that cert-manager manages in the cluster. The chart shortens a name above 52 characters, and then the name loses its suffix. For a release name longer than 34 characters, change the name patterns of these rules.

Alert	Signal	Why this threshold
HephaestusMetricsUnavailable	up	Several failed scrapes in a row, not one missed scrape
HephaestusConsumerPoison	integration_consumer_poison_total	Each poison event is a delivery that the application stopped retrying
HephaestusStreamPollStale	webhook_stream_poll_age_seconds	Three times the default 60-second monitor interval. NaN means a stream was never polled successfully, not an age of zero
HephaestusStreamPollMissing	No webhook_stream_poll_age_seconds from a scraped receiver	The stream monitor is gone, which a failed scrape does not show
HephaestusStreamBacklog	webhook_stream_consumer_pending: events that the broker did not deliver to a consumer	Under steady load, a few events wait between two fetches. A backlog above that level at each evaluation for 15 minutes shows consumers that fall behind or do not fetch. At its bound, the stream removes its oldest message, also when a consumer still needs it. The gauge counts an event once for each consumer that it matches
HephaestusWebhookPublishFailing	webhook_publish_total{outcome="failure"}	GitHub redelivery covers a short stall of the broker, not a long one
HephaestusWebhookRedeliveryNotRunning	github_webhook_redelivery_total	Redelivery requests that GitHub refused or that could not read the App's deliveries. One refusal can be a race. Three in 30 minutes are not
HephaestusWebhookRedeliveriesRequested	github_webhook_redelivery_total{outcome="requested"} divided by webhook_publish_total{stream="github",outcome="success"}, over one hour	The server asks for a redelivery when GitHub recorded no successful delivery of an event. This includes a delivery that the receiver never saw, for example at an ingress timeout or while the receiver restarted. One event can be asked for up to five times. A ratio scales with traffic: more than 1 request for each 20 received deliveries is more than occasional network faults. A minimum of three requests keeps a single redelivery in a quiet hour quiet. A client error from the receiver, such as from a wrong webhook secret, gives no request and no count. The alert stays up for up to one hour after the requests stop. Only GitHub reports its delivery outcomes: GitLab, Slack and Outline have no such metric
HephaestusWebhookEventsUnrecovered	github_webhook_unrecovered	Each counted event failed all six delivery attempts
HephaestusBudgetExhausted	llm_budget_exhausted_total	A workspace crossed the instance-funded monthly cap
HephaestusBudgetBlocked	llm_budget_blocked_total	Admission refused work at a cap. A single refusal is expected at a cap. Refusals that continue for 15 minutes need a decision on the budget
HephaestusAgentJobsFailing	agent_job_total with failed or timed_out	All terminal failures, whatever their cause. Three in one hour point to the platform also on a small instance that runs a few reviews each day. Cancellations do not count. The alert stays up for up to one hour after the last failure
HephaestusReviewSandboxFailing	agent_job_infra_retry_requeued_total{cause="sandbox"}	The sandbox could not start, initialize or finish, and the attempt runs again. Three in one hour point to the cluster: the sandbox namespace, its quota, the agent image or the nodes. The alert stays up for up to one hour after the last failure
HephaestusReviewUnreachable	agent_job_infra_retry_requeued_total with cause server_unreachable or provider_unreachable	The sandbox reached neither the server's gateway nor the model provider. The cause label tells which. Three in one hour point to the network policy or to the provider. The alert stays up for up to one hour after the last failure
HephaestusReviewPreparationFailing	agent_job_infra_retry_requeued_total{cause="preparation"}	An attempt failed before its sandbox started, for example because the reviewed source was not captured yet. A few retries are normal after a push. Retries that continue point to provider sync or the database
HephaestusReviewWorkerLost	agent_job_orphan_failed_total	Each count is a review that failed because a worker stopped during each of its attempts. Workers that stop that often are evicted, out of memory, or on lost nodes. The alert stays up for one hour after the count
HephaestusAgentQueueStalled	agent_queue_oldest_age_seconds	A practice review usually runs for 10 to 16 minutes. A wait of several review durations shows that no worker claims work
HephaestusSandboxIsolationFailed	sandbox_isolation_passed of a worker	A later probe failed, and the worker starts no sandbox until a probe passes. The wait covers the first probe at the start
HephaestusWalArchiveFailing	database_wal_archive_failing, and database_wal_archive_sample_failures_total	PostgreSQL retries a failed segment about once each minute, so a short fault of the bucket clears itself. A longer failure stops point-in-time recovery after the last archived segment. A server that cannot read the archiver for the same time cannot show that the archive works, so the alert also fires then
HephaestusWebhookReceiverDown	kube_deployment_status_replicas_available of the webhook Deployment, while its desired replicas are above 0	No available receiver means that every provider delivery fails. The wait covers the start of a replacement pod. A Deployment scaled to 0 for maintenance does not fire. A receiver that is unavailable for a shorter time than the wait, for example for 16 seconds to 2 minutes, does not fire. HephaestusWebhookRedeliveriesRequested can then fire for the GitHub deliveries that it lost, when they reach its threshold. Deliveries of other providers have no such alert
HephaestusWebhookReceiverUnready	kube_deployment_status_replicas_unavailable of the webhook Deployment, while another replica is available	The other replicas serve, so this is a warning. The wait is longer than the 10 minutes that a new pod waits for the schema
HephaestusBackupStale	The newest successful full or differential backup of a release, or the creation time of CronJobs with no successful backup	A differential backup runs each day. The hours above one day let a full backup run
HephaestusPostgresVolumeFilling	Free space of the PostgreSQL volume	A failed WAL archive fills the volume. The warning gives time to repair the archive. The critical level comes shortly before PostgreSQL stops. The warning covers only the band above the critical level, so the two never fire together
HephaestusJetStreamVolumeFilling	Free space of the JetStream volume	The volume is max_file plus 25 %, so a broker at its storage budget leaves 20 % free. Below the threshold, something outside the budget fills the volume. A full file system wedges the broker
HephaestusNodeDiskFilling	Free space of a writable node file system, not the volumes that the kubelet mounts for pods	The kubelet's image garbage collection starts at 85 % use and frees space to 80 %. When the space stays low, eviction comes next. The volume rules cover pod volumes
HephaestusDeployReconcileFailing	hephaestus_deploy_reconcile_success	The timer runs the reconciler each minute. A run that fails for 15 minutes does not clear itself. For example, a chart that refuses the host's values fails each run in the same way. The host then keeps its old release
HephaestusDeployReconcileStale	hephaestus_deploy_reconcile_timestamp_seconds	The reconciler writes the timestamp when a run ends, and one apply may take up to the unit's 30-minute start timeout. No finished run for 40 minutes means that the timer or the unit stopped
HephaestusDeployMetricsMissing	No hephaestus_deploy_reconcile_timestamp_seconds	The textfile collector, the metrics file or the reconciler is gone, which no other rule of the group shows
HephaestusDeployToolingPending	hephaestus_deploy_tooling_pending	The host runs tooling that no apply recorded. The next promotion of the release that the host runs ends it. An hour leaves time for that promotion
HephaestusCertificateExpiring	certmanager_certificate_expiration_timestamp_seconds	cert-manager renews a certificate when one third of its lifetime remains: 15 days of a 45-day certificate. For a certificate with a lifetime of less than 21 days, lower the threshold. cert-manager reports 0 for a certificate that it did not issue yet. Therefore, the alert also fires when a first issuance takes longer than its wait

The WAL archive rule reads the server's WAL archive metrics. It also applies to an external PostgreSQL that archives its WAL. No rule reads the age of the last archived segment. PostgreSQL switches a segment only after WAL activity, so an idle database archives nothing, and an age rule would fire falsely. An archive command that hangs without a failure stops the next backup, because pgBackRest waits for the backup's WAL in the archive. HephaestusBackupStale and HephaestusPostgresVolumeFilling then fire.

To adapt the rules:

1. Adjust windows, thresholds, and routing to your traffic and recovery objectives.

2. If you change the monitor interval, increase the poll-age threshold.

3. If you change postgresql.backup.schedule, change the age of HephaestusBackupStale.

4. Enable stream rules only where you expect the webhook stream monitor.

5. To detect an individually missing stream, add an absent_over_time rule for each expected stream and receiver target.

   The missing-series rule detects loss of the whole monitor. Counters created on their first event can be absent before that event. An increase rule needs at least two samples. It cannot recover an event that occurred before its first sample. Scraping does not replace warning and error logs.

   Poison counts mean the application stopped retrying a delivery. They are not a durable dead-letter queue. See Webhook ingestion operations.

6. Validate the config with:

   promtool check config prometheus.yml

7. Validate the rules and run their tests with:

   promtool check rules hephaestus-alerts.yml
   promtool test rules hephaestus-alerts.test.yml

   A changed threshold also needs a changed test.

8. Before you rely on the rules, test the notification route.

Prometheus evaluates rules. Your Alertmanager or equivalent controls notification delivery, grouping, and silences. See the Prometheus alerting guidance.

Health probes

Each probe group answers one question. The probes read only state inside the process. A slow or missing dependency therefore never restarts a Pod, and never takes every Pod out of rotation at once.

Group	Members	Question
Liveness (/actuator/health/liveness, /livez)	livenessState	Does the process run?
Readiness (/actuator/health/readiness, /readyz)	readinessState, and on a worker practiceReview and kubernetesSandbox	Can this Pod serve now?

NATS (integrationConsumer) and JetStream (webhook) are in neither group. They stay in /actuator/health and the metrics, which the alerts read. While the broker is slow, the receiver stays ready and answers deliveries with 503.

Database statement statistics

The bundled PostgreSQL image loads pg_stat_statements. It records the run count and time of each statement for all databases. To read the statistics, create the extension once in the postgres database. Then the application schema stays unchanged. From /opt/hephaestus/docker/self-host, run:

docker compose --env-file .env --env-file release-lock.env exec postgres \
  psql -U root -d postgres -c 'CREATE EXTENSION IF NOT EXISTS pg_stat_statements'
docker compose --env-file .env --env-file release-lock.env exec postgres \
  psql -U root -d postgres -c 'SELECT calls, round(mean_exec_time) AS mean_ms, round(total_exec_time) AS total_ms, left(query, 120) AS query FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20'

For most statements, the statistics replace constants with placeholders. A clean restart keeps them, and SELECT pg_stat_statements_reset() clears them.

Reading log records

Field	Meaning
@timestamp	UTC event time, in ISO-8601 format
log.level, log.logger, message	Severity, originating logger, and human-readable event description
service.name, service.version	Hephaestus and the release that the deployment names in APP_VERSION, the same values that traces carry
service.environment	DEPLOYMENT_ENVIRONMENT
service.node.name	The container ID, the same value as the X-Hephaestus-Replica response header
runtime.role	server, worker or webhook, or all for a local process that runs every role
context.event.name	Stable name on structured application events. Do not parse the message to identify these events
context.traceId, context.spanId	The active trace scope, when there is one
context.job.id, context.workspace.id	Thread-scoped context, when available
context.installation_id	The GitHub App installation of a workspace-scoped request, when it has one
context.integration.route, context.integration.delivery	On integration.message.failed: provider and event (github.push), and the delivery count
context.*	Every other event field and thread-scoped value
error.type, error.stack_trace	Exception class and frames when a call site includes the exception. Not present on every failure

Spring Boot writes a dotted name such as log.level as nested JSON objects. Event fields and thread-scoped values sit in the context object, so they cannot replace a field of the record. When an event names a field that the thread-scoped context also holds, for example job.id, the record carries the event's value once. This document names them without the context. prefix below. Read records with a JSON tool, not by text position. Event-specific numbers and booleans remain JSON numbers and booleans.

A JSON string can contain escaped newlines. Read logs as JSON, not as extra records from their rendered text.

A stack trace is at most 8192 characters and at most 32 frames for each exception. Docker splits a log line at 16 KiB, and the pieces are not JSON, so one record must stay below that. The trace starts with the root cause, and each exception that wraps it follows on a Wrapped by: line. When a long chain of wrappers reaches the limit, the outermost frames are cut, not the root cause. error.type is still the class of the outermost exception.

JSON escaping makes the written trace slightly longer than its limit. A message that a call site writes has no limit of its own. Each exception line carries a hash, so you can count repeated failures of one kind.

A record names each exception in a stack trace, including causes and suppressed exceptions, by its class only. It contains no error.message. A database driver quotes rejected row values in its messages, an HTTP client the request URL, and a parser the input that it rejected. These boundaries have their own guard:

• The PostgreSQL driver runs with logServerErrorDetail=false, so its messages omit the DETAIL line that quotes rejected row values. The primary message, the SQLSTATE and the constraint name remain. A primary message can still quote a value for some kinds of error.
• Hibernate's own log of the driver message for a failed statement, the org.hibernate.orm.jdbc.error category, is off. The application still logs the exception class and frames, and conflicts still record the SQLSTATE.
• Tomcat's request parser, org.apache.coyote.http11.Http11Processor, logs at WARN and above only. At INFO and DEBUG it writes the raw target of a request that it cannot parse, query string included.

When the database migration stops because the changelog fails validation or a changeset precondition fails, its APPLICATION FAILED TO START record gives the reason in its message. An example is a database from before v0.77.4 that needs the baseline synchronization runbook. A changeset precondition that cannot run, or a failed migration statement, quotes the driver message, so its record names the exception class only.

The application writes an exception's class, not its message, into its own log messages. Sync, provider, webhook and NATS failures log the class with the operation, the status where known, and opaque identifiers. A failed NATS delivery logs its route and delivery count, and attaches the stack trace on the first delivery only. The subject, which names an owner and a repository, stays out of the record. Workspaces are identified by workspace.id. The workspace slug is not a log field.

Run production at INFO. DEBUG on framework, client or driver packages can write request content, provider responses and container settings that no guard above covers.

The startup event runtime.roles.configured records runtime.server.enabled, runtime.worker.enabled, and runtime.webhook.enabled as booleans. It is informational when any role is enabled and warns when all are disabled. A role configuration log is not a readiness probe.

At the shared HTTP boundary, integration.request.failed identifies the request method, upstream host, and exception class without recording the full URL, headers, or exception message. database.constraint.conflict identifies the exception class and a valid five-character SQLSTATE (db.response.status_code) without recording SQL or rejected row values. Both remain warnings. The response still reports service unavailability or a conflict.

GitLab webhook registration failures expose the workspace and HTTP status in integration.webhook.registration.failed, never the provider's response body. Outline request diagnostics use integration.request.failed with the API operation, host, exception type, and HTTP status when available. They stay at debug level and do not include raw transport/decoding messages.

When a review sandbox fails, the log records the size of its transcript as sandbox.transcript.withheld, not its content. The transcript holds what the run was asked, what it read and the provider errors that it saw. No log level writes it. For a next-day investigation, use the worker's bounded debug session transcripts (retention decision).

The formatter masks secret field names (for example token, authorization, PRIVATE-TOKEN, X-Gitlab-Token and privateKey) and known credential formats. These formats are bearer and basic credentials, GitHub, GitLab and Slack tokens (including xoxe- and xapp-), JWTs, credentials in a URL, and signed or token query parameters. This masking is defense in depth, not redaction. It does not detect names, email addresses or private text that a call site writes into a message.

Keep raw requests, provider responses, signed URLs, and private runtime text out of logs. Restrict log access and retention as part of the instance's data-protection policy. Identifiers such as workspace.id, job.id, a trace ID or a container ID are not anonymous. They connect records to a workspace and its people, so they are personal data under the same policy.

Agent-job lifecycle

Lifecycle records contain event.name, job.id, workspace.id, job.type, job.phase, job.outcome, and duration.ms.

event.name	Emitted when	job.phase	job.outcome	duration.ms
agent.job.queued	A new job transaction commits	queue	queued	0
agent.job.started	A worker starts a claimed job	queue	started	Creation to start
agent.job.terminal	A terminal transition commits	total	completed, failed, timed_out, cancelled	Creation to terminal emission

An attempt that fails before its sandbox starts logs agent.job.preparation_failed at WARN. It has no phase, outcome or duration. job.preparation.stage names the stage that failed. job.preparation.kind and job.preparation.reason name the kind and the reason of the failure. exception.type and exception.cause.types name the exception classes. job.result names what the job does next.

Agent-job metrics

Prometheus name	Type	Labels	Meaning
agent_job_total	Counter	outcome	Committed terminal transitions
agent_job_duration_seconds	Histogram	phase	Queue, execution-attempt, and total durations

outcome is one of completed, failed, timed_out, or cancelled. phase is one of queue, execution, or total. Identifiers, model and provider names, worker IDs, and exception text are not labels. See Practice Review Operations for queue health and the existing execution metric.

Conversation turn metrics

Each claim of a conversation turn by a worker is one attempt. A turn has a second attempt only after a lease expiry (Runtime Roles).

Prometheus name	Type	Labels	Meaning
mentor_turn_started_total	Counter	none	Attempts that a worker claimed
mentor_turn_completed_total	Counter	outcome	Attempts that ended
mentor_turn_retried_total	Counter	none	Turns that went back to the queue after a lease expiry
mentor_turn_queued	Gauge	none	Turns that wait for a worker, sampled by the server every 30 seconds
mentor_turn_queue_wait_seconds	Histogram	none	From a question to the start of its first attempt, also while an earlier question of its conversation runs
llm_proxy_upstream_stalled_total	Counter	apiProtocol	conversation model calls that the worker's LLM proxy cut after the first-byte or the idle bound
resilience4j_circuitbreaker_state	Gauge	name, state	The provider breaker of each connection, named llm-connection-<scope>-<connection id>. state="open" is 1 while it refuses calls
resilience4j_circuitbreaker_not_permitted_calls_total	Counter	name, kind	Model calls that a provider breaker refused
mentor_turn_lease_reaper_failure_total	Counter	none	Expired leases that the lease reaper could not take back on one pass
mentor_turn_duration_seconds	Histogram	none	An attempt, from its claim to its end
mentor_turn_first_token_seconds	Histogram	none	From the start of an attempt to the first part of Hephaestus's answer
mentor_turn_cost_usd	Summary	none	The cost of one billed attempt in US dollars
mentor_host_ready	Gauge	none	1 while the conversation host of this worker has its command channel open
mentor_host_turns_active	Gauge	none	Turns that run in the conversation host of this worker now
mentor_turn_claim_failure_total	Counter	none	Claims and starts of a turn that failed with an error
mentor_slack_reply_total	Counter	state	Replies to questions asked in Slack, by the state they were left in
mentor_live_streams	Gauge	none	Live Hephaestus streams that the app holds open on this server

outcome is one of success, stopped, failed, or lease_expired. state is one of delivered, failed, or uncertain. failed includes a reply that Silent Mode kept from Slack. A turn with nothing to post, such as a turn that the developer stopped before Hephaestus wrote, is not counted.

The Slack reconciler takes each unsettled reply once and counts it again in its new state. It reads the Slack history once for a reply that a server tried to post. A reply that it finds is delivered. A reply that it does not find is failed, and Hephaestus never posts it again. A reply whose history it cannot read stays uncertain. Conversation runtime gives its limits.

Web and Slack turns both count. An attempt that ends before any part of an answer records no first-token time.

mentor_turn_cost_usd has _count, _sum and _max series. An attempt records a cost only when it made a model call and its model has a price.

Workers record most of these series. The server role records lease_expired outcomes, the two lease-reaper counters, the cost of an attempt that lost its lease, the Slack replies and the live streams. Thus, sum each series across all roles.

mentor_host_ready is 0 until the first turn on a worker starts its host. It is also 0 while a host starts and after a host is lost. Thus, a 0 alone does not show a fault.

The share of attempts that did not succeed shows problems that developers see. This query gives that share for the last hour:

sum(increase(mentor_turn_completed_total{outcome=~"failed|lease_expired"}[1h]))
  / sum(increase(mentor_turn_completed_total[1h]))

A high lease_expired share shows workers or conversation hosts that stop during turns. A high failed share with no lease expiries shows one of these causes:

• Admission declines turns.
• Turns reach their deadline.
• Turns cannot start a host.
• Turns get an error from the model.

The worker logs each failed attempt at WARN as conversation turn <id> attempt <n> failed: <reason>; last failed model call: <kind> <status>, or none. The reason tells which of these happened, for example its conversation host reported ERROR: model_error. The conversation host reports a kind of failure from a closed set, for example model_error, context_overflow or upstream_stalled. It does not report the error text of the model provider, a tool or the worker, or the messages of the conversation. A worker that ends an attempt itself names the class of the failure, for example starting it failed: IllegalStateException.

The worker logs each failed model call of a turn at WARN as conversation model call failed. The line has the turn id, the attempt, the kind of failure and the status that the provider or the proxy sent. It does not have the request or the response. It is the only WARN line for a failed model call of a conversation turn. A call that the end of its turn cancelled is not a failure.

The turn row keeps the cause, so the cause stays after the worker stops:

Column of mentor_turn	Holds
failure_cause	The reason of the log line
failed_call	The kind of the last failed model call of the attempt. From the provider: an exception class, upstream_status, upstream_stalled, upstream_paused or a stream outcome such as upstream_idle. From the proxy: a refusal such as refused_budget, or gateway_error.
failed_call_status	The HTTP status that the provider or the proxy sent for that call, if one was sent
llm_calls	The model calls of the attempt that completed

The conversation host's end, the worker's own end and a lease that expires write the first three columns. A lease that expires writes no failed_call, because the server role, not the worker, ends the turn. The columns are empty for a turn that did not fail, and for a turn that admission declines.

A turn that admission declines, for example because the budget is spent, is logged at INFO as was declined at its start. A turn whose second attempt loses its lease fails on the server role, which logs it at INFO as Mentor turn lease expired. Its failure_cause is its lease expired on its last attempt.

An increase in mentor_turn_claim_failure_total shows a worker that claims turns but cannot start them. The worker fails each such turn and continues to claim. Runtime roles tells when a worker replaces its conversation host, what it logs, and what happens to the turns of that host.

Call pool metrics

Each worker reports the call pools whose models it calls. A worker registers every series of a pool at start, so a rate starts from zero. The labels are pool (the pool's slug), class and reason, each from a closed set. class is turn, composition, request, event, schedule, or backfill. It identifies conversation, composition, or the origin of a review call. No series names a workspace, a model, a job or a turn.

Prometheus name	Type	Labels	Meaning
llm_pool_permits_active	Gauge	pool, class	Permits that this worker holds. Sum over the workers for the pool's calls in flight
llm_pool_permit_wait_seconds	Histogram	pool, class, reason	How long a queued call waited for its permit. reason=capacity includes concurrency, rate, and cooldown admission
llm_pool_permit_abandoned_total	Counter	pool, class, reason	Calls whose attempt ended, or whose work deadline passed, while they waited
llm_pool_upstream_throttled_total	Counter	pool	HTTP 429 answers from the pool's provider
llm_pool_cooldown_active	Gauge	pool	1 while this worker knows the pool to be paused after a 429. Take the maximum over the workers
llm_pool_self_fence_cancelled_total	Counter	pool, reason	Calls that a worker cancelled because it no longer held their permit. reason is no_renewal, lease_lost, or attempt_ended
llm_pool_lease_renewal_failures_total	Counter	none	Failed renewals of a worker's permits

Claim reservations do not add a second wait observation when the model call uses them.

Privacy-job metrics

Account erasure, export generation, export expiry and LLM usage retention each report one terminal outcome per run, published by the server role.

Prometheus name	Type	Labels	Meaning
privacy_job_completed_total	Counter	job, outcome	Terminal privacy-job executions
privacy_job_affected_total	Counter	job	Rows or subjects the run removed or produced

job is one of account_erasure, export_generation, export_retention, or llm_usage_retention. outcome is success, incomplete, or failure. An empty backlog is a success. No increase in the success counter therefore also warrants an alert. incomplete means the run ended without error but exhausted its time budget while eligible work remained.

The LLM usage retention sweep deletes in batches for at most five minutes. It reports incomplete when expired rows remain. A deployment that reports this each day never finishes deletion. Account, workspace, export, and source identifiers are not labels. The personal-data map lists the erasure and retention paths these metrics cover.

For export generation, success and the affected count are published only after the export commits. A failed attempt reports failure even if the database is unavailable and cannot save the failed status. Consult the server log when an export remains queued or processing after a reported failure.

WAL archive metrics

The server role reads pg_stat_archiver and archive_mode from the database every 30 seconds. A scrape does not query the database.

Prometheus name	Type	Meaning
database_wal_archive_failing	Gauge	1 when a failed archive attempt came after the last archived segment, else 0
database_wal_archive_sample_failures_total	Counter	Reads of the archiver statistics that failed

The gauge makes no claim that the server did not check:

• Before the first read, the gauge is NaN.
• While archive_mode is off, the gauge is NaN. A database without archiving is not reported as healthy or as failing.
• After a read that failed, the gauge is NaN until the next read, and the sample counter increases. The last value is not kept, because it could report a healthy archive that the server did not check. HephaestusWalArchiveFailing also fires on failed reads that continue, so a read outage does not resolve it.

The bundled PostgreSQL archives its WAL only when postgresql.backup.enabled is true. The Compose stacks archive with the backup overlay.

Correlating a request

When tracing is enabled, HTTP responses carry an X-Request-Id header containing the request's W3C trace ID. It is generated by tracing or extracted from valid incoming trace context, not derived from user attributes. While a trace scope is active, JSON logs carry its IDs as context.traceId and context.spanId. With tracing disabled, the correlation filter passes through without inventing an ID. Access logs stay off at every layer.

A queued review records its submission correlation, but an execution attempt starts its own trace. Join the request and subsequent lifecycle by job.id. Do not assume every log for a job shares the request's trace ID. Sandbox proxy calls propagate the execution trace. The trace export contract describes the separate submission attribute.

No collector or log aggregator is necessary. From the multi-file Compose stack:

1. Take the ID a user reports, or the X-Request-Id from a failed response.

2. Search the container logs:

   docker compose logs --no-color | grep '"traceId":"<the-id>"'

   That command finds records in the matching trace scope. To follow a queued job into later execution attempts:

3. Take its job.id from the queued event.

4. Search that identifier too.

Scheduler work can have execution traces without an original HTTP response. A trace ID proves that correlation context was present, not that a collector received the trace.

HTTP route metrics

http.server.requests uses route templates, not individual workspace slugs, message IDs, or query strings. Its URI-tag budget is 320: enough for the supported route inventory and the framework's bounded unmatched-request and management tags. A new URI beyond the budget is refused rather than creating unbounded time series. A limit warning therefore means metrics are incomplete and needs investigation, not a logging-level change.

Concrete IDs or URLs in uri tags indicate a tagging defect.

1. When the limit is reached, first inspect the actual uri tag values.
2. Fix a tagging defect before considering a larger budget.
3. Do not turn off the bound.

Collecting logs

Hephaestus sends no logs and no error reports to any service. The browser app reports nothing either.

What the host keeps by default

The supported Compose stacks use Docker's json-file driver with a size limit for each container:

Containers	Limit
application-server, application-worker, webapp, postgres, volume-init	5 files of 50 MB
webhook-server, nats-server	3 files of 10 MB

Docker writes these files in its default blocking mode. An application thread waits for the write, so no line is lost while the disk accepts writes. When a container reaches its limit, Docker deletes its oldest file. A line that no collector read before that deletion is lost. Sandbox containers use the host's default Docker logging configuration, which Hephaestus does not change.

Ship logs to Loki

This is optional. Grafana Alloy reads the host journal and sends the lines to a Loki that you operate. Grafana then queries that Loki. The repository ships an example only. It does not start Alloy, and it names no destination.

1. Upgrade to a release with this log format first. Lines written by an earlier release are not redacted after the fact, and nothing in this setup collects them.
2. Add docker/observability/compose.journald.yaml after the single-host stack's compose files. It needs Compose 2.24.4 or later. It moves the three Java runtime roles, application-server, application-worker and webhook-server, to the journald driver and tags their entries hephaestus-runtime. The tag is new, so journal entries written before the overlay never match it. The webapp keeps its local json-file log: nginx error lines can name a client address and a request with its query. PostgreSQL keeps json-file, because its error lines can quote rejected row values. Sandbox containers, NATS and the proxy are not tagged.
3. The multi-file Compose stack renders each stack as its own Compose project. Give each of the three Java services the same logging block in that stack's override.
4. Run Alloy with docker/observability/config.alloy. Set LOKI_PUSH_URL to your Loki push endpoint and DEPLOYMENT_ENVIRONMENT to the instance's name. The label reads unset when the variable is empty. Give Alloy read access to the journal through the systemd-journal group. That group can read the whole host journal, including sshd and sudo entries. The matches filter limits what Alloy ships, not what it can read, so treat the Alloy host account as privileged. Do not give it the Docker socket. The socket gives full control of the Docker daemon, also when you mount it read-only.
5. Set Loki's retention to the period in your record of processing.

The example reads only entries with the hephaestus-runtime tag. It sets two index labels, service_name and deployment_environment. Both come from a small fixed set.

The trace ID goes to structured metadata, so | trace_id="<id>" finds a request without a label per request. Workspace, job, user and repository identifiers stay in the line. Read them with | json. Do not make them labels: each value becomes its own stream (Loki label best practices).

Each stage has a bound, and each bound can lose lines:

Stage	Bound	When it is reached
Docker to journald	4 MiB buffer, non-blocking	Docker drops new lines. The app does not wait
journald	The host's RateLimitBurst and SystemMaxUse	journald drops or deletes old entries
Alloy start or long outage	max_age = "12h"	Older entries are not sent
Alloy to Loki	10 retries, at most 5 minutes apart	Alloy drops the batch

Hephaestus does not change journald.conf. Check its limits on the host before you rely on the journal as a buffer.

Exporting OpenTelemetry traces

Hephaestus uses Spring Boot's OpenTelemetry starter and the OTLP HTTP exporter. Export is disabled by default. Configure these settings on the runtime roles whose spans you want to collect:

Variable	Default	Purpose
TRACING_OTLP_ENABLED	false	Enable the OTLP trace exporter.
TRACING_OTLP_ENDPOINT	http://localhost:4318/v1/traces	Full OTLP HTTP receiver URL, reachable from the application container.
TRACING_SAMPLING_PROBABILITY	0.0	Fraction of new traces sampled. Use 1.0 for a dedicated evaluation instance.

The supported Compose stack forwards these settings to server, worker and webhook containers. Each span carries the running version as service.version and the deployment environment as deployment.environment.name. For authenticated collectors, configure Spring Boot's management.opentelemetry.tracing.export.otlp.headers through protected deployment configuration. Do not put collector credentials in a URL. OTLP log and metric export stay disabled: enabling traces does not start exporting application logs, private transcripts or a second metrics stream. Prometheus metrics are available through the private listener described in Scrape metrics.

Each practice-review attempt starts a practice_review.execute span. Its trace ID can differ from the job's submission correlation ID, which is carried as the hephaestus.job.submission_trace_id attribute rather than as a parent span. Sandbox proxy requests propagate the execution trace's sampling decision. A gen_ai.chat, gen_ai.decision, gen_ai.embeddings or gen_ai.rerank span covers each authenticated model request, streamed or not. It records the model, the HTTP outcome and the token usage that the provider reported, as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. A reranking call that a provider billed in search units records hephaestus.usage.search_units instead.

Framework HTTP spans describe the transport. Prompts, tool results, request bodies, headers and tokens used for authentication are not span attributes.

For streaming requests, hephaestus.stream.outcome records transport completion separately from the HTTP status. A client disconnect or stream failure marks the span as an error with a bounded error.type, even if HTTP 200 headers were already sent. No replacement response body is sent after the stream has been committed.

A trace ID is not proof of export. Sampling, exporter errors, collector retention and abrupt process termination can all leave partial coverage. A trace describes when calls happened and how they ended. It is not a record of what a review read or said.

Durable review records retain admitted results, their problems and citation checks, not raw inputs, model requests or session transcripts. The payload retention strips diagnostic job output.

The worker can separately retain bounded debug session transcripts. The retention decision defines their ownership checks, metadata fallback, erasure and expiry.