Log records
Each runtime role writes one JSON record per event to standard output, in the ECS format of Spring Boot. This page lists the fields and the events. Collect logs ships them to a log store.
Fields
| Field | Meaning |
|---|---|
@timestamp | UTC event time, in ISO-8601 format |
log.level, log.logger, message | Severity, originating logger, and human-readable event description |
service.name, service.version | Hephaestus and the release that the deployment names in APP_VERSION, the same values that traces carry. The chart sets APP_VERSION from release.version on every container of the server image, the migration Job included |
service.environment | DEPLOYMENT_ENVIRONMENT |
service.node.name | The container ID, the same value as the X-Hephaestus-Replica response header |
runtime.role | server, worker or webhook, or all for a local process that runs every role |
context.event.name | Stable name on structured application events. Do not parse the message to identify these events |
context.traceId, context.spanId | The active trace scope, when there is one |
context.job.id, context.workspace.id | Thread-scoped context, when available |
context.run.id, context.run.epoch, context.subject.key | The agent run whose holder writes the record, the epoch of its claim, and its subject, such as review:<id>. Two records of one run with different epochs come from different claims |
context.sandbox.name | The agent sandbox that made a gateway request: its claim, Sandbox and Pod name. Each gateway request also names its subject and workspace. A request for one run also names run.id and run.epoch, after the gateway checks that the run is the sandbox's own |
context.installation_id | The GitHub App installation of a sync run, when it has one |
context.integration.route, context.integration.delivery | On integration.message.failed: provider and event (github.push), and the delivery count |
context.* | Every other event field and thread-scoped value |
error.type, error.stack_trace | Exception class and frames when a call site includes the exception. Not present on every failure |
Spring Boot writes a dotted name such as log.level as nested JSON objects.
Event fields and thread-scoped values sit in the context object, so they cannot replace a field of the record.
When an event names a field that the thread-scoped context also holds, for example job.id, the record carries the event's value once.
This page names them without the context. prefix below.
Read records with a JSON tool, not by text position.
Event-specific numbers and booleans remain JSON numbers and booleans.
A JSON string can contain escaped newlines. Read logs as JSON, not as extra records from their rendered text.
A stack trace is at most 8192 characters and at most 32 frames for each exception.
Docker splits a log line at 16 KiB, and the pieces are not JSON, so one record must stay below that.
The trace starts with the root cause, and each exception that wraps it follows on a Wrapped by: line.
When a long chain of wrappers reaches the limit, the outermost frames are cut, not the root cause.
error.type is still the class of the outermost exception.
JSON escaping makes the written trace slightly longer than its limit. A message that a call site writes has no limit of its own. Each exception line carries a hash, so you can count repeated failures of one kind.
A record names each exception in a stack trace, including causes and suppressed exceptions, by its class only.
It contains no error.message.
A database driver quotes rejected row values in its messages, an HTTP client the request URL, and a parser the input that it rejected.
These boundaries have their own guard:
- The PostgreSQL driver runs with
logServerErrorDetail=false, so its messages omit theDETAILline that quotes rejected row values. The primary message, the SQLSTATE and the constraint name remain. A primary message can still quote a value for some kinds of error. - Hibernate's own log of the driver message for a failed statement, the
org.hibernate.orm.jdbc.errorcategory, is off. The application still logs the exception class and frames, and conflicts still record the SQLSTATE. - Tomcat's request parser,
org.apache.coyote.http11.Http11Processor, logs atWARNand above only. AtINFOandDEBUGit writes the raw target of a request that it cannot parse, query string included.
When the database migration stops because the changelog fails validation or a changeset precondition fails, its APPLICATION FAILED TO START record gives the reason in its message.
An example is a database that does not meet the database prerequisites.
A changeset precondition that cannot run, or a failed migration statement, quotes the driver message, so its record names the exception class only.
The application writes an exception's class, not its message, into its own log messages.
Sync, provider, webhook and NATS failures log the class with the operation, the status where known, and opaque identifiers.
A failed NATS delivery logs its route and delivery count, and attaches the stack trace on the first delivery only.
The subject, which names an owner and a repository, stays out of the record.
Workspaces are identified by workspace.id. The workspace slug is not a log field.
Run production at INFO.
DEBUG on framework, client or driver packages can write request content, provider responses and container settings that no guard above covers.
The startup event runtime.roles.configured records runtime.server.enabled,
runtime.worker.enabled, and runtime.webhook.enabled as booleans. It is informational when any
role is enabled and warns when all are disabled. A role configuration log is not a readiness probe.
At the shared HTTP boundary, integration.request.failed identifies the request method, upstream
host, and exception class without recording the full URL, headers, or exception message.
database.constraint.conflict identifies the exception class and a valid five-character SQLSTATE
(db.response.status_code) without recording SQL or rejected row values. Both remain warnings.
The response still reports service unavailability or a conflict.
GitLab webhook registration failures expose the workspace and HTTP status in
integration.webhook.registration.failed, never the provider's response body. Outline request
diagnostics use integration.request.failed with the API operation, host, exception type, and HTTP
status when available. They stay at debug level and do not include raw transport/decoding messages.
When GitLab already refused a connection's credential, each initialization logs integration.credential.refused as a warning, with integration.kind and workspace.id.
Initialization still inspects the token, because that call can see the credential accepted again.
It does not discover projects until then.
The admin sees the same state as an attention problem on the connection.
A deletion sweep that cannot prove the provider's listing complete deletes nothing and logs integration.deletion_sweep.skipped, with integration.kind and reason.
The record is a warning, except for reason=featureDisabled.
That reason means that the repository switched its issues or its pull requests off, which is an expected state, so the record is informational.
It also does not give the reconciliation job a warning. The sweep's summary counts these repositories.
When the worker cannot read the quotas of the sandbox namespace, it logs sandbox.quota.unreadable as a warning, and reviews wait.
When it can read them again, it logs sandbox.quota.readable with duration.ms, the time that reviews waited.
A short gap also occurs when the watch of the namespace cache reconnects.
When a review sandbox fails, the log records the size of its transcript as sandbox.transcript.withheld, not its content.
The transcript holds what the run was asked, what it read and the provider errors that it saw.
No log level writes it.
For a next-day investigation, use the worker's bounded debug session transcripts (retention decision).
The formatter masks secret field names (for example token, authorization, PRIVATE-TOKEN, X-Gitlab-Token and privateKey) and known credential formats.
These formats are bearer and basic credentials, GitHub, GitLab and Slack tokens (including xoxe- and xapp-), JWTs, credentials in a URL, and signed or token query parameters.
This masking is defense in depth, not redaction.
It does not detect names, email addresses or private text that a call site writes into a message.
Keep raw requests, provider responses, signed URLs, and private runtime text out of logs.
Restrict log access and retention as part of the instance's data-protection policy.
Identifiers such as workspace.id, job.id, a trace ID or a container ID are not anonymous.
They connect records to a workspace and its people, so they are personal data under the same policy.
The same holds for metrics.
When tracing samples a request, the server attaches its trace ID and span ID to some histograms as exemplars, for example to http_server_requests_seconds.
It does this only when a scraper asks for the OpenMetrics format, and with management.tracing.sampling.probability above 0.
The chart's monitoring stack keeps no exemplars: VictoriaMetrics stores the sample and drops its exemplar.
A Prometheus with exemplar storage keeps them.
Then its retention and access follow the policy for trace IDs.
Agent-job lifecycle
Lifecycle records contain event.name, job.id, workspace.id, review.id, job.phase,
job.outcome, and duration.ms.
event.name | Emitted when | job.phase | job.outcome | duration.ms |
|---|---|---|---|---|
agent.job.queued | A new job transaction commits | queue | queued | 0 |
agent.job.started | A worker starts a claimed job | queue | started | Creation to start |
agent.job.terminal | A terminal transition commits | total | completed, failed, timed_out, cancelled | Creation to terminal emission |
An attempt that fails before its sandbox starts logs agent.job.preparation_failed at WARN.
It has no phase, outcome or duration.
job.preparation.stage names the stage that failed.
job.preparation.kind and job.preparation.reason name the kind and the reason of the failure.
exception.type and exception.cause.types name the exception classes.
job.result names what the job does next.
Review and conversation events
Each record has an event.name and workspace.id.
event.name | Emitted when | Fields |
|---|---|---|
practice_review.stage.ended | A stage of a review attempt ends | review.stage, review.stage.outcome, duration.ms |
practice_review.ended | A review reaches its terminal status | review.id, review.kind, review.status, review.reason |
feedback.copy.settled | Feedback reaches a destination's final state | feedback.id, review.id, feedback.destination, feedback.state, feedback.reason |
mentor.turn.ended | An attempt of a conversation turn ends | turn.id, turn.attempt, turn.outcome |
practice_review.stage.ended has no review.id. It takes workspace.id, job.id and the trace ID from the log context of its attempt, so join it by job.id.
Agent run and outbox events
A review attempt runs under an agent run.
Each record that the attempt writes carries run.id, run.epoch and subject.key in its thread-scoped context, next to job.id.
Join the records of one run by run.id, and the records of one claim by run.id and run.epoch.
These are fields, not metric labels.
event.name | Emitted when | Fields |
|---|---|---|
agent.run.requeued | The reaper takes back an expired lease, and the run has retries left | run.id, run.epoch, subject.key, workspace.id, run.kind |
agent.run.ended | The reaper ends a run: its lease expired at its retry limit, or it passed its deadline | The same, and run.stop_reason |
agent.run.reaper.failed | The reaper cannot take back one run. It tries the run again on the next pass | The same, and error.type |
action.attempt.failed | An outbox adapter threw | action.id, action.kind, action.attempt, error.type, and workspace.id when the action has a workspace |
action.result.late | An attempt ended after another claim took its action, or after an erasure deleted it. Its result changes nothing | action.id, action.kind, action.attempt, and workspace.id when the action has a workspace |
action.ended.uncertain | An action ended UNCERTAIN: its write may have reached the provider, and nothing confirms it | action.id, action.kind, action.reason, and workspace.id when the action has a workspace |
action.batch.outlived | Outbox attempts of one replica ran longer than their lease. They keep running, and their late results change nothing | action.claims |
An outbox event names no person and no payload.
An account email has no workspace, so its events have no workspace.id.
Database logs
The bundled PostgreSQL image logs four kinds of events that show contention and maintenance:
| Event | Setting | Line |
|---|---|---|
A statement waits for a lock for longer than deadlock_timeout, 1 second | log_lock_waits=on | still waiting for ShareLock, then acquired, with the holding process and the relation |
| A query writes a temporary file of 10 MB or more | log_temp_files=10MB | temporary file, with its size |
| An autovacuum or autoanalyze takes 1 second or more | log_autovacuum_min_duration=1s | automatic vacuum or automatic analyze, with the table |
| A checkpoint | log_checkpoints=on, the PostgreSQL default | checkpoint complete, with its duration |
No line quotes a statement.
log_min_error_statement=panic keeps the statement text out of these lines and out of error lines, because the text can hold values from a person's work.
The image never sets log_statement or log_min_duration_statement.
To find slow statements, read the statement statistics, which hold no parameter values.
An error line can still quote a value in its DETAIL, for example the key of a unique constraint violation.
The chart does not expose values for these logging settings. The log collector of the chart does not read the PostgreSQL container.