Observability principles
A failure that nobody can see is a defect, also when the code around it works. Thus, each change in Hephaestus ships the signal that shows when it fails. ADR 0071 records the decision, the options that it rejects, and its order of work.
The constraints
- Operators run Hephaestus themselves. A small instance is one Kubernetes node. A signal must go to the log store and the Prometheus that the operator already has.
- Readers read feedback about their own work. A signal must not carry what they wrote.
- Feedback earns trust or it is not sent. A capture failure is a fact about Hephaestus. It is never a claim about the work of a person (Practice review pipeline).
The principles
- Every user-visible failure is observable, from the browser to the stage that caused it. An error view, a failed load or a page that does not respond must leave a signal on the server. That signal must lead to the request, the job or the stage that failed.
- Signals carry no content, browser reports carry no direct identifiers, and labels are closed sets. A signal has no reviewed text, prompts, model output, messages from the browser, page addresses or query strings. A browser report also has no person id, workspace id or user agent. A metric label is an enum. It is never an id, a model name or an address. A server log record can name a user or a repository, and its correlation ids are personal data. Log records gives the access and retention rules for log records.
- One id joins a browser event to its server trace. A browser event that a request caused carries the trace id of that request. Correlating a request is the contract for that id.
- Each change ships its signal, its alert or service level objective, and its runbook row. It ships them as a component ships its stories. A new failure mode without a signal is not done.
- The framework writes the signal. Do not add a second event or metrics abstraction. Structured logging gives the facilities to use.
- The server owns sampling. A caller's trace flags must not decide which traces the server keeps.
How the parts work together
Each runtime role writes its own signals. Logs go to standard output as ECS JSON. Metrics go to the management port. Traces go to an OTLP collector when the operator turns tracing on.
The browser sends only what an operator switch allows, to an endpoint of the server role. When this page was written, that was page experience metrics only. Browser fault reports follow ADR 0071. The server then writes each report as its own metric. The operator's tools collect all of it. Observability is the contract for each name that they read.
What a change ships
| Part | Where | What checks it |
|---|---|---|
| Signal | A log event with an event.name, a meter, or a span | Focused tests on the event or the meter |
| Name in the contract | Observability | Review |
| Alert | deploy/chart/hephaestus/files/hephaestus-alerts.yml | vp run test:alerts: each alert has a promtool case in which it fires and one in which it stays silent, a runbook row, and a runbook_url annotation |
| Runbook row | Alert response | vp run test:alerts: each alert has a row in that table |
| Metric that a rule reads | The meter in the code | ScrapeContractIntegrationTest. Each name that a rule selects with job="hephaestus" has a meter name in the code. The server exports it at start, or the test lists it as lazy or as a meter of another role, with the reason. Each histogram on the scrape offers every le bound that a rule reads, with the same text. A server at start stays within the series budget |
| Log record | The code that logs it | PersonalDataLogScan reads each record of a test run in the shipped encoding. A record must hold no email address, provider token, bearer value or fixture name. A match fails the run and prints only the logger, the event.name and the kind. It sees the levels that the test profile logs: INFO for this code, WARN for libraries. It does not see forked worker JVMs. The names are a fixed list |
The project's own instances
The project's test instance does not yet run the chart's monitoring stack.
Its vmalert loads a copy of the rule file by hand: the ConfigMap hephaestus-alert-rules in the namespace observability.
That copy does not follow the release, so replace it from the commit that the instance runs when the rules change.
The instance runs kube-state-metrics and node-exporter in that namespace, which allows host paths, with a ClusterRole for the kubelet scrape.
Thus, it evaluates the groups hephaestus-kubernetes and hephaestus-pull-deploy, which the chart's stack does not load.