Error reporting
How the server tells you about its own bugs and failures, where to read the reports on each platform, and how to send them to an error tracker. Metrics tell you that something is wrong; an error report tells you what failed, with its stack. The configuration reference lists every setting.
Three things happen to every report, whatever you configure:
- One
error reportline goes to standard error, with the stack (The log line). - The report is counted in an error group in the database. The administration screen shows the groups, and people with the
manage_errorspermission are notified of new and recurring ones (Error groups and notifications). - If you set
errors_exporter, the report also goes to an external service (Sending reports to a service).
Nothing leaves the server unless you set errors_exporter.
What is reported
Section titled “What is reported”| Reported | Not reported (the metrics and the other log lines cover them) |
|---|---|
| A panic in an HTTP handler, a job, or a background task | 4xx responses (bad input, authorization, rate limits) and the maintenance mode |
| An error behind a 5xx response of the client API, the admin API, the sign-in pages, ActivityPub, OAuth, or media | A request whose client went away |
A job that failed for good (dead) |
A job retry. It is reported when it runs out of retries |
| A failure to start: an invalid configuration, or a database or adapter that cannot be set up | A failed readiness check |
| An error the web client did not handle, sent by the browser to the server | Failures caused by other servers: delivery retries and give-ups, rejected signatures, failed remote fetches |
| The browser’s own noise: a lost network, errors from browser extensions, and chunks that a new deploy removed |
A failure of an outside service (mail, storage, payments, voice) is reported only when it makes a job fail for good or a request answer 5xx. A temporary failure shows in the my.adapter.* metrics instead (Running the server).
A panic in a background task stops the process after it is reported, so the server never keeps running in a broken state. Your platform restarts it.
The log line
Section titled “The log line”The line has msg = error report and level ERROR (severity is CRITICAL when the failure stops the process).
Its fields, and what scrubbing does to them, are in Running the server. In short:
my.error.fingerprintis the same for every occurrence of the same error. Search or group by it.my.error.wheresays where it happened, for exampleclient_api GET /api/v1/statuses/{id}.request_id,job_id,trace_id, andspan_idfind the other lines of the same request or job.- The line of an error of a request has
context.httpRequest: the method, the route as the URL (/api/v1/statuses/{id}, never the path the client asked for), and the status answered. Error Reporting shows them on the group and can filter by them. messageis last on the line: the error type and message, a newline, and the Go stack.- The stack starts where the error came from, not where it was reported: for a database error, the server function that ran the query. Errors from different places therefore fall into different groups, in Error Reporting and in the administration screen alike.
No other line carries a stack.
Google Cloud (Cloud Run)
Section titled “Google Cloud (Cloud Run)”Cloud Error Reporting reads the line without any setup and lists the errors in the console under Error Reporting. It is free; you pay for Cloud Logging only.
- It groups errors by the type and the top frames of the stack. It cannot take the server’s fingerprint, so a group in Error Reporting can differ from a group in the administration screen.
- It keeps groups for 30 days, and it notifies you of new errors and of resolved errors that happen again, at most 20 notifications an hour for the project. Set up its notification channel in the Google Cloud console.
- To find what happened around a report, search Cloud Logging for
jsonPayload."request_id"="<id>", or the same withtrace_id.
With the OpenTofu module mycroblog-gcp, the observability section sends the reports here by default (errors = "error-reporting"): it enables the Error Reporting API and leaves errors_exporter at none.
CloudWatch Logs reads the line as plain JSON.
- Turn on log anomaly detection for the log group. It flags a line pattern it has never seen before and can alarm to SNS. It reads only the first 1,500 characters of a line, which is why
messageis the last field. It does not keep a resolved state. - To count reports, make a metric filter such as
{ $.msg = "error report" }. Use no dimension taken frommy.error.fingerprint: every value of a dimension is billed as a metric of its own.my.error.sourceandmy.error.kindare safe. - Find the context with Logs Insights:
filter `request_id` = "<id>".
With the OpenTofu module mycroblog-aws, the observability section keeps the reports in CloudWatch Logs by default (errors = "cloudwatch"), with errors_exporter at none. Log anomaly detection stays a step of yours.
Self-hosted
Section titled “Self-hosted”Read the line from the container log: docker compose logs my | jq -c 'select(.msg == "error report")'.
With metrics_exporter = prometheus, the alert MyErrorReportsIncreasing (Alerts and runbook) warns when reports rise sharply; the counter behind it is my.errors.reported.
For notifications that need no outside service, use the administration screen and the webhooks below. For a tracker, use errors_exporter.
Error groups and notifications
Section titled “Error groups and notifications”The server counts the reports by fingerprint in the database. It needs no setting and no extra service, and it writes little: the first report of a fingerprint is written at once, and the following ones are added up in memory and written once a minute. A hard kill of the process can lose the counts of the last minute; the log lines are not lost that way. Counts older than 90 days are deleted every day. If the database is what failed, the report stays in the log line only.
The administration screen
Section titled “The administration screen”People whose role has manage_errors (the built-in admin role has it) see “Errors” in the administration menu (/admin/errors).
- The list shows the groups, the open ones by default, the most recently seen first. Filter by state (open, resolved, muted, every state) and by source (the server, the web client). Each row shows the first line of the message, where it happened, the counts of the last 24 hours and 7 days, when it was last seen, and its state.
- A group’s page starts with what you can do in its state (“Resolve” and “Mute” for an open group, “Mute” for a resolved one, “Unmute” for a muted one). It shows the group’s facts (where, the message, the first and last time and version, the fingerprint), its counts of the last 24 hours and 7 days, the latest occurrence (time, version, process role, the types, the request ID and the trace ID, and the stack), the occurrences by hour for the last 90 days, newest first, and the history of its state changes. The fingerprint, the request ID, the trace ID and the stack have a copy button: paste the IDs into your log search.
- “Resolve” is for an error you fixed. If it happens again on another version of the server, the group opens again and notifies again. The same error on the version that was resolved is only counted, so the last requests of the old version do not reopen it.
- “Mute” keeps counting but sends no notification or webhook event. “Unmute” opens the group again.
The same actions are available as GET /api/v1/admin/error_groups and POST /api/v1/admin/error_groups/{id}/resolve (/mute, /unmute).
Notifications and webhooks
Section titled “Notifications and webhooks”- People with
manage_errorsget the notificationadmin_error_groupfor a new error group and for a recurrence. At most 10 arrive an hour for the whole server; the rest of the hour fold into one notification that says more errors were reported. They can also get it by email (Moderation). - The webhook events
error_group.createdanderror_group.regressedcarry the same groups, so you can send them to Slack, Discord, or an automation tool (Server webhooks, Webhook reference). The webhook needsmanage_webhooks, and the events needmanage_errors. - A muted group never notifies.
Errors from the web client
Section titled “Errors from the web client”The web client sends the errors its screens did not handle to POST /api/v1/client_errors on your own server, so no third party sees them and no content security setting changes. The server answers 204 to everything and drops what it does not accept: a body over 16 KiB, a stack without a file of your own web client, and more than 10 reports a minute from one IP address. The same error is sent once per page view, and at most 20 per page view.
A report sent without a signed-in session is counted and logged but does not notify anyone by itself, so an anonymous script cannot fill your inbox. The group notifies when a signed-in person’s browser reports it too.
The stacks are those of the minified build and are not translated back to source, so read the route and the message first. The release is the version of the server that served the page. A page that opened offline, or after 3 seconds without an answer, comes from the browser’s copy, which names the version from when the browser stored the web client: an older version with the same web client.
UX measurements
Section titled “UX measurements”The web client sends samples of how fast it responds to POST /api/v1/client_measurements on your own server, the same way as its errors: how long a screen takes to appear (LCP) and to answer a press (INP) on each route, its long frames, each navigation by kind, each action by its ID (post.reply), and the round trip, jitter, and packet loss of voice calls. A sample names only the route, the kind, the action, or the call transport; it carries no account, address, or URL. The server answers 204 to everything and keeps at most 10 requests a minute from one IP address.
The server counts the samples by hour in the database, written once a minute, and deletes them after 30 days. GET /api/v1/admin/ux_measurements (manage_errors) sums the last 24 hours, or hours up to 720, with the 75th percentile of each name; the metrics my.ux.duration and my.ux.call.packet_loss export the same samples (Metrics). Each kind keeps its first 200 names in a process; the later ones count as other.
It is off by default; the instance setting ux_measurement turns it on, and while it is off the server takes no sample (Instance settings). While it is on, only a share of browser sessions sends samples, 10 % unless ux_measurement_share sets another, each session drawn once when the web client starts.
People whose role has manage_errors see “UX measurements” in the administration menu (/admin/ux-measurements). Choose the last 24 hours, 7 days or 30 days; each metric with samples has a table of its names with their samples and the 75th percentile, and, for LCP and INP, the share of samples that are good and that are poor (within 2.5 s and past 4 s for LCP, within 200 ms and past 500 ms for INP), a 75th percentile past the good threshold marked. “Measure the web client”, under the tables, is the setting ux_measurement, which only people whose role also has manage_settings change there, with its own “Save”.
Sending reports to a service
Section titled “Sending reports to a service”Set errors_exporter to send each report also to an error tracker. The log line and the groups in the database stay on. A report is sent once and never retried, and nothing waits for the service.
| Value | Sends to |
|---|---|
none (default) |
Nowhere |
sentry |
Any Sentry-compatible service at errors_sentry_dsn |
otlp |
An OpenTelemetry collector or backend, as log records |
errors_environment (default production) names the deployment on every report, such as staging.
Both exporters send the same scrubbed fields as the log line, with the fingerprint and the trace and span IDs, and nothing about the host: no host name, IP address, user, request body, header, or cookie.
The server limits what it sends so that the service is not flooded:
- The same fingerprint goes out at most once a minute per process, and all reports at most 60 a minute. The rest stay in the log and the groups.
- Up to 1,000 reports wait to be sent. A report that finds the queue full is dropped. When the server stops, it waits up to 2 seconds for the queue to empty.
- When the service answers
429or asks to slow down (Retry-After,X-Sentry-Rate-Limits), the server stops sending for that long. - The server never follows a redirect from the service, so that the DSN key and the report go nowhere else: a redirect counts as
failed. Use the DSN that the service shows, which points at its intake itself.
Watch the counter my.errors.exported: my.outcome is success, dropped (over a limit, the queue was full, or the service asked to slow down), or failed (the service could not be reached or refused the report) (Running the server).
Sentry, GlitchTip, and Datadog
Section titled “Sentry, GlitchTip, and Datadog”MY_ERRORS_EXPORTER=sentryMY_ERRORS_SENTRY_DSN_FILE=/run/secrets/sentry_dsn # or MY_ERRORS_SENTRY_DSN; the DSN is a secretWith the OpenTofu modules, leave the errors of the observability section to the settings, and the module creates the secret of the DSN for you to fill, which its output secrets_to_fill lists:
observability = { errors = "settings" } settings = { errors_exporter = "sentry" } secrets = ["errors_sentry_dsn"]The server speaks the Sentry protocol itself, so no Sentry SDK is involved and no SENTRY_* environment variable is read. It sets the fingerprint, so the service groups by it, and the tags service (my), my.role, my.error.where, and error.fingerprint. The release is the server’s version and the environment is errors_environment.
- Sentry (hosted or self-hosted): copy the DSN from the project’s client keys.
- GlitchTip: copy the DSN of the project. It is open source and runs on a small server. Check which alert rules your version offers for new and recurring issues.
- Datadog Error Tracking: use the Sentry DSN of your Datadog site,
https://<DD_API_KEY>@sentry-intake.<DD_SITE>/1. Datadog requires theservicetag, which the server always sets. Datadog stores these events as logs and groups them in Error Tracking. If you send the log line to Datadog with an agent or a forwarder instead (Error Tracking for Logs), the stack is inmessage; map it toerror.stackwith a log pipeline, or Datadog does not see it as a stack. Thesentryexporter needs no such mapping.
OTLP backends
Section titled “OTLP backends”MY_ERRORS_EXPORTER=otlpOTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 # or OTEL_EXPORTER_OTLP_LOGS_ENDPOINTOTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20...Each report is an OpenTelemetry log record named exception, with severity ERROR (FATAL when it stops the process), exception.type, exception.message, exception.stacktrace, my.error.fingerprint, my.error.where, my.error.source, my.error.kind, and the trace and span IDs. It goes over OTLP/HTTP (protobuf, gzip) with the same resource as the metrics and traces.
Whether the backend turns these records into grouped errors depends on the backend: some build their error views from trace data only, and Datadog needs error.stack. For Sentry, GlitchTip, and Datadog, use sentry. Use otlp when you want the records next to your other logs.
What scrubbing does and does not guarantee
Section titled “What scrubbing does and does not guarantee”Scrubbing runs once, when the report is created, so the log line, the database, and every exporter receive only the scrubbed text.
- In the message, URLs, email addresses,
@user@hosthandles, IP addresses, UUIDs and ULIDs, and numbers of 10 digits or more are replaced with<url>,<email>,<acct>,<ip>,<id>, and<n>, and the message is cut to 1 KiB. - In the stack, the values of function arguments are replaced with
(...)and the stack is cut to 16 KiB. File positions stay, relative to the source tree, so the paths of the build machine do not appear. - No report has a request body, header, query, cookie, account ID, user name, IP address, or User-Agent.
The scrubbing matches patterns, so it does not guarantee that a message holds no personal data. A name, a free text, or a number of fewer than 10 digits that an error message quotes can stay. The error field of the ordinary http request and job lines is not scrubbed this way. Treat the logs and the error tracker as places that may hold fragments of user content, and give them the same access control and retention as the rest of your logs. Once you set errors_exporter, the location, retention, and access rules of that service apply to the reports.
For LLMs and coding agents:this page as Markdown, llms.txt,llms-full.txt, llms-small.txt