Observability

Sable is a gateway that meters money, so an operator running it needs the ordinary things: a scrape endpoint and request traces. Both exist, both are off until you configure them, and both are aggregate-only — there is no per-account series and no request content anywhere in either stream, for the reasons in the privacy contract.

This page is for whoever runs the gateway. If you are a buyer wanting proof of what a particular request did, that is a receipt, not a metric.

Prometheus metrics

GET /metrics serves the Prometheus text exposition format at the root, beside /healthz.

Enabling it

SABLE_METRICS_TOKEN=$(openssl rand -hex 32)

Unset, the route is not mounted at all — a request to /metrics 404s exactly like any other unknown path. This is the same posture as SABLE_ADMIN_TOKEN: a route that exists and returns 401 still tells a prober the surface is there.

The token is not ceremony. The metric stream carries no user content, but it does carry request volume, token throughput and metered cost — which is to say it describes revenue. An open /metrics publishes your business volume to anyone who can reach the port.

curl -H "Authorization: Bearer $SABLE_METRICS_TOKEN" \
  https://api.buildsable.com/metrics

Scrape config

scrape_configs:
  - job_name: sable-gateway
    scheme: https
    metrics_path: /metrics
    scrape_interval: 30s
    authorization:
      type: Bearer
      credentials: "<SABLE_METRICS_TOKEN>"
    static_configs:
      - targets: ["api.buildsable.com"]

Database-backed gauges are cached for 15 seconds, so adding scrapers does not add database load — a monitoring system should never become a load generator against the thing it is monitoring.

What is exported

Counters:

Gauges:

The three alerts worth having

# The confidential tier has silently fail-closed. This has happened, more than
# once, and gone unnoticed for days each time.
- alert: SableConfidentialDown
  expr: sable_attestation_verified_backends == 0
  for: 5m

# Nobody is running the leader-gated background workers — health sampling,
# sweeps, purges and alerts have all stopped, while every replica still serves
# traffic and looks healthy.
- alert: SableNoLeader
  expr: max(sable_leader) == 0
  for: 2m

# Webhook delivery is falling behind rather than failing outright.
- alert: SableOutboxBacklog
  expr: sable_webhook_outbox_depth > 500
  for: 10m

OpenTelemetry traces

Set an OTLP/HTTP endpoint and the gateway exports spans to any OTel collector — Grafana Tempo, Datadog, Honeycomb, Langfuse:

SABLE_OTLP_ENDPOINT=https://collector.example.com/v1/traces
SABLE_OTLP_HEADERS=authorization=Bearer abc123,x-tenant=acme
SABLE_OTLP_SAMPLE_RATIO=0.1

Unset, no OpenTelemetry machinery is constructed at all and the log stack is byte-for-byte what it was before — the unconfigured path costs nothing.

SABLE_OTLP_SAMPLE_RATIO defaults to 0.1. Sampling is parent-based, so a trace your own service already sampled arrives whole instead of half-present. An unparseable or out-of-range value falls back to the default rather than refusing to boot; a bad sampling ratio is not a reason to take a gateway down.

Spans are exported in batches and flushed on graceful shutdown. That matters more than it sounds: an exporter that is never flushed drops the last seconds before every deploy, which is exactly the window you are looking at when a deploy goes wrong.

Resource attributes describe the deployment, never a user: service.name, service.version, deployment.region, and a per-process service.instance.id so replicas are distinguishable.

Span attributes follow OpenTelemetry semantic conventions where they exist — http.request.method, http.route, http.response.status_code. Note that otel.status_code is set to ERROR only for 5xx: a 4xx is the caller's error, not the server's, and marking those as failures makes a trace dashboard useless.

What Sable deliberately does not expose

This is a short list on purpose, and it is a design constraint rather than a gap to be filled later.

No content, ever. No prompt, no completion, no submitted code, no sandbox output. A span attribute and a metric label are both log lines that leave the process, so §3 governs them exactly as it governs the request log. There is no debug flag that turns this on.

No per-account series. Not account_id, not a key id, not a wallet address. Two independent reasons, either sufficient:

• Cardinality. One series per customer is unbounded by definition, and a metrics endpoint that grows a series per customer is the classic way a monitoring backend takes down the service it monitors.
• Disclosure. A per-account breakdown of spend and volume is a per-customer business report. Anyone who can scrape would be reading it.

Everything account-shaped is summed before it reaches the registry. If you need per-account numbers, they belong to the account holder: GET /v1/usage/events and the receipt stream, authenticated as that account.

No raw request paths. The route label is the matched pattern (/v1/receipts/:id), never the concrete path. A raw path carries resource identifiers, which would put ids into both metric labels and span names — one series per receipt, and identifiers shipped to a third-party collector. Unmatched requests are bucketed as route="unmatched" for the same reason.

No unbounded labels. Every label is drawn from a fixed vocabulary, and the registry enforces a hard ceiling on distinct label combinations per metric on top of that. Past the ceiling, new combinations collapse into __other__, so a future mistake degrades a metric instead of leaking memory.

Related

• Privacy contract — why the constraints above are constraints
• Verifiable receipts — per-request proof, for buyers
• Webhooks — including receipt_minted, if you want every metered call in your own audit stack