Observability#

serve is instrumented with Prometheus metrics and OpenTelemetry traces end to end, so a request can be watched as RPC → query → render.

Metrics (Prometheus)#

Metrics are exposed at /metrics on the HTTP port. The endpoint is unauthenticated — it sits outside the auth middleware so scrapers reach it without a token.

curl -s localhost:8081/metrics | grep rackattack_

What’s exported:

• Per-RPC rate / latency / error — rackattack_rpc_requests_total{method,code}, rackattack_rpc_duration_seconds (gRPC); plus rackattack_http_requests_total{route,status} for REST + GraphQL.
• DB pool saturation — rackattack_db_pool_{total,acquired,idle,max}_conns.
• Cache hit ratio — rackattack_cache_{hits,misses}_total.
• Go runtime + process collectors.

The k8s pod template carries prometheus.io/{scrape,port,path} annotations for annotation-based discovery (or use a ServiceMonitor with the Prometheus Operator).

Alerts & dashboard#

• Example alert rules — error rate, p99 latency, DB pool saturation, cache hit ratio — are in deploy/prometheus/alerts.yaml.
• A starter Grafana dashboard is deploy/grafana/rackattack-dashboard.json.

Tracing (OpenTelemetry)#

Each request gets an otelgrpc/otelhttp span with query.* and render.* child spans, so a trace reads RPC → query → render. Tracing is off by default (the global tracer is a no-op); point it at an OTLP collector to export:

export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
export OTEL_EXPORTER_OTLP_INSECURE=true   # for a local plaintext collector
rackattack serve

In Kubernetes, set those as env on the rackattack container (e.g. pointing at an OpenTelemetry Collector Service). The setup lives in internal/telemetry.