Skip to content

SLOs (Pyrra)

Service Level Objectives expressed as Kubernetes CRDs, managed by Pyrra. Each ServiceLevelObjective resource defines a target + indicator; Pyrra translates it into a PrometheusRule with multi-window multi-burn-rate alerts (Google SRE book methodology) that route through the existing Alertmanager → #monitoring Discord pipeline.

Why SLOs alongside rule-based alerts

Rule-based alerts (in manifests/alerting-rules.yaml, see alerting.md) fire on absolute thresholds: "p95 > 2s" → page. They're good for "is something broken right now."

SLOs are about long-term reliability: "we've burned X% of our 30-day error budget; at current rate we'll exhaust it in N hours." This catches slow degradation that absolute thresholds miss — a service that's been ~1% slower than usual for a week isn't tripping any alarm, but its error budget is bleeding.

Both signals matter. Rule-based alerts catch acute incidents; SLO burn-rate alerts catch chronic degradation. Together they form a coherent "is the service healthy on the timescales the team cares about" picture.

Pyrra architecture

ServiceLevelObjective CRs (k8s/envs/platform/manifests/slos/*.yaml)
        │
        │  watched by
        ▼
Pyrra operator (monitoring/ ns, single replica)
        │
        │  writes
        ▼
PrometheusRule CRs (auto-generated, named after the SLO)
        │
        │  picked up by kube-prometheus-stack
        ▼
Prometheus evaluates burn-rate expressions every 30s
        │
        │  fires
        ▼
Alertmanager (existing routing in monitoring/values.yaml)
        │
        │  group_by + inhibition rules, same as manual alerts
        ▼
Discord #monitoring channel

The Pyrra API server pod also exposes a UI at pyrra.<cluster-domain> (currently disabled — see "UI access" below) showing current compliance + burn-rate per SLO.

The five starter SLOs

All live in k8s/envs/platform/manifests/slos/. Targets reflect what we can reasonably promise without over- or under-engineering.

SLO Target Window Indicator Metric source
API availability 99.5% 30d non-5xx / total request ratio tomoda_http_requests_total{status_class}
API latency (p95) 95% 30d requests ≤ 500 ms / total tomoda_http_request_duration_seconds_bucket{le="0.5"}
WS broadcast latency (p99) 99% 30d broadcasts ≤ 100 ms / total tomoda_ws_broadcast_duration_seconds_bucket{le="0.1"}
Async task success 99% 30d non-failed / total Asynq tasks tomoda_async_task_total{status}
Login success 99.9% 30d system success / total login attempts tomoda_auth_logins_total{status}

Error budget interpretation

A 30-day window means at 99.5% target, you have 3h 36m of "downtime" per month to spend. The error budget concept means: as long as you're under that, you can take risks (deploy fast, accept brief regressions); when you're over it, freeze and focus on reliability.

Burn-rate alerts fire when the budget is being consumed at a faster-than-acceptable pace, not when it's exhausted. The standard multi-window scheme:

Window Burn rate threshold Severity Meaning
1h 14.4× critical 2% of monthly budget burned in last hour → if sustained, budget exhausted in <3 days
6h 6× critical 5% burned in 6h → page within hours
3d 1× warning 10% burned over 3 days → slow drift, address next business day

Pyrra emits all three for every SLO automatically. Alertmanager's existing config (group + inhibit + Discord receiver) handles them without extra setup.

Adding a new SLO

  1. Pick a metric exported by the backend (or anything Prometheus scrapes).
  2. Create a new file under k8s/envs/platform/manifests/slos/<name>.yaml:
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: my-new-slo
  namespace: monitoring
spec:
  target: "99.5"     # percentage as a string
  window: 30d
  description: "Plain-English statement of what this guarantees."
  indicator:
    # Either ratio (errors/total) or latency (success bucket / total count)
    ratio:
      errors:
        metric: my_metric_total{status="failed"}
      total:
        metric: my_metric_total
  1. Commit → Argo CD syncs → Pyrra writes a PrometheusRule within ~30s → burn-rate alerts active.

Verification

# 1. Pyrra operator + API server running
kubectl get pods -n monitoring -l app.kubernetes.io/name=pyrra
#   pyrra-operator-...     1/1   Running
#   pyrra-api-server-...   1/1   Running

# 2. SLOs registered
kubectl get servicelevelobjective -n monitoring
#   NAME                       TARGET   WINDOW
#   api-availability           99.5     30d
#   api-latency-p95            95       30d
#   ws-broadcast-latency-p99   99       30d
#   async-task-success         99       30d
#   login-success              99.9     30d

# 3. PrometheusRules auto-generated
kubectl get prometheusrule -n monitoring -l app.kubernetes.io/name=pyrra
#   one PrometheusRule per SLO

# 4. Burn-rate alerts loaded in Prometheus
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090
# Open http://localhost:9090/alerts
# Search for "ErrorBudgetBurn" — should see 3 alerts per SLO (1h critical, 6h critical, 3d warning)

# 5. Pyrra UI (when exposed)
kubectl port-forward -n monitoring svc/pyrra-api-server 9099:9099
# Open http://localhost:9099 — list of SLOs + current compliance + burn-rate graphs

UI access

Pyrra's API server includes a small UI showing per-SLO compliance + burn-rate per window. Currently exposed only via port-forward (kubectl port-forward -n monitoring svc/pyrra-api-server 9099:9099).

To make it accessible at https://pyrra.tomoda.life via Cloudflare + oauth2-proxy (same pattern as Grafana / Argo CD UI), add a Traefik IngressRoute in k8s/envs/platform/manifests/ referencing the existing oauth2-proxy middleware. Deferred for now — port-forward is sufficient and adds one less public hostname to manage.

Tuning targets

Start with these numbers and adjust based on observed reality:

  • If an SLO is consistently green (95%+ budget remaining), the target may be too lax. Tighten.
  • If an SLO is consistently red (budget exhausted), either the system needs reliability work OR the target is unrealistic — pick one. Don't ratchet targets down to "always green" or SLOs become meaningless.
  • Window: 30d is the default. Shorter windows (7d) react faster to recent changes; longer windows (90d) smooth over noise. Start at 30d and only change when there's a clear reason.