Synthetic monitoring

External uptime + health probes via a Cloudflare Worker, running every minute from Cloudflare's edge. Catches "the cluster is down" / "DNS is broken" / "the public hostname stopped resolving" scenarios that in-cluster Prometheus can't see (it can't probe itself if the cluster is down).

Architecture

Cloudflare Worker (tomoda-synthetic)         Cloudflare Tunnel + Access
runs every 1 min via cron                    (loki-push.tomoda.life)
   │                                                │
   ├─► probe N URLs in parallel                     │
   │      (fetch with 5-8s timeout)                 │
   │                                                │
   ├─► reconcile against Workers KV state           │
   │   (healthy ↔ failed transitions)               │
   │                                                │
   ├─► state-change notifications ───► Discord #monitoring
   │                                    (via Tomodachi webhook)
   │                                                │
   └─► every probe result ──── HTTPS POST ──────────┤
                                                    │
                                                    ▼
                                    Cloudflare Access validates
                                    CF-Access-Client-Id / -Secret
                                                    │
                                                    ▼
                                    cloudflared (monitoring ns)
                                                    │
                                                    ▼
                                    loki.monitoring.svc.cluster.local:3100
                                                    │
                                                    ▼
                                    Loki ingests log line per probe

Probes (current set)

Defined in synthetic/worker.js:

Probe	URL	Allowed status	Timeout
api	https://api.tomoda.life/health	200	5s
api-dev	https://api-dev.tomoda.life/health	200	5s
app	https://app.tomoda.life	200	5s
app-dev	https://app-dev.tomoda.life	200	5s
assets	https://assets.tomoda.life	200, 403	5s
argocd	https://argo-app.tomoda.life	200, 302	8s
docs	https://docs.tomoda.life	200, 302	5s
login	https://api.tomoda.life/api/v1/auth/login (POST, real password)	200	10s
login-bad-password	same URL (POST, deliberate wrong password)	401	10s
auth-middleware	https://api.tomoda.life/api/v1/users/me (GET, no auth header)	401	5s

302 is accepted for argocd and docs because those are gated by oauth2-proxy / Cloudflare Access — a 302 to the auth flow IS the healthy response.

login (positive path)

POSTs {email, password} JSON to the real auth endpoint with the correct password, exercising the full critical path (handler → JWT signing → DB read). Credentials bound to the Worker at deploy time via two env vars:

• LOGIN_PROBE_EMAIL — plain text (synthetic+probe@tomoda.life by default), Terraform variable synthetic_probe_email
• LOGIN_PROBE_PASSWORD — secret, Terraform variable synthetic_probe_password (sourced from GCP SM tomoda-synthetic-probe-password at apply time)

The synthetic test user is seeded into the database by the backend on boot (backend/internal/database/seed.go in the tomoda repo), reading the same password from its own SYNTHETIC_PROBE_PASSWORD env var (projected via ESO from the same GCP SM secret). The two systems stay in sync because they read the same source of truth.

login-bad-password (negative path — auth rejection)

Same URL as login, same real email, but a hardcoded wrong password. The backend MUST reject with 401. This catches a critical security regression: if a code change somehow accepts any password (bypassed bcrypt comparison, broken middleware, dev mode flag flipped in prod), this probe goes from healthy → failed and fires 🚨 login-bad-password DOWN — that's the canary.

Wrong password is hardcoded in worker.js, not env-bound — deterministic + can't accidentally match a real user's password.

auth-middleware (negative path — middleware enforcement)

GET to a protected endpoint (/api/v1/users/me) without an Authorization header. The auth middleware MUST reject with 401. Goes failed if:

• Middleware regression lets unauthenticated requests through (CRITICAL)
• The endpoint is no longer protected (auth annotation accidentally removed)
• The path was renamed / removed (update the probe URL)

A 200 here means anyone on the public internet can call protected endpoints. Treat any failure of this probe as a security incident until proven otherwise.

Status semantics summary

For login (positive): - 200 → healthy. Backend issued a JWT. - 401 → failed. Either the seed user doesn't exist yet (backend hasn't deployed the migration), the password was rotated in GCP SM without re-deploying the backend, or someone toggled the user inactive. - 5xx → failed. Backend is broken. - Timeout → failed.

Alert behaviour — state-change only

Probe state lives in Workers KV (state/<probe> = healthy | failed):

Transition	Discord behaviour
healthy → failed	🚨 DOWN message with the probe name + failure detail
failed → healthy	✅ RECOVERED message with downtime duration
failed → failed	Silent — except every 60 min while still down, fire a STILL-DOWN reminder so a long incident doesn't fall off the radar
healthy → healthy	Silent (the common case)

Result: a 6-hour outage produces ~7 messages per probe (1 DOWN + 5 hourly STILL-DOWN reminders + 1 RECOVERED), not 360. Matches the design committed-to in alerting.md.

Loki integration

In parallel with Discord, every probe result is pushed to Loki via the tunneled loki-push.tomoda.life endpoint:

• Stream labels: job="synthetic", probe=<name>, status=healthy|failed
• Log body: JSON with probe, url, ok, status, duration_ms, error

Query in Grafana → Explore → Loki:

# All failed probes in the last hour
{job="synthetic", status="failed"} | json

# Specific probe latency over time
{job="synthetic", probe="api"} | json | line_format "{{.duration_ms}}ms {{.status}}"

A future Grafana panel could chart success rate per probe + latency p95 from this data. Deferred to the dashboards follow-up.

Auth chain — why three Cloudflare features

The Worker → Loki push path goes through three Cloudflare products:

1. Cloudflare Workers — the Worker itself, running on Cloudflare's edge. Authenticated to Cloudflare's API at deploy time via the Terraform Cloudflare provider's API token.
2. Cloudflare Tunnel — tomoda-prod-tunnel, with the in-cluster cloudflared connector. The Worker doesn't authenticate to the tunnel directly; the tunnel just provides the transport (loki-push.tomoda.life → in-cluster Loki).
3. Cloudflare Access — gates loki-push.tomoda.life before traffic enters the tunnel. Service-token policy: only requests with valid CF-Access-Client-Id + CF-Access-Client-Secret headers pass. Without Access, anyone with the tunnel URL could push fake logs.

The Worker presents Access headers via two secret env vars (LOKI_ACCESS_CLIENT_ID, LOKI_ACCESS_CLIENT_SECRET) bound at deploy time by Terraform.

Secrets used

All projected via existing patterns — no new GCP SM entries beyond what's already in manual-setup.md:

Secret	Where it's used	How it gets there
tomoda-alert-webhook-primary	Worker → Discord post	TF_VAR_discord_webhook_url passed at terraform apply time (read from GCP SM via gcloud), binds as Worker secret DISCORD_WEBHOOK_URL
tomoda-cloudflare-tunnel-token	cloudflared in-cluster auth to the tunnel	ESO → K8s Secret cloudflared-tunnel-credentials in monitoring/ → TUNNEL_TOKEN env var on the cloudflared Deployment
tomoda-cloudflare-access-client-id	Worker → Loki push auth	Terraform-generated by the cloudflare_zero_trust_access_service_token resource, bound as Worker secret
tomoda-cloudflare-access-client-secret	Same as above	Same
tomoda-cloudflare-api-token	Terraform Cloudflare provider	Standard — TF_VAR_cloudflare_api_token

Note: the Access service token's client-id/secret are created by Terraform itself (not pulled from GCP SM). The Terraform state holds them as outputs. If we want to rotate them outside of a terraform apply cycle, we'd need to swap to a pre-created service token + variable-fed binding — out of scope for v1.

Deployment

Three places touch this system:

1. infrastructure/cloudflare/ — Terraform for the Worker, KV namespace, cron trigger, tunnel, tunnel routing config, Access app, Access policy, service token, DNS record.
2. k8s/envs/platform/cloudflared/ — Argo Application for the in-cluster cloudflared Deployment (plain manifest, not a Helm chart — the chart only did credentials-file auth; we run cloudflared tunnel run --token with the ESO-projected token).
3. synthetic/worker.js — the actual Worker code (vanilla JS, single file).

Initial deploy flow:

# 1. Manual setup (one-time per environment) — see manual-setup.md
#    Ensure all relevant tomoda-* secrets exist in GCP SM

# 2. Terraform apply for the Cloudflare resources
cd infrastructure/cloudflare
export TF_VAR_cloudflare_api_token="$(gcloud secrets versions access latest --secret=tomoda-cloudflare-api-token)"
export TF_VAR_discord_webhook_url="$(gcloud secrets versions access latest --secret=tomoda-alert-webhook-primary)"
export TF_VAR_synthetic_probe_password="$(gcloud secrets versions access latest --secret=tomoda-synthetic-probe-password)"
terraform init
terraform plan -out=plan.out
terraform apply plan.out

# 3. Push the auto-generated tunnel token to GCP SM
#    (re-uses the same secret slot manual-setup.md mentions, but the value
#    comes from Terraform now)
terraform output -raw tunnel_token | gcloud secrets versions add \
  tomoda-cloudflare-tunnel-token --project=development-485000 --data-file=-

# 4. Argo CD picks up the new k8s/envs/platform/cloudflared/ Application
#    automatically on next sync. cloudflared comes up, ESO projects the
#    tunnel token, tunnel goes online.

# 5. Worker starts firing the 1-min cron immediately after the
#    cloudflare_workers_cron_trigger lands. No further action needed.

Verification

# 1. Cron trigger active
gcloud auth print-identity-token   # warmth check that gcloud works
curl -fsS "https://api.cloudflare.com/client/v4/accounts/<account_id>/workers/scripts/tomoda-synthetic/schedules" \
  -H "Authorization: Bearer <token>"
# Should list "* * * * *"

# 2. KV state populating (after first few cron runs)
#    Cloudflare dashboard → Workers & Pages → tomoda-synthetic → KV → tomoda-synthetic-probe-state
#    Should see one 'state/<probe>' entry per probe, value "healthy" or "failed".

# 3. cloudflared in-cluster
kubectl get pods -n monitoring -l app=cloudflared
#   Running, 4 connections established to Cloudflare edge

# 4. Tunnel route resolves
curl -fsS https://loki-push.tomoda.life/ready
#   Without Access headers, returns 403 — that's the gate working correctly.
curl -fsS https://loki-push.tomoda.life/ready \
  -H "CF-Access-Client-Id: <id>" -H "CF-Access-Client-Secret: <secret>"
#   With headers, returns 200 from Loki.

# 5. Loki has synthetic logs
#    Grafana → Explore → Loki → query: {job="synthetic"} | json
#    Should see one entry per minute, per probe.

# 6. Trigger a test alert
#    Manually delete a Cloudflare DNS record for one of the probed hostnames
#    (or just unplug a probe URL). Within ~1 min, Discord #monitoring fires
#    🚨 DOWN. Restore the record → ✅ RECOVERED within another minute.

Rotation

Secret	When to rotate	How
Discord webhook URL	If leaked	Generate new webhook on the channel, push to GCP SM, terraform apply rebinds it
Access service-token client-id/secret	Annually (Cloudflare expires after 1 year)	terraform apply -replace=cloudflare_zero_trust_access_service_token.synthetic_worker — generates new pair, rebinds to Worker
Tunnel token	If leaked	terraform apply -replace=cloudflare_zero_trust_tunnel_cloudflared.loki_push — generates new tunnel, push new token to GCP SM, ESO syncs in 1h (or restart cloudflared Deployment)

Editing the probe list

Add or remove entries in the PROBES array at the top of synthetic/worker.js. Commit the change. Run:

cd infrastructure/cloudflare
terraform plan    # should show 1 change to cloudflare_workers_script.synthetic
terraform apply

The Worker code is re-uploaded; next cron tick uses the new probe list. No downtime — the upload is atomic on Cloudflare's edge.

Related

• Alerting — the Alertmanager + Discord pipeline that the synthetic probe shares (both post to #monitoring)
• Manual setup — provisioning the Cloudflare API token + GCP SM secrets that this stack depends on
• Source code: synthetic/worker.js and infrastructure/cloudflare/