Cost Monitoring¶
Prepaid API balances (DeepSeek, Serper), usage-billed Google Places and the flat monthly hosting bills as Prometheus series, a Grafana Costs dashboard, and Discord alerts on the same pipe as every other alert.
Series and alerts¶
All in PrometheusRule cost-alerts (k8s/envs/platform/manifests/cost-alerts.yaml), shown on the Grafana Costs dashboard (k8s/apps/grafana-dashboards/costs/tomoda-costs.json). Alerts post to #monitoring through the Discord receiver.
| Series / alert | Meaning |
|---|---|
cost:fixed_monthly{item,currency} |
One flat monthly bill, tax included, in its own currency. OVH server: 151.41 CAD (133.99 + 13% tax) |
cost:fx_rate{from="USD",to="CAD"} |
Manual USD to CAD rate |
cost:usage_monthly{item,currency} |
One usage-billed API over the trailing 30d. Google Places: places_estimated_cost_usd_total in USD, at list price before the monthly free cap, so it reads high at low volume |
cost:monthly_total_cad |
Sum of cost:fixed_monthly and cost:usage_monthly, USD items converted through cost:fx_rate |
deepseek_balance_usd{currency} |
Balance of the one DeepSeek account dev and prod share, exported by the async worker in both envs (hourly, one replica per run) |
serper_credits_remaining |
Serper pack size minus serper_credits_used_total over 30d |
DeepSeekBalanceLow |
Balance below USD 10 (warning) / USD 3 (critical), the highest value from any env's pods whose last successful fetch is under 2h old |
DeepSeekBalanceCheckStale |
No successful balance fetch from any async pod, dev or prod, in 2h, held for 30m. Counts pods a rollout just replaced (max_over_time over 3h), so new pods reporting 0 before their first hourly run do not fire it; the balance alert is blind |
SerperCreditsLow |
Below 5000 (warning) / 1000 (critical) |
Thresholds are constants in the cost.thresholds group.
Third-party usage¶
The Third-party usage row on the Costs dashboard follows the env picker (prod by default); the Costs row above it stays cluster-wide. Fed by the backend counters listed in the tomoda repo's third-party usage metrics page.
| Panel | Source |
|---|---|
| DeepSeek tokens, cache-hit ratio, est. USD (30d) | llm_tokens_total; USD at the same per-token rates as the LLMDailySpendHigh alert |
| Serper credits (30d), credits per day | serper_credits_used_total by operation |
| Places requests (30d), requests per day by SKU | places_requests_total by sku |
| Places est. USD (30d), estimated USD per day | places_estimated_cost_usd_total, plus DeepSeek tokens at list price |
| Requests per hour, errors per hour | llm_requests_total, places_requests_total, search_enrichment_requests_total{provider="serper"}; errors are status!="ok" |
Monthly figures are the trailing 30 days. Places prices live in the backend config, so a Google price change is a backend change.
Currency¶
CAD is the reporting currency: the dashboard headline is cost:monthly_total_cad. Every cost series keeps a currency label and is never summed across currencies except through cost:fx_rate. Balances and their alerts stay in their native currency (DeepSeek in USD) and are not spend, so they stay out of the total. Serper is a prepaid pack too, so it is tracked in credits, not dollars.
- Update the FX rate when it drifts more than about 5%: edit
cost:fx_ratein thecost.monthlygroup and its date in the comment and in the dashboard's rate panel title. - Add a fixed bill: add a
cost:fixed_monthlyrule to thecost.monthlygroup withitemandcurrencylabels and the tax-included amount. A USD item is converted automatically; another currency needs its owncost:fx_rateand a term incost:monthly_total_cad.
Top-up workflow¶
- Discord posts
DeepSeekBalanceLow/SerperCreditsLowto#monitoring. - Top up the provider account.
- For Serper: reset the pack size (
cost:serper_pack_credits) incost-alerts.yaml.
Cloudflare¶
Cloudflare evaluates its own usage; nothing passes through Prometheus. infrastructure/cloudflare/notifications.tf declares one billing_usage_alert per product, posting to #monitoring through the Discord #monitoring webhook destination (built from GSM tomoda-alert-webhook-primary). Each fires once per billing period.
| Product | Alert at | Free tier | Price above |
|---|---|---|---|
r2_storage |
400 GB | 10 GB | USD 0.015 / GB-month |
r2_class_a_operations |
1M | 1M / month | USD 4.50 / million |
r2_class_b_operations |
10M | 10M / month | USD 0.36 / million |
R2 holds about 290 GB, so storage already bills a few dollars a month. Retune in billing_usage_alerts. The account-wide spend alert (auto-created, USD 10, email) lives in the dashboard under Notifications: the provider has no budget alert type.
GCP spend guards¶
GCP still bills for Secret Manager, the Terraform state bucket and, most importantly, the Places API, which is the one usage-based cost there that grows with users. FCM (push) is free. Nothing in the cluster reads GCP billing; these are the guard:
- Budgets with alerts.
infrastructure/gcp-org/billing.tf: CAD 25/month ondevelopment-485000and CAD 100/month on the whole billing account, each emailing billing admins at 50%, 90% and 100% of actual spend. A budget only notifies; it never stops spend. See GCP organization. - Places API quota caps. APIs & Services → Places API (New) → Quotas & system limits: set a per-day cap on the request quotas the backend uses (Nearby Search, Place Details, Text Search) slightly above normal daily use, so a bug or abuse loop stops at the cap instead of billing. Raise the caps deliberately as usage grows.
- Restrict the API key (
tomoda-google-places-api-key) to the Places API (Credentials → the key → API restrictions), so a leaked key can't call other paid APIs.
Revisit the budget and caps whenever Places usage grows; the alert email is the trigger.