Cost Monitoring

Cloud spend (AWS + GCP) and prepaid API balances (DeepSeek, Serper) as Prometheus gauges, a Grafana Costs dashboard, and Discord alerts on the same pipe as every other alert. Terraform budgets on each cloud are a second, independent backstop.

What it monitors

Signal	Source	Metric
AWS month-to-date cost by service	Cost Explorer GetCostAndUsage (global, us-east-1)	aws_cost_mtd_usd{service}
GCP month-to-date cost by service	BigQuery billing export	gcp_cost_mtd_usd{service}
DeepSeek prepaid balance	GET api.deepseek.com/user/balance	deepseek_balance_usd{currency}
Serper credits remaining (est.)	backend serper_requests_total counter (separate change)	serper_credits_remaining (recording rule)

The exporter

A small Go program (k8s/envs/platform/cost-exporter/) exposing /metrics (promhttp) and /healthz, refreshing on an hourly ticker (REFRESH_INTERVAL, configurable). It runs in the monitoring namespace; Prometheus scrapes its ServiceMonitor (labelled release: monitoring) cluster-wide.

Each provider fails independently: a provider erroring bumps cost_exporter_scrape_errors_total{provider} and leaves its last-good gauges in place, never blanking the others. A provider whose secret is absent stays disabled (the exporter runs with any subset wired).

Env var	Default	Purpose
REFRESH_INTERVAL	1h	Scrape cadence
SCRAPE_TIMEOUT	30s	Per-provider timeout
AWS_REGION	us-east-1	Cost Explorer is global; pinned here
GCP_BILLING_TABLE	development-485000.billing_export.gcp_billing_export_v1_01CBC0_F49F57_C4BF88	project.dataset.table; presence enables GCP provider
GCP_PROJECT	development-485000	BigQuery client project
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY	(from ESO)	Presence enables the AWS provider
DEEPSEEK_API_KEY	(from ESO)	Presence enables the DeepSeek provider

GCP auth is Workload Identity (KSA monitoring/cost-exporter → GCP SA cost-exporter, BigQuery Data Viewer + Job User). AWS + DeepSeek creds are projected by ESO from GCP Secret Manager. See Secrets Management and IAM.

Build + push the image:

The image is built and pushed automatically by Cloud Build trigger (cost-exporter-push-trigger) on merges to main touching k8s/envs/platform/cost-exporter/**. Manual build/push is also available:

task cost-exporter:push   # gcloud builds submit --config=cloudbuild-cost-exporter.yaml

Dashboard

Costs folder in Grafana (k8s/.../manifests/dashboards/tomoda-costs.yaml, sidecar ConfigMap):

• AWS + GCP MTD stacked by service, with totals and a straight-line month-end projection (MTD / day_of_month × days_in_month).
• DeepSeek balance gauge + trend.
• Serper credits-remaining gauge + burn rate (requests/day).
• Prepaid runway: days-to-empty and top-up-within countdowns (balance ÷ 7-day burn).

Alerts

PrometheusRule cost-alerts (k8s/.../manifests/cost-alerts.yaml). Thresholds live in a cost.thresholds recording group as constant series, so retuning is a one-line edit, not an expression rewrite.

Alert	Warning	Critical
AWSCostMTDHigh	MTD > $200	MTD > $400
GCPCostMTDHigh	MTD > $300	MTD > $600
DeepSeekBalanceLow	balance < $20	balance < $5
SerperCreditsLow	credits < 10000	credits < 3000

Thresholds are placeholders

The values above are starting points — tune them in the cost.thresholds group. SerperCreditsLow stays dormant until the backend serper_requests_total counter ships (a separate tomoda-repo change); the recording rule references it and evaluates to the full pack size until then.

Alerts carry severity + summary/description annotations, so the existing Discord receiver renders and delivers them with no routing change.

Top-up workflow

1. Discord posts DeepSeekBalanceLow / SerperCreditsLow to #monitoring.
2. Check the Costs dashboard runway panels for days-to-empty.
3. Top up the provider account.
4. For Serper: reset the pack size (cost:serper_pack_credits) and the counter window in cost-alerts.yaml at each top-up.

Terraform budgets (second backstop)

Independent of the exporter, cloud-native budgets alert on spend directly:

• AWS (infrastructure/aws/cost_budget.tf): aws_budgets_budget → SNS topic → email to the owner. Discord-via-Lambda is a future option; the SNS topic is the seam. Gated on cost_budget_owner_email.
• GCP (infrastructure/gcp/cost_budget.tf): google_billing_budget with threshold rules → a Pub/Sub topic (cost-budget-alerts) + owner email. Consuming the topic for Discord is future. Gated on billing_account.

Operator prerequisites

The exporter runs immediately but reads real data only once these are done:

1. GCP billing → BigQuery export (Manual Console step). In the Cloud Console, enable the standard usage cost billing export into the billing_export dataset (created by infrastructure/gcp/cost_exporter_iam.tf). Then set GCP_BILLING_TABLE on the Deployment to the export table (development-485000.billing_export.gcp_billing_export_v1_<BILLING_ACCOUNT>). GCP does not provide an API for Terraform to toggle billing export.
2. DeepSeek key (IaC + Secret Value). Terraform provisions tomoda-deepseek-api-key in GCP Secret Manager (infrastructure/gcp/cost_exporter_iam.tf). Populate its secret value. ESO projects it into monitoring.
3. AWS Cost Explorer creds (Fully Automated via IaC). terraform apply on infrastructure/aws/cost_explorer_iam.tf creates the scoped IAM User and automatically provisions tomoda-aws-cost-explorer-creds in GCP Secret Manager, so ESO projects it directly into monitoring.

Cost Explorer request cost

GetCostAndUsage bills ~$0.01/request. Hourly refresh is a few cents a month.