Skip to content

Loading the location seed set

How to publish a generated location seed set to GCS and load it into a target Postgres on demand. The seed set is the location backbone: countries, the admin-area hierarchy, the city gazetteer, and curated collections (UNESCO, Michelin, 50 Best). Loading it populates locations + related tables so discovery and search have data to resolve against.

The pipeline has four stages, three of which live here in devops:

generate
app catalogseeder
→
publish
seed-publish.sh
→
seed bucket
gs://…/seed/<version>
→
load Job
on-demand, per env
→
Postgres
tomoda-{dev,prod}
The generate step runs in the app repo; publish, store, and load live here.

Prerequisites

  • Seed bucket + loader SA exist. Terraformed in infrastructure/gcp/location_seed.tf: bucket tomoda-location-seed-development-485000, GCP SA location-seed-loader@… (read-only), and the Workload Identity bindings to the location-seed-loader K8s SA in tomoda-dev / tomoda-prod.
  • Publisher IAM. The identity running seed-publish.sh must be var.location_seed_publisher (writer on the bucket).
  • gcloud authed against development-485000, including ADC: gcloud auth login && gcloud auth application-default login && gcloud config set project development-485000.
  • A generated seed set. Produced by the app's catalogseeder (app repo). The output directory holds NDJSON collections plus a checksummed manifest.json.
  • The tomoda-backend image. The load Job runs the loader binary shipped inside it at /app/catalogseeder (validate then load). See Image below.

1. Generate (app repo)

Run the app's catalogseeder to produce the seed set into a local directory. See the app repo's catalog-seeding docs for source selection and tuning. The output must contain manifest.json. The seed set is organized as:

seed/<version>/global/...   planet-wide backbone (countries, cities, curated)
seed/<version>/local/...    region-scoped supplements
seed/<version>/manifest.json

<version> is an operator-chosen tag: a date (2026-07-17) or a git sha. It names an immutable prefix in the bucket.

2. Publish

scripts/seed-publish.sh validates the local directory and uploads it under seed/<version>/:

./scripts/seed-publish.sh --version 2026-07-17 --dir ./seed-out

It refuses to publish when:

  • manifest.json is missing from the directory.
  • catalogseeder validate fails (only when the binary is on PATH; skipped with a warning otherwise).
  • seed/<version>/ already exists, unless you pass --force. Prefer a fresh --version over --force so published versions stay immutable.

The bucket defaults to tomoda-location-seed-<project> from the active gcloud project; override with --bucket.

3. Load (on demand, dev first)

The load is a Kubernetes Job, not a CronJob. It is never scheduled and never auto-applied: the manifests under k8s/apps/tomoda/location-seed-load/ are a template that is not referenced by any Argo CD Application, so Argo leaves them alone. An operator renders and applies a run by hand.

The Job's initContainer pulls gs://<bucket>/seed/<version> into an emptyDir; the main container runs /app/catalogseeder validate --from /seed then /app/catalogseeder load --from /seed --city-highlights <N> against the env's Postgres, with DB creds from that env's backend-secrets-<env> + backend-config-<env>.

Two knobs per run: the seed version and city-highlights N (top-N backbone attractions eager-promoted per city; 0 skips the pass).

Render, set the knobs, apply

Build the dev overlay, substitute the version + N, and apply with a unique name (timestamp keeps successive runs distinct):

TS=$(date -u +%Y%m%d-%H%M%S)
SEED_VERSION=2026-07-17
CITY_HIGHLIGHTS=25

kubectl kustomize k8s/apps/tomoda/location-seed-load/overlays/dev \
  | sed \
      -e "s/REPLACE_ME_SEED_VERSION/${SEED_VERSION}/" \
      -e "s/REPLACE_ME_CITY_HIGHLIGHTS/${CITY_HIGHLIGHTS}/" \
      -e "s/name: location-seed-load$/name: location-seed-load-${TS}/" \
  | kubectl apply -f -

This is the one sanctioned direct kubectl apply

The GitOps rule forbids kubectl apply for Argo-managed resources. This Job is deliberately outside Argo, so applying it by hand is the intended trigger, not a bypass. The manifests carry no schedule and no Argo Application.

Watch it:

kubectl -n tomoda-dev get job -l app=location-seed-load
kubectl -n tomoda-dev logs -f job/location-seed-load-${TS}

The Job cleans itself up 48h after finishing (ttlSecondsAfterFinished). backoffLimit: 0 means a failed load is inspected and re-run by hand rather than retried blindly.

4. Load prod

Repeat step 3 against the prod overlay once the dev load looks right. Prod targets tomoda_prod via postgres-prod and pins an immutable image tag rather than :latest:

TS=$(date -u +%Y%m%d-%H%M%S)
kubectl kustomize k8s/apps/tomoda/location-seed-load/overlays/prod \
  | sed \
      -e "s/REPLACE_ME_SEED_VERSION/${SEED_VERSION}/" \
      -e "s/REPLACE_ME_CITY_HIGHLIGHTS/${CITY_HIGHLIGHTS}/" \
      -e "s/name: location-seed-load$/name: location-seed-load-${TS}/" \
  | kubectl apply -f -

Verify after a load

  • Job succeeded: kubectl -n tomoda-<env> get job -l app=location-seed-load shows COMPLETIONS 1/1.
  • Logs: the loader prints per-collection inserted/skipped/upgraded counts and, when --city-highlights > 0, a promoted= line.
  • Row counts (read-only): connect via pgAdmin or psql and check SELECT count(*) FROM locations; and SELECT count(*) FROM countries; moved as expected. The load is checksum-gated per collection (seed_load_state), so re-running the same version is a near no-op.

Image

The loader ships inside the tomoda-backend image as a second binary at /app/catalogseeder (no dedicated image; backend/data is go:embed'd, so no data mount). Each overlay overrides the base placeholder IMAGE_PLACEHOLDER_CATALOGSEEDER to its env's …/tomoda-{dev,prod}-repo/tomoda-backend, and the Job command invokes /app/catalogseeder validate --from /seed then /app/catalogseeder load …. Dev tracks :latest; prod pins an immutable tag per run so a load is reproducible.

Deletion protection

The seed bucket, the two data buckets (tomoda-db-backups-*, tomoda-observability-*), and the CloudFront-signing secrets carry lifecycle { prevent_destroy = true } in Terraform. A terraform destroy or a resource removal that would delete one of them now fails the plan. Removing a protected resource is a two-step, deliberate act: delete the prevent_destroy block in one commit, then destroy in another. This guards the source-of-truth data (seed sets, backups, trace/log chunks) and the signing keys against an accidental destroy. See IAM for the loader Workload Identity binding and Secrets Management for the protected secrets.