Skip to content

Catalog Seeding

Purpose

How the global place catalog (countries, regions, cities, notable places and curated lists such as UNESCO World Heritage, Michelin and the "50 Best" family) is generated from open data and loaded into locations. This is the cold-start data layer: it fills the map, search and discovery surfaces before any user content exists. Seeded rows are ordinary Location rows and flow through the same read paths as resolve-minted places.

Mental model

Build offline, then load. Checksummed files are the handoff, and everything a row needs is baked into the file before load, so loading is a straight insert:

Wikidata · GeoNames · UNESCO DataHub
→
catalogseedergenerate → enrich → validate
→
NDJSON + manifestsha256, in R2
→
loaderupsert, on-demand Job
→
locations + collections
Generation is offline and occasional; enrich resolves geography and 28-locale localizations against Photon at build time; a validate gate blocks an invalid set from being published; loading is a manual, checksum-driven, idempotent Job.
Stage Command Does
Generate catalogseeder --source <name\|all> / --tier <tier> Fetches each source, normalizes to SeedPlace, writes one NDJSON per collection plus manifest.json (format version 2, per-file sha256, rows, source, license)
Merge (optional) catalogseeder merge --from <dir> Replaces wikidata-notable-places with the same rows as one pois entry (tier density) and folds UNESCO sites onto matching places. A set carries the notable layer as wikidata-notable-places or as pois, never both
Enrich catalogseeder enrich --from <dir> Photon reverse-geocode across 28 locales, canonical country, Serper address and photo for the prestige lists, side files for the spare Serper results
Backfill country catalogseeder backfill-country --from <dir> Point-in-polygon country / country_code for enriched rows Photon left blank
Validate catalogseeder validate --from <dir> Read-only gate: manifest version, sha256, row counts, providers, enrichment invariants, side files, cover manifest and image refs
Republish catalogseeder republish --from <manifest> --out <manifest> Turns an existing version's manifest into a current one without regenerating data
Load catalogseeder load --from <dir> Checkpointed insert through the ingest funnel, then geo covers, then the post-load task chain
Geo covers catalogseeder geocover ..., load --geo-covers-only Cover photo generation and the independent cover load

Dataset inventory

One table, so refreshing the numbers is a single edit. Current published version: seed/2026-11-08/.

File Tier Rows Size Notes
world-countries reference 258 4.9 MB ISO 3166-1 plus boundary polygons
world-admin-areas reference 51,414 46 MB GeoNames ADM1 and ADM2
world-cities gazetteer 170,342 136 MB GeoNames settlements
wikidata-notable-places curated 583,065 602 MB Top tracks only (per-city 15, per-country 250, global 100k); no long tail
geonames-natural-places curated 226,364 241 MB Physical features with a Wikidata QID, absent from notable
unesco-world-heritage curated 1,272 2.2 MB
michelin-guide curated 7,709 16 MB Curated rows are enriched, Google refs on about 75 to 87%
9 "50 Best" and vineyard lists curated about 100 each small
serper-side-pois, serper-spend side per enrich run small Present only when the enrich pass paid for Serper
curated-photos.ndjson not a collection 9,252 5.1 MB Maps place_id or seed_ref to a captured photo
place-photos/ not a collection 17,164 objects 6.3 GB Curated and enrich-captured photos, plus place-photos/geo/ cover images
geo-covers.ndjson not a collection 8,119 3.1 MB Cover manifest

Every manifest entry carries a tier: reference (countries, admin areas), gazetteer (cities), curated (browsable lists and the notable overlay), side (spare Serper results and spend log) and the retired density (the merged pois set; the Overture-based variant is retired and archived). The loader orders reference, gazetteer, curated, density, side so city ids resolve before the places that use them. curated-photos.ndjson, place-photos/ and the cover manifest are read by the loader but are not manifest entries.

R2 layout

Bucket tomoda-location-seed, four independent trees:

Prefix Holds Mutable
seed/<version>/ A generated place set: manifest.json, NDJSON, place-photos/, curated-photos.ndjson, optionally a cover manifest No
geo-covers/<set>/ A cover set: geo-covers.ndjson (or an approved manifest) plus place-photos/geo/<kind>/<key>.jpg No
simulation/<version>/ The small bundle behind SIM=1 (below) No
seed-archive/ Retired data kept for reference, never loaded: the Overture source and the Overture density file (pois.ndjson, 2.82M rows) pulled from each seed version and simulation bundle No

Covers live outside seed versions so a new cover set ships without republishing the place set, and a place set ships without covers. Versions and sets are immutable: a change is a new directory, never an overwrite.

Operating the pipeline

Generation is offline, and loading is always a manual, on-demand Job, never a cron. A refresh republishes files and runs the load by hand, with no deploy. The publish and load Jobs live in the DevOps repo; the cluster-side runbook (R2 bucket, the Job overlays for dev and prod) is DevOps → Loading the location seed set.

task db:seed:build chains merge, dedupe, enrich, backfill-country and validate over a seed-out/ dir (it needs PHOTON_URL, for example kubectl port-forward -n platform svc/photon 2322:2322).

Generate

cd backend
go run ./cmd/catalogseeder --source unesco --out ./seed-out      # one collection
go run ./cmd/catalogseeder --tier curated --out ./seed-out       # a whole tier
go run ./cmd/catalogseeder --source all   --out ./seed-out       # every regular source

Generating one source at a time accumulates into one manifest. A manifest that already lists entries from an older generator keeps its old version, so it stays refused by the validate gate and the loader until regenerated or republished.

notable-dump is an explicit source: --source all and --tier never select it. It streams the ~95 GB Wikidata JSON dump offline and keeps only the top tracks, so no long tail reaches the set:

go run ./cmd/catalogseeder --source notable-dump --dump-path ~/Downloads/latest-all.json.bz2 \
  --closure-cache ./notable-closure.json --out ./seed-out
Flag Default Purpose
--dump-path required The dump (latest-all.json.bz2 or a plain .json)
--closure-cache none Caches the P279* class closure; built from WDQS once
--per-city 15 Top-N per city (P131)
--per-country 250 Per-country floor (P17) so thin countries still get coverage
--per-cat-per-city 2 Max of one category per city (0 = uncapped)
--global-top 100000 Global prominence track, by sitelinks

It writes the same wikidata-notable-places collection as the WDQS notable source (selectNotable in source/notable/notable_dump_select.go bounds the keep-set by the three tracks). Running --source all regenerates that file through WDQS and replaces a dump-built one, so run the dump last.

Landmark classes. A skyscraper, office building, town square, public building, building complex or great wall is none of the core classes, so the first dump matched no root for Taipei 101, the Empire State Building or Red Square. landmarkClasses in source/notable/notable.go adds them after every core class (so they only claim places nothing else did), each with a sitelink floor of 25 to 60 well above the dump's global-track cutoff. A full dump run picks them up. To extend an existing notable file without the dump, use the explicit notable-topup source: it keeps every base row and appends the landmark classes from WDQS (read-only, paced), skipping places the base holds:

go run ./cmd/catalogseeder --source notable-topup --topup-base ./seed/wikidata-notable-places.ndjson --out ./seed-out

--topup-gap-floor N also re-queries the core classes at that sitelink floor. A coverage fixture of about 130 world landmarks lives in notable_landmarks_test.go; run it against a seed dir with NOTABLE_COVERAGE_DIR=<dir> go test ./cmd/catalogseeder/source/notable -run Coverage -v.

Enrich, backfill, validate, publish

export PHOTON_URL=http://localhost:2322
export SERPER_API_KEY=...        # optional, paid, prestige lists only
go run ./cmd/catalogseeder enrich           --from ./seed-out --workers 16
go run ./cmd/catalogseeder backfill-country --from ./seed-out
go run ./cmd/catalogseeder validate         --from ./seed-out

Enrich is parallel, cached and resumable (see Build-time enrichment). Each paid prestige /maps call returns about 20 other places; enrich keeps them as the side files serper-side-pois.ndjson and serper-spend.ndjson, which backfill-country leaves untouched.

Validate prints a per-collection PASS/FAIL summary and exits non-zero on any failure. It rejects a manifest older than format version 2, and checks every file's sha256 and row count, that every primary provider is one the loader maps, the enrichment invariants (an enriched row carries a country; every coordinate-bearing pois row is enriched), the side files, and the cover manifest (known kind, unique key, image exists under the dir). It touches no database and makes no network calls.

Publish uploads the validated directory to seed/<version>/ with the DevOps seed-publish.sh, which re-runs validate and refuses to overwrite a version.

Republish an existing version

When the data is good but the manifest is outdated (no format version, a retired entry), build a new version without regenerating anything. Data files are copied inside the bucket and only the manifest is rewritten, so every per-file sha256 is kept:

rclone copy --server-side-across-configs r2:tomoda-location-seed/seed/<old> r2:tomoda-location-seed/seed/<new> \
  --exclude /manifest.json --exclude /pois.ndjson
rclone copyto r2:tomoda-location-seed/seed/<old>/manifest.json /tmp/manifest.old.json
go run ./cmd/catalogseeder republish --from /tmp/manifest.old.json --out /tmp/manifest.json
rclone copyto /tmp/manifest.json r2:tomoda-location-seed/seed/<new>/manifest.json

Rows whose primary provider is retired (an old UNESCO merge picked an overture match) are repaired with --repair-from <local copy of the version> --repair-out <empty dir>: the first known ref is promoted to primary (wikidata from supersedes, then from secondary, then any other known provider; google_places is never a primary), retired refs are dropped everywhere, and a row with nothing to promote fails the run. Only the changed files are written to --repair-out, with the manifest carrying their new sha256 and row count; upload those files plus the manifest over the copied version.

republish drops the retired entries (tier density, slug pois), stamps format version 2 and logs which files not to copy. It refuses to drop pois when the manifest has no wikidata-notable-places, since that would remove the notable layer. Pull the new version and run validate before the first load. The DevOps seed-publish.sh flow wraps these steps.

Add --prune-orphan-photos --photos-from <local copy> --photos-out <dir> to drop the curated photos that no seed row refers to, by Google place id or by seed ref. It writes a new curated-photos.ndjson and orphan-photos.txt (the object paths under place-photos/) into --photos-out and deletes nothing: upload the file over the version and move the listed objects to the archive. curated-photos.ndjson is not a manifest entry, so the manifest is unchanged.

Load

Target How
Dev, then prod A Kubernetes Job (not a CronJob) applied by hand per environment. It pulls seed/<version>, runs catalogseeder validate, then catalogseeder load --from /seed --city-highlights <N> against that environment's Postgres. The binary ships in the tomoda-backend image at /app/catalogseeder. Re-running the same version is a no-op
Local task db:seed:catalog (below)
Covers only catalogseeder load --geo-covers-only --from <covers dir> (Geo covers)

The loader refuses a manifest older than format version 2 and fails any place file with a provider it does not know (for example the retired overture) before writing a row; the provider scan covers every file first. Secondary and superseded ids from retired providers on otherwise valid rows are dropped, so no overture row reaches location_providers.

Local dev. task db:seed:catalog pulls the latest (or SEED_VERSION=) set from r2:tomoda-location-seed/seed/<version>/ into local Postgres, validates it, and loads it. It uses an rclone remote named r2 when configured; otherwise it builds r2 from the tomoda-r2-cluster-credentials Secret Manager token (needs gcloud auth login).

Mode Command Loads
lite (default) task db:seed:catalog Everything except the heavy collections: pois, wikidata-notable-places, geonames-natural-places, and the geo cover manifest and photos
full FULL=1 task db:seed:catalog The complete set
sim SIM=1 task db:seed:catalog The simulation bundle
covers COVERS=<set> task db:seed:catalog After the seed, pulls geo-covers/<set>/ and loads it alone with --geo-covers-only

Downloads are cached in .seed-cache/<version>-<mode>/ and marked complete only after the whole download finishes. One list of excluded slugs drives the download filter, the manifest prune and the cache cleanup. Each run removes older caches of the same mode.

Simulation bundle

backend/scripts/seed/build-sim-seed.py builds simulation/<version>/ from a full seed cache plus the cover set, for local dev and task test:simulation (SIM=1):

  • full countries and regions;
  • top cities by population;
  • notable and natural places scoped to a radius around the top metros (the retired pois is never emitted);
  • covers for all countries, every ADM1 region (matched by GeoNames feature code, so both US-CA and geonames_<id> keys count) and every kept city the cover set has, downscaled to about 512px;
  • curated and prestige place photos, compressed;
  • a version 2 manifest listing only real collections.

backend/scripts/seed/gen-sim-hotspots.py regenerates the major-city list the cover scope derives from. Rebuild both when the seed refreshes, then upload to simulation/<version>/.

Load design

  • Chunks and checkpoints. streamFile reads an NDJSON file in chunks and hands each to the one write path, Ingest (a transaction per chunk). After a chunk, seed_load_state stores (slug, sha256, version, next_row, complete). The same checksum left incomplete resumes at next_row; a changed checksum restarts at row 0; a complete one is skipped (--force reloads). A crash between a chunk's commit and its checkpoint replays that chunk, which reconciles by source ref.
  • Curated lists. Pruning memberships needs every row of the file, so a resumed load first resolves the earlier chunks' refs to place ids, then prunes after the last chunk.
  • Per-chunk round trips. Everything a curated file does around the ingest is batched per chunk: the misattached-ref check, the membership upsert and the harvested-row promotion lookup each run as one statement per 3,000 rows.
  • No embedding during load. The post-load embed task computes vectors through the embedding gate. The Job logs the row count per provenance and how many rows are embed-eligible.
  • Side files. serper-side-pois goes through the harvest candidate gates (focus, quality, distance from the call centre) as harvested rows with origin serper_enrich, so they never embed. serper-spend is cumulative: a load from row 0 replaces the earlier trigger = 'seed' rows in location_provider_spend in the same transaction as its first chunk, so a forced reload or a new version cannot double-count.
  • Photos. thumbAttacher rehosts photos under places-images/ (see Curated photos). A store failure is logged once and photos are skipped for the rest of the run; after fixing the store run load --reattach-thumbnails --from <dir>, which re-attaches photos to loaded places without reloading rows.
  • Geo covers run after the locations in the same Job, or alone, through one code path (see Geo covers).

Post-load tasks

The Job does not run these itself: it enqueues the first step on the low queue and exits, and each step enqueues the next, paced. A worker with Redis must be running or the chain never progresses. A step already queued under the same cursor collapses, and the log says so when that happens.

Step Does
link_cities Sets city_id on every non-City place to the nearest city within 50 km, preferring the same country (city_id linking). Four id-range shards run in parallel on the low queue, each paging by cursor; partial GiST indexes on City rows serve the lookup
highlights Per-city top-N promotion from --city-highlights N (0 skips), ranked within each city, so it runs after linking. It pages by city id, 300 cities per task, each page enqueuing the next, and queues enrichment only for the places that page actually promoted, so the fan-out is paced by the low queue. Idempotent: a retried or re-run page promotes nothing twice
embed The same four shards embed the rows the embedding gate lets through: curated provenances (wikidata_seed, unesco_seed, promoted; natural features are not curated) and any row with prominence at or above 0.35 (0.57 for GeoNames cities, population about 15,000) or user engagement
enrich Fan-out of the runtime enricher for seed rows that were not pre-enriched

Sources and tiers

Every source declares a tier; a run targets one source or a whole tier (--source <name>|all, --tier <tier>).

Tier Source License Contents
reference Wikidata (SPARQL) CC0 countries (ISO 3166-1)
reference GeoNames (downloaded dump) CC BY 4.0 administrative areas (states/provinces, counties/districts)
gazetteer GeoNames (downloaded dump) CC BY 4.0 cities / settlements by population
curated Wikidata + UNESCO DataHub CC0 UNESCO World Heritage
curated Michelin dataset (ngshiheng/michelin-my-maps) + Wikidata MIT (dataset); awards © Michelin, facts only Michelin Guide restaurants (stars, Bib Gourmand, Green Star)
curated theworlds50best.com + Wikidata (seed); enriched by the shared load-time path editorial ranking, facts only the "50 Best" family: World's / Asia's / Latin America's / North America's / MENA restaurants, World's / Asia's / North America's bars, World's hotels, World's Best Vineyards
curated Wikidata (SPARQL) CC0 notable places overlay via WDQS (notable, ~80K): museums, landmarks, parks, monuments, temples with rich multilingual labels
curated Wikidata JSON dump (latest-all.json.bz2) CC0 notable places from the offline dump (notable-dump, explicit source): per-city 15, per-country 250 and a global top 100k by sitelinks, about 583k rows, no long tail
curated GeoNames (downloaded dump) + Wikidata CC BY 4.0 + CC0 natural attractions (geonames-natural-places): physical features (mountains, lakes, waterfalls, islands, parks) carrying a Wikidata QID and absent from the notable set

Wikidata is the backbone: it supplies the curated + reference tiers and is the identity hub. GeoNames is a downloadable file (enumerated locally, avoiding query timeouts) for the high-cardinality gazetteer.

Administrative hierarchy

The regions source fills the mid-tier between country and city, so the area tree can be reconstructed for gazetteer recentering, not just display. It reads the GeoNames allCountries dump for feature class A, feature codes ADM1 (states/provinces), ADM2 (counties/regions/districts), and ADM3 (districts in the countries that model them). Each row becomes a geonames:<geonameid> SeedPlace with category administrative_area_level_1|2|3, its GeoNames population, and 28-language localizations folded in from alternateNamesV2 (the same path cities use). Hierarchy linkage lives in metadata: geonames_feature_code plus admin1_code (and admin2_code for an ADM2), which combined with the first-class country_code locate an area within its parent. Generate with --source regions; a full run is ~220k areas.

Boundary polygons. The countries source attaches a GeoNames shapes_all_low boundary (a GeoJSON MultiPolygon) to each country, joined through countryInfo.txt's ISO to geonameid map. The loader writes it to the boundary geometry(MultiPolygon,4326) column, which is write-only: no query selects it (sqlc maps geometry to a string it never reads), it exists for ST_Contains point-in-polygon country assignment. GeoNames shapes_all_low is country-level only, so ADM1/ADM2 boundaries are not yet populated: the regions.boundary column and its write path are ready for an admin polygon source (geoBoundaries CGAZ) when that ingester ships; until then admin assignment falls back to nearest-centroid. A boundary fetch failure is non-fatal, so it never blocks the gazetteer itself.

GeoNames cross-references. Cities and admin areas carry more than their own name from the same GeoNames pass. Both attach the IANA timezone and (from alternateNamesV2) a wikidata_qid in metadata plus a wikidata Secondary, so a settlement or area dedups exactly against the notable set on QID. Cities additionally carry iata / icao airport codes in metadata. Countries carry capital, currency_code, calling_code, languages, and area_sqkm from countryInfo.txt (LoadCountryFacts in backend/cmd/catalogseeder/source/gazetteer/boundaries.go); continent is still derived by the loader from the ISO code, not stored on the seed row.

Natural attractions

geonames-natural-places (--source natural) emits GeoNames physical features (mountains, volcanoes, lakes, waterfalls, glaciers, islands, canyons, parks, nature reserves, and the rest) that carry a Wikidata QID and are not already in the notable set. Identity is the Wikidata QID, so a row dedups exactly against notable on QID; the GeoNames id rides as a geonames Secondary. It runs after notable in the same seed dir, reading the notable file's QIDs to skip anything already covered. FetchNatural in backend/cmd/catalogseeder/source/gazetteer/natural.go keeps only whitelisted attraction feature codes (featureCategory, mapping each code to a display category); minor landforms (hills, points, ridges) are dropped as the tail a Wikidata id alone does not make notable. Mountains and peaks (MT/PK/PKS) are elevation-gated to prominent summits (>= mountainMinElevation, 2000m), the famous low peaks already arriving through the sitelink-ranked notable set. 28-locale localizations come from alternateNamesV2; the row then flows through the same Photon enrich (region / city / country / address + 28 locales) as notable.

Identity, dedup, provenance

  • Canonical identity is the (provider, provider_id) pair on Location: wikidata:Q… for curated/reference + the notable overlay, geonames:<geonameid> for cities, photon:{type}:{id} for OSM, unesco:<id_no> for World Heritage sites Wikidata lacks. Cross-provider ids for the same place live in secondary_providers.
  • Dedup ladder (before every insert): canonical pair → secondary_providers cross-id → shared Wikidata QID → spatial + fuzzy-name proximity. A match merges (appends the id, keeps user/admin/enriched fields) rather than inserting a twin.
  • provenance records how a row's data was sourced (wikidata_seed, natural_seed, geonames_seed, osm_seed, unesco_seed, provider_api, serper_maps, harvested) and drives merge precedence: user/admin edits and provider-API enrichment outrank a re-seed. harvested rows (machine-harvested on demand) are excluded from paid enrichment.
  • prominence (0..1) is a popularity prior the ranker boosts on, derived from raw signals (Wikidata sitelinks, then city population). It stands in for user engagement until that accrues.
  • metadata (jsonb) holds place-intrinsic source extras with no first-class column (e.g. a city's GeoNames feature code). Collection-specific facts live on the collection membership, not here.

city_id linking (nearest city)

Every non-City POI references its city through city_id, a self-FK to the city's own location row (cities are locations). It is assigned in one set-based pass after the whole dataset is loaded (a worker task the load Job enqueues), not per row during ingest, so a planet-scale load is a bounded sweep of UPDATE statements instead of tens of millions of round-trips.

The pass walks POIs by primary-key cursor in chunks (cityLinkChunkSize) so no single statement locks or bloats the table. Each chunk runs a LATERAL nearest-city join: for each still-unlinked POI it picks the nearest City within 50km, preferring the same country_code (an empty POI country_code matches any country). The KNN order-by (coordinates <-> …) is served by the GiST idx_coordinates index, so it is an index scan, not a per-city sequential scan over the backbone. Advancing by id rather than by "rows still NULL" keeps the loop moving past POIs with no city within 50km. It is idempotent: only NULL city_id rows are touched, so a re-load leaves already-linked rows unchanged.

city_id is what read-time card assembly follows for the localized city name (see Localization) and what the per-city highlight promotion pass ranks within, so the linking sweep always runs before promotion.

Localization

Localization is first-class: every place displays fully localized (name + city + country + region) in all supported languages, and is searchable by a localized brief, the rule is store each localized geography name once, reference it, and assemble at read:

  • A location stores its own name localizations (Location.Localizations, a BCP-47 map) — unique per place, unavoidable, small.
  • Shared geography is referenced, not copied: city_id (a self-reference to the city's own location row — cities are locations), region_id, and country_code. Each localized geography name is stored once, in the gazetteer.
  • Read assembly resolves a card for the viewer's locale from the row's own localizations plus the referenced city/country/region names. Countries (a small fixed set) are cached in memory; cities are resolved by batched lookup.

Localizations start from each source's native multilingual data (Wikidata labels for curated + reference, GeoNames alternateNamesV2 for cities) and are then completed by the build-time enrich stage, which reverse-geocodes each POI against Photon across all 28 supported locales to fill the name, address, city, and country a source did not cover. The published rows are therefore fully localized before load.

Timezone is not a localization or a stored field — it is derived from coordinates. See Timezones.

Data model

Seeded rows populate locations (see Locations for the full model); the seeding-relevant columns are provider / provider_id, secondary_providers, provenance, prominence, localizations, metadata, is_active, and the geography references. Curated lists add location_collections + collection_memberships.

Invariants & gotchas

  • Seeded rows are indistinguishable to the app from resolve-minted rows except by provenance and is_active; they use the same search/map/discovery reads.
  • is_active gates map + search visibility. Seeded rows may stage inactive and flip active after a coverage check.
  • Reconciliation deactivates, never deletes. A place dropped from a source flips inactive or loses a membership; its history and attached moments survive.
  • License is tracked per collection, never per row (location_collections carries it); required attribution is surfaced once at the app level.
  • Names are cleaned at ingest with the same isMintablePlaceName gate the mint path uses, so seed data does not pollute search or the trigram index.