Skip to content

Catalog Seeding

Purpose

How the global place catalog — countries, cities, and curated lists like UNESCO World Heritage — is generated from open data and loaded into locations. This is the cold-start data layer: it fills the map, search, and discovery surfaces before any user-generated content exists. Seeded rows are ordinary Location rows and flow through the same read paths as resolve-minted places.

Mental model

Two stages, with checksummed files as the handoff:

Overture · Wikidata · GeoNames · UNESCO DataHub
→
catalogseedergenerate → validate
→
NDJSON + manifestsha256, in GCS
→
loaderupsert, on-demand Job
→
locations + collections
Generation is offline and occasional; a validate gate blocks a bad set from ever being published or loaded; loading is a manual, checksum-driven, idempotent Job. Republishing data needs no deploy.
  • Generate — backend/cmd/catalogseeder fetches each source, normalizes to a common SeedPlace, deduplicates within the set, and writes one NDJSON file per collection plus a manifest.json carrying each file's sha256, row count, source, and license.
  • Validate — catalogseeder validate --from <dir> proves the generated set is complete and solid before it is ever published or loaded (see Operating the pipeline). It is read-only (no DB, no network) and exits non-zero on any problem, so it gates the publish and load steps.
  • Load — reads the manifest, and for any file whose checksum changed since the last load, streams it through the dedup ladder and a provenance-precedence upsert into Postgres, then records the new checksum. Once every file is loaded, a single set-based pass links each non-City POI to its nearest city (city_id), so city-linking is one chunked, index-served UPDATE sweep rather than a per-row query in the ingest loop (see city_id linking). catalogseeder load --city-highlights N then runs the per-city top-N attraction promotion pass (see Locations → Per-city highlight promotion), so each city's marquee sights are eager-enriched once the density backbone is in place. Omit the flag (default 0) to skip it. The promotion pass ranks per city_id, so it always runs after the city-linking sweep.

Operating the pipeline

Generate, validate, publish, load. The four stages are decoupled by the checksummed files: generation is offline, and loading is always a manual, on-demand Job, never a cron. A refresh republishes files and runs the load Job by hand; it needs no deploy. The publish and load stages live in the DevOps repo; this repo owns generate and validate. The full cluster-side runbook (GCS bucket, loader service account, the Job overlays for dev and prod) is DevOps → Loading the location seed set.

1. Generate

Produce a seed set into a local directory. Target one source, or a whole tier, or everything:

cd backend
go run ./cmd/catalogseeder --source unesco --out ./seed-out          # one collection
go run ./cmd/catalogseeder --tier curated --out ./seed-out           # a whole tier
go run ./cmd/catalogseeder --source all   --out ./seed-out           # everything

Each run writes <slug>.ndjson per collection and folds an entry into seed-out/manifest.json (slug, file, rows, sha256, source, license, generated_at). Generating one source at a time accumulates into the single manifest, so a set can be built incrementally. Source-specific tuning (--overture-bbox, --notable-min-sitelinks, --michelin-no-backing, and the rest) is in Sources and tiers.

notable-dump is an operator-run source, not part of --source all: it streams a ~95 GB Wikidata JSON dump offline (hours) rather than hitting WDQS, so it is run on its own when the notable backbone needs a refresh:

go run ./cmd/catalogseeder --source notable-dump \
  --dump-path ~/Downloads/latest-all.json.bz2 \
  --closure-cache ./notable-closure.json \
  --out ./seed-out
Flag Default Purpose
--dump-path (required) Path to the Wikidata dump (latest-all.json.bz2 or a plain .json)
--closure-cache (none) File to cache the P279* class closure; loaded if present, else built from WDQS once and written here
--per-city 12 Keep the top-N notable places per city (P131)
--global-top 100000 Size of the global prominence track (by sitelinks)
--max-records 5000000 Hard ceiling on emitted rows

It writes wikidata-notable-places.ndjson (a recent run emitted ~677k rows). The overture density source is likewise operator-run and currently inactive (the notable backbone comes from notable-dump); regenerate it only for an archival refresh.

2. Validate

Prove the set is complete and solid before it leaves the machine:

go run ./cmd/catalogseeder validate --from ./seed-out

For every manifest entry it checks: the file exists, its recomputed sha256 matches the manifest, its line count equals rows, and a cheap per-source sanity pass (every row parses as JSON, has a non-empty name and canonical identity, and carries coordinates where the source promises them, the "50 Best" rankings being the one coordless-by-design exemption). It prints a per-collection PASS/FAIL summary and exits non-zero on any failure, so it gates the steps that follow. It touches no database and makes no network calls.

3. Publish

Upload the validated directory to the seed bucket under an immutable seed/<version>/ prefix (<version> is a date or git sha the operator picks). The DevOps seed-publish.sh script re-runs validate and refuses to overwrite an existing version:

./scripts/seed-publish.sh --version 2026-07-17 --dir ./seed-out   # in the DevOps repo

4. Load (on demand, dev then prod)

Loading is a Kubernetes Job, not a CronJob: it is applied by hand per environment, dev first, then prod once dev looks right. The Job pulls seed/<version> from the bucket, runs catalogseeder validate then catalogseeder load --from /seed --city-highlights <N> against that environment's Postgres. The load is checksum-gated per collection (seed_load_state), so re-running the same version is a near no-op. See the DevOps runbook above for the render-and-apply commands and the two per-run knobs (seed version, --city-highlights N).

The loader binary ships inside the tomoda-backend image at /app/catalogseeder (a static, pure-Go build; backend/data is go:embed'd, so no runtime data mount), so the Job runs that image with the entrypoint overridden rather than pulling a second image.

Local dev skips the Job: task db:seed:catalog pulls the latest (or SEED_VERSION=) set from gs://tomoda-location-seed-<project>/seed/<version>/ into local Postgres (needs gcloud auth login), excluding the ~2 GB overture-places file so a laptop load stays quick. task test:simulation runs it after the schema reset. Versions are immutable, so db:seed:catalog always gets exactly what was published.

Sources and tiers

Every source declares a tier; a run targets one source or a whole tier (--source <name>|all, --tier <tier>).

Tier Source License Contents
reference Wikidata (SPARQL) CC0 countries (ISO 3166-1)
reference GeoNames (downloaded dump) CC BY 4.0 administrative areas (states/provinces, counties/districts)
gazetteer GeoNames (downloaded dump) CC BY 4.0 cities / settlements by population
curated Wikidata + UNESCO DataHub CC0 UNESCO World Heritage
curated Michelin dataset (ngshiheng/michelin-my-maps) + Wikidata MIT (dataset); awards © Michelin, facts only Michelin Guide restaurants (stars, Bib Gourmand, Green Star)
curated theworlds50best.com + Wikidata (seed); enriched by the shared load-time path editorial ranking, facts only the "50 Best" family: World's / Asia's / Latin America's / North America's / MENA restaurants, World's / Asia's / North America's bars, World's hotels, World's Best Vineyards
curated Wikidata (SPARQL) CC0 notable places overlay via WDQS (notable, ~80K): museums, landmarks, parks, monuments, temples with rich multilingual labels
curated Wikidata JSON dump (latest-all.json.bz2) CC0 notable places at scale (notable-dump, up to ~5M): top-N per city worldwide + a global top ~100k by sitelinks. The offline dump avoids WDQS result caps; a one-off operator run streams it
density Overture Maps Places (GeoParquet on S3, via DuckDB) CDLA Permissive 2.0 the overture source is retained but inactive: the notable backbone comes from Wikidata (notable-dump), not Overture

Wikidata is the backbone: it supplies the curated + reference tiers and is the identity hub. GeoNames is a downloadable file (enumerated locally, avoiding query timeouts) for the high-cardinality gazetteer.

Administrative hierarchy

The regions source fills the mid-tier between country and city, so the area tree can be reconstructed for gazetteer recentering, not just display. It reads the GeoNames allCountries dump for feature class A, feature codes ADM1 (states/provinces), ADM2 (counties/regions/districts), and ADM3 (districts in the countries that model them). Each row becomes a geonames:<geonameid> SeedPlace with category administrative_area_level_1|2|3, its GeoNames population, and 28-language localizations folded in from alternateNamesV2 (the same path cities use). Hierarchy linkage lives in metadata: geonames_feature_code plus admin1_code (and admin2_code for an ADM2), which combined with the first-class country_code locate an area within its parent. Generate with --source regions; a full run is ~220k areas.

Identity, dedup, provenance

  • Canonical identity is the (provider, provider_id) pair on Location: wikidata:Q… for curated/reference + the notable overlay, overture:<gers-id> for the bulk backbone, geonames:<geonameid> for cities, photon:{type}:{id} for OSM, unesco:<id_no> for World Heritage sites Wikidata lacks. Cross-provider ids for the same place live in secondary_providers.
  • Dedup ladder (before every insert): canonical pair → secondary_providers cross-id → shared Wikidata QID → spatial + fuzzy-name proximity. A match merges (appends the id, keeps user/admin/enriched fields) rather than inserting a twin.
  • provenance records how a row's data was sourced (wikidata_seed, geonames_seed, osm_seed, unesco_seed, provider_api, serper_maps, bulk_import) and drives merge precedence: user/admin edits and provider-API enrichment outrank a re-seed. bulk_import also excludes the density backbone from paid enrichment.
  • prominence (0..1) is a popularity prior the ranker boosts on, derived from raw signals (Wikidata sitelinks, city population). It stands in for user engagement until that accrues.
  • metadata (jsonb) holds place-intrinsic source extras with no first-class column (e.g. a city's GeoNames feature code). Collection-specific facts live on the collection membership, not here.

city_id linking (nearest city)

Every non-City POI references its city through city_id, a self-FK to the city's own location row (cities are locations). The loader assigns it in one set-based pass after the whole dataset is loaded, not per row during ingest, so a planet-scale load is a bounded sweep of UPDATE statements instead of tens of millions of round-trips.

The pass walks POIs by primary-key cursor in chunks (cityLinkChunkSize) so no single statement locks or bloats the table. Each chunk runs a LATERAL nearest-city join: for each still-unlinked POI it picks the nearest City within 50km, preferring the same country_code (an empty POI country_code matches any country). The KNN order-by (coordinates <-> …) is served by the GiST idx_coordinates index, so it is an index scan, not a per-city sequential scan over the backbone. Advancing by id rather than by "rows still NULL" keeps the loop moving past POIs with no city within 50km. It is idempotent: only NULL city_id rows are touched, so a re-load leaves already-linked rows unchanged.

city_id is what read-time card assembly follows for the localized city name (see Localization) and what the per-city highlight promotion pass ranks within, so the linking sweep always runs before promotion.

Curated collections

A curated list is both a browsable filter and, optionally, an earnable stamp, off one seeded dataset. location_collections names the list; collection_memberships ties a location to it with the source's external_id, an edition, an is_current flag, and a distinction (e.g. UNESCO category + inscription year). Living collections reconcile by diff each release; editions are retained per the collection's retention policy.

UNESCO is fact-checked against UNESCO's own open-data portal (data.unesco.org, whose id_no equals Wikidata property P757): only authoritatively inscribed sites are kept, each enriched with the official category and inscription year, one row per site.

Prestige lists (Michelin, 50 Best)

The Michelin and "50 Best" sources are curated-tier lists meant to be dedup-merged onto backbone POIs by the ingest funnel, contributing prestige tags plus prominence. They are derivative datasets: we take facts only (name, award, rank, coords), never editorial descriptions or images.

  • Michelin consumes a maintained scrape of the guide (ngshiheng/michelin-my-maps, MIT). Awarded rows only (stars + Bib Gourmand; Selected Restaurants are skipped). Each row is provider = michelin, the guide URL slug its provider_id, category Restaurant. The award maps to a tag (michelin_1_star / michelin_2_star / michelin_3_star / bib_gourmand, plus michelin_green_star) and a michelin distinction block (stars, award, year, cuisine, price, url).
  • 50 Best scrapes the published William Reed rankings (name + city + rank) into one source per list, one per ranking in the family: restaurants (World's, Asia's, Latin America's, North America's, MENA), bars (World's, Asia's, North America's), hotels (World's), and vineyards (World's Best Vineyards). Each entry carries a ranking tag (e.g. worlds_50_best_restaurants, worlds_50_best_bars, worlds_50_best_hotels, worlds_best_vineyards), the list's category (Restaurant / Bar / Hotel / Vineyard), and a distinction block with rank + year. The 1-50 page embeds the full 1-100 depth, so a ranking with an extended list is captured to rank 100 in one fetch; rankings that publish only a top-50 (North America's restaurants, MENA) stop at 50. The vineyards ranking lives on its own domain that redirects into the same site structure, so one parser handles the whole family.

Michelin backs each row with Wikidata by coordinate proximity (the guide rows are address-precise), attaching the canonical QID as a secondary provider id plus multilingual labels and a sitelinks prominence signal. The whole backing step runs under a single WDQS budget (default 12 min) threaded as a context deadline through every WDQS call, so a stalled request is cancelled the moment the budget expires rather than hanging the run; rows past the budget (or once WDQS hard-throttles) keep their Michelin-native identity, the documented fallback. Pass --michelin-no-backing to skip the backing step entirely for a fast michelin-only seed when WDQS is degraded.

The "50 Best" seeder resolves coordinates at seed time against a planet Photon index, then hands the rest to the shared funnel. Prestige is a global set, so it geocodes each ranked row (name + city context) against the configured planet Photon (PHOTON_URL, falling back to the komoot public planet endpoint), behind a validation guardrail: it accepts a hit only when the name matches (accent-folded, token-level) and the coords fall within a metro radius of the expected city's own geocoded center, so a bare-name query cannot mis-geocode to a same-named place on another continent (a bare "Aman Venice" resolving to Amman, Jordan is rejected). On a miss or a rejected hit the row is dropped from the generated seed and logged, so a prestige collection ships only rows with real coordinates. Everything else, category, tags ([bucket facet, ranking tag]), collection membership, the rank + year distinction, goes through the common ingest funnel. There are three ingest paths (runtime search/create, capture URL into Places/Serper, and this batch seed) and they all converge on the same funnel Ingest and the same async enricher: any remaining coordinate gap-fill and all commercial detail are filled by the shared enricher.

Prestige coords are solid by construction. A prestige row that fails resolution is dropped at generation rather than emitted at the null island, so a loaded prestige collection has no lat=0/lng=0 rows to reconcile after the fact. Point PHOTON_URL at the in-cluster planet Photon when generating (the local Photon index is region-limited); the komoot public Photon is the fallback.

Localizations are the one thing the seeder adds itself, because Wikidata is a free additive data source rather than a resolution path:

  • Seed-time, from Wikidata. Each entry is matched to a Wikidata QID by name (city-disambiguated), and its labels across the full 28-language photonLanguages set populate Localizations (kept only when they differ from the base name). This attaches the QID as a wikidata secondary id.
  • Load-time, from Photon. Entries Wikidata cannot match arrive with thin localizations, so the async enrichment sweep's gapFillLocalizations reverse-geocodes their coordinates against the self-hosted Photon index across the app's SupportedLocalizationLanguages, bringing them up to the same coverage as any runtime-minted row.

If neither Wikidata nor Photon has a localization, the field stays empty.

Coordinates and commercial detail come from the one shared enricher (EnrichLocation in backend/internal/services/location/location_enricher.go), the same path a runtime row and a capture row take. It fires once per first-time-created location (enrichment_at NULL) and is two-tier and provider-agnostic:

  1. Tier 1, Photon gap-fill (cheap, self-hosted), for every row. A coordless row is forward-geocoded against Photon (photonForwardGeocode), so a seed row that loaded without coordinates gets them from name + city.
  2. Tier 2, paid commercial enrichment (Serper /maps, then the Google Places API fallback). Its gate is tagsAreRichDetail(loc.Tags) AND NOT isBulkImport(loc) AND NOT already enriched (hasFreshGoogleFields). The rich-detail category gate alone is not enough: the high-cardinality density backbone (the future millions of Overture / Wikidata POIs, stamped bulk_import provenance) must never be paid-enriched, so isBulkImport excludes it and it gets Tier 1 only. Prestige, runtime, and capture rows are not bulk, so they enrich normally; a capture-path row that arrived Serper-enriched is skipped by the already-enriched guard. The mapper fills the first-class location columns (website, phone_number, international_phone_number, price_level, business_status, google_maps_uri, opening_hours, place_types, full_address, district).
  3. Localization gap-fill (Photon) and embedding generation run for every row, independent of the commercial gate, so a row still gets embedded even when the commercial budget is exhausted.

A prestige row that merges into an existing location does not re-resolve. Serper returns Google Maps data, so it carries Google's terms; we store facts (coords, address, contact, hours) under standard usage.

Unmatched entries keep their native identity (Michelin) or their name + city (50 Best) so the funnel's fuzzy dedup can place them onto an existing backbone POI. La Liste is a follow-up: its ranking is a client-rendered app with no server HTML or open feed.

The POI backbone (Overture) and notable overlay (Wikidata)

The density tier is a two-layer backbone: Overture supplies bulk breadth, Wikidata supplies notable depth and multilingual labels, and the funnel's dedup folds the two into one row where they overlap (Overture coords + Wikidata labels/prominence).

  • Overture Places is the bulk backbone. It is a CDLA-permissive GeoParquet release on S3, extracted with the DuckDB CLI (INSTALL httpfs; INSTALL spatial;) so a confidence floor and an optional market bbox are predicate-pushed into the scan rather than downloading the whole planet. The generator shells out to duckdb, projects id / geometry / names.primary + names.common / category / confidence / country / website to NDJSON, then maps each row: names.primary -> Name, names.common -> Localizations, and the Overture category code -> a taxonomy subcategory + its bucket facet tag. A leaf with no targeted rule climbs to its Overture top-level group via a committed leaf -> group map (backend/data/overture_category_ancestry.json, generated from Overture's category CSV). All 22 Overture top-level groups carry a group-default rule, so a row whose category resolves to any known group lands on a real leaf. Only a row Overture itself left categoryless (its categories.primary is empty, a few percent of the set) becomes Other Location, which still stays lexically searchable. Overture rows carry provider = overture and load as bulk_import provenance, so the enricher gives them Photon gap-fill only and the cold long tail is never embedded until a user engages it. Tune the run with --overture-confidence, --overture-bbox west,south,east,north, and --overture-parquet (a local file / glob, for where S3 egress is unavailable).
  • Wikidata notable places is the overlay. A per-class SPARQL (one bounded P31/P279* walk per place class, so WDQS does not time out) pulls notable destinations across what a traveler wants to see, do, eat, and shop: nature and wonders (mountain, volcano, waterfall, lake, island, canyon, cave, viewpoint, national park, nature reserve, garden), landmarks and architecture (museum, monument, memorial, castle, fortress, bridge, tower, lighthouse), entertainment (amusement park, water park, zoo, aquarium, casino, stadium), market destinations (night market, hawker centre, market hall, marketplace), shopping (mall, department store, shopping street, bazaar, arcade), and religious sites (temple, shrine, pagoda, stupa, church, mosque). Individual restaurants, cafes, and shops stay out of this overlay: those are the Overture backbone plus the dining lists, so the eager-enriched embedded tier stays bounded. Classes are most-specific first so the first class to claim a QID wins. Entities need coordinates and must clear a sitelink floor (--notable-min-sitelinks, default 5, low enough to reach mid-tier cities' notable places); each class contributes up to 25000 entities. Ultra-common natural classes (mountain, island) carry a higher per-class floor so their capped set surfaces marquee entities rather than an obscure tail. Each entity's class maps to a taxonomy leaf, sitelinks become the prominence signal, and wikidataLabels attaches the full 28-language photonLanguages label set. These rows carry provider = wikidata and load as the normal enrichable wikidata_seed, so they enrich eagerly and overlay onto the Overture backbone the funnel dedups them against. This is the layer that gives great multilingual coverage where Overture is thin. A single class query failure is skipped and logged, not fatal, so one transient WDQS hiccup does not lose the other classes' work; the run still aborts without writing when more than a quarter of the classes fail or zero entities are collected, so a systemic WDQS outage cannot clobber a good notable file with a near-empty one.

The full multi-million bulk run is a separate, occasional execution (it needs the DuckDB CLI plus S3 egress); the pipeline is validated on a bounded sample, one city bbox or a small --limit, before a wide run.

Maintenance and annual refresh

The rankings re-publish on an annual cadence (each "50 Best" award ceremony, Michelin's yearly guide), so these collections are refreshed by re-running the generator per source and re-loading. Re-running is cheap: the seed-time resolver memoizes by (name, city) within a run, and the load path dedups by source ref, so a venue that also appeared last edition reconciles rather than duplicates. The paid step (Serper /maps) only fires for entries Photon cannot place, and only for venues not already resolved, so a refresh bills roughly for the year's new entries, not the whole list.

A refresh must reconcile collection membership, not just upsert locations: collection_memberships carries the edition + rank + is_current, so a re-load updates ranks, marks dropped venues non-current, and stamps the new edition per the collection's retention policy. Locations dropped from a ranking are deactivated by the funnel, never deleted (see the reconciliation invariant below).

Localization

Localization is first-class: every place displays fully localized (name + city + country + region) in all supported languages, and is searchable by a localized query. The full contract is Localization; in brief, the rule is store each localized geography name once, reference it, and assemble at read:

  • A location stores its own name localizations (Location.Localizations, a BCP-47 map) — unique per place, unavoidable, small.
  • Shared geography is referenced, not copied: city_id (a self-reference to the city's own location row — cities are locations), region_id, and country_code. Each localized geography name is stored once, in the gazetteer.
  • Read assembly resolves a card for the viewer's locale from the row's own localizations plus the referenced city/country/region names. Countries (a small fixed set) are cached in memory; cities are resolved by batched lookup.

Localizations are captured at generation time from each source's native multilingual data — Wikidata labels (curated + reference) and GeoNames alternateNamesV2 (cities). Photon reverse-geocoding is an optional gap-filler for places a source did not cover; it is never required.

Timezone is not a localization or a stored field — it is derived from coordinates. See Timezones.

Data model

Seeded rows populate locations (see Locations for the full model); the seeding-relevant columns are provider / provider_id, secondary_providers, provenance, prominence, localizations, metadata, is_active, and the geography references. Curated lists add location_collections + collection_memberships.

Invariants & gotchas

  • Seeded rows are indistinguishable to the app from resolve-minted rows except by provenance and is_active; they use the same search/map/discovery reads.
  • is_active gates map + search visibility. Seeded rows may stage inactive and flip active after a coverage check.
  • Reconciliation deactivates, never deletes. A place dropped from a source flips inactive or loses a membership; its history and attached moments survive.
  • License is tracked per collection, never per row (location_collections carries it); required attribution is surfaced once at the app level.
  • Names are cleaned at ingest with the same isMintablePlaceName gate the mint path uses, so seed data does not pollute search or the trigram index.