Catalog Seeding¶

Purpose¶

How the global place catalog — countries, cities, and curated lists like UNESCO World Heritage — is generated from open data and loaded into locations. This is the cold-start data layer: it fills the map, search, and discovery surfaces before any user-generated content exists. Seeded rows are ordinary Location rows and flow through the same read paths as resolve-minted places.

Mental model¶

Build offline, then load. Checksummed files are the handoff, and everything a row needs is baked into the file before load, so loading is a straight insert:

Overture · Wikidata · GeoNames · UNESCO DataHub

→

catalogseedergenerate → merge → enrich → validate

→

NDJSON + manifestsha256, in GCS

→

loaderupsert, on-demand Job

→

locations + collections

Generation is offline and occasional; merge folds the notable + Overture layers into one deduped POI set; enrich resolves geography and 28-locale localizations against Photon at build time; a validate gate blocks a half-enriched set from being published; loading is a manual, checksum-driven, idempotent Job.

Generate — backend/cmd/catalogseeder fetches each source, normalizes to a common SeedPlace, deduplicates within the set, and writes one NDJSON file per collection plus a manifest.json carrying each file's sha256, row count, source, and license.
Merge — catalogseeder merge --from <dir> folds wikidata-notable-places and overture-places into one deduped pois.ndjson: a notable place and its Overture twin collapse to one row carrying both provider ids, and unmatched Overture rides along as density, capped to the top rows per spatial cell by confidence. The two raw sources are dropped from the manifest (see Unified POI generator).
Enrich — catalogseeder enrich --from <dir> resolves every POI against Photon at build time, filling region / city / country (canonical) / address and the full 28-locale localizations, plus Serper commercial detail for the prestige lists. The result is baked into the NDJSON and marked enriched, so the loader inserts it straight and the async worker skips it (see Build-time enrichment).
Backfill country — catalogseeder backfill-country --from <dir> fills country / country_code on enriched rows that Photon left blank (rural, offshore, remote coordinates) by point-in-polygon against the country boundary polygons already in world-countries, so a validate pass no longer flags "enriched but missing country" (see Country backfill).
Validate — catalogseeder validate --from <dir> proves the built set is complete and solid before it is ever published or loaded (see Operating the pipeline). It is read-only (no DB, no network), checks that every pois row is enriched, and exits non-zero on any problem, so it gates the publish and load steps.
Load — reads the manifest, and for any file whose checksum changed since the last load, streams it through the dedup ladder and a provenance-precedence upsert into Postgres, then records the new checksum. Once every file is loaded, a single set-based pass links each non-City POI to its nearest city (city_id), so city-linking is one chunked, index-served UPDATE sweep rather than a per-row query in the ingest loop (see city_id linking). catalogseeder load --city-highlights N then runs the per-city top-N attraction promotion pass (see Locations → Per-city highlight promotion), so each city's marquee sights are eager-enriched once the density backbone is in place. Omit the flag (default 0) to skip it. The promotion pass ranks per city_id, so it always runs after the city-linking sweep.

Operating the pipeline¶

Generate, merge, enrich, backfill-country, validate, publish, load. The stages are decoupled by the checksummed files: generation is offline, and loading is always a manual, on-demand Job, never a cron. A refresh republishes files and runs the load Job by hand; it needs no deploy. The publish and load stages live in the DevOps repo; this repo owns generate, merge, enrich, and validate. The full cluster-side runbook (GCS bucket, loader service account, the Job overlays for dev and prod) is DevOps → Loading the location seed set.

task db:seed:build chains merge, enrich, and validate over a seed-out/ dir in one command (it needs PHOTON_URL pointed at an in-cluster Photon, e.g. via kubectl port-forward -n platform svc/photon 2322:2322).

1. Generate¶

Produce a seed set into a local directory. Target one source, or a whole tier, or everything:

cd backend
go run ./cmd/catalogseeder --source unesco --out ./seed-out          # one collection
go run ./cmd/catalogseeder --tier curated --out ./seed-out           # a whole tier
go run ./cmd/catalogseeder --source all   --out ./seed-out           # everything

Each run writes <slug>.ndjson per collection and folds an entry into seed-out/manifest.json (slug, file, rows, sha256, source, license, generated_at). Generating one source at a time accumulates into the single manifest, so a set can be built incrementally. Source-specific tuning (--overture-bbox, --notable-min-sitelinks, --michelin-no-backing, and the rest) is in Sources and tiers.

notable-dump is an operator-run source, not part of --source all: it streams a ~95 GB Wikidata JSON dump offline (hours) rather than hitting WDQS, so it is run on its own when the notable backbone needs a refresh:

go run ./cmd/catalogseeder --source notable-dump \
  --dump-path ~/Downloads/latest-all.json.bz2 \
  --closure-cache ./notable-closure.json \
  --out ./seed-out

Flag	Default	Purpose
`--dump-path`	(required)	Path to the Wikidata dump (`latest-all.json.bz2` or a plain `.json`)
`--closure-cache`	(none)	File to cache the `P279*` class closure; loaded if present, else built from WDQS once and written here
`--per-city`	15	Keep the top-N notable places per city (`P131`)
`--per-country`	250	Floor of top-N notable places per country (`P17`), so thin countries still get coverage
`--per-cat-per-city`	2	Max places of one category per city; de-skews so churches cannot swamp a city's slots (0 = uncapped)
`--global-top`	100000	Size of the global prominence track (by sitelinks)
`--max-records`	5000000	Hard ceiling on emitted rows

A candidate is kept if it fits any track (global, per-city, or per-country); a global flagship still counts toward its city and country floors rather than sitting on top of them (selectNotable in backend/cmd/catalogseeder/source/notable/notable_dump_select.go). It writes wikidata-notable-places.ndjson, which the merge stage then folds into the unified POI set.

2. Merge¶

Fold the notable + Overture layers into one deduped POI collection:

go run ./cmd/catalogseeder merge --from ./seed-out --per-cell 15 --match-meters 75

See Unified POI generator for the dedup and density capping. It rewrites the manifest, replacing wikidata-notable-places and overture-places with a single pois entry.

3. Enrich¶

Resolve geography and localizations against Photon at build time so the loader has nothing left to do:

export PHOTON_URL=http://localhost:2322   # kubectl port-forward -n platform svc/photon 2322:2322
export SERPER_API_KEY=…                    # optional; enables the prestige commercial tier
go run ./cmd/catalogseeder enrich --from ./seed-out --workers 16

It reverse-geocodes each POI across the 28 supported locales, fills region/city/country/address, canonicalizes the country name against the countries gazetteer, and (for the prestige lists only) adds Serper address + Google place ids. The pass is parallel, cached, and resumable: a crash or a Serper credit-out never re-pays for work already on disk, and re-running picks up where it stopped. See Build-time enrichment.

4. Backfill country¶

Fill a country onto enriched rows that Photon left without one:

go run ./cmd/catalogseeder backfill-country --from ./seed-out   # [--slug X] [--drop-unresolved]

Runs after enrich, before validate. See Country backfill. Validate requires both country and country_code on every enriched row, which this satisfies.

5. Validate¶

Prove the set is complete and solid before it leaves the machine:

go run ./cmd/catalogseeder validate --from ./seed-out

For every manifest entry it checks: the file exists, its recomputed sha256 matches the manifest, its line count equals rows, and a cheap per-source sanity pass (every row parses as JSON, has a non-empty name and canonical identity, and carries coordinates where the source promises them, the "50 Best" rankings being the one coordless-by-design exemption). It also enforces the post-enrichment invariants: a row stamped enriched must carry a resolved country, and every coordinate-bearing row in the unified pois collection must be enriched, so a half-run enrichment can never slip into publish. It prints a per-collection PASS/FAIL summary and exits non-zero on any failure, so it gates the steps that follow. It touches no database and makes no network calls.

6. Publish¶

Upload the validated directory to the seed bucket under an immutable seed/<version>/ prefix (<version> is a date or git sha the operator picks). The DevOps seed-publish.sh script re-runs validate and refuses to overwrite an existing version:

./scripts/seed-publish.sh --version 2026-07-17 --dir ./seed-out   # in the DevOps repo

7. Load (on demand, dev then prod)¶

Loading is a Kubernetes Job, not a CronJob: it is applied by hand per environment, dev first, then prod once dev looks right. The Job pulls seed/<version> from the bucket, runs catalogseeder validate then catalogseeder load --from /seed --city-highlights <N> against that environment's Postgres. The load is checksum-gated per collection (seed_load_state), so re-running the same version is a near no-op. See the DevOps runbook above for the render-and-apply commands and the two per-run knobs (seed version, --city-highlights N).

The loader binary ships inside the tomoda-backend image at /app/catalogseeder (a static, pure-Go build; backend/data is go:embed'd, so no runtime data mount), so the Job runs that image with the entrypoint overridden rather than pulling a second image.

Local dev skips the Job: task db:seed:catalog pulls the latest (or SEED_VERSION=) set from gs://tomoda-location-seed-<project>/seed/<version>/ into local Postgres (needs gcloud auth login). By default it runs a lite load that skips the three heavy global collections (overture-places, wikidata-notable-places, geonames-natural-places) so a laptop load stays quick; pass FULL=1 to load the complete published set. Downloads are cached in .seed-cache/<version>-<mode>/, keyed by version and mode, so lite and full never clobber each other and re-runs skip the download. Versions are immutable, so db:seed:catalog always gets exactly what was published.

Simulation bundle (fast local seed)¶

The full catalog is multi-GB (the POI set alone) and the geo-cover set is tens of GB, far more than a local dev or a task test:simulation run needs. A small prebuilt simulation bundle is published under simulation/<version>/ in the seed bucket and pulled with SIM=1:

task db:seed:catalog SIM=1     # pull + load the simulation subset
task test:simulation           # defaults to SIM=1 (pass FULL=1 for the full catalog)

The bundle has the same layout as a seed version (manifest, NDJSON, place-photos/, geo-covers.ndjson), subset for size:

Full countries and regions, so travel, atlas, and passport testing traverse every country and region.
Top cities by population (rows), enough for global discovery coverage.
POIs scoped to the top metros (a bounded radius around the highest-population cities), so discovery and search return real places where the sim operates without loading the multi-million global tail.
Covers for all countries, all ADM1 regions (states/provinces), and the major cities per country (every city above a population floor, capped, with a per-country minimum so small markets still appear). Covers are downscaled to ~512px for the bundle so coverage is comprehensive while the download stays a few hundred MB; dev/prod S3 keep the full-res originals. The ADM2+ region long tail and minor cities fall back to no cover.
Curated/prestige place photos (Michelin, the 50 Best lists, etc.) compressed into the bundle, so the sim is self-contained: local loads never fetch the full catalog's place-photos/. The loader reads place-photos/<place_id>.jpg from the seed dir; the full catalog serves the full-res originals from seed/<version>/place-photos/.

backend/scripts/seed/build-sim-seed.py builds the bundle from a full seed cache plus the geo-cover set (paths and tuning are flags; see --help). backend/scripts/seed/gen-sim-hotspots.py regenerates cmd/test/global-hotspots/hotspots.json, the major-city list the sim seeds activity at (and the cover scope derives from). Rebuild both when the seed refreshes, then upload to simulation/<version>/.

Sources and tiers¶

Every source declares a tier; a run targets one source or a whole tier (--source <name>|all, --tier <tier>).

Tier	Source	License	Contents
`reference`	Wikidata (SPARQL)	CC0	countries (ISO 3166-1)
`reference`	GeoNames (downloaded dump)	CC BY 4.0	administrative areas (states/provinces, counties/districts)
`gazetteer`	GeoNames (downloaded dump)	CC BY 4.0	cities / settlements by population
`curated`	Wikidata + UNESCO DataHub	CC0	UNESCO World Heritage
`curated`	Michelin dataset (`ngshiheng/michelin-my-maps`) + Wikidata	MIT (dataset); awards © Michelin, facts only	Michelin Guide restaurants (stars, Bib Gourmand, Green Star)
`curated`	theworlds50best.com + Wikidata (seed); enriched by the shared load-time path	editorial ranking, facts only	the "50 Best" family: World's / Asia's / Latin America's / North America's / MENA restaurants, World's / Asia's / North America's bars, World's hotels, World's Best Vineyards
`curated`	Wikidata (SPARQL)	CC0	notable places overlay via WDQS (`notable`, ~80K): museums, landmarks, parks, monuments, temples with rich multilingual labels
`curated`	Wikidata JSON dump (`latest-all.json.bz2`)	CC0	notable places at scale (`notable-dump`, up to ~5M): top-N per city worldwide + a global top ~100k by sitelinks. The offline dump avoids WDQS result caps; a one-off operator run streams it
`curated`	GeoNames (downloaded dump) + Wikidata	CC BY 4.0 + CC0	natural attractions (`geonames-natural-places`): physical features (mountains, lakes, waterfalls, islands, parks) carrying a Wikidata QID and absent from the notable set
`density`	Overture Maps Places (GeoParquet on S3, via DuckDB)	CDLA Permissive 2.0	bulk POI breadth; the merge stage folds it with the notable overlay into one `pois` collection

Wikidata is the backbone: it supplies the curated + reference tiers and is the identity hub. GeoNames is a downloadable file (enumerated locally, avoiding query timeouts) for the high-cardinality gazetteer.

Administrative hierarchy¶

The regions source fills the mid-tier between country and city, so the area tree can be reconstructed for gazetteer recentering, not just display. It reads the GeoNames allCountries dump for feature class A, feature codes ADM1 (states/provinces), ADM2 (counties/regions/districts), and ADM3 (districts in the countries that model them). Each row becomes a geonames:<geonameid> SeedPlace with category administrative_area_level_1|2|3, its GeoNames population, and 28-language localizations folded in from alternateNamesV2 (the same path cities use). Hierarchy linkage lives in metadata: geonames_feature_code plus admin1_code (and admin2_code for an ADM2), which combined with the first-class country_code locate an area within its parent. Generate with --source regions; a full run is ~220k areas.

Boundary polygons. The countries source attaches a GeoNames shapes_all_low boundary (a GeoJSON MultiPolygon) to each country, joined through countryInfo.txt's ISO to geonameid map. The loader writes it to the boundary geometry(MultiPolygon,4326) column, which is write-only: no query selects it (sqlc maps geometry to a string it never reads), it exists for ST_Contains point-in-polygon country assignment. GeoNames shapes_all_low is country-level only, so ADM1/ADM2 boundaries are not yet populated: the regions.boundary column and its write path are ready for an admin polygon source (geoBoundaries CGAZ) when that ingester ships; until then admin assignment falls back to nearest-centroid. A boundary fetch failure is non-fatal, so it never blocks the gazetteer itself.

GeoNames cross-references. Cities and admin areas carry more than their own name from the same GeoNames pass. Both attach the IANA timezone and (from alternateNamesV2) a wikidata_qid in metadata plus a wikidata Secondary, so a settlement or area dedups exactly against the notable set on QID. Cities additionally carry iata / icao airport codes in metadata. Countries carry capital, currency_code, calling_code, languages, and area_sqkm from countryInfo.txt (LoadCountryFacts in backend/cmd/catalogseeder/source/gazetteer/boundaries.go); continent is still derived by the loader from the ISO code, not stored on the seed row.

Natural attractions¶

geonames-natural-places (--source natural) emits GeoNames physical features (mountains, volcanoes, lakes, waterfalls, glaciers, islands, canyons, parks, nature reserves, and the rest) that carry a Wikidata QID and are not already in the notable set. Identity is the Wikidata QID, so a row dedups exactly against notable on QID; the GeoNames id rides as a geonames Secondary. It runs after notable in the same seed dir, reading the notable file's QIDs to skip anything already covered. FetchNatural in backend/cmd/catalogseeder/source/gazetteer/natural.go keeps only whitelisted attraction feature codes (featureCategory, mapping each code to a display category); minor landforms (hills, points, ridges) are dropped as the tail a Wikidata id alone does not make notable. Mountains and peaks (MT/PK/PKS) are elevation-gated to prominent summits (>= mountainMinElevation, 2000m), the famous low peaks already arriving through the sitelink-ranked notable set. 28-locale localizations come from alternateNamesV2; the row then flows through the same Photon enrich (region / city / country / address + 28 locales) as notable.

Identity, dedup, provenance¶

Canonical identity is the (provider, provider_id) pair on Location: wikidata:Q… for curated/reference + the notable overlay, overture:<gers-id> for the bulk backbone, geonames:<geonameid> for cities, photon:{type}:{id} for OSM, unesco:<id_no> for World Heritage sites Wikidata lacks. Cross-provider ids for the same place live in secondary_providers.
Dedup ladder (before every insert): canonical pair → secondary_providers cross-id → shared Wikidata QID → spatial + fuzzy-name proximity. A match merges (appends the id, keeps user/admin/enriched fields) rather than inserting a twin.
provenance records how a row's data was sourced (wikidata_seed, geonames_seed, osm_seed, unesco_seed, provider_api, serper_maps, bulk_import) and drives merge precedence: user/admin edits and provider-API enrichment outrank a re-seed. bulk_import also excludes the density backbone from paid enrichment.
prominence (0..1) is a popularity prior the ranker boosts on, derived from raw signals (Wikidata sitelinks, then city population, then Overture confidence for the density rows that carry neither). It stands in for user engagement until that accrues.
metadata (jsonb) holds place-intrinsic source extras with no first-class column (e.g. a city's GeoNames feature code). Collection-specific facts live on the collection membership, not here.

`city_id` linking (nearest city)¶

Every non-City POI references its city through city_id, a self-FK to the city's own location row (cities are locations). The loader assigns it in one set-based pass after the whole dataset is loaded, not per row during ingest, so a planet-scale load is a bounded sweep of UPDATE statements instead of tens of millions of round-trips.

The pass walks POIs by primary-key cursor in chunks (cityLinkChunkSize) so no single statement locks or bloats the table. Each chunk runs a LATERAL nearest-city join: for each still-unlinked POI it picks the nearest City within 50km, preferring the same country_code (an empty POI country_code matches any country). The KNN order-by (coordinates <-> …) is served by the GiST idx_coordinates index, so it is an index scan, not a per-city sequential scan over the backbone. Advancing by id rather than by "rows still NULL" keeps the loop moving past POIs with no city within 50km. It is idempotent: only NULL city_id rows are touched, so a re-load leaves already-linked rows unchanged.

city_id is what read-time card assembly follows for the localized city name (see Localization) and what the per-city highlight promotion pass ranks within, so the linking sweep always runs before promotion.

Curated collections¶

A curated list is both a browsable filter and, optionally, an earnable stamp, off one seeded dataset. location_collections names the list; collection_memberships ties a location to it with the source's external_id, an edition, an is_current flag, and a distinction (e.g. UNESCO category + inscription year). Living collections reconcile by diff each release; editions are retained per the collection's retention policy.

UNESCO is fact-checked against UNESCO's own open-data portal (data.unesco.org, whose id_no equals Wikidata property P757): only authoritatively inscribed sites are kept, each enriched with the official category and inscription year, one row per site.

Prestige lists (Michelin, 50 Best)¶

The Michelin and "50 Best" sources are curated-tier lists meant to be dedup-merged onto backbone POIs by the ingest funnel, contributing prestige tags plus prominence. They are derivative datasets: we take facts only (name, award, rank, coords), never editorial descriptions or images.

Michelin consumes a maintained scrape of the guide (ngshiheng/michelin-my-maps, MIT). Awarded rows only (stars + Bib Gourmand; Selected Restaurants are skipped). Each row is provider = michelin, the guide URL slug its provider_id, category Restaurant. The award maps to a tag (michelin_1_star / michelin_2_star / michelin_3_star / bib_gourmand, plus michelin_green_star) and a michelin distinction block (stars, award, year, cuisine, price, url).
50 Best scrapes the published William Reed rankings (name + city + rank) into one source per list, one per ranking in the family: restaurants (World's, Asia's, Latin America's, North America's, MENA), bars (World's, Asia's, North America's), hotels (World's), and vineyards (World's Best Vineyards). Each entry carries a ranking tag (e.g. worlds_50_best_restaurants, worlds_50_best_bars, worlds_50_best_hotels, worlds_best_vineyards), the list's category (Restaurant / Bar / Hotel / Vineyard), and a distinction block with rank + year. The 1-50 page embeds the full 1-100 depth, so a ranking with an extended list is captured to rank 100 in one fetch; rankings that publish only a top-50 (North America's restaurants, MENA) stop at 50. The vineyards ranking lives on its own domain that redirects into the same site structure, so one parser handles the whole family.

Michelin backs each row with Wikidata by coordinate proximity (the guide rows are address-precise), attaching the canonical QID as a secondary provider id plus multilingual labels and a sitelinks prominence signal. The whole backing step runs under a single WDQS budget (default 12 min) threaded as a context deadline through every WDQS call, so a stalled request is cancelled the moment the budget expires rather than hanging the run; rows past the budget (or once WDQS hard-throttles) keep their Michelin-native identity, the documented fallback. Pass --michelin-no-backing to skip the backing step entirely for a fast michelin-only seed when WDQS is degraded.

The "50 Best" seeder resolves coordinates at seed time against a planet Photon index, then hands the rest to the shared funnel. Prestige is a global set, so it geocodes each ranked row (name + city context) against the configured planet Photon (PHOTON_URL, falling back to the komoot public planet endpoint), behind a validation guardrail: it accepts a hit only when the name matches (accent-folded, token-level) and the coords fall within a metro radius of the expected city's own geocoded center, so a bare-name query cannot mis-geocode to a same-named place on another continent (a bare "Aman Venice" resolving to Amman, Jordan is rejected). On a miss or a rejected hit the row is dropped from the generated seed and logged, so a prestige collection ships only rows with real coordinates. Everything else, category, tags ([bucket facet, ranking tag]), collection membership, the rank + year distinction, goes through the common ingest funnel. There are three ingest paths (runtime search/create, capture URL into Places/Serper, and this batch seed) and they all converge on the same funnel Ingest and the same async enricher: any remaining coordinate gap-fill and all commercial detail are filled by the shared enricher.

Prestige coords are solid by construction. A prestige row that fails resolution is dropped at generation rather than emitted at the null island, so a loaded prestige collection has no lat=0/lng=0 rows to reconcile after the fact. Point PHOTON_URL at the in-cluster planet Photon when generating (the local Photon index is region-limited); the komoot public Photon is the fallback.

Localizations are the one thing the seeder adds itself, because Wikidata is a free additive data source rather than a resolution path:

Seed-time, from Wikidata. Each entry is matched to a Wikidata QID by name (city-disambiguated), and its labels across the full 28-language photonLanguages set populate Localizations (kept only when they differ from the base name). This attaches the QID as a wikidata secondary id.
Load-time, from Photon. Entries Wikidata cannot match arrive with thin localizations, so the async enrichment sweep's gapFillLocalizations reverse-geocodes their coordinates against the self-hosted Photon index across the app's SupportedLocalizationLanguages, bringing them up to the same coverage as any runtime-minted row.

If neither Wikidata nor Photon has a localization, the field stays empty.

Coordinates and commercial detail come from the one shared enricher (EnrichLocation in backend/internal/services/location/location_enricher.go), the same path a runtime row and a capture row take. It fires once per first-time-created location (enrichment_at NULL) and is two-tier and provider-agnostic:

Tier 1, Photon gap-fill (cheap, self-hosted), for every row. A coordless row is forward-geocoded against Photon (photonForwardGeocode), so a seed row that loaded without coordinates gets them from name + city.
Tier 2, paid commercial enrichment (Serper /maps, then the Google Places API fallback). Its gate is tagsAreRichDetail(loc.Tags) AND NOT isBulkImport(loc) AND NOT already enriched (hasFreshGoogleFields). The rich-detail category gate alone is not enough: the high-cardinality density backbone (the future millions of Overture / Wikidata POIs, stamped bulk_import provenance) must never be paid-enriched, so isBulkImport excludes it and it gets Tier 1 only. Prestige, runtime, and capture rows are not bulk, so they enrich normally; a capture-path row that arrived Serper-enriched is skipped by the already-enriched guard. The mapper fills the first-class location columns (website, phone_number, international_phone_number, price_level, business_status, google_maps_uri, opening_hours, place_types, full_address, district).
Localization gap-fill (Photon) and embedding generation run for every row, independent of the commercial gate, so a row still gets embedded even when the commercial budget is exhausted.

A prestige row that merges into an existing location does not re-resolve. Serper returns Google Maps data, so it carries Google's terms; we store facts (coords, address, contact, hours) under standard usage.

Unmatched entries keep their native identity (Michelin) or their name + city (50 Best) so the funnel's fuzzy dedup can place them onto an existing backbone POI. La Liste is a follow-up: its ranking is a client-rendered app with no server HTML or open feed.

Curated photos¶

The curated prestige collections (michelin, unesco, 50-best) carry one Google Places photo per venue. At generation the seeder captures a single photo per unique place_id into place-photos/<place_id>.<ext> and writes a curated-photos.ndjson mapping (place_id -> file). At load time thumbAttacher (backend/cmd/catalogseeder/command/load/loader_thumbnails.go) rehosts each matched photo to the object-store key item-images/seed/<place_id>.<ext> and stamps the location's thumbnail_key (served via storage.PublicURL). The key is deterministic, so a re-load overwrites the same object rather than orphaning a new one.

It is best-effort: no object store configured or no place-photos/ directory in the seed means thumbnails are skipped and the load still succeeds (as it did before photos were captured). A missing file or upload error is logged per-row and skipped, never failing the load. db:seed:catalog pulls place-photos/ alongside the NDJSON set so a local load can rehost too.

Geo covers¶

Every country, admin region, and city carries a cover photo for the atlas, map, and discovery surfaces, self-hosted under a deterministic key so the image can be swapped later without a reseed.

Generate. catalogseeder geocover --seed-dir <dir> --out <dir> resolves one cover per entity from a source ladder and downloads it to place-photos/geo/<kind>/<key>.jpg, appending a resumable geo-covers.ndjson manifest row (kind, key, name, source, image_file, and author / license / source_url). Keys are natural and stable: country ISO2 (FI), region <provider>_<id> or ISO 3166-2, city GeoNames id.

Pixabay is the primary source (no attribution required, commercial use and rehosting allowed), tried for every entity. Each entity runs an ordered query ladder from most specific to bare name (coverQueries in backend/cmd/catalogseeder/command/geocover/geocover.go), so an ambiguous name (Georgia, Chad, Turkey) biases toward a landmark or landscape hit before falling back to the plain name. One API key per worker; a wide crawl shards the city file across keys for throughput.
Wikipedia lead image is the keyless fallback when Pixabay returns nothing.

--only country|region|city scopes a run to one kind, --tier-only limits the long tail, and the done-check skips entities already in the manifest, so a run resumes without re-spending API budget.

Load. loader_geocover.go uploads each manifest image to geo-covers/<kind>/<key>.jpg in the object store and stamps the matching row's cover_image_key plus cover_author / cover_license / cover_source_url, matched by natural key: countries.code, regions.(provider, provider_id) or iso_3166_2, and locations.id via the location_providers GeoNames junction for cities. It is best-effort (a missing file, upload error, or no-match row is logged and skipped) and runs once entity rows exist. At read time the cover URL composes from cover_image_key.

Public serving. Geo covers are generic and non-sensitive, so they are served as plain unsigned CDN URLs rather than the short-lived signed URLs private assets use. storage.Prefix.IsPublic() marks the geo-covers/ prefix and assetService.URLFromKey returns {baseURL}/{key} for it instead of signing, so CloudFront caches them at the edge. The geo-covers/* cache behavior in the DevOps CloudFront config serves those paths unsigned even under signed-URL enforcement. See Storage.

Swap a cover without reseeding. The key is deterministic, so a better photo is dropped at the same object key (geo-covers/country/FR.jpg) and picked up straight away: the DB stores only the key, the URL composes from it, and nothing in Postgres changes. Two caveats. The cover_author / cover_license columns stay as last loaded, so update them if the new photo's source differs. Because the public cache TTL is long, invalidate the path on the CDN (or wait out the TTL) for the new image to appear.

The POI backbone (Overture) and notable overlay (Wikidata)¶

The density tier is a two-layer backbone: Overture supplies bulk breadth, Wikidata supplies notable depth and multilingual labels, and the funnel's dedup folds the two into one row where they overlap (Overture coords + Wikidata labels/prominence).

Overture Places is the bulk backbone. It is a CDLA-permissive GeoParquet release on S3, extracted with the DuckDB CLI (INSTALL httpfs; INSTALL spatial;) so a confidence floor and an optional market bbox are predicate-pushed into the scan rather than downloading the whole planet. The generator shells out to duckdb, projects id / geometry / names.primary + names.common / category / confidence / country / website to NDJSON, then maps each row: names.primary -> Name, names.common -> Localizations, and the Overture category code -> a taxonomy subcategory + its bucket facet tag. A leaf with no targeted rule climbs to its Overture top-level group via a committed leaf -> group map (backend/data/overture_category_ancestry.json, generated from Overture's category CSV). All 22 Overture top-level groups carry a group-default rule, so a row whose category resolves to any known group lands on a real leaf. Only a row Overture itself left categoryless (its categories.primary is empty, a few percent of the set) becomes Other Location, which still stays lexically searchable. Overture rows carry provider = overture and load as bulk_import provenance, so the enricher gives them Photon gap-fill only and the cold long tail is never embedded until a user engages it. Tune the run with --overture-confidence, --overture-bbox west,south,east,north, and --overture-parquet (a local file / glob, for where S3 egress is unavailable).
Wikidata notable places is the overlay. A per-class SPARQL (one bounded P31/P279* walk per place class, so WDQS does not time out) pulls notable destinations across what a traveler wants to see, do, eat, and shop: nature and wonders (mountain, volcano, waterfall, lake, island, canyon, cave, viewpoint, national park, nature reserve, garden), landmarks and architecture (museum, monument, memorial, castle, fortress, bridge, tower, lighthouse), entertainment (amusement park, water park, zoo, aquarium, casino, stadium), market destinations (night market, hawker centre, market hall, marketplace), shopping (mall, department store, shopping street, bazaar, arcade), and religious sites (temple, shrine, pagoda, stupa, church, mosque). Individual restaurants, cafes, and shops stay out of this overlay: those are the Overture backbone plus the dining lists, so the eager-enriched embedded tier stays bounded. Classes are most-specific first so the first class to claim a QID wins. Entities need coordinates and must clear a sitelink floor (--notable-min-sitelinks, default 5, low enough to reach mid-tier cities' notable places); each class contributes up to 25000 entities. Ultra-common natural classes (mountain, island) carry a higher per-class floor so their capped set surfaces marquee entities rather than an obscure tail. Each entity's class maps to a taxonomy leaf, sitelinks become the prominence signal, and wikidataLabels attaches the full 28-language photonLanguages label set. These rows carry provider = wikidata and load as the normal enrichable wikidata_seed, so they enrich eagerly and overlay onto the Overture backbone the funnel dedups them against. This is the layer that gives great multilingual coverage where Overture is thin. A single class query failure is skipped and logged, not fatal, so one transient WDQS hiccup does not lose the other classes' work; the run still aborts without writing when more than a quarter of the classes fail or zero entities are collected, so a systemic WDQS outage cannot clobber a good notable file with a near-empty one.

The full multi-million bulk run is a separate, occasional execution (it needs the DuckDB CLI plus S3 egress); the pipeline is validated on a bounded sample, one city bbox or a small --limit, before a wide run.

Unified POI generator (merge)¶

catalogseeder merge (backend/cmd/catalogseeder/command/merge) folds the two density layers into one deduped pois.ndjson so the density backbone is a single collection rather than two overlapping raw files. It holds the notable set in memory behind a spatial grid, streams the far larger Overture file, and for each Overture row:

Twin found (within --match-meters, default 75m, and a fuzzy name match via seed.NameMatches): the Overture id is appended to the notable row as a secondary provider ref, and the Overture row is not emitted. The notable row's Wikidata prominence + 28-language labels stay; it simply gains Overture's identity.
No twin: the row is density. Each ~5km cell keeps only its top --per-cell rows by Overture confidence (a bounded per-cell bucket evicts the lowest when full), so a busy cell contributes its best rows rather than whichever streamed first. An optional --per-country cap bounds density per country.

It then emits every notable row (now carrying any absorbed refs) plus the kept density, streaming to a .partial sidecar it renames on success, and rewrites the manifest to drop wikidata-notable-places + overture-places in favor of one pois entry (tier density). Overture confidence rides through on SeedPlace.Confidence and becomes the density row's prominence at load.

Build-time enrichment (enrich)¶

catalogseeder enrich (backend/cmd/catalogseeder/command/enrich) moves the geography + localization work that the load-time enricher used to do forward to build time, so the published NDJSON is already complete and the loader inserts it straight. For every coordinate-bearing row in the curated + density tiers it:

reverse-geocodes the coordinate against Photon once per locale across the 28 SupportedLocalizationLanguages, filling region / city / country / address from the English result and per-locale name + address + city + country into localizations, keeping any source-provided names as the base;
canonicalizes country to the countries gazetteer's single spelling (so the seed never carries "USA" next to "United States");
for the "50 Best" family only (serperEligible: restaurants, bars, hotels, vineyards, which are Google-listed businesses), adds the postal address + Google place id from Serper; Michelin already ships addresses and UNESCO sites are not Google businesses, so both take the free Photon tier only;
stamps the row enriched, which the loader reads to set enrichment_at, so the async worker skips it.

The pass is parallel (a worker pool over rows, a global cap on concurrent Photon calls) and content-addressed cached + resumable: each Photon and Serper response is keyed to disk, so a resumed run reads cache instead of the network, and a Serper credit-out checkpoints the .partial and exits rather than losing progress. The load-time enricher (EnrichLocation) remains the path for runtime-minted and capture rows and for any seed row that was not pre-enriched.

Country backfill (backfill-country)¶

catalogseeder backfill-country (backend/cmd/catalogseeder/command/backfill/backfill.go) fills country / country_code on rows Photon left blank (rural, offshore, or remote coordinates), so the validate invariant "an enriched row carries a country" comes back clean without re-geocoding. It runs after enrich and before validate. For each coordinate-bearing row missing a country it does a point-in-polygon test against the country boundary polygons already carried on world-countries (loaded once into github.com/paulmach/orb geometries, each behind a bbox prefilter), then canonicalizes the country name from the resolved code. A point inside no polygon takes a nearest-country fallback within ~1 degree (nearestCapDeg, ~110km), so a coastal or offshore feature still resolves while a mid-ocean point does not.

go run ./cmd/catalogseeder backfill-country --from ./seed-out --slug pois --drop-unresolved

--slug restricts to one collection (default: every collection); it rewrites each touched file in place and refolds its checksum into the manifest. With --drop-unresolved, an enriched row still without a country after the fill is dropped: these are the genuine off-Earth features (Martian and lunar volcanoes, international-waters shipwrecks) that carry a Wikidata QID and coordinates but sit inside no country polygon.

Maintenance and annual refresh¶

The rankings re-publish on an annual cadence (each "50 Best" award ceremony, Michelin's yearly guide), so these collections are refreshed by re-running the generator per source and re-loading. Re-running is cheap: the seed-time resolver memoizes by (name, city) within a run, and the load path dedups by source ref, so a venue that also appeared last edition reconciles rather than duplicates. The paid step (Serper /maps) only fires for entries Photon cannot place, and only for venues not already resolved, so a refresh bills roughly for the year's new entries, not the whole list.

A refresh must reconcile collection membership, not just upsert locations: collection_memberships carries the edition + rank + is_current, so a re-load updates ranks, marks dropped venues non-current, and stamps the new edition per the collection's retention policy. Locations dropped from a ranking are deactivated by the funnel, never deleted (see the reconciliation invariant below).

Localization¶

Localization is first-class: every place displays fully localized (name + city + country + region) in all supported languages, and is searchable by a localized brief, the rule is store each localized geography name once, reference it, and assemble at read:

A location stores its own name localizations (Location.Localizations, a BCP-47 map) — unique per place, unavoidable, small.
Shared geography is referenced, not copied: city_id (a self-reference to the city's own location row — cities are locations), region_id, and country_code. Each localized geography name is stored once, in the gazetteer.
Read assembly resolves a card for the viewer's locale from the row's own localizations plus the referenced city/country/region names. Countries (a small fixed set) are cached in memory; cities are resolved by batched lookup.

Localizations start from each source's native multilingual data (Wikidata labels for curated + reference, GeoNames alternateNamesV2 for cities) and are then completed by the build-time enrich stage, which reverse-geocodes each POI against Photon across all 28 supported locales to fill the name, address, city, and country a source did not cover. The published rows are therefore fully localized before load.

Timezone is not a localization or a stored field — it is derived from coordinates. See Timezones.

Data model¶

Seeded rows populate locations (see Locations for the full model); the seeding-relevant columns are provider / provider_id, secondary_providers, provenance, prominence, localizations, metadata, is_active, and the geography references. Curated lists add location_collections + collection_memberships.

Invariants & gotchas¶

Seeded rows are indistinguishable to the app from resolve-minted rows except by provenance and is_active; they use the same search/map/discovery reads.
is_active gates map + search visibility. Seeded rows may stage inactive and flip active after a coverage check.
Reconciliation deactivates, never deletes. A place dropped from a source flips inactive or loses a membership; its history and attached moments survive.
License is tracked per collection, never per row (location_collections carries it); required attribution is surfaced once at the app level.
Names are cleaned at ingest with the same isMintablePlaceName gate the mint path uses, so seed data does not pollute search or the trigram index.

Cross-links¶

The place model and read paths: Locations.
Timezone derivation: Timezones.
Data model: Data model.