Curated collections and geo covers¶
Part of Catalog seeding: the curated lists (UNESCO, Michelin, the "50 Best" family), their photos, the geo cover lifecycle, enrichment and refresh.
Curated collections¶
A curated list is both a browsable filter and, optionally, an earnable
stamp, off one seeded dataset. location_collections names the list;
collection_memberships ties a location to it with the source's external_id,
an edition, an is_current flag, and a distinction (e.g. UNESCO category +
inscription year). Living collections reconcile by diff each release; editions are
retained per the collection's retention policy.
UNESCO is fact-checked against UNESCO's own open-data portal
(data.unesco.org, whose id_no equals Wikidata property P757): only
authoritatively inscribed sites are kept, each enriched with the official category
and inscription year, one row per site.
Category. A site's category comes from its type, not its UNESCO class: its
direct Wikidata instance-of classes first (the notable overlay's class table plus
canal), then whole-word keywords in its names (cathedral, bridge, canal, fortress,
old town, archaeological, national park, lake, and so on), each mapped onto a
place-taxonomy leaf. Cultural sites only try the cultural keywords and Natural
sites only the natural ones. Sites that match nothing fall back to Cultural ->
Historical Site, Natural -> National Park, Mixed -> Landmark.
Folded into existing places. The merge stage attaches each site to the place
that already exists instead of leaving a second row. A site matches the notable
rows by exact id (the same Wikidata QID), then by normalized name
(ignoring "World Heritage Site" / "National Historic Site" noise) within a radius
sized from the site's inscribed area (2km to 30km). A longer candidate name matches
only when its extra words are designations ("Butrint National Park"), never a
venue at the site ("Delos Synagogue", "Great Zimbabwe Hotel"); a shorter one must
be at least two words ("Durham Castle" for "Durham Castle and Cathedral"). Only an
equal multi-word name in the same country may match up to 60km away, for linear or
large sites such as a canal. Lodging, food, transport, office and retail places
are never fold targets. On a match the site row takes that place's
identity, name, coordinates and category and keeps its own external_id and
distinction; it also lists its own old identity in supersedes. Nothing matching
leaves the site as minted. The rewrite is in place and repeatable: merging again
gives the same result.
Reload. The loader writes the folded row through the normal ingest funnel, so it
attaches to the existing place and creates the collection membership there. For
each supersedes entry it folds the old row into the kept place with the same
MergeInto an admin merge uses: memberships the kept place already has are
dropped, and everything else (source ids, including enrichment and cross-source
ids the old row gained, moments, check-ins, every location reference) moves over
before the old row is removed. Re-running the load is a no-op, since the old ids
now resolve to the kept place.
Twins outside the seed. A place minted at runtime (a search or capture hit) can already hold a curated row's Google place id, like a "Rideau Canal National Historic Site" next to the World Heritage row. After ingesting a curated row the loader folds such a holder into it when the names match (designation phrases aside), it is within 25km (60km for an equal multi-word heritage name), and it is not itself on the same list.
False-merge repair. Before ingesting a curated row, the loader checks the place that already holds its id. When that place also holds another id from the same provider and does not match the row (a different name, or more than 25km away), the row's ids are released from it, so the row lands on its own venue. This undoes merges an older seed caused through ids from unvalidated lookups.
Prestige lists (Michelin, 50 Best)¶
The Michelin and "50 Best" sources are curated-tier lists meant to be dedup-merged onto backbone POIs by the ingest funnel, contributing prestige tags plus prominence. They are derivative datasets: we take facts only (name, award, rank, coords), never editorial descriptions or images.
- Michelin consumes a maintained scrape of the guide
(
ngshiheng/michelin-my-maps, MIT). Awarded rows only (stars + Bib Gourmand; Selected Restaurants are skipped). Each row isprovider = michelin, the guide URL slug itsprovider_id, categoryRestaurant. The award maps to a tag (michelin_1_star/michelin_2_star/michelin_3_star/bib_gourmand, plusmichelin_green_star) and amichelindistinction block (stars,award,year,cuisine,price,url). - 50 Best scrapes the published William Reed rankings (name + city + rank)
into one source per list, one per ranking in the family: restaurants (World's,
Asia's, Latin America's, North America's, MENA), bars (World's, Asia's, North
America's), hotels (World's), and vineyards (World's Best Vineyards). Each entry
carries a ranking tag (e.g.
worlds_50_best_restaurants,worlds_50_best_bars,worlds_50_best_hotels,worlds_best_vineyards), the list's category (Restaurant/Bar/Hotel/Vineyard), and a distinction block withrank+year. The1-50page embeds the full1-100depth, so a ranking with an extended list is captured to rank 100 in one fetch; rankings that publish only a top-50 (North America's restaurants, MENA) stop at 50. The vineyards ranking lives on its own domain that redirects into the same site structure, so one parser handles the whole family.
Michelin backs each row with Wikidata by coordinate proximity (the guide rows are
address-precise), attaching the canonical QID as a secondary provider id plus
multilingual labels and a sitelinks prominence signal. The whole backing step runs
under a single WDQS budget (default 12 min) threaded as a context deadline through
every WDQS call, so a stalled request is cancelled the moment the budget expires
rather than hanging the run; rows past the budget (or once WDQS hard-throttles)
keep their Michelin-native identity, the documented fallback. Pass
--michelin-no-backing to skip the backing step entirely for a fast michelin-only
seed when WDQS is degraded.
The "50 Best" seeder resolves coordinates at seed time against a planet Photon
index, then hands the rest to the shared funnel. Prestige is a global set, so it
geocodes each ranked row (name + city context) against the configured planet Photon
(PHOTON_URL, falling back to the komoot public planet endpoint), behind a
validation guardrail: it accepts a hit only when the name matches (accent-folded,
token-level) and the coords fall within a metro radius of the expected city's own
geocoded center, so a bare-name query cannot mis-geocode to a same-named place on
another continent (a bare "Aman Venice" resolving to Amman, Jordan is rejected). On
a miss or a rejected hit the row is dropped from the generated seed and logged, so a
prestige collection ships only rows with real coordinates. Everything else, category, tags
([bucket facet, ranking tag]), collection membership, the rank + year distinction,
goes through the common ingest funnel. There are three ingest paths (runtime
search/create, capture URL into Places/Serper, and this batch seed) and they all
converge on the same funnel Ingest and the same async enricher: any remaining
coordinate gap-fill and all commercial detail are filled by the shared enricher.
Prestige coords are solid by construction. A prestige row that fails resolution
is dropped at generation rather than emitted at the null island, so a loaded prestige
collection has no lat=0/lng=0 rows to reconcile after the fact. Point PHOTON_URL at
the in-cluster planet Photon when generating (the local Photon index is region-limited);
the komoot public Photon is the fallback.
Localizations are the one thing the seeder adds itself, because Wikidata is a free additive data source rather than a resolution path:
- Seed-time, from Wikidata. Each entry is matched to a Wikidata QID by name
(city-disambiguated: a candidate in another city is taken only when it lies
within the metro radius of the entry's coords, so a same-name venue elsewhere
never lends its QID), and its labels across the full 28-language
photonLanguagesset populateLocalizations(kept only when they differ from the base name). This attaches the QID as awikidatasecondary id. - Load-time, from Photon. Entries Wikidata cannot match arrive with thin
localizations, so the async enrichment sweep's
gapFillLocalizationsreverse-geocodes their coordinates against the self-hosted Photon index across the app'sSupportedLocalizationLanguages, bringing them up to the same coverage as any runtime-minted row.
If neither Wikidata nor Photon has a localization, the field stays empty.
Coordinates and commercial detail come from the one shared enricher
(EnrichLocation in backend/internal/services/location/location_enricher.go),
the same path a runtime row and a capture row take. It fires once per
first-time-created location (enrichment_at NULL) and is two-tier and
provider-agnostic:
- Tier 1, Photon gap-fill (cheap, self-hosted), for every row. A coordless
row is forward-geocoded against Photon (
photonForwardGeocode), so a seed row that loaded without coordinates gets them from name + city. - Tier 2, paid commercial enrichment (Serper
/maps, then the Google Places API fallback). Its gate istagsAreRichDetail(loc.Tags)AND NOTisHarvested(loc)AND NOT already enriched (hasFreshGoogleFields). The rich-detail category gate alone is not enough: the high-cardinality density backbone (machine-harvested POIs, stampedharvestedprovenance) must never be paid-enriched, soisHarvestedexcludes it and it gets Tier 1 only. Prestige, runtime, and capture rows are not bulk, so they enrich normally; a capture-path row that arrived Serper-enriched is skipped by the already-enriched guard. The mapper fills the first-class location columns (website,phone_number,international_phone_number,price_level,business_status,google_maps_uri,opening_hours,place_types,full_address,district). - Localization gap-fill (Photon) and embedding generation run for every row, independent of the commercial gate, so a row still gets embedded even when the commercial budget is exhausted.
A prestige row that merges into an existing location does not re-resolve. Serper returns Google Maps data, so it carries Google's terms; we store facts (coords, address, contact, hours) under standard usage.
Unmatched entries keep their native identity (Michelin) or their name + city (50 Best) so the funnel's fuzzy dedup can place them onto an existing backbone POI. La Liste is a follow-up: its ranking is a client-rendered app with no server HTML or open feed.
Curated photos¶
The curated prestige collections (michelin, unesco, 50-best) carry one Google
Places photo per venue. At generation the seeder captures a single photo per
unique place_id into place-photos/<place_id>.<ext> and writes a
curated-photos.ndjson mapping (place_id -> file). At load time
thumbAttacher (backend/cmd/catalogseeder/command/load/loader_thumbnails.go)
rehosts each matched photo to the object-store key places-images/<place_id>.<ext>
and stamps the location's thumbnail_key (served via storage.PublicURL). The
key is deterministic, so a re-load overwrites the same object rather than orphaning
a new one.
A venue photographed without a validated Google id is keyed by its seed identity
instead: a curated-photos.ndjson row with seed_ref (<provider>:<provider_id>,
e.g. michelin:alici) in place of place_id. The loader tries the row's Google id
first, then its seed ref.
It is best-effort: no object store configured or no place-photos/ directory
in the seed means thumbnails are skipped and the load still succeeds (as it did
before photos were captured). A missing file or upload error is logged per-row and
skipped, never failing the load. db:seed:catalog pulls place-photos/ alongside
the NDJSON set so a local load can rehost too.
Geo covers¶
Every country, admin region, and city carries a cover photo for the atlas, map, and discovery surfaces, self-hosted under a deterministic key so the image can be swapped later without a reseed.
Generate. catalogseeder geocover --seed-dir <dir> --out <dir> resolves one
cover per entity from a source ladder and downloads it to
place-photos/geo/<kind>/<key>.jpg, appending a resumable geo-covers.ndjson
manifest row (kind, key, name, source, image_file, and author / license /
source_url). Keys are natural and stable: country ISO2 (FI), region
<provider>_<id> or ISO 3166-2, city GeoNames id.
- Unsplash is the primary headline-tier source (highest-quality editorial
photography), tried first for countries, ADM1 regions and top/airport cities when
UNSPLASH_ACCESS_KEYis set. A relevance guard requires the photo's tags or description to mention a significant token of the place name, so a query Unsplash can't match falls through to Pixabay rather than borrowing an unrelated photo. Photos tagged as conflict or weapons imagery are dropped. Images are fetched at 1600px wide (JPEG) rather than the multi-MB original. The requireddownload_locationendpoint is triggered on selection. Calls are paced to the 1000 requests/hour production cap with a small random jitter;UNSPLASH_RATE_MSoverrides the spacing for a one-shot batch that fits inside one hour. - Pixabay is the fallback headline-tier source (no attribution required,
commercial use and rehosting allowed), tried when Unsplash yields nothing. Each
entity runs an ordered query ladder from most specific to bare name (
coverQueriesinbackend/cmd/catalogseeder/command/geocover/geocover.go), so an ambiguous name (Georgia, Chad, Turkey) biases toward a landmark or landscape hit before falling back to the plain name. One API key per worker; a wide crawl shards the city file across keys for throughput. - Wikipedia lead image is the keyless fallback when both API sources return nothing, and the sole source for the long tail.
Each manifest row carries a confidence review signal:
| Level | Meaning |
|---|---|
override |
A curated pick from the overrides file |
high |
Unsplash hit whose tags or description confirm the place name |
medium |
First usable Unsplash scene with no name confirmation, or a Pixabay hit |
low |
Later-rung Unsplash fallback or flag/symbol, or a Wikipedia lead image |
--overrides <file> pins hand-picked photos: a JSON object of
"<kind>:<key>": "<unsplash photo id>" (for example "country:FR"). A pinned entity
bypasses search entirely, and a missing file is treated as no overrides.
--only country|region|city scopes a run to one kind, --tier-only limits the long
tail, and the done-check skips entities already in the manifest, so a run resumes
without re-spending API budget.
Load. loader_geocover.go uploads each manifest image to a content-addressed key,
geo-covers/<kind>/<key>-<hash>.jpg, where <hash> is the first 12 hex characters of
the SHA-256 of the image bytes. It stamps the matching row's cover_image_key plus
cover_author / cover_license / cover_source_url, matched by natural key:
countries.code, regions.(provider, provider_id) or iso_3166_2, and
locations.id via the location_providers GeoNames junction for cities. It is
best-effort per row (a missing file, upload error, or no-match row is logged and
counted) and needs the entity rows to exist. At read time the cover URL composes from cover_image_key.
load --geocover-manifest <file> picks the manifest, relative to --from (default
geo-covers.ndjson; a path outside the seed dir is rejected). Each row's image_file
resolves under the seed dir, so images need not live in place-photos/geo/.
Seed bucket layout. The tomoda-location-seed bucket holds four independent trees:
| Prefix | Holds |
|---|---|
seed/<version>/ |
An immutable generated place set (manifest, NDJSON, place-photos/) |
seed-archive/ |
Retired seed versions, never loaded |
geo-covers/<set>/ |
An immutable cover set: the manifest (geo-covers.ndjson or approved-manifest.ndjson) plus place-photos/geo/<kind>/<key>.jpg |
simulation/<version>/ |
The small self-contained bundle behind SIM=1, covers downscaled to 512px included |
Covers live at the bucket root, not inside a seed version, so a new cover set ships without republishing the multi-GB place set and a place set ships without covers. A seed version that does carry a cover manifest still loads it.
Cover lifecycle. Covers are content-addressed: the object key embeds a hash of the image bytes, and a row is written only when its key or credit differs from the manifest.
- Rows whose entity is not loaded yet (a city that loads later) upload nothing and are
logged as unmatched. The manifest then stays incomplete in
seed_load_state, so the next run re-checks it from the start and attaches them once the entity exists. - The checkpoint checksum covers the manifest and every image's key, so replacing an image under an unchanged manifest counts as a new set. A killed run resumes at its saved row.
- An unchanged, fully attached set is skipped without touching the database or the
bucket.
--forcere-checks every row and re-uploads only objects missing from the bucket. - A replaced cover leaves its previous object in place until a prune removes it, so clients holding the old URL keep working until you choose to clean up.
Update geo covers (runbook).
- Publish the reviewed set as a new
geo-covers/<set>/tree in the seed bucket and pull it to a local directory (rclone copy r2:tomoda-location-seed/geo-covers/<set> ./covers). The manifest may be the generator'sgeo-covers.ndjsonor an approved subset; pick it with--geocover-manifest. - Dry run, dev first. Without prune flags the load only adds and replaces:
cd backend
go run ./cmd/catalogseeder load --geo-covers-only --from ./covers
The log reports stamped, unchanged, unmatched and failed counts. A second
run of the same set reports a skip and writes nothing. A store or database error
is logged once, stops the cover step at that row, leaves the checkpoint incomplete and
fails the process, so the Job shows as failed; rerun resumes at that row.
3. To remove covers the set no longer names, add the prune. It is a dry run that only
reports counts and the unreferenced objects it would delete until --prune-apply is
passed:
go run ./cmd/catalogseeder load --geo-covers-only --from ./covers --prune-geo-covers # report
go run ./cmd/catalogseeder load --geo-covers-only --from ./covers --prune-geo-covers --prune-apply
The prune treats the manifest as the full desired set: rows it does not name lose
their cover. For a partial manifest, narrow it with --prune-source <host substring>
or skip it. After the row clear, it deletes the geo-covers/ objects that no country,
region or city row references and the manifest does not name, re-checking each key
against the database first. An object any row still uses is never deleted.
4. Repeat against prod with its config. For the Job, run the same binary with
--geo-covers-only instead of the full load; no seed version is involved.
catalogseeder geocover prune --source <host substring> is the standalone form: it
clears the four cover columns on rows whose cover_source_url host contains the
substring (for example pixabay), sparing rows named by --exclude-manifest <ndjson>.
It is a dry run that prints counts per table unless --apply is passed, and it leaves
objects in the bucket.
A place with no manifest row keeps no cover and the clients render a placeholder.
Public serving. Geo covers are generic and non-sensitive, so they are served as
plain unsigned CDN URLs rather than the short-lived signed URLs private assets use.
storage.Prefix.IsPublic() marks the geo-covers/ prefix and
assetService.URLFromKey returns {baseURL}/{key} for it instead of signing, so
the edge caches them. The assets Worker serves those paths unsigned. See
Storage.
Swap a cover without reseeding. Replace the image in the cover set and run the
runbook above. Update cover_author / cover_license through the
manifest row so the credit matches the new photo.
The notable overlay (Wikidata)¶
Wikidata supplies notable depth and multilingual labels for the curated tier.
- Wikidata notable places. A per-class SPARQL (one bounded
P31/P279*walk per place class, so WDQS does not time out) pulls notable destinations across what a traveler wants to see, do, eat, and shop: nature and wonders (mountain, volcano, waterfall, lake, island, canyon, cave, viewpoint, national park, nature reserve, garden), landmarks and architecture (museum, monument, memorial, castle, fortress, bridge, tower, lighthouse), entertainment (amusement park, water park, zoo, aquarium, casino, stadium), market destinations (night market, hawker centre, market hall, marketplace), shopping (mall, department store, shopping street, bazaar, arcade), and religious sites (temple, shrine, pagoda, stupa, church, mosque). Individual restaurants, cafes, and shops stay out of this overlay: those arrive through on-demand harvesting plus the dining lists, so the eager-enriched embedded tier stays bounded. Classes are most-specific first so the first class to claim a QID wins. Entities need coordinates and must clear a sitelink floor (--notable-min-sitelinks, default 5, low enough to reach mid-tier cities' notable places); each class contributes up to 25000 entities. Ultra-common natural classes (mountain, island) carry a higher per-class floor so their capped set surfaces marquee entities rather than an obscure tail. Each entity's class maps to a taxonomy leaf, sitelinks become the prominence signal, andwikidataLabelsattaches the full 28-languagephotonLanguageslabel set. These rows carryprovider = wikidataand load as the normal enrichablewikidata_seed, so they enrich eagerly. A single class query failure is skipped and logged, not fatal, so one transient WDQS hiccup does not lose the other classes' work; the run still aborts without writing when more than a quarter of the classes fail or zero entities are collected, so a systemic WDQS outage cannot clobber a good notable file with a near-empty one.
Unified POI generator (merge)¶
catalogseeder merge (backend/cmd/catalogseeder/command/merge) is optional. It
writes the notable rows as pois.ndjson and replaces the wikidata-notable-places
manifest entry with one pois entry (tier density), streaming to a .partial
sidecar it renames on success; it preserves the manifest's format version. The Overture
backbone that used to share pois is retired, so the rows are the notable layer
unchanged. A version carries the notable layer as wikidata-notable-places or as
pois, never both; the published seed/2026-11-08/ keeps wikidata-notable-places.
When unesco-world-heritage.ndjson is present, merge also folds each World Heritage
site into the place it matches and rewrites that file and its manifest entry in place.
The folded UNESCO row and the notable row share an identity, so the loader reconciles
them into one place instead of inserting a twin. See
Curated collections for the matching rules.
Build-time enrichment (enrich)¶
catalogseeder enrich (backend/cmd/catalogseeder/command/enrich) moves the
geography + localization work that the load-time enricher used to do forward to
build time, so the published NDJSON is already complete and the loader inserts it
straight. For every coordinate-bearing row in the curated + density tiers it:
- reverse-geocodes the coordinate against Photon once per locale across the 28
SupportedLocalizationLanguages, fillingregion/city/country/addressfrom the English result and per-locale name + address + city + country intolocalizations, keeping any source-provided names as the base; - canonicalizes
countryto the countries gazetteer's single spelling (so the seed never carries "USA" next to "United States"); - for the curated prestige lists (Michelin, UNESCO, the "50 Best" family), adds
the postal address, Google place id and commercial detail from Serper. Serper
returns its top place even when that is a different venue, so a hit is kept only
when its title matches the row's name and it sits close enough (
policyFor): within 1.5km for Michelin (address-precise), 30km for UNESCO, and within the metro (120km) for "50 Best", whose rows then move to the hit's position because their seed coords are a city-validated Photon guess. Serper runs before Photon so the reverse-geocode uses the final position; - stamps the row
enriched, which the loader reads to setenrichment_at, so the async worker skips it.
The pass is parallel (a worker pool over rows, a global cap on concurrent Photon
calls) and content-addressed cached + resumable: each Photon and Serper response
is keyed to disk, so a resumed run reads cache instead of the network, and a Serper
credit-out checkpoints the .partial and exits rather than losing progress. Only
fresh paid calls feed the side files (via .serper-calls.ndjson in the seed dir),
so replaying cached responses adds no spend. The
load-time enricher (EnrichLocation) remains the path for runtime-minted and
capture rows and for any seed row that was not pre-enriched.
Country backfill (backfill-country)¶
catalogseeder backfill-country
(backend/cmd/catalogseeder/command/backfill/backfill.go) fills country /
country_code on rows Photon left blank (rural, offshore, or remote coordinates),
so the validate invariant "an enriched row carries a country" comes back clean
without re-geocoding. It runs after enrich and before validate. For each
coordinate-bearing row missing a country it does a point-in-polygon test against
the country boundary polygons already carried on world-countries (loaded once
into github.com/paulmach/orb geometries, each behind a bbox prefilter), then
canonicalizes the country name from the resolved code. A point inside no polygon
takes a nearest-country fallback within ~1 degree (nearestCapDeg, ~110km), so a
coastal or offshore feature still resolves while a mid-ocean point does not.
go run ./cmd/catalogseeder backfill-country --from ./seed-out --slug wikidata-notable-places --drop-unresolved
--slug restricts to one collection (default: every collection); it rewrites each
touched file in place and refolds its checksum into the manifest. With
--drop-unresolved, an enriched row still without a country after the fill is
dropped: these are the genuine off-Earth features (Martian and lunar volcanoes,
international-waters shipwrecks) that carry a Wikidata QID and coordinates but sit
inside no country polygon.
Maintenance and annual refresh¶
The rankings re-publish on an annual cadence (each "50 Best" award ceremony,
Michelin's yearly guide), so these collections are refreshed by re-running the
generator per source and re-loading. Re-running is cheap: the seed-time resolver
memoizes by (name, city) within a run, and the load path dedups by source ref, so
a venue that also appeared last edition reconciles rather than duplicates. The
paid step (Serper /maps) only fires for entries Photon cannot place, and only
for venues not already resolved, so a refresh bills roughly for the year's new
entries, not the whole list.
Each curated load prunes its own collection: once the file has loaded, memberships of that collection whose place the file no longer lists are removed, so a venue that fell off a ranking loses its badge. Other collections and user data are never touched, and a reload of the same file removes nothing.
A refresh must reconcile collection membership, not just upsert locations:
collection_memberships carries the edition + rank + is_current, so a re-load
updates ranks, marks dropped venues non-current, and stamps the new edition per
the collection's retention policy. Locations dropped from a ranking are
deactivated by the funnel, never deleted (see the reconciliation invariant below).