Building the Photon multilang index¶
How to produce a multilingual Photon index tarball and publish it to R2 so the in-cluster Photon Deployment picks it up. The index is what gives ?lang=ja, ?lang=zh-Hans etc. real localized names instead of English fallbacks.
Output of a successful build is four objects in r2:tomoda-photon-index/<region>/ (public at https://photon-index.tomoda.life/<region>/):
photon-db-<region>-multilang-<YYYY-MM>.tar.bz2: the dated artifactphoton-db-<region>-multilang-<YYYY-MM>.tar.bz2.md5: its checksumphoton-db-<region>-multilang-latest.tar.bz2: thelatestalias prod pollsphoton-db-<region>-multilang-latest.tar.bz2.md5: alias checksum
Photon polls the latest MD5 every 24h. Prod runs UPDATE_STRATEGY=SEQUENTIAL (the server disk can't hold two planet indexes), so a changed checksum means geocoding is down while the new index downloads and unpacks (an hour or two). Upload new planet indexes in a quiet window.
Companion: dev reference
k8s/envs/platform/photon-indexer/README.md is the script reference (flags, env knobs, troubleshooting table, the "Resuming a partially-indexed Nominatim" recipe, the 28-language rationale). This page is the operator walkthrough.
Prerequisites¶
- A large build machine. Canada and Asia regional builds run on a beefy laptop (32 GB RAM minimum, ~250 GB free disk for Asia). Planet needs 32 vCPU, 256 GB RAM and ~1.5 TB of local NVMe for ~5 days; rent one by the hour and delete it afterwards. Never use the cluster server.
- rclone with a remote named
r2whose token can writetomoda-photon-index. - Docker + Compose v2 on the build host.
What the build does¶
PBF download (host, resumable)
│
▼
Stage 1: Nominatim mediagis/nominatim
osm2pgsql → Postgres → Nominatim rank indexing (hours to days)
│
▼
Stage 2: Photon import photon-indexer:local
Nominatim Postgres → index in photon_data/ (minutes to an hour)
│
▼
Stage 3: tar + upload host script
pbzip2 tar → md5 → rclone to R2 → server-side copy to latest (md5 last)
scripts/photon-index-local.sh (preflight + tuning defaults) execs into k8s/envs/platform/photon-indexer/build-index.sh (the orchestrator).
Build procedure¶
1. Build the indexer image¶
cd k8s/envs/platform/photon-indexer
docker build -t photon-indexer:local .
The Dockerfile copies photon.jar out of rtuszik/photon-docker:2.2.0 so indexer and runtime stay byte-identical. When bumping Photon, bump that tag here, in k8s/envs/platform/photon/manifests.yaml, and in the app repo's docker-compose.dev.yml together; the index format changes across majors.
2. (Planet only) Prepare the build machine¶
Copy k8s/envs/platform/photon-indexer/ to /workspace/indexer and scripts/photon-index-local.sh to /workspace/, put Docker's data root on the NVMe scratch, build the image there, and configure the r2 rclone remote. Run everything inside tmux so SSH drops don't kill it.
3. Pre-stage the PBF (recommended for planet)¶
nohup curl -L -C - --retry 10 --retry-delay 30 --retry-all-errors \
-o /workspace/pbf/planet-latest.osm.pbf \
https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf \
> /workspace/pbf.log 2>&1 &
Pass the path via PBF_HOST_PATH=... so the orchestrator skips its own download.
4. Kick off the build¶
# Laptop: canada
./scripts/photon-index-local.sh canada
# Build machine: planet
echo y | PHOTON_INDEXER_DIR=/workspace/indexer \
PBF_HOST_PATH=/workspace/pbf/planet-latest.osm.pbf \
/workspace/photon-index-local.sh planet 2>&1 | tee /workspace/planet.log
The script prints region, bucket, languages and JVM/Postgres tuning before prompting; read it. Defaults are sized for the 256 GB machine; for a 32 GB laptop, lower JAVA_HEAP, POSTGRES_SHARED_BUFFERS, POSTGRES_MAINTENANCE_WORK_MEM and OSM2PGSQL_CACHE (see the comment block in the script).
5. Wait¶
| Region | Stage 1 (Nominatim) | Stage 2 (Photon) | Stage 3 (tar+upload) | Total |
|---|---|---|---|---|
| Canada | 1-2h | ~10 min | ~5 min | 2-3h |
| Asia | 5-8h | ~30 min | ~30 min | 6-9h |
| Planet | ~4 days | ~1h | ~30 min | ~5 days |
docker compose -f k8s/envs/platform/photon-indexer/docker-compose.build.yml logs -f nominatim shows live progress.
Resuming a failed planet build
Do not docker compose down -v after a failure; the Nominatim volume is days of work. Follow the README's "Resuming a partially-indexed Nominatim" recipe.
6. Verify the upload¶
rclone ls r2:tomoda-photon-index/<region>/
You want all four files. If only the dated pair is there, re-point latest as below.
Regional builds vs the planet¶
The planet index is the production target. Regional builds (canada, asia) exist to iterate on the pipeline cheaply and to feed local dev (docker-compose.dev.yml pulls Canada from photon-index.tomoda.life). They use a short language list; planet uses the full 28-language set.
To chain regions on one machine, run them sequentially and wipe the compose volumes in between (docker compose ... down -v), because they share volume names. Never overlap another build with planet's Nominatim phase; it needs all the RAM.
Flipping latest¶
To roll back to an older artifact or recover from a partial upload (copy the tarball first, the .md5 last):
REGION=planet SNAPSHOT=2026-05 BASE="photon-db-${REGION}-multilang"
rclone copyto r2:tomoda-photon-index/${REGION}/${BASE}-${SNAPSHOT}.tar.bz2 \
r2:tomoda-photon-index/${REGION}/${BASE}-latest.tar.bz2
rclone copyto r2:tomoda-photon-index/${REGION}/${BASE}-${SNAPSHOT}.tar.bz2.md5 \
r2:tomoda-photon-index/${REGION}/${BASE}-latest.tar.bz2.md5
To pick it up now instead of at the next daily poll:
kubectl rollout restart -n platform deployment/photon
kubectl logs -n platform -f deployment/photon
Smoke test multilingual coverage¶
kubectl port-forward -n platform svc/photon 2322:2322 &
sleep 2
for lang in en ja ko zh-Hans zh-Hant ar es fr de hi th; do
name=$(curl -s "http://localhost:2322/reverse?lat=35.6762&lon=139.6503&lang=${lang}" \
| jq -r '.features[0].properties.name')
printf " %-10s %s\n" "${lang}" "${name}"
done
kill %1
Expected for Tokyo: en → Tokyo, ja → 東京都, ko → 도쿄도, zh-Hans → 东京, zh-Hant → 東京, ar → طوكيو. Any English fallback means the index isn't multilang; check LANGUAGES in the build run.
After the build¶
Delete the rented build machine once the four objects are in R2; the scratch data isn't needed.
Related¶
- Photon (Kubernetes): the in-cluster Deployment that consumes the index
- Cloudflare → R2: the bucket and its public domain
k8s/envs/platform/photon-indexer/README.md: script reference and recovery recipes