Skip to content

Building the Photon multilang index

How to produce a multilingual Photon index tarball and publish it to R2 so the in-cluster Photon Deployment picks it up. The index is what gives ?lang=ja, ?lang=zh-Hans etc. real localized names instead of English fallbacks.

Output of a successful build is four objects in r2:tomoda-photon-index/<region>/ (public at https://photon-index.tomoda.life/<region>/):

  • photon-db-<region>-multilang-<YYYY-MM>.tar.bz2: the dated artifact
  • photon-db-<region>-multilang-<YYYY-MM>.tar.bz2.md5: its checksum
  • photon-db-<region>-multilang-latest.tar.bz2: the latest alias prod polls
  • photon-db-<region>-multilang-latest.tar.bz2.md5: alias checksum

Photon polls the latest MD5 every 24h. Prod runs UPDATE_STRATEGY=SEQUENTIAL (the server disk can't hold two planet indexes), so a changed checksum means geocoding is down while the new index downloads and unpacks (an hour or two). Upload new planet indexes in a quiet window.

Companion: dev reference

k8s/envs/platform/photon-indexer/README.md is the script reference (flags, env knobs, troubleshooting table, the "Resuming a partially-indexed Nominatim" recipe, the 28-language rationale). This page is the operator walkthrough.

Prerequisites

  • A large build machine. Canada and Asia regional builds run on a beefy laptop (32 GB RAM minimum, ~250 GB free disk for Asia). Planet needs 32 vCPU, 256 GB RAM and ~1.5 TB of local NVMe for ~5 days; rent one by the hour and delete it afterwards. Never use the cluster server.
  • rclone with a remote named r2 whose token can write tomoda-photon-index.
  • Docker + Compose v2 on the build host.

What the build does

PBF download (host, resumable)
        │
        ▼
Stage 1: Nominatim         mediagis/nominatim
   osm2pgsql → Postgres → Nominatim rank indexing   (hours to days)
        │
        ▼
Stage 2: Photon import     photon-indexer:local
   Nominatim Postgres → index in photon_data/       (minutes to an hour)
        │
        ▼
Stage 3: tar + upload      host script
   pbzip2 tar → md5 → rclone to R2 → server-side copy to latest (md5 last)

scripts/photon-index-local.sh (preflight + tuning defaults) execs into k8s/envs/platform/photon-indexer/build-index.sh (the orchestrator).

Build procedure

1. Build the indexer image

cd k8s/envs/platform/photon-indexer
docker build -t photon-indexer:local .

The Dockerfile copies photon.jar out of rtuszik/photon-docker:2.2.0 so indexer and runtime stay byte-identical. When bumping Photon, bump that tag here, in k8s/envs/platform/photon/manifests.yaml, and in the app repo's docker-compose.dev.yml together; the index format changes across majors.

2. (Planet only) Prepare the build machine

Copy k8s/envs/platform/photon-indexer/ to /workspace/indexer and scripts/photon-index-local.sh to /workspace/, put Docker's data root on the NVMe scratch, build the image there, and configure the r2 rclone remote. Run everything inside tmux so SSH drops don't kill it.

nohup curl -L -C - --retry 10 --retry-delay 30 --retry-all-errors \
    -o /workspace/pbf/planet-latest.osm.pbf \
    https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf \
    > /workspace/pbf.log 2>&1 &

Pass the path via PBF_HOST_PATH=... so the orchestrator skips its own download.

4. Kick off the build

# Laptop: canada
./scripts/photon-index-local.sh canada

# Build machine: planet
echo y | PHOTON_INDEXER_DIR=/workspace/indexer \
    PBF_HOST_PATH=/workspace/pbf/planet-latest.osm.pbf \
    /workspace/photon-index-local.sh planet 2>&1 | tee /workspace/planet.log

The script prints region, bucket, languages and JVM/Postgres tuning before prompting; read it. Defaults are sized for the 256 GB machine; for a 32 GB laptop, lower JAVA_HEAP, POSTGRES_SHARED_BUFFERS, POSTGRES_MAINTENANCE_WORK_MEM and OSM2PGSQL_CACHE (see the comment block in the script).

5. Wait

Region Stage 1 (Nominatim) Stage 2 (Photon) Stage 3 (tar+upload) Total
Canada 1-2h ~10 min ~5 min 2-3h
Asia 5-8h ~30 min ~30 min 6-9h
Planet ~4 days ~1h ~30 min ~5 days

docker compose -f k8s/envs/platform/photon-indexer/docker-compose.build.yml logs -f nominatim shows live progress.

Resuming a failed planet build

Do not docker compose down -v after a failure; the Nominatim volume is days of work. Follow the README's "Resuming a partially-indexed Nominatim" recipe.

6. Verify the upload

rclone ls r2:tomoda-photon-index/<region>/

You want all four files. If only the dated pair is there, re-point latest as below.

Regional builds vs the planet

The planet index is the production target. Regional builds (canada, asia) exist to iterate on the pipeline cheaply and to feed local dev (docker-compose.dev.yml pulls Canada from photon-index.tomoda.life). They use a short language list; planet uses the full 28-language set.

To chain regions on one machine, run them sequentially and wipe the compose volumes in between (docker compose ... down -v), because they share volume names. Never overlap another build with planet's Nominatim phase; it needs all the RAM.

Flipping latest

To roll back to an older artifact or recover from a partial upload (copy the tarball first, the .md5 last):

REGION=planet SNAPSHOT=2026-05 BASE="photon-db-${REGION}-multilang"
rclone copyto r2:tomoda-photon-index/${REGION}/${BASE}-${SNAPSHOT}.tar.bz2 \
              r2:tomoda-photon-index/${REGION}/${BASE}-latest.tar.bz2
rclone copyto r2:tomoda-photon-index/${REGION}/${BASE}-${SNAPSHOT}.tar.bz2.md5 \
              r2:tomoda-photon-index/${REGION}/${BASE}-latest.tar.bz2.md5

To pick it up now instead of at the next daily poll:

kubectl rollout restart -n platform deployment/photon
kubectl logs -n platform -f deployment/photon

Smoke test multilingual coverage

kubectl port-forward -n platform svc/photon 2322:2322 &
sleep 2
for lang in en ja ko zh-Hans zh-Hant ar es fr de hi th; do
    name=$(curl -s "http://localhost:2322/reverse?lat=35.6762&lon=139.6503&lang=${lang}" \
        | jq -r '.features[0].properties.name')
    printf "  %-10s %s\n" "${lang}" "${name}"
done
kill %1

Expected for Tokyo: en → Tokyo, ja → 東京都, ko → 도쿄도, zh-Hans → 东京, zh-Hant → 東京, ar → طوكيو. Any English fallback means the index isn't multilang; check LANGUAGES in the build run.

After the build

Delete the rented build machine once the four objects are in R2; the scratch data isn't needed.

  • Photon (Kubernetes): the in-cluster Deployment that consumes the index
  • Cloudflare → R2: the bucket and its public domain
  • k8s/envs/platform/photon-indexer/README.md: script reference and recovery recipes