Building the Photon multilang index

How to produce a multilingual Photon index tarball and publish it to GCS so the in-cluster Photon Deployment picks it up. The index is what gives ?lang=ja, ?lang=zh-Hans etc. real localized names instead of English fallbacks — without it Photon answers everything in English.

Output of a successful build is four objects in gs://${PROJECT_ID}-photon-index-usc1/<region>/:

• photon-db-<region>-multilang-<YYYY-MM>.tar.bz2 — the dated artifact
• photon-db-<region>-multilang-<YYYY-MM>.tar.bz2.md5 — its checksum
• photon-db-<region>-multilang-latest.tar.bz2 — the latest alias prod polls
• photon-db-<region>-multilang-latest.tar.bz2.md5 — alias checksum

Photon pods poll the latest MD5 every 24h and atomically swap when it changes — no Photon-side action needed once the tarball is uploaded.

Companion: dev reference

k8s/envs/platform/photon-indexer/README.md is the dev/script reference (script flags, env knobs, file layout, troubleshooting table, the "Resuming a partially-indexed Nominatim" recovery recipe, the 28-language rationale). This page is the operator walkthrough — read both if you're touching the pipeline.

Prerequisites

• GCS bucket and photon-indexer SA exist. One-time per project, provisioned in bootstrap → Step 2. Don't re-do that here.
• A large host. Canada and Asia regional builds can run on a beefy laptop (32 GB RAM minimum, ~250 GB free disk for Asia). Planet is GCE-only — n2-highmem-32 with 4 × 375 GB local NVMe SSDs in RAID-0 is the tested shape (~1.4 TB scratch, ~5 days wall-clock, ~$200-400/run).
• gcloud authed against development-485000, including ADC: gcloud auth login && gcloud auth application-default login && gcloud config set project development-485000. The wrapper script uses ADC for the final gsutil cp.
• Docker + Compose v2 on the build host.

What the build actually does

Photon can't read OSM .osm.pbf directly — it imports from a populated Nominatim Postgres. So the pipeline is two heavy stages glued by docker-compose.build.yml, plus an upload step:

PBF download (host, resumable)
        │
        ▼
Stage 1 — Nominatim         mediagis/nominatim
   osm2pgsql → Postgres → Nominatim's own rank indexing
   (hours-to-days; dominates wall-clock)
        │
        ▼
Stage 2 — Photon import     photon-indexer:local
   streams from Nominatim Postgres → Lucene index in photon_data/
   (minutes-to-an-hour)
        │
        ▼
Stage 3 — tar + upload      host script
   pbzip2-tar photon_data/ → md5 → gsutil cp → refresh latest aliases

The whole pipeline is driven by scripts/photon-index-local.sh (preflight + tuning defaults) which execs into k8s/envs/platform/photon-indexer/build-index.sh (the orchestrator).

Build procedure

1. Build the indexer image

cd k8s/envs/platform/photon-indexer
docker build -t photon-indexer:local .

The Dockerfile copies photon.jar straight out of rtuszik/photon-docker:2.2.0 so indexer and runtime stay byte-identical. When bumping the Photon version, bump that tag here, in k8s/envs/platform/photon/manifests.yaml, and in the app repo's docker-compose.dev.yml together — index file format changes across majors.

2. (Planet only) Provision the build VM

Skip for laptop builds. For planet, run through the steps below.

The SSD_TOTAL_GB quota covers both pd-ssd and pd-balanced (they share one limit, default 500 GB on a new project). The walkthrough uses local SSDs (separate LOCAL_SSD_TOTAL_GB quota, faster storage, ~same price). If you have SSD_TOTAL_GB ≥ 1500 GB, you can skip the RAID dance and use a single pd-ssd boot disk — see "pd-ssd variant" at the bottom.

# Check both quotas first
gcloud compute regions describe us-central1 --format="json" \
    | jq '.quotas[] | select(.metric == "SSD_TOTAL_GB" or .metric == "LOCAL_SSD_TOTAL_GB")'

One-time firewall rule for IAP SSH (only needed once per project):

gcloud services enable iap.googleapis.com
gcloud compute firewall-rules create allow-ssh-from-iap \
    --direction=INGRESS --action=ALLOW --rules=tcp:22 \
    --source-ranges=35.235.240.0/20 \
    --description="Allow SSH from IAP TCP forwarding"

IAP routes SSH through Google's authenticated tunnel — safer than 0.0.0.0/0:22. Every subsequent gcloud compute ssh/scp needs --tunnel-through-iap.

Provision the VM with the photon-indexer SA attached so gsutil cp works at the end. Boot disk on pd-standard (cheap, plentiful quota); the four local NVMe SSDs hold the actual scratch:

gcloud compute instances create photon-build --machine-type=n2-highmem-32 --boot-disk-size=50GB --boot-disk-type=pd-standard --local-ssd=interface=NVME --local-ssd=interface=NVME --local-ssd=interface=NVME --local-ssd=interface=NVME --image-family=debian-12 --image-project=debian-cloud --service-account=photon-indexer@development-485000.iam.gserviceaccount.com --scopes=cloud-platform --zone=us-central1-b

Two foot-guns at VM-create time

• Service account: without --service-account=photon-indexer@… the VM uses the default compute SA, which has no GCS write on the index bucket. Build runs ~5 days, then 403s at upload. Smoke-test with gsutil cp /tmp/x gs://bucket/y before kicking off the long build.
• Line continuations: the \ line continuation breaks easily on copy-paste (trailing spaces eat the newline). If you see command not found: --machine-type=…, paste the whole command as one physical line.

Upload the indexer code + wrapper script:

gcloud compute scp --tunnel-through-iap --recurse \
    ~/workspace/devops/k8s/envs/platform/photon-indexer photon-build:/tmp/indexer \
    --zone=us-central1-b
gcloud compute scp --tunnel-through-iap \
    ~/workspace/devops/scripts/photon-index-local.sh photon-build:/tmp/ \
    --zone=us-central1-b

Don't put a trailing / on the destination of a recursive scp — modern OpenSSH refuses to create the target if it doesn't exist.

Stripe the four local SSDs into one volume (RAID-0 → XFS → /workspace):

gcloud compute ssh --tunnel-through-iap photon-build --zone=us-central1-b -- '
sudo apt-get update && sudo apt-get install -y mdadm xfsprogs
sudo mdadm --create --verbose --run /dev/md0 --level=0 --raid-devices=4 \
    /dev/nvme0n1 /dev/nvme0n2 /dev/nvme0n3 /dev/nvme0n4
sudo mkfs.xfs /dev/md0
sudo mkdir -p /workspace
sudo mount /dev/md0 /workspace
sudo chmod 777 /workspace
df -h /workspace
'

The --run flag is required for non-interactive creation; without it mdadm waits for a confirmation prompt that never comes.

Install Docker on /workspace + tmux + pbzip2 (parallel bzip2 saves ~2h of compression on 32 vCPU):

gcloud compute ssh --tunnel-through-iap photon-build --zone=us-central1-b -- '
curl -fsSL https://get.docker.com | sudo sh
sudo apt-get install -y tmux bzip2 pbzip2
sudo systemctl stop docker
sudo mkdir -p /workspace/docker
echo "{\"data-root\": \"/workspace/docker\"}" | sudo tee /etc/docker/daemon.json
sudo systemctl start docker
docker info 2>/dev/null | grep "Docker Root Dir"
'

Expect Docker Root Dir: /workspace/docker — confirms Docker is using local SSD for image layers + volumes.

Move the indexer in place and build its image:

gcloud compute ssh --tunnel-through-iap photon-build --zone=us-central1-b -- '
sudo gcloud auth configure-docker -q
sudo mv /tmp/indexer /workspace/indexer
sudo mv /tmp/photon-index-local.sh /workspace/photon-index-local.sh
sudo chmod +x /workspace/photon-index-local.sh /workspace/indexer/build-index.sh
cd /workspace/indexer && sudo docker build -t photon-indexer:local .
'

Verify before the long-running build:

gcloud compute ssh --tunnel-through-iap photon-build --zone=us-central1-b -- '
df -h /workspace
docker info 2>/dev/null | grep "Docker Root Dir"
docker images photon-indexer:local
'

Expect ~1.5 TB free on /workspace, Docker rooted there, and a recent photon-indexer:local image.

Never gcloud compute instances stop the VM mid-build

Local SSD data vanishes on stop (not just on delete). Detach tmux and exit the SSH session if you want to step away, but never instances stop.

3. Pre-stage the PBF (recommended for planet)

The orchestrator can download the PBF itself, but for planet (~86 GB) it's smart to kick the download off separately so it can resume across SSH sessions without holding the orchestrator hostage. curl -C - resumes from disk on retry:

sudo nohup curl -L -C - --retry 10 --retry-delay 30 --retry-all-errors \
    -o /workspace/pbf/planet-latest.osm.pbf \
    https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf \
    > /workspace/pbf.log 2>&1 &
disown

Pass the resulting path via PBF_HOST_PATH=... when kicking off the build so the orchestrator skips its own download.

4. Kick off the build

# Laptop — canada
cd scripts
./photon-index-local.sh canada

# Cloud VM — planet (in a tmux or with nohup so SSH drops don't kill it)
sudo bash -c "echo y | env \
    PHOTON_INDEXER_DIR=/workspace/indexer \
    PBF_HOST_PATH=/workspace/pbf/planet-latest.osm.pbf \
    nohup /workspace/photon-index-local.sh planet \
    > /workspace/planet.log 2>&1 &"

scripts/photon-index-local.sh confirms region, bucket, languages, and JVM/Postgres tuning before prompting — read the printout, it's your last chance to catch a wrong region or stale gcloud config. Defaults are sized for n2-highmem-32; for a 32 GB laptop, override JAVA_HEAP, POSTGRES_SHARED_BUFFERS, POSTGRES_MAINTENANCE_WORK_MEM, and OSM2PGSQL_CACHE downward (see the comment block in the script).

PHOTON_INDEXER_DIR is required on the cloud VM where the layout is flattened to /workspace/indexer. Locally the script resolves it relative to the devops repo and you can omit it.

5. Wait

Rough wall-clock per region on n2-highmem-32:

Region	Stage 1 (Nominatim)	Stage 2 (Photon)	Stage 3 (tar+upload)	Total
Canada	1-2h	~10 min	~5 min	2-3h
Asia	5-8h	~30 min	~30 min	6-9h
Planet	~4 days	~1h	~30 min	~5 days

The orchestrator prints heartbeats every 5 min during Stage 1. Stage 1 is almost always the long pole — Postgres's CREATE INDEX during Nominatim's rank-indexing phase dominates. If you suspect it's hung, docker compose -f k8s/envs/platform/photon-indexer/docker-compose.build.yml logs -f nominatim shows live osm2pgsql / nominatim index progress.

Resuming a failed planet build

If a planet build dies mid-way (OOM, healthcheck flip, accidental teardown), do not docker compose down -v — the Nominatim data volume is hours-to-days of work. The README has a "Resuming a partially-indexed Nominatim" recipe that touches import-finished, bypasses mediagis's start.sh (its --warm step fails on partial data), and resumes with nominatim index --minrank N. Saves ~24h on a planet rebuild.

6. Verify upload

PROJECT_ID="development-485000"
gsutil ls gs://${PROJECT_ID}-photon-index-usc1/<region>/

You want all four files (dated + latest, each with its .md5). If only the dated pair appears, the latest alias copy failed — re-run just that step (see "Flipping latest" below).

Regional builds vs the planet

The planet index is the production target. It's what prod Photon serves and what gives the app worldwide multilang coverage.

Regional builds (canada, asia) exist mainly to iterate on the pipeline cheaply — verifying the indexer image, the compose wiring, a new Photon version bump, or a new language in the LANGUAGES list, without burning days on a planet rebuild. Once a working planet artifact is in GCS, the regional artifacts rarely need to change.

Regional builds also use a shorter language list (en,ja,zh,zh-Hans,zh-Hant — just enough to exercise the multilang code path). Planet uses the full 28-language set — see "Why these languages" in k8s/envs/platform/photon-indexer/README.md for the rationale.

Chaining regions on the same VM

The n2-highmem-32 is sized for planet, so canada and asia have huge headroom on it. You can chain regions on the same VM rather than spinning up separate ones — the cheap way to refresh dev's smaller indexes alongside a production planet build.

Two safe modes:

1. Sequential, same compose stack. Run canada → wipe volumes → run asia → wipe → run planet. Each does a full Nominatim import. ~25h total (2-3h canada + 8-9h asia + ~20h planet).
2. Small region overlapped with planet's PBF download. The host curl for the planet PBF uses ~0 RAM/CPU, so a small build can have the full VM during that window. Do NOT overlap any other build with planet's Nominatim phase — it needs all 256 GB. Watch free -h; if usage > 200 GB or swap kicks in, kill the small build immediately.

Critical cleanup between regions — canada/asia/planet share the same compose project so the same volume names. Before starting the next region, wipe the previous run's volumes or Nominatim will refuse to re-import on top of stale data:

sudo docker compose -f /workspace/indexer/docker-compose.build.yml down -v
sudo docker volume ls | grep photon-build       # should be empty
sudo df -h /workspace                            # should drop ~30-300 GB

Flipping latest

Photon pods read <basename>-latest.tar.bz2 and re-check its MD5 every 24h. A successful build sets the alias automatically. To re-point it manually — for rollback to an older artifact, or recovering from a partial upload:

BUCKET="${PROJECT_ID}-photon-index-usc1"
REGION="planet"
SNAPSHOT="2026-05"
BASE="photon-db-${REGION}-multilang"

gsutil cp gs://${BUCKET}/${REGION}/${BASE}-${SNAPSHOT}.tar.bz2 \
          gs://${BUCKET}/${REGION}/${BASE}-latest.tar.bz2
gsutil cp gs://${BUCKET}/${REGION}/${BASE}-${SNAPSHOT}.tar.bz2.md5 \
          gs://${BUCKET}/${REGION}/${BASE}-latest.tar.bz2.md5

To force prod to re-pull immediately instead of waiting up to 24h for the next poll:

kubectl rollout restart -n data deployment/photon
kubectl logs -n data -f deployment/photon   # watch "loaded successfully"

If you've never built before and the PVC has stale data from a previous failed pull, also kubectl delete pvc -n data photon-data-pvc before the restart so the new pod downloads from scratch.

Smoke test multilingual coverage

After prod picks up the new index, port-forward Photon and confirm a few non-English locales return localized names:

kubectl port-forward -n data svc/photon 2322:2322 &
sleep 2

for lang in en ja ko zh-Hans zh-Hant ar es fr de hi th; do
    name=$(curl -s "http://localhost:2322/reverse?lat=35.6762&lon=139.6503&lang=${lang}" \
        | jq -r '.features[0].properties.name')
    printf "  %-10s %s\n" "${lang}" "${name}"
done

kill %1

Expected for Tokyo: en → Tokyo, ja → 東京都, ko → 도쿄도, zh-Hans → 东京, zh-Hant → 東京, ar → طوكيو, etc. If any locale falls back to English the index isn't multilang — re-check the LANGUAGES env var in the build run.

pd-ssd variant (when SSD quota allows)

If SSD_TOTAL_GB ≥ 1500 GB (after a quota bump), you can skip the local-SSD RAID and use a single pd-ssd boot disk:

gcloud compute instances create photon-build --machine-type=n2-highmem-32 --boot-disk-size=1500GB --boot-disk-type=pd-ssd --image-family=debian-12 --image-project=debian-cloud --service-account=photon-indexer@development-485000.iam.gserviceaccount.com --scopes=cloud-platform --zone=us-central1-b

Skip the mdadm + Docker-data-root steps. Docker stays on the default /var/lib/docker (which is on the big pd-ssd boot). All other steps are identical. Tradeoff: pd-ssd is slightly slower than local NVMe and ~2× the cost per GB-month, but bootstrap is shorter.

After the build (cloud VM only)

Once you've verified the four GCS objects exist, delete the VM:

gcloud compute instances delete photon-build --zone=us-central1-b --quiet

The local SSD scratch space vanishes with the VM. That's fine — the artifact is in GCS.

Related

• Photon Indexer infra — bucket, SA, IAM bindings (the resource view)
• Photon (Kubernetes app) — the in-cluster Deployment that consumes the index
• Bootstrap — Step 2 — one-time bucket + SA creation
• k8s/envs/platform/photon-indexer/README.md — dev reference: script flags, env knobs, troubleshooting table, "Resuming a partially-indexed Nominatim" recovery recipe, and the 28-language rationale