Async (Asynq)¶
All background work runs through one system — Asynq, a Redis-backed task queue. The same Asynq deployment runs the server (consumes event-driven and cron-enqueued tasks) and the scheduler (enqueues cron-driven tasks on a wall-clock schedule).
Source: backend/internal/async/. Layout:
internal/async/
├── tasks_event.go ← event task constants + payloads (location enrich, engine signals)
├── tasks_item.go ← plan-item task constants (item:parse, link:enrich)
├── tasks_cron.go ← cron task constants
├── tasks_cleanup.go ← cleanup task constants + payloads (S3 delete, orphan sweep)
├── client.go ← Asynq client (event-driven enqueue)
├── server.go ← Asynq server + Handler interface
├── scheduler.go ← Asynq scheduler + cronEntries
├── app.go ← lifecycle wrapper (Start / Shutdown)
└── handlers/
├── handlers.go ← HandlerService struct + constructor + Register
├── event.go ← event handler interfaces (LocationEnricher, EngineEvaluator,
│ EngineExpirer, EngineReplayer) + methods + registerEventHandlers
├── item.go ← plan-item handler methods
├── cleanup.go ← S3 delete + orphan-sweep handler methods
└── cron.go ← cron handler methods + registerCronHandlers
internal/async/ stays free of services imports so service callers (which already use services) can enqueue without an import cycle. The handlers/ sub-package owns the services-aware code.
The event/cron file split is a consistent convention — anything event-driven lives in *_event.go / event.go; anything cron-driven lives in *_cron.go / cron.go. Same HandlerService type in both handler files (Go methods can live across files in the same package).
Server¶
// async/server.go
asynq.NewServer(redisOpt, asynq.Config{
Concurrency: cfg.Worker.Concurrency,
Queues: map[string]int{
"critical": 6,
"default": 3,
"low": 1,
},
ErrorHandler: ..., // logs failed tasks
})
Weights are relative dequeue ratios — Asynq pulls 6:3:1 from critical:default:low. Concurrency is how many tasks the server processes simultaneously across all queues. Per-task duration + outcome is recorded against the central Prometheus registry via observability.AsyncMetricsMiddleware.
Retries, exponential backoff, and the archived (dead-letter) queue are handled by Asynq itself. Failed tasks land in asynq:{<queue>}:archived with the captured error.
Scheduler¶
// async/scheduler.go
sched.Register("@every 5m", asynq.NewTask(TaskStatusUpdate, nil),
asynq.MaxRetry(3),
asynq.Unique(4*time.Minute),
asynq.Retention(24*time.Hour),
)
asynq.Unique(window) is the cross-replica dedup primitive — every replica's scheduler tries to enqueue at the cron tick; the first to acquire Asynq's unique-task lock wins, the rest get ErrDuplicateTask and silently skip. The window is set shorter than the cron interval so the lock expires before the next tick.
Cron entries¶
Registered in async/scheduler.go::cronEntries:
| Task | Interval | What it does |
|---|---|---|
cron:message_expiry |
30s | Hard-deletes disappearing chat messages past their expires_at |
cron:status_update |
5m | Recomputes event status (upcoming → live → finished) |
cron:redis_sync |
5m | Re-syncs event geo locations into Redis from Postgres |
cron:moment_cleanup |
5m | Soft-deletes expired non-journaled moments |
cron:token_cleanup |
1h | Sweeps expired credential material: refresh tokens, OTPs, idle sessions. See retention windows |
cron:account_suspension |
12h | Suspends accounts flagged for moderation |
cron:purge |
24h | Cleans orphaned events + purges archived events older than 14 days |
cron:deactivated_acct_purge |
24h | Hard-deletes accounts after the deactivation grace period |
cron:moment_purge |
24h | Hard-deletes soft-deleted moments older than 7 days |
cron:challenge_archive |
24h | Flips status=published → archived for time-bounded challenges past their window |
cron:user_stats_refresh |
1m | Recomputes tomoda_users_* Prometheus gauges (total / active_24h / new_24h / by_country) for the analytics dashboard |
cron:notification_purge |
24h | Deletes old read / expired notifications |
cron:parsed_link_sweep |
24h | GCs shared parsed_links rows no item references (item hard-deletes cascade item_links at the DB level, bypassing the inline cleanup) |
cron:location_enrich_sweep |
1m | Enqueues up to 2000 never-enriched locations (enrichment_at IS NULL) for the async equalizer. The self-heal net: seed rows leave enrichment_at nil and are drained in rate-limited waves; also backstops any runtime fire that failed. Each run also queues up to 2 enriched rows with no photo as photo_backfill (100/day, see Place photos). Asynq dedups per location id |
cron:location_embedded_count |
5m | Samples the count of active rows carrying a semantic embedding into the tomoda_locations_embedded_total gauge. A cheap COUNT; the graduation signal for the two-stage vector recall (SearchLocationsSemantic). As the embedded set grows, the coarse binary quantization behind stage 1 starts dropping true neighbors at a fixed rerank pool size, so watch this gauge to know when the binary-HNSW index should move to a disk-backed quantized index |
cron:event_embed_sweep |
1m | Embeds up to 64 live events whose search vector is missing or stale (embedded_at IS NULL: new, edited, or never backfilled) in batches of 8. A no-op when embeddings are disabled; an unreachable server ends the tick quietly (the embeddings breaker logs it once) and the next one retries |
cron:provider_spend_purge |
24h | Folds location_provider_spend rows older than 30 days into location_provider_spend_daily (dropping user, cell and text hash) and deletes the daily totals after 400 days, then deletes location_seed_tiles and location_seed_texts rows expired for over 7 days (the grace lets a failing tile reach its failure limit) and settled tiles older than twice the tile TTL, 5000 rows per statement |
cron:harvest_outbox_sweep |
5m | Replays paid harvest answers stored in the Redis outbox (harvest:outbox:*) for more than 3 minutes, up to 50 per run, through Engine.RunHandoff, which never calls the provider. An answer is stored before its write and deleted once committed, so a crash, a deploy or exhausted worker retries lose nothing. Sets the harvest_outbox_pending gauge |
cron:ranking_label_prune |
24h | Deletes ranking impressions + selections older than 180 days, keeping the durable search-ranking label sink bounded |
cron:activity_rollup |
5m | Folds new activity_events into interaction_edges (the ranker's affinity signal), advancing a per-job watermark. A lag holds the window back from now() so in-flight inserts are not skipped |
cron:activity_partition_maintain |
24h | Provisions the upcoming monthly activity_events partitions and drops any fully older than the 30-day retention window (retention by partition drop, not row deletes) |
cron:security_log_purge |
24h | Bounds the trails that carry an IP and a user-agent (login history, audit logs) plus expired plan share tokens. See retention windows |
cron:chat_message_purge |
24h | The two chat hard-deletes the expires_at path never reaches: tombstoned messages past their window, and event-chat history past its event's retention_days |
cron:deepseek_balance |
1h | Reads the DeepSeek prepaid balance (GET {semantic.base_url}/user/balance) into the deepseek_balance_usd gauge behind the DeepSeekBalanceLow alert. Skips with a debug log unless semantic.base_url is the DeepSeek API (api.deepseek.com) and SEMANTIC_API_KEY is set. See semantic metrics |
cron:insights_build |
daily 03:30 UTC | Rebuilds yesterday's Backstage insights (insights_daily, insights_member_daily) from activity_events and the content tables, fills any day missed in the last 14, and trims per-member rows older than 180 days. See Backstage service |
Adding a cron task means appending one entry to cronEntries AND registering a handler in async/handlers/handlers.go::Register.
Retention windows¶
Personal data kept with no stated period is itself a compliance problem, so every table below has a purge and a documented window. Each window is a named constant next to the service that owns the sweep, never a literal at the call site.
| Data | Window | Constant | Why this long |
|---|---|---|---|
| OTP codes | 24h past expiry | otpRetention (auth/session_service.go) |
Codes expire in minutes. The grace only lets support answer "did my code arrive" before the row, and the address on it, goes. Per-address attempt counters live in Redis and are unaffected |
| Sessions | 90d idle | sessionRetention (auth/session_service.go) |
Cut on last_active_at. The row carries an IP and a device string, and a session idle this long can't be resumed: its refresh token expired well before |
| Refresh tokens | past expires_at |
— | A dead token has no residual use |
| Login history | 180d | loginHistoryRetention (auth/session_service.go) |
Long enough to investigate an account takeover reported months later, short enough that a login trail isn't a permanent record. Risk scoring reads the most recent 50 rows, so the window never starves it |
| Audit logs | 365d | auditLogRetention (platform/audit/audit.go) |
Covers the forensic window for an incident surfaced long after the fact and the period a compliance review asks for |
| Plan share tokens | 30d past expiry | shareTokenRetention (plan/plan_collaboration_service.go) |
Reads already reject an expired token, so the row is inert; the grace only explains why a link stopped working |
| Deleted chat messages | 30d past deleted_at |
deletedMessageRetention (chat/service.go) |
The tombstone renders "message deleted" in an open thread and is what a moderation report is checked against. Attached images are already released at soft-delete time |
| Event chat history | per-event retention_days (default 14) |
events.retention_days |
Set per event by its host; the sweep respects end_time + retention_days |
| Notifications | 90d | notificationRetentionDays (async/handlers/cron.go) |
Inbox rows go regardless of read state |
| Ranking labels | 180d | rankingLabelRetentionDays (async/handlers/cron.go) |
A retune trains on recent selection behavior |
Account deletion is separate from retention: it's driven by the ON DELETE CASCADE FKs on every user-owned table rather than by a sweep. See Storage → Purge ordering and Data model.
Event-driven tasks¶
| Task | Enqueued by | Notes |
|---|---|---|
item:parse |
plan.ItemService via async.Client.EnqueueItemParse |
Default queue, per-item dedup ID, 3-minute task timeout. Deterministic capture parse; links fan out concurrently inside the handler. |
link:enrich |
plan.ItemService via async.Client.EnqueueLinkEnrich |
Low queue, per-item dedup ID (force reparse undeduped), 3-minute task timeout. Additionally guarded by the enrich:item:lock:* Redis lock so a double fire can't double-spend the enrichment quota. |
location:enrich |
location.LocationService via async.Client.EnqueueEnrichment |
Low queue, per-location dedup ID — back-to-back enqueues for the same location collapse into one execution |
location:burst_seed |
harvest.Engine via async.Client.EnqueueBurstHandoff, run by LocationService.RunBurstHandoff |
Default queue (kept out from behind the enrichment backlog on low), up to 12 retries; an archived task counts into tomoda_locations_harvest_handoffs_archived_total and its answer stays in the outbox for the sweeper. An undecodable payload is SkipRetry. Payload is a harvest.Handoff: places the Serper call already returned, never the query. The worker writes them and stamps the tile; it never calls the provider, so a retry cannot buy twice. Sent when the in-process write fails or the server is draining |
location:promote |
maps.Service and others via async.Client.PromoteOnEngagement, run by LocationService.PromoteOnEngagement |
Low queue, 3 retries, per-place dedup ID held for an hour. Graduates a harvested place a user saved or attached to an event (see Demand-driven promotion) |
location:rehost_photos |
location.LocationService.RehostPlacePhotos via async.Client.EnqueueRehostPhotos |
Default queue, one task per harvest or search-more mint, one retry. Payload is a list of {location_id, url} validated in the handler (1 to 200 http(s) items). Rehosts provider thumbnails of harvested places under the burst:photos:{yyyymmdd} daily download ceiling; failures are dropped |
location:seed_postload |
catalogseeder load via async.Client.EnqueueSeedPostLoad, run by LocationService.RunSeedPostLoad |
Low queue, up to 5 retries, 10 min timeout. One task per step of a chain, each enqueuing the next after a short pace: link_cities (fans out to 4 id-range shards that run in parallel and page by cursor; the last shard to finish starts the next step, detected through a Redis set, seed:postload:done:{step}), highlights (paged by city id, 300 cities per task with the cursor in the payload; skipped when the per-city count is 0), embed (the same 4 shards, batches of 200 through the embed gate; skipped when embeddings are off), enrich (one sweep of never-enriched rows; the enrichment sweep cron drains the rest). Task ids are step, shard and cursor, so a re-run of the load Job does not double a queued step. An unknown step is SkipRetry |
cleanup:s3_delete |
assets.Service.EnqueueDelete (backend/internal/platform/assets) |
Hard-delete cascades fan out one task per asset key. Low queue. The handler refuses curated-prefix keys (stamps/, stamps/bespoke/, curios/) as defence-in-depth. |
cleanup:s3_orphan_sweep |
Admin via POST /api/v1/backstage/async/s3-orphan-sweep?prefix=… |
One prefix per task; enumeration is bounded. Not scheduled — see orphan sweep rationale. |
engine:eval_travel_log, engine:eval_user_activated |
gameengine.Signal on travel-log write / user activation |
Evaluate the challenge + reward rule trees and grant earned curios. See Services → Game Engine. |
engine:expire_challenge, engine:replay_challenge, engine:replay_stamp |
gameengine expiry scheduler / admin replay |
Async challenge cleanup and rule replay. |
legal:policy_notice |
Worker startup (HandlerService.OnStart) via async.Client.EnqueuePolicyNotice, when legal.policy_notices_enabled |
One task per pending policy notice with someone left to notify. Low queue, task id legal-notice:{notice} so every worker's startup enqueue collapses to one (a duplicate is success), 12-hour timeout, 20 retries. Each send is recorded as a notified legal_acceptances row; send failures delete their rows and return an error, so the retry mails only them. See Auth → Policy acceptance. |
Cleanup tasks¶
Both cleanup task types live in async/tasks_cleanup.go; handlers in async/handlers/cleanup.go.
type S3DeletePayload struct {
Key string `json:"key"`
Triggered time.Time `json:"triggered"`
}
type S3OrphanSweepPayload struct {
Prefix string `json:"prefix,omitempty"`
Triggered time.Time `json:"triggered"`
}
| Task | Handler invariants |
|---|---|
cleanup:s3_delete |
Rejects empty keys and curated-prefix keys with asynq.SkipRetry. S3 errors propagate so asynq retries per MaxRetry, then archives on exhaustion. |
cleanup:s3_orphan_sweep |
Looks up the prefix's owning column in orphanColumns() (no entry for curated prefixes), SELECT DISTINCT col FROM table ... Unscoped() to build the "owned" set (soft-deleted rows still own their assets), then ListObjectsV2 paginated against the prefix. Each unowned key gets enqueued as a cleanup:s3_delete so the sweep stays read-only and per-key retries apply uniformly. Skips with nil (not retry) when the bound FileStorage isn't an *S3Storage — local dev / RustFS. |
The orphan sweep's per-prefix DB lookup map is the source of truth for "which DB column owns this prefix's keys" — adding a user-generated prefix means adding both a storage.Prefix constant and an entry to orphanColumns().
Lifecycle¶
async.App owns the runtime. App.Start() registers handlers + cron entries then runs Server and Scheduler in goroutines, plus Handler.OnStart (one-off startup enqueues, such as pending policy notices) with a one-minute deadline. App.Shutdown() stops them in reverse order. Both are idempotent.
The single binary serves multiple roles via SERVER_MODE; async.App.Start is only invoked when modes.runAsync is true (modes full and async). See cmd/server/main.go.
Registering a new task¶
Pick the path that matches your trigger.
Event-driven (request handler / service enqueues)¶
- Define the task type and payload in
async/tasks_event.go:
const TaskMyEvent = "my_domain:event"
type MyEventPayload struct {
SomeID uuid.UUID `json:"some_id"`
}
- Write the handler in
async/handlers/event.go(methods on the existingHandlerService):
func (h *HandlerService) HandleMyEvent(ctx context.Context, t *asynq.Task) error {
var payload async.MyEventPayload
if err := json.Unmarshal(t.Payload(), &payload); err != nil {
return fmt.Errorf("unmarshal: %v: %w", err, asynq.SkipRetry)
}
return h.someService.DoThing(ctx, payload.SomeID)
}
If the handler needs a service that isn't on HandlerService yet, declare a narrow interface in event.go (see LocationEnricher / EngineEvaluator for the pattern) and bind the concrete domain type to it in wiring/providers.go (AsyncSet or AppSet).
- Wire it on the mux by adding one line to
registerEventHandlersinevent.go:
mux.HandleFunc(async.TaskMyEvent, h.HandleMyEvent)
- Enqueue from a service via the injected
*async.Client:
asyncClient.Enqueue(asynq.NewTask(async.TaskMyEvent, body), asynq.Queue("default"))
Cron-driven (scheduler enqueues on a wall-clock tick)¶
- Define the task type in
async/tasks_cron.go(cron tasks typically have no payload):
const TaskMyCron = "cron:my_cron"
- Write the handler in
async/handlers/cron.go:
func (h *HandlerService) HandleMyCron(ctx context.Context, _ *asynq.Task) error {
return h.someService.DoSweep(ctx)
}
- Wire it on the mux by adding one line to
registerCronHandlersincron.go:
mux.HandleFunc(async.TaskMyCron, h.HandleMyCron)
- Append a
cronEntrytocronEntriesinasync/scheduler.go. Use a cron spec (@every 1h,0 3 * * *) and auniqueWindowstrictly shorter than the interval so the cross-replica dedup lock expires before the next tick:
{"@every 1h", asynq.NewTask(TaskMyCron, nil), 55 * time.Minute},
Common notes¶
Return asynq.SkipRetry from a handler to send the task straight to the archived queue — useful for un-recoverable errors like unmarshal failures or invalid IDs.
Operations¶
- Recovery — failed tasks land in Asynq's
archivedqueue per queue. Inspect / requeue via the admin DLQ endpoints below, or viaasynqctl/ theasynqmonweb UI (neither deployed today, both compatible with our data structures). - Scaling —
tomoda-asyncdeployment runs N replicas. All replicas consume from the shared Asynq queues; the scheduler'sasynq.Uniquededup ensures cron tasks only fire once per tick regardless of replica count.
Admin DLQ endpoints¶
backstage.AsyncHandler (backend/internal/services/backstage/async_handler.go) wraps the asynq Inspector + Client so an on-call admin can triage archived tasks without shelling into the cluster.
| Method | Path | What it does |
|---|---|---|
GET |
/api/v1/backstage/async/queues |
Snapshot of every queue (size / active / pending / scheduled / retry / archived / completed / latency / paused) |
GET |
/api/v1/backstage/async/queues/:queue/archived |
Paginated archived listing — page, page_size (max 200); shows payload + last error |
POST |
/api/v1/backstage/async/queues/:queue/archived/:id/run |
Pull a single archived task back into pending |
POST |
/api/v1/backstage/async/queues/:queue/archived/run-all |
Bulk replay every archived task in a queue |
DELETE |
/api/v1/backstage/async/queues/:queue/archived/:id |
Drop a poison-payload task |
POST |
/api/v1/backstage/async/s3-orphan-sweep?prefix=… |
Enqueue an on-demand orphan sweep for one prefix (curated prefixes return 400) |
Endpoints are generic across task types; the cleanup queue is not the only DLQ in play. The two reads need ops.queues and every action needs ops.action.replay (operators and admins), checked fresh per request by the Backstage route table. Each read and action writes an audit row before it runs. Backstage's Ops section is the UI over these routes.
See also¶
- Redis — the underlying store
docs/reference/redis-keys.md—asynq:*keys are managed internally by the library; nothing inasync/writes to Redis outside the library calls