Object Storage¶
User-uploaded assets live in a private Cloudflare R2 bucket (tomoda-assets-{env}), accessed through the S3 API. The backend never proxies uploads from the client: the frontend PUTs directly to R2 with a short-lived presigned URL the backend issues. Reads go through the assets Worker on assets[-dev].tomoda.life, which checks signed URLs and caches at the edge.
backend/internal/storage/ is the S3 / local driver (the FileStorage contract); backend/internal/platform/assets wraps it as assets.Service, the single write / delete entry point. Handlers and other services go through assets.Service, never FileStorage directly.
Mental model¶
RN / web
assets.Service.IssueUploadURL
private
CDN domain
Bucket layout¶
Every prefix is a typed storage.Prefix constant. The set is closed: adding a location means adding a constant in backend/internal/storage/prefixes.go and an AssetKind in backend/internal/platform/assets/assets.go, never passing a raw string.
| Prefix | Owning column | Lifecycle |
|---|---|---|
avatars/ |
users.asset_key |
Replace on rotation; delete on user purge |
group-avatars/ |
chat_rooms.asset_key |
Replace on rotation; delete on room purge |
chat-images/ |
chat_messages.metadata->'images'->'keys' (JSONB array; 1..N per message) |
Delete on message hard-delete (sender-delete propagates) + disappearing-message TTL |
moments/ |
moments.asset_key |
Delete on moment hard-delete; will be ref-counted once moment tagging ships |
event-covers/ |
events.asset_key |
Delete on event hard-delete / cancel |
plan-covers/ |
plans.asset_key |
User-uploaded cover; overrides the parsed-item thumbnail fallback |
support-screenshots/ |
support_cases.screenshot_key (a screenshot attached to a problem report) |
Kept with the case; signed URLs only, Backstage-facing; the orphan sweep checks support_cases |
item-images/ |
item_photos.asset_key (user-owned photos), item_links.thumbnail_key (rehosted link thumbnails) |
Delete on item / link hard-delete; not tied to source URL rot |
user-stamps/ |
stamp_mints.asset_key (pixelated PNG) |
Delete when the last stamp_mint_holders row drops — see DeleteMintIfOrphan |
user-stamps-raw/ |
stamp_mints.raw_asset_key |
Cascades with user-stamps/ — same mint owns both keys |
stamps/ |
tomoda_stamps.asset_key |
Curated — permanent |
stamps/bespoke/ |
tomoda_stamps.bespoke_asset_key |
Curated — permanent |
curios/ |
curios.asset_key |
Curated — permanent |
places-images/ |
locations.thumbnail_key, locations.photos |
Curated — permanent — public. Prestige place photos captured during catalog seeding (Michelin, the 50 Best, etc.) and Serper place thumbnails rehosted under content-hashed keys (assets.UploadContentHashed, kind place-image; see Place photos) |
geo-cover-submissions/ |
geo_cover_submissions.asset_key |
Member photo staged for review (owner-prefixed, private). Deleted when the submission is approved or rejected; removed with the member's account |
geo-covers/ |
{countries,regions,locations}.cover_image_key |
Curated — permanent — public. Country / region / city cover photos (Unsplash then Pixabay for the headline tier, Wikipedia for the long tail), and member photos a reviewer approved |
Per-prefix S3 lifecycle rules (transitions, multipart-abort) live in devops/infrastructure/aws/s3.tf — moments/ transitions to STANDARD_IA at 30 days; every prefix shares the 24h abort-incomplete-multipart rule.
Curated vs. user-generated¶
Prefix.IsCurated() returns true for stamps/, stamps/bespoke/, curios/, places-images/, and geo-covers/. These hold admin-managed assets seeded by the curated-import path and are never deleted by user-facing code. Both assets.Service.Delete / assets.Service.EnqueueDelete and the async cleanup handler refuse curated keys; the orphan sweep skips them entirely.
Public vs. signed¶
Prefix.IsPublic() returns true for geo-covers/ and places-images/. These hold non-sensitive, shared catalog imagery, so assets.Service.URLFromKey composes a plain {public_url}/{key} instead of a short-TTL signed URL: the CDN caches them at the edge and unauthenticated clients (and, later, link-share crawlers) can fetch them. The assets Worker allows unsigned GETs on these paths (its public path list in the devops repo's assets-worker/, deployed by infrastructure/cloudflare/assets_worker.tf). Every other prefix is served as a signed URL.
User-generated prefixes follow the app-cascade-delete pattern: when the owning DB row is hard-deleted, the service enqueues an assets.Service.EnqueueDelete for each key. No DB foreign key, no S3 lifecycle rule — the deletion is an explicit cascade in service code.
Direct-to-S3 uploads¶
The frontend uploads bytes straight to S3 via presigned PUT. Backend egress stays out of the hot path.
POST /uploads/<kind> → presigned PUT to S3 → finalise on the owning resource
| Start endpoint | Issues for | Owner-scoped key | Finalise on |
|---|---|---|---|
POST /api/v1/uploads/avatar |
AssetAvatar |
user ID | PUT /api/v1/users/me/avatar |
POST /api/v1/uploads/group-avatar |
AssetGroupAvatar |
— (caller supplies room_id, membership-checked) |
PUT /api/v1/dm/rooms/{id}/avatar |
POST /api/v1/uploads/chat-image |
AssetChatImage |
room ID (nested: chat-images/{room}/…) |
POST /chats/:id/messages with image_keys: [key] |
POST /api/v1/uploads/chat-image/batch |
AssetChatImage × N |
room ID (nested) | POST /chats/:id/messages with image_keys: [...] (1..10, MIME per key) |
POST /api/v1/uploads/moment |
AssetMoment |
user ID | POST /api/v1/moments |
POST /api/v1/uploads/event-cover |
AssetEventCover |
user ID | POST / PATCH /api/v1/events (cover_image_url = key) |
POST /api/v1/uploads/plan-cover |
AssetPlanCover |
user ID | PATCH /api/v1/plans/{id} (cover_image_key) |
POST /api/v1/uploads/item-image |
AssetItemImage |
user ID | POST /api/v1/items (image_keys) or POST /api/v1/items/{id}/photos |
Each start endpoint returns { key, upload_url }. The TTL is fixed at 5 minutes (issueUploadURLTTL in backend/internal/platform/assets/assets.go). Two things are bound into the signature so a client can't rewrite them at upload time:
- Content-Type — signed; S3 rejects a PUT whose
Content-Typediffers. - Key prefix — generated server-side from the kind's
storage.Prefix; the client can't pick the key.
Room-scoped kinds are membership-gated at issue time. chat-image, chat-image/batch, and group-avatar take a room_id; the handler parses it as a UUID and calls CheckParticipation (via the ChatMembership port in backend/internal/services/media/deps.go) before signing. Without that gate the room segment of a chat-images/{room}/… key would be attacker-chosen, letting any authenticated caller write into another room's prefix.
Size is not bound into the signature. Binding Content-Length would force the client to send exactly that byte count, which the issuer can't know in advance. Instead the per-kind cap (spec.maxBytes in backend/internal/platform/assets/assets.go) is enforced post-upload by AssertUploaded's HeadObject check, with a best-effort delete on oversize.
After the PUT, the client calls the finalise endpoint with the key. The handler routes to assets.Service.AssertUploaded, which:
- verifies the key lives under the kind's prefix (catches a client trying to POST somebody else's
moments/key to the avatar finalise); - for owner-scoped kinds, checks that the owner appears where the kind puts it: as a name prefix (
{owner}_{ts}) for flat kinds, or as the first path segment ({owner}/{ts}) for nested kinds likechat-image; - HEADs the object to confirm it exists and re-checks size against the cap.
The finalise step then persists the key on the owning model and enqueues the old key (if any) for async cleanup.
Every presigned kind runs the finalise check
AssertUploaded is the only place spec.maxBytes is enforced, so the finalise path is what stops an oversized object. Every presigned kind calls it: avatar, group-avatar, chat-image, moment, item-image, event-cover, and plan-cover. event-cover finalises in EventHandler (create and update), plan-cover in PlanService (create, update, and the explicit cover on promote). chat-image is additionally checked in Service.validateImageKey (backend/internal/services/chat/service.go) on the send path.
group-avatar is registered with scopeByOwner: false, so its key carries no owner and the presign passes an empty ownerID: the room owns the asset, not the uploader, and both the presign and the finalise endpoint gate on room membership. event-cover and plan-cover use scopeByOwner: true, so their keys embed the uploader ({owner}_{ts}) and AssertUploaded binds the key to that user, preventing one user from claiming another's uploaded cover.
The user-stamp catalog compose UI still uploads through the server-side UploadRaw path rather than a presigned PUT, since it needs server-side processing alongside the persist.
Local dev / RustFS falls back to the same presign code path; the endpoint override is object_storage.endpoint (STORAGE_ENDPOINT).
Signed reads through the assets Worker¶
The bucket is private. A Cloudflare Worker bound to it serves object_storage.public_url (e.g. https://assets.tomoda.life); assets.Service.URLFromKey composes the URL on read, called via the AfterFind hooks on each model with an asset.
- Public prefixes get a plain
{public_url}/{key}. - Everything else gets
{public_url}/{key}?exp=<unix>&sig=<mac>fromstorage.URLSigner(backend/internal/storage/url_signer.go): HMAC-SHA256 over"<escaped path>\n<exp>", base64url without padding, keyed byASSET_SIGNING_KEY. The Worker recomputes it and rejects expired or tampered URLs. Both sides share a fixed test vector, so a format change breaks a test on each side. - Signed and unsigned requests for the same object share one edge cache entry; the signature is checked before the cache.
When public_url is empty (local dev), URLFromKey falls back to FileStorage.GetURL which returns the direct RustFS URL — same shape, different host.
Cascade delete¶
User-row hard-deletes fan out async S3 deletes. The DB transaction commits immediately; the S3 work runs on the asynq low queue.
| Trigger | Cascade |
|---|---|
MomentService.PurgeSoftDeleted |
For each purged row, assets.Service.EnqueueDelete(moment.AssetKey) |
UserStampService.UntagSelf / replace path |
repo.DeleteMintIfOrphan returns both keys when no holders remain; service calls EnqueueDelete on each |
UserService.DeleteUser / GDPR purge |
Enumerates every key the account owns before any row is touched: avatar, moment media, direct-chat avatars, hosted-event and hosted-plan covers, and plan item photos. Group-chat avatars are excluded, since the room outlives the departing member. See the purge ordering below |
AuthService.UpdateAvatar (replace) / UpdateGroupAvatar |
Async assets.Service.EnqueueDelete on the old key after the new key persists — keeps the request path off S3 |
ChatService.DeleteMessage (sender hard-delete) |
Each key in metadata.images.keys enqueued so every attached image disappears for everyone |
ChatService.CleanupExpiredMessages (disappearing TTL) |
Batch enqueue every key in each expired row's metadata.images.keys |
The task type is cleanup:s3_delete; payload is the key + a timestamp. Handler: HandleS3Delete in backend/internal/async/handlers/cleanup.go. The handler re-checks the curated-prefix guard before calling the S3 backend.
Purge ordering¶
DeleteUser (backend/internal/services/user/user_service.go) sequences the account purge so a partial failure can't strand personal data:
- Collect asset keys. The row cascade destroys the only record of which keys the account owned, so enumeration happens first. A failure here aborts the purge, leaving the account intact for the next run.
- Chat policy. Direct rooms are deleted outright; group history is anonymized so it survives for the remaining members. This is the one part of the cascade the schema can't express, so a failure aborts.
- Hard-delete the user row. Every remaining table reaches
usersthrough anON DELETE CASCADEFK, so this single statement is the row erasure. - Enqueue asset deletes. Last, because enqueuing before the row delete would destroy a live user's media if the delete then failed. A key that fails to enqueue is named in the log and the run returns an error, since the user row is already gone and the purge can't be retried from the top.
Steps whose failure costs nothing (session revocation, the post-delete orphaned-event sweep) log and continue.
Failed-delete recovery¶
Failed cleanup:s3_delete tasks retry per the asynq retry policy, then land in the archived queue. The admin DLQ inspector surfaces them:
| Endpoint | What it does |
|---|---|
GET /api/v1/backstage/async/queues |
Snapshot per queue: size, active, pending, scheduled, retry, archived |
GET /api/v1/backstage/async/queues/:queue/archived |
Paginated listing of archived tasks with payload + last error |
POST /api/v1/backstage/async/queues/:queue/archived/:id/run |
Pull a single task back into pending |
POST /api/v1/backstage/async/queues/:queue/archived/run-all |
Bulk replay every archived task in a queue |
DELETE /api/v1/backstage/async/queues/:queue/archived/:id |
Drop a poison-payload task |
See backend/internal/services/backstage/async_handler.go.
Orphan sweep¶
Crash windows between a row's hard-delete and the cleanup:s3_delete task can leave an object in S3 with no DB owner. The orphan-sweep handler enumerates each user-generated prefix, cross-checks every key against the owning column (using Unscoped() so soft-deleted rows still count as owners), and queues a delete for any unowned key. Chat-image keys live inside chat_messages.metadata->'images'->'keys', so that prefix uses a raw SQL jsonb_array_elements_text pluck instead of a direct column read.
The sweep is admin-triggered only. It is intentionally not on the scheduler — at scale, the enumeration cost on S3 plus the Distinct pluck to build the "owned" set is dominated by moments/ and chat-images/ (millions of objects after a few months). Cost beats value for a routine cron.
Admins kick it off via POST /api/v1/backstage/async/s3-orphan-sweep?prefix=moments/ — one prefix per call, bounded scope. Curated prefixes return 400. The handler routes the task onto the low queue with MaxRetry=2 and 24h retention.
See HandleS3OrphanSweep + orphanColumns in backend/internal/async/handlers/cleanup.go for the per-prefix DB lookup map.
Configuration¶
Construction lives in wiring.ProvideFileStorage; the only backend in production is S3Storage.
object_storage key |
Purpose |
|---|---|
bucket |
Bucket name |
region |
auto for R2 (the default when blank) |
public_url |
Assets Worker URL (e.g. https://assets.tomoda.life), used by URLFromKey |
endpoint |
https://<account>.r2.cloudflarestorage.com when deployed, http://<lan-ip>:9000 for RustFS |
use_path_style |
true for RustFS, false for R2 |
Env overrides: STORAGE_BUCKET, STORAGE_REGION, STORAGE_ENDPOINT, STORAGE_PUBLIC_URL, STORAGE_USE_PATH_STYLE.
Env only (never in YAML): STORAGE_ACCESS_KEY_ID / STORAGE_SECRET_ACCESS_KEY (the R2 token; empty falls back to the SDK default chain) and ASSET_SIGNING_KEY (the HMAC key shared with the assets Worker). With no signing key, as in local dev, signed URLs fall back to S3 presigned GETs. Signed URLs expire after 30 minutes. The SDK computes request checksums only when required, because S3-compatible stores reject the default flexible checksums on presigned PUTs.
Uploads use the managed s3/manager.Uploader (multipart-aware) and set Cache-Control: public, max-age=31536000, immutable so the CDN can pin objects indefinitely. Content-Type comes from the asset-side sniff (assets.Service.Upload) or the presigned signature (direct PUT).
Local dev (RustFS)¶
docker-compose.dev.yml runs RustFS on:
- API:
http://localhost:9000 - Console:
http://localhost:9001/rustfs/console/(login:tomoda/tomoda123)
Bucket tomoda-local is created on startup. S3.use_path_style: true is required.
See also¶
backend/internal/platform/assets— the single write / delete entry point that wraps the storage driver- Async —
cleanup:s3_delete+cleanup:s3_orphan_sweeptask definitions, admin DLQ endpoints