Start with one bounded export queue, a resident export worker and a separate public verifier process. This is a deployment recipe around the existing SDK, not a built-in durable queue or a claim of horizontal scaling.
A small startup pipeline
generation → final rendition → bounded job queue → resident SDK exporter
→ accepted staging file + receipt
→ one publication step
public upload → separate verifier process → reviewed registry / signed report
Keep publisher private keys with the exporter. Give the verifier its own report key and read-only registry access. One SDK shares its Node worker across video profiles and verification; an export can hold that worker lock for the whole job. A public verifier in another process avoids waiting behind your export queue.
Job states and back pressure
| State | Product behavior |
|---|---|
| Pending | Keep job ID, generation time, input hash, profile and intended rendition in durable application storage |
| Queue full | Decline admission or return a bounded retry response; do not create an unlimited in-memory queue |
| Running | Lease one job to one worker; create a unique staging path for this attempt |
| Accepted | Retain the accepted receipt and reconcile registry/output state before publishing |
| Published | Record publication against the same idempotency key so a retry cannot publish twice |
| Failed or interrupted | Preserve the failure; inspect completed artifacts before retrying; never publish unmarked input as fallback |
Set your own maximum queue length, lease timeout and retry cap from the workload. A 2-vCPU benchmark is not a measured concurrency limit. The owned-file worker processes a bounded manifest serially; your application supplies durable admission and job state.
Handle a busy verifier
The reference verifier has one inference slot, four upload slots and no queue. Handle HTTP 429 and its Retry-After response with bounded backoff. Do not repeatedly upload a large file in a tight retry loop. Match the application upload limit and your reverse proxy limit; the documented defaults are 512 MiB and 128 MiB respectively.
Size positive, negative and incomplete searches separately. On the ten-second 720p engineering fixture, warm positive verification took 2.04 seconds and a completed negative took 40.32 seconds. Longer searches can exhaust the budget; not_checked is not a completed negative. See the full timing and resource scope.
Keep registry state consistent
Retain the registry, manifests, receipts and application job state on persistent storage. Use a controlled writer and give SQLite readers access to the database directory and its supporting files. Do not copy only a live database file and assume it is a consistent backup.
For a small SQLite deployment, an online backup can use the standard backup API:
from pathlib import Path
import sqlite3
source = Path("registry/assets.sqlite3").resolve()
backup = Path("backups/assets.snapshot.sqlite3")
backup.parent.mkdir(parents=True, exist_ok=True)
# Use a fresh destination name for each snapshot; retain keys separately.
if backup.exists():
raise FileExistsError(backup)
with sqlite3.connect(source.as_uri() + "?mode=ro", uri=True) as src:
with sqlite3.connect(backup) as dst:
src.backup(dst)
assert dst.execute("PRAGMA integrity_check").fetchone()[0] == "ok"
The database snapshot does not automatically back up referenced manifests or keys. Coordinate their retention, protect private material, and test a restore in isolation. Check that an accepted receipt and media file resolve to the intended public record after restart. Reconcile orphan staging files against your job ledger before applying a retention policy; do not delete unknown files automatically.
A first operational acceptance pass
- Submit one image and one video through the same application queue.
- Trigger a controlled failed export; confirm no unmarked publication fallback.
- Restart the worker and retry the same application job ID; confirm one publication.
- Exercise a busy verifier and a clean negative, recording retry behavior and memory.
- Restore an isolated registry/manifests snapshot and independently verify an accepted output.
These checks complement the SDK qualification. They do not establish your production traffic capacity or a multi-replica storage design.
First integration · Reference limits · Production setup · Costs