Skip to content

The submit-poll-postprocess-review pipeline

Problem

Generative media services are asynchronous, slow, and priced per attempt. A request takes minutes, may fail, and costs money whether or not the result is usable. Written as a synchronous call, the integration blocks, loses work on restart, and produces artifacts nobody can trace back to what made them.

Every such integration converges on the same five stages — and because each is written for one vendor, the shape gets rebuilt each time and the lessons do not transfer.

Technique

Name the five stages, and treat the last two as non-optional.

  1. Submit — send the request, persist the job handle immediately. A handle held only in memory is a paid job lost on restart.
  2. Poll — with backoff, a timeout, and a terminal-state check. Distinguish failed from still running: a job that will never finish must not be polled forever.
  3. Post-process — normalise into house format. Transcode, resample, trim, re-encode. Vendors return what suits them, not what your library expects.
  4. Provenance sidecar — write a companion record: which service, which model and version, the exact prompt and parameters, the job id, the timestamp. Without it an asset is unreproducible, and six months later nobody can tell whether it can be regenerated or must be recreated by hand.
  5. Human review gate — generated media reaches a person before it ships.

The sidecar is the stage people skip, and the one that costs most later. The review gate is the stage people skip second, usually after a run that looked fine.

When it applies

Any paid, asynchronous, generative service: image, video, audio, 3-D, voice, long-running text.

When it does NOT apply

Fast synchronous APIs, where polling machinery is overhead. And anywhere provenance is genuinely irrelevant — a throwaway preview regenerated on demand does not need a sidecar, though it is worth being honest about whether it is truly throwaway, because artifacts have a way of getting reused.

The review gate is skippable only when the output is not published under your name.

Evidence

Four independent media pipelines — 3-D models, audio effects, image-to-video, storyboard assembly — were built at different times against different vendors. All four ended up with these five stages. None of them named the shape, so each rediscovered the same things: that job handles must be persisted before the first poll, that a terminal-failure state needs its own branch, and that a sidecar written later cannot recover parameters nobody stored.

That is the argument for writing a pattern down. The technique was already proven four times over and still cost full price on the fifth.

See also