Skip to content
Sayiir

Tutorial: Video Processing Pipeline (Rust)

The problem

Section titled “The problem”

You run a video platform. When a user uploads a video, your pipeline needs to download it, transcode to multiple resolutions in parallel, generate thumbnails, run content moderation, upload approved assets to CDN, and notify the user. Simple enough on paper.

Here’s what happens in production:

  • The 4K transcode OOMs at minute 28. Your retry logic restarts all three transcodes — burning 45 minutes of completed 720p and 1080p work.
  • The moderation API returns a 503. Your pipeline gives up. The video is stuck in “processing” forever. The user files a support ticket.
  • Moderation rejects a video, but the transcoded files are already on CDN. Nobody cleans them up. You’re paying for storage on content that should never have been published.
  • You deploy mid-pipeline. All in-flight state is gone. Dozens of videos need to be manually re-queued.

These aren’t hypothetical — they’re production incidents that cost real companies real money.

How Sayiir solves each one

Section titled “How Sayiir solves each one”

Each step in a Sayiir workflow persists its result. When a step fails, only that step retries. Completed work is never lost.

workflow! {
    name: "video-pipeline",
    codec: JsonCodec,
    steps: [
        download_video,
        validate_upload,
        (transcode_720p || transcode_1080p || transcode_4k || generate_thumbnails || moderate_content),
        merge_results,
        route check_moderation -> Verdict {
            Approved => [upload_to_cdn, update_database, notify_user],
            Rejected => [cleanup_artifacts, notify_rejection],
        }
    ]
}

The || operator runs five tasks in parallel — three ffmpeg transcodes, thumbnail extraction, and content moderation — each with its own retry policy tuned for its failure mode. The 4K transcode gets 3 retries with 10s exponential backoff. The moderation API gets 3 retries with 2s backoff for transient 503s.

The merge_results task is a smart fan-in gate: it requires 720p and 1080p but treats 4K as best-effort. If 4K fails, the pipeline continues with two resolutions instead of crashing.

The route sends approved videos to CDN upload and rejected videos to a cleanup branch that deletes the transcoded artifacts — no orphaned files on CDN.

And if you deploy mid-pipeline? Sayiir resumes from exactly where it left off.

Production nightmareSayiir feature
4K OOM restarts all transcodesStep-level retries — only the failed branch retries
Moderation API 503 → stuck foreverPer-task backoff — retries with exponential delay
4K failure blocks the whole pipelineBest-effort tasks — return None, merge handles it
Rejected video, assets left on CDNConditional routing — rejected branch cleans up
Deploy mid-pipeline, state lostDurable execution — each step result is persisted

Try it yourself

Section titled “Try it yourself”

The full example uses real ffmpeg for transcoding and thumbnail extraction — no mocks. It downloads a 30 MB test video from the Blender Foundation, transcodes to 720p and 1080p in parallel, and produces real output files you can inspect.

Terminal window
cargo run
Full example: Video Pipeline (Rust) Complete Cargo project with real ffmpeg transcoding, 14 tasks, typed routing, and structured logging.

Next steps

Section titled “Next steps”