Document type: How-to. Reader: you run a pipeline with slow, expensive
steps (encodes, uploads, agent runs) and want to reuse prior work, resume after
crashes, and never accept a truncated artifact. Source: course-video-manager
snapshot 4c1f3f5d49417e54b185bfe737b1e1a56f29c7b8.
Step 1 — Separate the recipe hash from the actual-byte hash
When a destination must be known before rendering, separate recipe identity from output identity (ADR 0027). The recipe hash (here, the Export Hash) names what the renderer was told to do — inputs, order, format — and is knowable before any work starts, so it addresses the destination and lets work overlap. The actual-byte hash (Byte Hash) is the digest of what was produced, knowable only after the run, and is the only thing that can say whether this run's output is byte-identical to something you already shipped.
Illustrative generic contract:
type RecipeHash = string;
type ByteHash = string;
type Address = { recipeHash: RecipeHash }; // known before the work
type Artifact = { recipeHash: RecipeHash; byteHash: ByteHash; bytes: number };
type PriorLocation = { url: string };
type ReusePlan = ReadonlyMap<ByteHash, PriorLocation>;
// In this design, address by recipe; reuse only after comparing actual bytes.
The source also records why an export-generation counter was rejected: a counter records that somebody deleted a file; only the bytes record what the file contains. (source) Indexing the reuse plan by byte hash additionally means identical outputs can copy from any prior location, not just their own old address.
Reproducibility caution. ADR 0027's measurement was short synthetic footage on one machine, one ffmpeg/driver combination, including six concurrent encodes agreeing with a serial run. That is evidence the copy saving is plausible on that setup — not proof of bit-for-bit determinism across ffmpeg or driver upgrades. Do not lift its encoder flags or concurrency as best practice; measure your own toolchain, and re-measure after any upgrade. The symptom of losing reproducibility is benign (a full re-upload), not a wrong release — size your tolerance accordingly.
Step 2 — Check duration/completeness before acceptance
Accept an artifact on measured properties, not on "the tool exited zero". In the source, three of 93 exports were silently short (by 9.6s, 34.3s, 71.3s) and one was live; the fix is a pure rule: compare the measured duration to the summed clip durations, refuse only a shortfall greater than a tolerance (1s there), always accept overruns, and refuse a file with no playable duration at all. Choose your tolerance from your own product's data — do not copy 1s.
Cache the measurement so it costs once per artifact lifetime: a sidecar beside
the artifact holding the byte hash, size, and measured duration, validated
against the file's current size on every read; an unparseable sidecar or a size mismatch is treated as absent and re-derived. This is not a complete freshness check: replacing an artifact with different bytes of the same size can leave this cached digest stale. Invalidate it with the artifact generation, or recompute the digest when outside modifications are possible. (source: export-sha256-sidecar.ts)
Step 3 — Give resumable work stable identities
Give resumable stages identities that survive the process. The recipe hash can locate an intended artifact, but a corrected renderer can produce different bytes at the same recipe address. Replace the artifact, its sidecar and its publication receipt consistently; the address alone cannot prove freshness.
For agent runs, capture the session id at each stage. The source resumes the session whose extraction failed so the correction retains that context. If the identity is missing, report the recovery limit and inspect persisted outputs before choosing an authorized restart. Do not silently repeat side effects.
Step 4 — Snapshot immutable state before asynchronous work
Decide reuse from an immutable snapshot taken up front, never from state that the running pipeline is mutating. The source draws its reuse plan from the previous release's committed receipt plus a storage listing, and never from older releases; a missing, unparseable, or archived-away receipt simply yields a smaller plan (everything uploads), never an error. The snapshot must be taken before the pool drains: which outputs will be copyable is unknown until each one lands, so (a) snapshot first, (b) decide the copy batch once, at the earliest moment the full set is known, and (c) fall back to upload if the batch is refused.
Step 5 — Repair output serialization separately from repeating side effects
When an agent both does side-effectful work and must emit rigid structured
output, split the phases: run the work without the output schema (so a
malformed JSON never aborts the work), keep its session id and its commits as
the source of truth, then resume that session with an extraction prompt and the
schema, retrying in-context against the failed extraction's own session. The intended repair changes only serialization, preserving the producer's completed work. This requires verification; the prompt alone cannot guarantee that side effects are never repeated. (source:
run-with-extraction.ts, run-with-retry.ts)
Cautions the source docstrings overstate:
- A comment saying the extraction pass does not commit is not mechanically enforced. The retry wrapper spreads the produce options through, so the extraction run inherits the same tool options. Advise: where the host offers an admitted read-only mode for receipt extraction, use it; otherwise treat "extraction does no work" as an invariant to verify, not a guarantee.
- Do not copy this repo's default retry cap (3), its fleet counts, or its model strings as best practices. Use actual host limits, classified failures and evidence of progress; do not introduce arbitrary counts into your own fleet.
Verify
- Reuse: hash an actual artifact, confirm the plan matches only on the actual-byte hash, and confirm a re-run with changed inputs lands at a new address and uploads. Verification means comparing actual output hashes — do not blindly hash every output to prior bytes; choose checks and tolerances appropriate to your product.
- Truncation: feed the acceptance rule a short file and confirm refusal, an overrun and confirm the product's chosen policy, and an unreadable file and confirm refusal.
- Recovery: crash between produce and extract; resume must reuse the session and produce no duplicate commits.
Related PRs: #1591 (open, not merged); #1542 + #1550 (one transport, then
fixing deploy-time migration behavior); #1564 (byte hashes); #1501 (pipelined
publish); #1557 (the live truncated export). CI limitation: PR #1601 omitted
.github/workflows/test.yml; it is absent at this snapshot, so none of the
above behavior is asserted as covered by CI.