Address the completion review findings on top of the upstream review
fixes:
- Store runs on the engine's checkpoint goroutine without the task
lock and serializes the returned value afterwards: take the fetcher
mutex and return a deep copy instead of the live state pointer,
matching the HTTP protocol's Store pattern
- Load the journal and per-run base counters before publishing the run
as active, so Stats/Progress never observe half-initialized fields
- Bind the journal fingerprint to the key IV as well: a rotated IV on
an otherwise identical plan would silently decrypt to garbage
- Keep every EXT-X-MEDIA member per group instead of only the first
one, so a group with a URI on a later member is still detected
- After a merge that raced with a pause, keep the staged segments and
exit silently instead of cleaning them up; drop a dead context
check and an unused helper, and make the MAP identity and merge
part-file names unambiguous
The merge pipeline concatenates stored bytes, so features it cannot
reproduce faithfully are now detected and rejected with a clear error
instead of silently succeeding with a broken output:
- EXT-X-GAP (a hole in the merged file) and EXT-X-DISCONTINUITY
(a broken timeline)
- separate audio renditions referenced by the selected variant: the
master EXT-X-MEDIA groups are parsed and an AUDIO group with its own
URI fails the resolve; unreferenced groups are ignored
- changing init sections: a repeated EXT-X-MAP is compared by all of
its attributes (URI and BYTERANGE), not just the URI
- encrypted EXT-X-MAP without an explicit IV: the spec does not define
a sequence-derived IV for init sections
- non-identity AES-128 KEYFORMAT values
The supported matrix is documented in the package doc.
Add regression tests for every rejection plus real media validation:
ffmpeg-generated TS segments and fMP4 init/fragments are committed as
fixtures, downloaded through the fetcher (including an AES-128 key
rotation variant), and the merged outputs are validated structurally -
TS on the 188 byte grid with continuity counter rules, PAT/PMT and PES
presence, fMP4 by walking the ISO-BMFF box hierarchy.
- Give every task a random staging identity persisted in the saved
state: two tasks downloading the same playlist into the same
directory get separate segment folders and journals instead of
deleting each other's progress
- Bind the journal to a fingerprint of the full download plan (media
URL, ordered segments, ranges, keys); foreign or unreadable journals
are discarded rather than applied to the wrong segments, and the
ledger is rewritten through a temp file + rename
- Legacy states without a staging identity migrate to a fresh folder
instead of reusing the shared URL-hash folder
- Merge into a unique temporary file next to the target, check write
and close errors, then remove-then-rename over the final name: a
failed merge preserves the staged segments for a retry and re-merging
after a crash stays idempotent
- Drop the staging folder when a URL patch invalidates the plan
- Parse the request extra in Setup so restored fetchers resend Referer,
Cookie and other configured headers on resumed segment/key requests;
the store/restore regression test round-trips the state and the task
meta through JSON exactly like the engine storage does
- Rebuild the resolved plan on Start when the resource survived but the
plan was lost to a crash between the task save and the first
checkpoint, keeping the existing output name
- Patch now merges partial updates instead of replacing the request and
options wholesale: URL-only patches keep the output path and renamed
file, header-only patches keep the resolved plan and take effect on
the next run, Labels merge per key, Proxy swaps rebuild the client,
and nil arguments are no longer a panic. Editing a running task is
rejected, consistent with the ed2k protocol
- Cover retry-after-failure counter resets, running-task patch
rejection and the missing-state self-heal with regression tests
- Give every Start its own run context, WaitGroup (including the queue
producer), supervisor done channel and snapshots of the plan, request
headers, HTTP client and merge target; Pause/Close wait for all run
goroutines, including a merge in flight, before returning
- Build the HTTP client in Resolve/Start before workers launch instead of
lazily inside do(), which raced between concurrent segment downloads
- Reject task edits while a run is active instead of mutating shared state
- Reset counters per run and seed them from the completed-segment journal;
count only stored output bytes as progress, never discarded retry bytes
- Report Res.Size only when every segment is unencrypted with a known
length; ciphertext sums overstate the final output, indeterminate plans
rely on the engine's TotalDownloaded backfill at completion
New hls protocol in internal/protocol/hls (registered ahead of http):
- master playlist variant picking (highest bandwidth), RFC3986 segment
URL resolution, fMP4 init segments, EXT-X-BYTERANGE, EXT-X-GAP skip
- AES-128 decryption with mid-playlist key rotation and sequence-derived
IVs; explicit errors for live streams, SAMPLE-AES and multi-period fMP4
- concurrent segment downloads with retry, journal-based resume across
pause/app restarts, automatic in-order merge into a single .ts/.mp4
- best-effort Content-Length probing for determinate progress
- anti-hotlink Referer/Cookie/UA via task extra headers (Go transport
does not filter them)
Core: fix SchemeFilter extension match for URLs with query strings
(also benefits .torrent links); Flutter: Protocol.hls enum, hls protocol
config model (prevents config round-trip wipe) and HLS settings block
with localized labels in all 22 locales.
Tests: parser/crypto unit tests, httptest e2e (Referer-enforced CDN,
encrypted streams, byterange, pause/resume, store/restore) and an
opt-in live-site acceptance gate (HLS_LIVE_URL).
Levi
sun
Wu Shuwen
Levi
ggg
Copilot
Matteo Pietro Dazzi
Krau
Charles Huang