Compare commits

...
Author SHA1 Message Date
Levi c03132d423 fix(windows): scope MSVC coroutine compatibility to affected plugins 2026-10-13 20:26:36 +08:00
Levi 128fada585 fix(ci): pin conforming Windows headers and keep desktop jobs independent 2026-10-13 20:24:47 +08:00
Levi 7dd704fc71 fix(ci): migrate Windows plugin forks to standard coroutines 2026-10-13 20:02:31 +08:00
Ashish Goswami 6f20790b17 feat(protocol/http, mcpserver, download): add checksum verification, MCP speed tool, pause webhook, and CLI flags (#1542) 2026-10-13 08:28:56 +08:00
Kunpeng Xie fa622a4063 fix(http): skip prefetch wait for restored downloads (#1584) 2026-10-12 11:13:42 +08:00
Levi c6619794fc fix: preserve Cookie headers in extension requests (#1581) 2026-10-10 22:10:08 +08:00
Levi 87a8f87b70 feat(ui): rework extension update detection and entry point (#1535) 2026-10-10 22:09:47 +08:00
WuFeng ba84a04a43 fix(settings): use Save as the primary label on settings form dialogs (#1571) 2026-10-08 16:31:16 +08:00
Levi 224b488087 feat(task): record and display task completion time (#1536) 2026-09-25 19:15:31 +08:00
Levi 7ce4fb7f0e fix(ui): smooth task progress bar updates (#1532) 2026-09-25 19:08:20 +08:00
UltraGeek0102 8d1ae93b03 refactor(ios): clean up continued processing task handling (#1547) 2026-09-25 16:21:09 +08:00
JiangJiang 363153fa33 fix(http): use Content-Range total size for 206 responses (#1545) 2026-09-23 09:52:59 +08:00
Levi f0dab90e09 feat(extension): expose request mutation methods on task.meta.req (#1538) 2026-09-23 09:33:28 +08:00
doctor-martin-sowa 6e60889ce8 feat(lang): Add Slovak localization (app_sk.arb) (#1541) 2026-09-23 09:32:07 +08:00
Levi 88d150b933 test(bt): isolate cleanup fixtures from part-file promotion (#1531) 2026-09-21 22:30:54 +08:00
Levi 694b43e3f9 docs: update beta download link to v2.0.0-beta.3 2026-09-19 08:16:57 +08:00
93 changed files with 4364 additions and 412 deletions

No files matched your search

+1 -1
View File
@@ -63,7 +63,7 @@ Connect Gopeed to an AI agent and manage downloads with natural language. For ex
Gopeed 2.0.0 is currently in public beta, introducing a redesigned interface, a native communication architecture that connects desktop and mobile clients directly to the Go core through FFI, a more consistent cross-platform experience, improved task management, more flexible API support, and MCP-based AI agent integration. Some features may still be incomplete or unstable, so please try it and report any issues you encounter. Gopeed 2.0.0 is currently in public beta, introducing a redesigned interface, a native communication architecture that connects desktop and mobile clients directly to the Go core through FFI, a more consistent cross-platform experience, improved task management, more flexible API support, and MCP-based AI agent integration. Some features may still be incomplete or unstable, so please try it and report any issues you encounter.
- [Download Gopeed 2.0.0 Beta 2](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.2) - [Download Gopeed 2.0.0 Beta 3](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.3)
Once the features and stability meet our release standards, we will publish the official Gopeed 2.0.0 release. Beta users will be able to upgrade directly to the final release, while existing stable users will not be automatically moved onto the beta channel. Once the features and stability meet our release standards, we will publish the official Gopeed 2.0.0 release. Beta users will be able to upgrade directly to the final release, while existing stable users will not be automatically moved onto the beta channel.
+1 -1
View File
@@ -63,7 +63,7 @@ Gopeed を AI Agent と連携すると、自然言語でダウンロードを管
Gopeed 2.0.0 は現在パブリックベータ版です。刷新されたインターフェース、デスクトップおよびモバイルから FFI で Go コアへ直接接続するネイティブ通信アーキテクチャ、より統一されたクロスプラットフォーム体験、強化されたタスク管理、より柔軟な API、MCP による AI Agent 連携機能を導入しています。一部の機能はまだ不完全または不安定な場合がありますので、ぜひお試しいただき、問題があればフィードバックをお寄せください。 Gopeed 2.0.0 は現在パブリックベータ版です。刷新されたインターフェース、デスクトップおよびモバイルから FFI で Go コアへ直接接続するネイティブ通信アーキテクチャ、より統一されたクロスプラットフォーム体験、強化されたタスク管理、より柔軟な API、MCP による AI Agent 連携機能を導入しています。一部の機能はまだ不完全または不安定な場合がありますので、ぜひお試しいただき、問題があればフィードバックをお寄せください。
- [Gopeed 2.0.0 Beta 2 をダウンロード](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.2) - [Gopeed 2.0.0 Beta 3 をダウンロード](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.3)
機能と安定性が正式リリースの基準に達した時点で、Gopeed 2.0.0 正式版を公開します。ベータ版のユーザーは正式版へ直接アップデートできます。既存の安定版ユーザーが自動的にベータチャンネルへ移行することはありません。 機能と安定性が正式リリースの基準に達した時点で、Gopeed 2.0.0 正式版を公開します。ベータ版のユーザーは正式版へ直接アップデートできます。既存の安定版ユーザーが自動的にベータチャンネルへ移行することはありません。
+1 -1
View File
@@ -63,7 +63,7 @@ Sau khi kết nối Gopeed với AI Agent, bạn có thể quản lý tải xu
Gopeed 2.0.0 hiện đang trong giai đoạn beta công khai, với giao diện được thiết kế lại, kiến trúc giao tiếp native kết nối trực tiếp ứng dụng desktop và di động với lõi Go qua FFI, trải nghiệm đa nền tảng nhất quán hơn, khả năng quản lý tác vụ được cải thiện, API linh hoạt hơn và khả năng tích hợp AI Agent qua MCP. Một số tính năng có thể vẫn chưa hoàn thiện hoặc chưa ổn định, vì vậy hãy dùng thử và gửi phản hồi nếu bạn gặp vấn đề. Gopeed 2.0.0 hiện đang trong giai đoạn beta công khai, với giao diện được thiết kế lại, kiến trúc giao tiếp native kết nối trực tiếp ứng dụng desktop và di động với lõi Go qua FFI, trải nghiệm đa nền tảng nhất quán hơn, khả năng quản lý tác vụ được cải thiện, API linh hoạt hơn và khả năng tích hợp AI Agent qua MCP. Một số tính năng có thể vẫn chưa hoàn thiện hoặc chưa ổn định, vì vậy hãy dùng thử và gửi phản hồi nếu bạn gặp vấn đề.
- [Tải Gopeed 2.0.0 Beta 2](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.2) - [Tải Gopeed 2.0.0 Beta 3](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.3)
Khi các tính năng và độ ổn định đáp ứng tiêu chuẩn phát hành, chúng tôi sẽ phát hành Gopeed 2.0.0 chính thức. Người dùng bản beta có thể nâng cấp trực tiếp lên bản chính thức, trong khi người dùng bản ổn định hiện tại sẽ không tự động được chuyển sang kênh beta. Khi các tính năng và độ ổn định đáp ứng tiêu chuẩn phát hành, chúng tôi sẽ phát hành Gopeed 2.0.0 chính thức. Người dùng bản beta có thể nâng cấp trực tiếp lên bản chính thức, trong khi người dùng bản ổn định hiện tại sẽ không tự động được chuyển sang kênh beta.
+1 -1
View File
@@ -63,7 +63,7 @@ Gopeed(全称 **Go Speed**,也叫“够快下载器”)是一款使用 Go
Gopeed 2.0.0 目前处于公开 Beta 测试阶段,引入了全新界面、通过 FFI 直连 Go 核心的原生通信架构、更一致的跨平台体验、更完善的任务管理、更灵活的 API,以及用于 AI Agent 接入的 MCP 能力。部分功能仍可能不够完善或稳定,欢迎提前体验并向我们反馈问题。 Gopeed 2.0.0 目前处于公开 Beta 测试阶段,引入了全新界面、通过 FFI 直连 Go 核心的原生通信架构、更一致的跨平台体验、更完善的任务管理、更灵活的 API,以及用于 AI Agent 接入的 MCP 能力。部分功能仍可能不够完善或稳定,欢迎提前体验并向我们反馈问题。
- [下载 Gopeed 2.0.0 Beta 2](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.2) - [下载 Gopeed 2.0.0 Beta 3](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.3)
当功能完整度和稳定性达到正式发布标准后,我们会发布 Gopeed 2.0.0 正式版。已安装 Beta 版本的用户可以直接升级到最终正式版,现有稳定版用户则不会被自动切换到 Beta 渠道。 当功能完整度和稳定性达到正式发布标准后,我们会发布 Gopeed 2.0.0 正式版。已安装 Beta 版本的用户可以直接升级到最终正式版,现有稳定版用户则不会被自动切换到 Beta 渠道。
+1 -1
View File
@@ -63,7 +63,7 @@ Gopeed(**Go Speed** 的縮寫)是一款以 Go 與 Flutter 開發的高速、
Gopeed 2.0.0 目前處於公開 Beta 測試階段,引入了全新介面、透過 FFI 直連 Go 核心的原生通訊架構、更一致的跨平台體驗、更完善的任務管理、更靈活的 API,以及透過 MCP 串接 AI Agent 的能力。部分功能仍可能不夠完善或穩定,歡迎搶先體驗並向我們回報問題。 Gopeed 2.0.0 目前處於公開 Beta 測試階段,引入了全新介面、透過 FFI 直連 Go 核心的原生通訊架構、更一致的跨平台體驗、更完善的任務管理、更靈活的 API,以及透過 MCP 串接 AI Agent 的能力。部分功能仍可能不夠完善或穩定,歡迎搶先體驗並向我們回報問題。
- [下載 Gopeed 2.0.0 Beta 2](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.2) - [下載 Gopeed 2.0.0 Beta 3](https://github.com/GopeedLab/gopeed/releases/tag/v2.0.0-beta.3)
當功能完整度與穩定性達到正式發布標準後,我們會發布 Gopeed 2.0.0 正式版。已安裝 Beta 版本的使用者可以直接升級至最終正式版,現有穩定版使用者則不會被自動切換至 Beta 頻道。 當功能完整度與穩定性達到正式發布標準後,我們會發布 Gopeed 2.0.0 正式版。已安裝 Beta 版本的使用者可以直接升級至最終正式版,現有穩定版使用者則不會被自動切換至 Beta 頻道。
+4
View File
@@ -10,6 +10,8 @@ type args struct {
url string url string
connections *int connections *int
dir *string dir *string
checksum *string
checksumAlgo *string
} }
func parse() *args { func parse() *args {
@@ -20,6 +22,8 @@ func parse() *args {
var args args var args args
args.connections = flag.Int("C", 16, "Concurrent connections.") args.connections = flag.Int("C", 16, "Concurrent connections.")
args.dir = flag.String("D", dir, "Store directory.") args.dir = flag.String("D", dir, "Store directory.")
args.checksum = flag.String("checksum", "", "Expected checksum hash (hex encoded).")
args.checksumAlgo = flag.String("checksum-algo", "md5", "Checksum algorithm: md5, sha1, or sha256.")
flag.Parse() flag.Parse()
t := flag.Args() t := flag.Args()
if len(t) > 0 { if len(t) > 0 {
+16 -1
View File
@@ -19,6 +19,18 @@ func main() {
var wg sync.WaitGroup var wg sync.WaitGroup
wg.Add(1) wg.Add(1)
var checksumOpt *http.ChecksumOption
if args.checksum != nil && *args.checksum != "" {
algo := "md5"
if args.checksumAlgo != nil && *args.checksumAlgo != "" {
algo = *args.checksumAlgo
}
checksumOpt = &http.ChecksumOption{
Algorithm: algo,
Expected: *args.checksum,
}
}
_, err := download.Boot(). _, err := download.Boot().
URL(args.url). URL(args.url).
Listener(func(event *download.Event) { Listener(func(event *download.Event) {
@@ -44,7 +56,10 @@ func main() {
}). }).
Create(&base.Options{ Create(&base.Options{
Path: *args.dir, Path: *args.dir,
Extra: http.OptsExtra{Connections: *args.connections}, Extra: http.OptsExtra{
Connections: *args.connections,
Checksum: checksumOpt,
},
}) })
if err != nil { if err != nil {
panic(err) panic(err)
+7 -2
View File
@@ -83,10 +83,15 @@ func TestWaitCleansUnselectedFilesFromSharedPiece(t *testing.T) {
{Length: 4, Path: []string{"keep.txt"}}, {Length: 4, Path: []string{"keep.txt"}},
{Length: 4, Path: []string{"nested", "skip.txt"}}, {Length: 4, Path: []string{"nested", "skip.txt"}},
}} }}
store := storage.NewFileOpts(storage.NewFileClientOpts{ storageOpts := storage.NewFileClientOpts{
ClientBaseDir: dir, ClientBaseDir: dir,
TorrentDirMaker: func(baseDir string, _ *metainfo.Info, _ metainfo.Hash) string { return baseDir }, TorrentDirMaker: func(baseDir string, _ *metainfo.Info, _ metainfo.Hash) string { return baseDir },
}) }
// VerifyDataContext can return before storage finishes marking the piece
// complete. Disable automatic part-file promotion so it cannot rename
// the .part fixtures that this test creates to exercise cleanup.
storageOpts.UsePartFiles.Set(false)
store := storage.NewFileOpts(storageOpts)
defer store.Close() defer store.Close()
config := torrent.TestingConfig(t) config := torrent.TestingConfig(t)
config.DisableTCP = true config.DisableTCP = true
+152 -22
View File
@@ -2,8 +2,13 @@ package http
import ( import (
"context" "context"
"crypto/md5"
"crypto/sha1"
"crypto/sha256"
"encoding/hex"
"errors" "errors"
"fmt" "fmt"
"hash"
"io" "io"
"net/http" "net/http"
"net/url" "net/url"
@@ -280,6 +285,12 @@ type Fetcher struct {
primaryReadyOnce sync.Once primaryReadyOnce sync.Once
primaryReadyCh chan struct{} primaryReadyCh chan struct{}
// checksumFailed marks that the last completed download failed
// checksum verification, so the next retry must force a full
// re-download instead of skipping connections that already
// reported Completed.
checksumFailed atomic.Bool
// Start pending mechanism // Start pending mechanism
startPending atomic.Bool startPending atomic.Bool
resolvedCh chan struct{} // Signal when resolve completes resolvedCh chan struct{} // Signal when resolve completes
@@ -417,12 +428,26 @@ func (f *Fetcher) Resolve(req *base.Request, opts *base.Options) error {
res.Range = true res.Range = true
} }
// Get content length from Content-Length header // A partial response describes one segment, not the whole resource.
reuseResolveBody := true
if resp.StatusCode == base.HttpCodePartialContent {
start, _, total, err := parseRangeResponse(resp)
if err != nil {
resp.Body.Close()
f.setState(stateError)
return err
}
res.Size = total
res.Range = true
// Prefetch is a contiguous prefix copied at offset zero. A segment
// starting elsewhere must be fetched again through the Range path.
reuseResolveBody = start == 0
} else {
contentLength := resp.Header.Get(base.HttpHeaderContentLength) contentLength := resp.Header.Get(base.HttpHeaderContentLength)
if contentLength != "" { if contentLength != "" {
parse, err := strconv.ParseInt(contentLength, 10, 64) if size, err := strconv.ParseInt(contentLength, 10, 64); err == nil {
if err == nil { res.Size = size
res.Size = parse }
} }
} }
@@ -467,7 +492,12 @@ func (f *Fetcher) Resolve(req *base.Request, opts *base.Options) error {
// IMPORTANT: Keep the response body open for downloading in Start phase // IMPORTANT: Keep the response body open for downloading in Start phase
// This is crucial for one-time URLs that can only be accessed once // This is crucial for one-time URLs that can only be accessed once
f.resolveRespLock.Lock() f.resolveRespLock.Lock()
if reuseResolveBody {
f.resolveResp = resp f.resolveResp = resp
} else {
resp.Body.Close()
f.prefetchDone.Store(true)
}
f.resolveRespLock.Unlock() f.resolveRespLock.Unlock()
f.setState(stateResolved) f.setState(stateResolved)
@@ -479,7 +509,7 @@ func (f *Fetcher) Resolve(req *base.Request, opts *base.Options) error {
// Start async prefetch in background (only for range-supported resources) // Start async prefetch in background (only for range-supported resources)
// For non-range resources, the response will be used directly in Start // For non-range resources, the response will be used directly in Start
if res.Range { if res.Range && reuseResolveBody {
f.prefetchStopCh = make(chan struct{}) f.prefetchStopCh = make(chan struct{})
f.resolveFallback.Store(true) f.resolveFallback.Store(true)
go f.asyncPrefetch() go f.asyncPrefetch()
@@ -590,7 +620,7 @@ func (f *Fetcher) stopPrefetchAndCopyData() int64 {
} }
// Wait for prefetch to finish (with timeout) // Wait for prefetch to finish (with timeout)
for i := 0; i < 1000 && !f.prefetchDone.Load(); i++ { for i := 0; f.prefetchStopCh != nil && i < 1000 && !f.prefetchDone.Load(); i++ {
time.Sleep(10 * time.Millisecond) time.Sleep(10 * time.Millisecond)
} }
@@ -645,6 +675,22 @@ func (f *Fetcher) cleanupPrefetchFile() {
f.resolveFallback.Store(false) f.resolveFallback.Store(false)
} }
// prepareFullRedownload resets all fetcher connection and prefetch state
// after a checksum mismatch so the subsequent Start takes the fresh-start path.
func (f *Fetcher) prepareFullRedownload() {
f.connMu.Lock()
if f.meta.Res.Range {
f.connections = nil
f.resolveConn = nil
f.rangeReprobeEligible = false
f.rangeValidatorPinned = false
}
f.connMu.Unlock()
f.cleanupPrefetchFile()
f.resolveDataPos.Store(0)
}
// Options may be replaced between Resolve and Start when creating a resolved task. // Options may be replaced between Resolve and Start when creating a resolved task.
func (f *Fetcher) initOptions() error { func (f *Fetcher) initOptions() error {
opts := f.meta.Opts opts := f.meta.Opts
@@ -662,6 +708,9 @@ func (f *Fetcher) initOptions() error {
extra.Connections = 1 extra.Connections = 1
} }
} }
if err := extra.Checksum.Validate(); err != nil {
return err
}
return nil return nil
} }
@@ -716,11 +765,26 @@ func (f *Fetcher) doStart() error {
// If retrying after error, reset connection states for retry // If retrying after error, reset connection states for retry
if state == stateError { if state == stateError {
fullRedownload := f.checksumFailed.Swap(false)
if fullRedownload {
// Checksum mismatch: the file content itself is wrong even
// though every connection reported Completed. Truncate the
// corrupt file so stale bytes cannot linger if the retry is
// interrupted before every connection re-writes its chunk.
if err := os.Truncate(f.meta.SingleFilepath(), 0); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to reset file after checksum mismatch: %w", err)
}
f.prepareFullRedownload()
}
f.connMu.Lock() f.connMu.Lock()
for _, conn := range f.connections { for _, conn := range f.connections {
// Reset connections that can be retried // On a normal error, only reset connections that didn't
if !conn.Completed && conn.State != connCompleted { // finish. On a checksum mismatch, reset EVERY connection —
if !f.hasSequentialPrefixLocked(conn) { // "Completed" only means the download loop finished, not
// that the bytes were verified correct.
shouldReset := fullRedownload || (!conn.Completed && conn.State != connCompleted)
if shouldReset {
if fullRedownload || !f.hasSequentialPrefixLocked(conn) {
f.resetConnectionForRestart(conn) f.resetConnectionForRestart(conn)
} }
conn.State = connNotStarted conn.State = connNotStarted
@@ -951,8 +1015,14 @@ func (f *Fetcher) expandConnections() {
} }
f.fileMu.Unlock() f.fileMu.Unlock()
var err error
if chkErr := f.verifyChecksum(); chkErr != nil {
err = chkErr
f.setState(stateError)
} else {
f.setState(stateDone) f.setState(stateDone)
f.doneCh <- nil }
f.doneCh <- err
return return
} }
@@ -1544,27 +1614,41 @@ func (f *Fetcher) completeFromResolveFallback(conn *connection) bool {
return true return true
} }
func validateRangeResponse(resp *http.Response, requestedStart int64, requestedEnd *int64, totalSize int64) (int64, int64, error) { // parseRangeResponse validates a single byte range with a known complete length.
// Unknown totals are rejected: a segment length cannot stand in for resource size.
func parseRangeResponse(resp *http.Response) (int64, int64, int64, error) {
contentRange := resp.Header.Get(base.HttpHeaderContentRange) contentRange := resp.Header.Get(base.HttpHeaderContentRange)
fields := strings.Fields(contentRange) fields := strings.Fields(contentRange)
if len(fields) != 2 || !strings.EqualFold(fields[0], base.HttpHeaderBytes) { if len(fields) != 2 || !strings.EqualFold(fields[0], base.HttpHeaderBytes) {
return 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange) return 0, 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange)
} }
rangeAndTotal := strings.Split(fields[1], "/") rangeAndTotal := strings.Split(fields[1], "/")
if len(rangeAndTotal) != 2 { if len(rangeAndTotal) != 2 {
return 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange) return 0, 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange)
} }
bounds := strings.Split(rangeAndTotal[0], "-") bounds := strings.Split(rangeAndTotal[0], "-")
if len(bounds) != 2 { if len(bounds) != 2 {
return 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange) return 0, 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange)
} }
start, startErr := strconv.ParseInt(bounds[0], 10, 64) start, startErr := strconv.ParseInt(bounds[0], 10, 64)
end, endErr := strconv.ParseInt(bounds[1], 10, 64) end, endErr := strconv.ParseInt(bounds[1], 10, 64)
total, totalErr := strconv.ParseInt(rangeAndTotal[1], 10, 64) total, totalErr := strconv.ParseInt(rangeAndTotal[1], 10, 64)
if startErr != nil || endErr != nil || totalErr != nil || start < 0 || end < start || total <= end { if startErr != nil || endErr != nil || totalErr != nil || start < 0 || end < start || total <= end {
return 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange) return 0, 0, 0, fmt.Errorf("%w: malformed Content-Range %q", errInvalidRangeResponse, contentRange)
}
expectedLength := end - start + 1
if resp.ContentLength >= 0 && resp.ContentLength != expectedLength {
return 0, 0, 0, fmt.Errorf("%w: Content-Length %d does not match range length %d", errInvalidRangeResponse, resp.ContentLength, expectedLength)
}
return start, end, total, nil
}
func validateRangeResponse(resp *http.Response, requestedStart int64, requestedEnd *int64, totalSize int64) (int64, int64, error) {
start, end, total, err := parseRangeResponse(resp)
if err != nil {
return 0, 0, err
} }
if start != requestedStart || (requestedEnd != nil && end != *requestedEnd) || (requestedEnd == nil && end != total-1) { if start != requestedStart || (requestedEnd != nil && end != *requestedEnd) || (requestedEnd == nil && end != total-1) {
return 0, 0, fmt.Errorf("%w: Content-Range %d-%d does not match requested range", errInvalidRangeResponse, start, end) return 0, 0, fmt.Errorf("%w: Content-Range %d-%d does not match requested range", errInvalidRangeResponse, start, end)
@@ -1573,11 +1657,7 @@ func validateRangeResponse(resp *http.Response, requestedStart int64, requestedE
return 0, 0, fmt.Errorf("%w: Content-Range total %d does not match resource size %d", errInvalidRangeResponse, total, totalSize) return 0, 0, fmt.Errorf("%w: Content-Range total %d does not match resource size %d", errInvalidRangeResponse, total, totalSize)
} }
expectedLength := end - start + 1 return end - start + 1, total, nil
if resp.ContentLength >= 0 && resp.ContentLength != expectedLength {
return 0, 0, fmt.Errorf("%w: Content-Length %d does not match range length %d", errInvalidRangeResponse, resp.ContentLength, expectedLength)
}
return expectedLength, total, nil
} }
func (f *Fetcher) resolveOpenEndedRange(conn *connection, totalSize int64) error { func (f *Fetcher) resolveOpenEndedRange(conn *connection, totalSize int64) error {
@@ -2038,7 +2118,7 @@ func (f *Fetcher) resetConnectionForRestart(conn *connection) {
// so pause/retry must restart instead of continuing from the old offset. // so pause/retry must restart instead of continuing from the old offset.
if conn.Chunk == nil { if conn.Chunk == nil {
conn.Chunk = newChunk(0, 0) conn.Chunk = newChunk(0, 0)
} else { } else if !f.meta.Res.Range {
conn.Chunk.Begin = 0 conn.Chunk.Begin = 0
conn.Chunk.setEnd(0) conn.Chunk.setEnd(0)
conn.Chunk.Downloaded = 0 conn.Chunk.Downloaded = 0
@@ -2209,11 +2289,16 @@ func (f *Fetcher) onDownloadComplete() {
} }
f.fileMu.Unlock() f.fileMu.Unlock()
if finalErr != nil { if finalErr == nil {
if chkErr := f.verifyChecksum(); chkErr != nil {
finalErr = chkErr
f.setState(stateError) f.setState(stateError)
} else { } else {
f.setState(stateDone) f.setState(stateDone)
} }
} else {
f.setState(stateError)
}
select { select {
case f.doneCh <- finalErr: case f.doneCh <- finalErr:
@@ -2221,6 +2306,51 @@ func (f *Fetcher) onDownloadComplete() {
} }
} }
func (f *Fetcher) verifyChecksum() error {
if f.meta == nil || f.meta.Opts == nil || f.meta.Opts.Extra == nil {
return nil
}
extra, ok := f.meta.Opts.Extra.(*fhttp.OptsExtra)
if !ok || extra.Checksum == nil {
return nil
}
chk := extra.Checksum
if chk.Algorithm == "" && chk.Expected == "" {
return nil
}
var h hash.Hash
switch strings.ToLower(strings.TrimSpace(chk.Algorithm)) {
case "md5":
h = md5.New()
case "sha1", "sha-1":
h = sha1.New()
case "sha256", "sha-256":
h = sha256.New()
default:
return fmt.Errorf("unsupported checksum algorithm: %s", chk.Algorithm)
}
filePath := f.meta.SingleFilepath()
file, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("open file for checksum verification failed: %w", err)
}
defer file.Close()
buf := make([]byte, 64*1024)
if _, err := io.CopyBuffer(h, file, buf); err != nil {
return fmt.Errorf("calculate checksum failed: %w", err)
}
actual := hex.EncodeToString(h.Sum(nil))
expected := strings.ToLower(strings.TrimSpace(chk.Expected))
if !strings.EqualFold(actual, expected) {
f.checksumFailed.Store(true)
return fmt.Errorf("checksum mismatch: expected %s, got %s", expected, actual)
}
return nil
}
func (f *Fetcher) checkCompletion() bool { func (f *Fetcher) checkCompletion() bool {
// Check if all data has been downloaded // Check if all data has been downloaded
f.connMu.Lock() f.connMu.Lock()
@@ -0,0 +1,388 @@
package http
import (
"crypto/sha256"
"encoding/hex"
"fmt"
gohttp "net/http"
"net/http/httptest"
"os"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/GopeedLab/gopeed/pkg/base"
"github.com/GopeedLab/gopeed/pkg/protocol/http"
)
// serveRangeContent serves byte range requests or full responses from data.
func serveRangeContent(w gohttp.ResponseWriter, r *gohttp.Request, data []byte) {
w.Header().Set(base.HttpHeaderAcceptRanges, base.HttpHeaderBytes)
rangeHdr := r.Header.Get(base.HttpHeaderRange)
total := int64(len(data))
if rangeHdr == "" {
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", total))
w.WriteHeader(gohttp.StatusOK)
_, _ = w.Write(data)
return
}
// Parse bytes=start-end or bytes=start-
if !strings.HasPrefix(rangeHdr, "bytes=") {
w.WriteHeader(gohttp.StatusBadRequest)
return
}
spec := strings.TrimPrefix(rangeHdr, "bytes=")
parts := strings.Split(spec, "-")
start, err := strconv.ParseInt(parts[0], 10, 64)
if err != nil || start < 0 || start >= total {
w.WriteHeader(gohttp.StatusRequestedRangeNotSatisfiable)
return
}
end := total - 1
if len(parts) > 1 && parts[1] != "" {
if parsedEnd, err := strconv.ParseInt(parts[1], 10, 64); err == nil && parsedEnd >= start && parsedEnd < total {
end = parsedEnd
}
}
contentLength := end - start + 1
w.Header().Set(base.HttpHeaderContentRange, fmt.Sprintf("bytes %d-%d/%d", start, end, total))
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", contentLength))
w.WriteHeader(gohttp.StatusPartialContent)
_, _ = w.Write(data[start : end+1])
}
func TestFetcher_ChecksumMismatch_Retry(t *testing.T) {
corruptContent := []byte("Corrupt Initial Download Data 33B")
correctContent := []byte("Correct Retried Download Data 33B")
correctSha256Hasher := sha256.New()
correctSha256Hasher.Write(correctContent)
correctSha256 := hex.EncodeToString(correctSha256Hasher.Sum(nil))
var phase atomic.Int32 // 0: serve corruptContent, 1: serve correctContent
server := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) {
var served []byte
if phase.Load() == 0 {
served = corruptContent
} else {
served = correctContent
}
serveRangeContent(w, r, served)
}))
defer server.Close()
downloadDir := t.TempDir()
fileName := "test_checksum_retry.txt"
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: downloadDir,
Name: fileName,
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "sha256",
Expected: correctSha256,
},
},
}
// 1. Initial attempt -> will download corruptContent, failing checksum verification
if err := f.Resolve(&base.Request{URL: server.URL + "/retry_test.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
err := f.Wait()
if err == nil {
t.Fatal("expected checksum mismatch error on initial attempt, got nil")
}
if !strings.Contains(err.Error(), "checksum mismatch") {
t.Fatalf("expected 'checksum mismatch' in error, got %v", err)
}
if f.getState() != stateError {
t.Fatalf("expected stateError after checksum mismatch, got %v", f.getState())
}
// Flip phase to 1 after observing checksum mismatch
phase.Store(1)
// 2. Retry download -> should trigger forced full reset, truncate file, and re-download correctContent
if err := f.Start(); err != nil {
t.Fatalf("Start error on retry: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error on retry (expected success): %v", err)
}
if f.getState() != stateDone {
t.Fatalf("expected stateDone after successful retry, got %v", f.getState())
}
// 3. Verify on-disk file content matches the re-downloaded bytes
filePath := f.meta.SingleFilepath()
fileBytes, err := os.ReadFile(filePath)
if err != nil {
t.Fatalf("failed to read completed file: %v", err)
}
if string(fileBytes) != string(correctContent) {
t.Fatalf("on-disk file content mismatch:\ngot: %q\nwant: %q", string(fileBytes), string(correctContent))
}
// 4. Compute hash independently and verify
actualHasher := sha256.New()
actualHasher.Write(fileBytes)
actualHash := hex.EncodeToString(actualHasher.Sum(nil))
if actualHash != correctSha256 {
t.Fatalf("re-computed on-disk hash mismatch: got %s, want %s", actualHash, correctSha256)
}
}
func TestFetcher_ChecksumMismatch_Retry_PartialPrefetch(t *testing.T) {
corruptContent := []byte("Corrupt Initial Download Data 33B")
correctContent := []byte("Correct Retried Download Data 33B")
correctSha256Hasher := sha256.New()
correctSha256Hasher.Write(correctContent)
correctSha256 := hex.EncodeToString(correctSha256Hasher.Sum(nil))
var phase atomic.Int32 // 0: corrupt, 1: correct
var isFirstRequest atomic.Bool
isFirstRequest.Store(true)
releasePrefetch := make(chan struct{})
var closeReleaseOnce sync.Once
var mu sync.Mutex
var phase1Ranges []string
server := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) {
curPhase := phase.Load()
if curPhase == 1 {
mu.Lock()
phase1Ranges = append(phase1Ranges, r.Header.Get(base.HttpHeaderRange))
mu.Unlock()
serveRangeContent(w, r, correctContent)
return
}
// Phase 0: First request is Resolve GET without Range header.
// Write exactly 7 bytes and flush, then block until release channel closes or context is done.
if isFirstRequest.CompareAndSwap(true, false) {
w.Header().Set(base.HttpHeaderAcceptRanges, base.HttpHeaderBytes)
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(corruptContent)))
w.WriteHeader(gohttp.StatusOK)
_, _ = w.Write(corruptContent[:7])
if flusher, ok := w.(gohttp.Flusher); ok {
flusher.Flush()
}
select {
case <-releasePrefetch:
case <-r.Context().Done():
}
return
}
// Subsequent requests in Phase 0 (such as Range request from Start)
serveRangeContent(w, r, corruptContent)
}))
defer server.Close()
defer closeReleaseOnce.Do(func() { close(releasePrefetch) })
downloadDir := t.TempDir()
fileName := "test_checksum_prefetch_retry.txt"
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: downloadDir,
Name: fileName,
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "sha256",
Expected: correctSha256,
},
},
}
// 1. Resolve phase
if err := f.Resolve(&base.Request{URL: server.URL + "/prefetch_retry.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
// Wait until prefetch has received exactly 7 bytes
for i := 0; i < 200; i++ {
if f.prefetchSize.Load() >= 7 {
break
}
time.Sleep(10 * time.Millisecond)
}
if f.prefetchSize.Load() < 7 {
t.Fatalf("expected at least 7 prefetched bytes, got %d", f.prefetchSize.Load())
}
// Release prefetch handler right before Start
closeReleaseOnce.Do(func() { close(releasePrefetch) })
// 2. Start initial download attempt (will complete with corruptContent and fail checksum)
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
err := f.Wait()
if err == nil {
t.Fatal("expected checksum mismatch error on initial attempt, got nil")
}
if !strings.Contains(err.Error(), "checksum mismatch") {
t.Fatalf("expected 'checksum mismatch' in error, got %v", err)
}
if f.getState() != stateError {
t.Fatalf("expected stateError after checksum mismatch, got %v", f.getState())
}
// 3. Flip to phase 1 (correctContent) and retry
phase.Store(1)
if err := f.Start(); err != nil {
t.Fatalf("Start error on retry: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error on retry (expected success): %v", err)
}
if f.getState() != stateDone {
t.Fatalf("expected stateDone after successful retry, got %v", f.getState())
}
// Verify that in Phase 1 at least one Range request started at byte 0
mu.Lock()
rangesSeen := append([]string(nil), phase1Ranges...)
mu.Unlock()
hasByteZero := false
for _, r := range rangesSeen {
if strings.HasPrefix(r, "bytes=0-") {
hasByteZero = true
break
}
}
if !hasByteZero {
t.Fatalf("expected at least one Phase 1 Range request starting at byte 0, saw: %v", rangesSeen)
}
// Verify on-disk content matches correctContent exactly
filePath := f.meta.SingleFilepath()
fileBytes, err := os.ReadFile(filePath)
if err != nil {
t.Fatalf("failed to read completed file: %v", err)
}
if string(fileBytes) != string(correctContent) {
t.Fatalf("on-disk file content mismatch:\ngot: %q\nwant: %q", string(fileBytes), string(correctContent))
}
// Compute hash independently and verify
actualHasher := sha256.New()
actualHasher.Write(fileBytes)
actualHash := hex.EncodeToString(actualHasher.Sum(nil))
if actualHash != correctSha256 {
t.Fatalf("re-computed on-disk hash mismatch: got %s, want %s", actualHash, correctSha256)
}
}
func TestFetcher_ChecksumMismatch_Retry_NonRange(t *testing.T) {
corruptContent := []byte("Corrupt Non-Range Initial Data 33B")
correctContent := []byte("Correct Non-Range Retried Data 33B")
correctSha256Hasher := sha256.New()
correctSha256Hasher.Write(correctContent)
correctSha256 := hex.EncodeToString(correctSha256Hasher.Sum(nil))
var phase atomic.Int32 // 0: corrupt, 1: correct
server := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) {
// Server does not support Range (always returns full content 200 OK)
var served []byte
if phase.Load() == 0 {
served = corruptContent
} else {
served = correctContent
}
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(served)))
w.WriteHeader(gohttp.StatusOK)
_, _ = w.Write(served)
}))
defer server.Close()
downloadDir := t.TempDir()
fileName := "test_checksum_nonrange_retry.txt"
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: downloadDir,
Name: fileName,
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "sha256",
Expected: correctSha256,
},
},
}
// 1. Resolve and start initial attempt
if err := f.Resolve(&base.Request{URL: server.URL + "/nonrange_retry.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
err := f.Wait()
if err == nil {
t.Fatal("expected checksum mismatch error on initial non-range attempt, got nil")
}
if !strings.Contains(err.Error(), "checksum mismatch") {
t.Fatalf("expected 'checksum mismatch' in error, got %v", err)
}
if f.getState() != stateError {
t.Fatalf("expected stateError after checksum mismatch, got %v", f.getState())
}
// 2. Switch phase and retry
phase.Store(1)
if err := f.Start(); err != nil {
t.Fatalf("Start error on non-range retry: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error on non-range retry (expected success): %v", err)
}
if f.getState() != stateDone {
t.Fatalf("expected stateDone after successful non-range retry, got %v", f.getState())
}
// 3. Verify on-disk file content
filePath := f.meta.SingleFilepath()
fileBytes, err := os.ReadFile(filePath)
if err != nil {
t.Fatalf("failed to read completed file: %v", err)
}
if string(fileBytes) != string(correctContent) {
t.Fatalf("on-disk file content mismatch:\ngot: %q\nwant: %q", string(fileBytes), string(correctContent))
}
// 4. Compute hash independently and verify
actualHasher := sha256.New()
actualHasher.Write(fileBytes)
actualHash := hex.EncodeToString(actualHasher.Sum(nil))
if actualHash != correctSha256 {
t.Fatalf("re-computed on-disk hash mismatch: got %s, want %s", actualHash, correctSha256)
}
}
@@ -0,0 +1,304 @@
package http
import (
"crypto/md5"
"crypto/sha256"
"encoding/hex"
"fmt"
gohttp "net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"github.com/GopeedLab/gopeed/internal/test"
"github.com/GopeedLab/gopeed/pkg/base"
"github.com/GopeedLab/gopeed/pkg/protocol/http"
)
func TestFetcher_ChecksumVerification(t *testing.T) {
content := []byte("Hello, Gopeed Checksum Verification Test Data!")
md5Hasher := md5.New()
md5Hasher.Write(content)
correctMd5 := hex.EncodeToString(md5Hasher.Sum(nil))
sha256Hasher := sha256.New()
sha256Hasher.Write(content)
correctSha256 := hex.EncodeToString(sha256Hasher.Sum(nil))
wrongHash := "0123456789abcdef0123456789abcdef"
server := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) {
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(content)))
w.Header().Set(base.HttpHeaderAcceptRanges, base.HttpHeaderBytes)
w.WriteHeader(gohttp.StatusOK)
_, _ = w.Write(content)
}))
defer server.Close()
t.Run("Valid MD5", func(t *testing.T) {
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_valid_md5.txt",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "md5",
Expected: correctMd5,
},
},
}
if err := f.Resolve(&base.Request{URL: server.URL + "/test.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error (expected success): %v", err)
}
})
t.Run("Valid SHA256", func(t *testing.T) {
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_valid_sha256.txt",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "sha256",
Expected: correctSha256,
},
},
}
if err := f.Resolve(&base.Request{URL: server.URL + "/test.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error (expected success): %v", err)
}
})
t.Run("Wrong MD5 mismatch", func(t *testing.T) {
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_wrong_md5.txt",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "md5",
Expected: wrongHash,
},
},
}
if err := f.Resolve(&base.Request{URL: server.URL + "/test.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
err := f.Wait()
if err == nil {
t.Fatal("Wait expected checksum mismatch error, got nil")
}
if !strings.Contains(err.Error(), "checksum mismatch") {
t.Fatalf("expected 'checksum mismatch' in error, got %v", err)
}
})
t.Run("Unsupported Algorithm", func(t *testing.T) {
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_unsupported.txt",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "invalid-algo",
Expected: wrongHash,
},
},
}
err := f.Resolve(&base.Request{URL: server.URL + "/test.txt"}, opts)
if err == nil {
t.Fatal("Resolve expected error for unsupported algorithm, got nil")
}
if !strings.Contains(err.Error(), "unsupported checksum algorithm") {
t.Fatalf("expected 'unsupported checksum algorithm' in error, got %v", err)
}
if f.checksumFailed.Load() {
t.Fatal("checksumFailed flag should NOT be set on unsupported algorithm error")
}
})
t.Run("Case-Insensitive Hash Comparison", func(t *testing.T) {
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_case_insensitive.txt",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "sha256",
Expected: strings.ToUpper(correctSha256),
},
},
}
if err := f.Resolve(&base.Request{URL: server.URL + "/test.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error (expected success with upper case hash): %v", err)
}
})
t.Run("Empty Expected Checksum (Skipped)", func(t *testing.T) {
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_empty_expected.txt",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "",
Expected: "",
},
},
}
if err := f.Resolve(&base.Request{URL: server.URL + "/test.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error (expected skip and success): %v", err)
}
})
t.Run("Large File Multi-Chunk Streaming (>64KB)", func(t *testing.T) {
largeData := make([]byte, 128*1024)
for i := range largeData {
largeData[i] = byte(i % 256)
}
largeSha256Hasher := sha256.New()
largeSha256Hasher.Write(largeData)
largeSha256 := hex.EncodeToString(largeSha256Hasher.Sum(nil))
largeServer := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) {
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(largeData)))
w.Header().Set(base.HttpHeaderAcceptRanges, base.HttpHeaderBytes)
w.WriteHeader(gohttp.StatusOK)
_, _ = w.Write(largeData)
}))
defer largeServer.Close()
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_large_file.bin",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "sha256",
Expected: largeSha256,
},
},
}
if err := f.Resolve(&base.Request{URL: largeServer.URL + "/large.bin"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error (expected large file checksum success): %v", err)
}
})
t.Run("No Checksum (Opt-in backward compatibility)", func(t *testing.T) {
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: test.DownloadName,
Extra: &http.OptsExtra{
Connections: 1,
},
}
if err := f.Resolve(&base.Request{URL: server.URL + "/test.txt"}, opts); err != nil {
t.Fatalf("Resolve error: %v", err)
}
if err := f.Start(); err != nil {
t.Fatalf("Start error: %v", err)
}
if err := f.Wait(); err != nil {
t.Fatalf("Wait error: %v", err)
}
})
t.Run("Invalid Checksum In Resolve Makes Zero Requests", func(t *testing.T) {
var reqReceived atomic.Int32
customServer := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) {
reqReceived.Add(1)
w.WriteHeader(gohttp.StatusOK)
_, _ = w.Write([]byte("some data"))
}))
defer customServer.Close()
f := buildFetcher()
defer f.Close()
opts := &base.Options{
Path: t.TempDir(),
Name: "test_invalid_checksum.txt",
Extra: &http.OptsExtra{
Connections: 1,
Checksum: &http.ChecksumOption{
Algorithm: "sha256",
Expected: "invalid-hex-chars-not-valid-hex-00000000000000000000000000000000",
},
},
}
err := f.Resolve(&base.Request{URL: customServer.URL + "/test.txt"}, opts)
if err == nil {
t.Fatal("Resolve expected error for invalid checksum hash, got nil")
}
if reqReceived.Load() != 0 {
t.Fatalf("expected 0 requests to server on invalid checksum in Resolve, got %d", reqReceived.Load())
}
})
}
+13
View File
@@ -13,6 +13,7 @@ import (
"net/url" "net/url"
"os" "os"
"strings" "strings"
"sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time" "time"
@@ -853,22 +854,34 @@ func TestFetcher_InitialIgnoredRangeUsesResponseSize(t *testing.T) {
resolvePayload := bytes.Repeat([]byte("resolve"), 64*1024) resolvePayload := bytes.Repeat([]byte("resolve"), 64*1024)
replacement := bytes.Repeat([]byte("replacement"), 24*1024) replacement := bytes.Repeat([]byte("replacement"), 24*1024)
var rangeRequests atomic.Int32 var rangeRequests atomic.Int32
rangeStarted := make(chan struct{})
releaseResolve := sync.OnceFunc(func() { close(rangeStarted) })
server := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) { server := httptest.NewServer(gohttp.HandlerFunc(func(w gohttp.ResponseWriter, r *gohttp.Request) {
w.Header().Set(base.HttpHeaderAcceptRanges, base.HttpHeaderBytes) w.Header().Set(base.HttpHeaderAcceptRanges, base.HttpHeaderBytes)
if r.Header.Get(base.HttpHeaderRange) == "" { if r.Header.Get(base.HttpHeaderRange) == "" {
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(resolvePayload))) w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(resolvePayload)))
w.WriteHeader(gohttp.StatusOK) w.WriteHeader(gohttp.StatusOK)
w.(gohttp.Flusher).Flush()
// Keep Resolve prefetch incomplete until the Range request starts.
// Otherwise Start may reuse the completed body and skip this scenario.
select {
case <-rangeStarted:
case <-r.Context().Done():
return
}
_, _ = w.Write(resolvePayload) _, _ = w.Write(resolvePayload)
return return
} }
rangeRequests.Add(1) rangeRequests.Add(1)
releaseResolve()
w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(replacement))) w.Header().Set(base.HttpHeaderContentLength, fmt.Sprintf("%d", len(replacement)))
w.WriteHeader(gohttp.StatusOK) w.WriteHeader(gohttp.StatusOK)
_, _ = w.Write(replacement) _, _ = w.Write(replacement)
})) }))
defer server.Close() defer server.Close()
defer releaseResolve()
f := buildFetcher() f := buildFetcher()
if err := f.Resolve(&base.Request{URL: server.URL + "/ignored-range.data"}, &base.Options{ if err := f.Resolve(&base.Request{URL: server.URL + "/ignored-range.data"}, &base.Options{
@@ -0,0 +1,119 @@
package http
import (
"bytes"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"os"
"testing"
"github.com/GopeedLab/gopeed/pkg/base"
fhttp "github.com/GopeedLab/gopeed/pkg/protocol/http"
)
func TestResolveResponseSize(t *testing.T) {
for _, tc := range []struct {
name string
status int
length, contentRange string
size int64
invalid bool
}{
{"full", 200, "801571015", "", 801571015, false},
{"partial_prefix", 206, "472321339", "bytes 0-472321338/801571015", 801571015, false},
{"partial_nonzero", 206, "100000", "bytes 100000-199999/801571015", 801571015, false},
{"second_report", 206, "335134409", "bytes 0-335134408/552140234", 552140234, false},
{"length_mismatch", 206, "10", "bytes 0-19/100", 0, true},
{"unknown_total", 206, "472321339", "bytes 0-472321338/*", 0, true},
{"missing_range", 206, "10", "", 0, true},
{"malformed_range", 206, "10", "bytes invalid", 0, true},
{"invalid_bounds", 206, "10", "bytes 20-10/100", 0, true},
{"invalid_total", 206, "10", "bytes 0-9/9", 0, true},
{"overflow", 206, "10", "bytes 0-9/9223372036854775808", 0, true},
{"chunked_partial", 206, "", "bytes 0-9/100", 100, false},
{"unknown_full", 200, "", "", 0, false},
} {
t.Run(tc.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if tc.length != "" {
w.Header().Set("Content-Length", tc.length)
}
if tc.contentRange != "" {
w.Header().Set("Content-Range", tc.contentRange)
}
w.WriteHeader(tc.status)
w.(http.Flusher).Flush()
}))
defer server.Close()
f := buildFetcher()
defer f.Close()
err := f.Resolve(&base.Request{URL: server.URL + "/file"}, &base.Options{Path: t.TempDir()})
if tc.invalid {
if !errors.Is(err, errInvalidRangeResponse) {
t.Fatalf("Resolve error = %v, want invalid range", err)
}
if f.meta.Res != nil || f.resolveResp != nil || f.getState() != stateError {
t.Fatal("invalid response was retained or published")
}
return
}
if err != nil {
t.Fatal(err)
}
if f.meta.Res.Size != tc.size || f.meta.Res.Files[0].Size != tc.size {
t.Fatalf("resource/file size = %d/%d, want %d", f.meta.Res.Size, f.meta.Res.Files[0].Size, tc.size)
}
if tc.status == 206 && !f.meta.Res.Range {
t.Fatal("partial response must enable Range downloads")
}
})
}
}
func TestDownloadPartialResolve(t *testing.T) {
payload := make([]byte, 256*1024)
for i := range payload {
payload[i] = byte(i*31 + i/251)
}
for _, start := range []int{0, 100000} {
for _, connections := range []int{1, 4} {
t.Run(fmt.Sprintf("start_%d_connections_%d", start, connections), func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
begin, end := start, start+9999
if h := r.Header.Get("Range"); h != "" {
if _, err := fmt.Sscanf(h, "bytes=%d-%d", &begin, &end); err != nil {
t.Errorf("invalid request range %q: %v", h, err)
w.WriteHeader(400)
return
}
}
w.Header().Set("Content-Length", fmt.Sprint(end-begin+1))
w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", begin, end, len(payload)))
w.WriteHeader(206)
w.Write(payload[begin : end+1])
}))
defer server.Close()
f := buildFetcher()
defer f.Close()
if err := f.Resolve(&base.Request{URL: server.URL + "/file"}, &base.Options{Path: t.TempDir(), Name: "file", Extra: &fhttp.OptsExtra{Connections: connections}}); err != nil {
t.Fatal(err)
}
if err := f.Start(); err != nil {
t.Fatal(err)
}
if err := f.Wait(); err != nil {
t.Fatal(err)
}
got, err := os.ReadFile(f.meta.SingleFilepath())
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("download content mismatch: got %d bytes, want %d", len(got), len(payload))
}
})
}
}
}
@@ -0,0 +1,79 @@
package http
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"sync/atomic"
"testing"
"time"
"github.com/GopeedLab/gopeed/internal/controller"
"github.com/GopeedLab/gopeed/internal/fetcher"
"github.com/GopeedLab/gopeed/internal/test"
"github.com/GopeedLab/gopeed/pkg/base"
fhttp "github.com/GopeedLab/gopeed/pkg/protocol/http"
)
func TestFetcher_RestoredStartSkipsAbsentPrefetch(t *testing.T) {
data := bytes.Repeat([]byte("0123456789abcdef"), 64*1024)
var firstRequest atomic.Int64
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
firstRequest.CompareAndSwap(0, time.Now().UnixNano())
http.ServeContent(w, r, "restored.bin", time.Time{}, bytes.NewReader(data))
}))
defer server.Close()
dir := t.TempDir()
size := int64(len(data))
half := size / 2
file := filepath.Join(dir, "restored.bin")
if err := os.WriteFile(file, data[:half], 0600); err != nil {
t.Fatal(err)
}
conn := &connection{ID: 0, Role: rolePrimary, State: connNotStarted, Chunk: newChunk(0, size-1), Downloaded: half}
conn.Chunk.Downloaded = half
rangeMode := true
saved := &fetcherData{Connections: []*connection{conn}, Range: &rangeMode, ResourceSize: &size, FileSize: &size}
meta := &fetcher.FetcherMeta{
Req: &base.Request{URL: server.URL + "/restored.bin"},
Res: &base.Resource{Size: size, Range: true, Files: []*base.FileInfo{{Name: "restored.bin", Size: size}}},
Opts: &base.Options{Path: dir, Name: "restored.bin", Extra: &fhttp.OptsExtra{Connections: 1}},
}
fm := new(FetcherManager)
_, restore := fm.Restore()
f := restore(meta, saved).(*Fetcher)
ctl := controller.NewController()
ctl.GetConfig = func(v any) {
if err := json.Unmarshal([]byte(test.ToJson(fm.DefaultConfig())), v); err != nil {
t.Error(err)
}
}
f.Setup(ctl)
defer f.Close()
started := time.Now()
if err := f.Start(); err != nil {
t.Fatal(err)
}
if err := f.Wait(); err != nil {
t.Fatal(err)
}
actual, err := os.ReadFile(file)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(actual, data) {
t.Fatal("restored download differs from the complete server content")
}
if firstRequest.Load() == 0 {
t.Fatal("restored download did not request the remaining content")
}
delay := time.Duration(firstRequest.Load() - started.UnixNano())
t.Logf("first restored request after %s", delay)
if delay > 2*time.Second {
t.Errorf("restored download waited %s before its first request", delay)
}
}
-18
View File
@@ -25,24 +25,6 @@ type Request struct {
SkipVerifyCert bool `json:"skipVerifyCert"` SkipVerifyCert bool `json:"skipVerifyCert"`
} }
// SetLabels replaces all labels on the request.
func (r *Request) SetLabels(labels map[string]string) {
r.Labels = labels
}
// PutLabel sets a label on the request.
func (r *Request) PutLabel(key, value string) {
if r.Labels == nil {
r.Labels = make(map[string]string)
}
r.Labels[key] = value
}
// DelLabel deletes a label from the request.
func (r *Request) DelLabel(key string) {
delete(r.Labels, key)
}
func (r *Request) Validate() error { func (r *Request) Validate() error {
if r.URL == "" { if r.URL == "" {
return fmt.Errorf("invalid request url") return fmt.Errorf("invalid request url")
+1
View File
@@ -1912,6 +1912,7 @@ func (d *Downloader) doPause(task *Task) (err error) {
} }
if handled { if handled {
d.emit(EventKeyPause, task) d.emit(EventKeyPause, task)
d.triggerWebhooks(WebhookEventDownloadPause, task, nil)
} }
return nil return nil
} }
+26 -6
View File
@@ -45,6 +45,32 @@ type generationTestManager struct {
func TestExtensionTaskMethodsMatchEventCapabilities(t *testing.T) { func TestExtensionTaskMethodsMatchEventCapabilities(t *testing.T) {
taskType := reflect.TypeOf((*Task)(nil)) taskType := reflect.TypeOf((*Task)(nil))
for _, typ := range []reflect.Type{taskType, reflect.TypeOf((*ExtensionTask)(nil)), reflect.TypeOf((*OnErrorExtensionTask)(nil))} {
for _, name := range []string{"SetUrl", "SetExtra", "PatchExtra", "PatchHeaders", "SetMethod", "SetBody", "SetHeaders", "PutHeader", "DelHeader", "SetTrackers"} {
if _, ok := typ.MethodByName(name); ok {
t.Fatalf("%v unexpectedly exposes %s", typ, name)
}
}
}
requestType := reflect.TypeOf((*ExtensionTaskRequest)(nil))
for _, name := range []string{"SetUrl", "SetLabels", "PutLabel", "DelLabel", "SetMethod", "SetBody", "SetHeaders", "PutHeader", "DelHeader", "SetTrackers"} {
if _, ok := requestType.MethodByName(name); !ok {
t.Fatalf("request wrapper does not expose %s", name)
}
}
for _, typ := range []reflect.Type{reflect.TypeOf(ExtensionTask{}), reflect.TypeOf(OnErrorExtensionTask{})} {
if _, ok := typ.FieldByName("Req"); ok {
t.Fatalf("%v exposes the removed task.req entry", typ)
}
field, ok := typ.FieldByName("Meta")
if !ok || field.Type != reflect.TypeOf((*ExtensionTaskMeta)(nil)) {
t.Fatalf("%v has no metadata wrapper", typ)
}
}
field, ok := reflect.TypeOf(ExtensionTaskMeta{}).FieldByName("Req")
if !ok || field.Type != requestType {
t.Fatal("meta.req has no request wrapper")
}
if _, ok := taskType.MethodByName("Continue"); ok { if _, ok := taskType.MethodByName("Continue"); ok {
t.Fatal("regular task unexpectedly exposes Continue") t.Fatal("regular task unexpectedly exposes Continue")
} }
@@ -53,17 +79,11 @@ func TestExtensionTaskMethodsMatchEventCapabilities(t *testing.T) {
} }
extensionTaskType := reflect.TypeOf((*ExtensionTask)(nil)) extensionTaskType := reflect.TypeOf((*ExtensionTask)(nil))
if _, ok := extensionTaskType.MethodByName("SetUrl"); !ok {
t.Fatal("extension task does not expose SetUrl")
}
if _, ok := extensionTaskType.MethodByName("Continue"); ok { if _, ok := extensionTaskType.MethodByName("Continue"); ok {
t.Fatal("non-error extension task unexpectedly exposes Continue") t.Fatal("non-error extension task unexpectedly exposes Continue")
} }
errorTaskType := reflect.TypeOf((*OnErrorExtensionTask)(nil)) errorTaskType := reflect.TypeOf((*OnErrorExtensionTask)(nil))
if _, ok := errorTaskType.MethodByName("SetUrl"); !ok {
t.Fatal("onError task does not expose SetUrl")
}
if _, ok := errorTaskType.MethodByName("Continue"); !ok { if _, ok := errorTaskType.MethodByName("Continue"); !ok {
t.Fatal("onError task does not expose Continue") t.Fatal("onError task does not expose Continue")
} }
@@ -0,0 +1,222 @@
package engine_test
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/GopeedLab/gopeed/pkg/download/engine"
)
// Exercise server-side header permissions, including formerly forbidden names,
// download authentication, metadata and non-safelisted no-cors values.
var extensionRequestHeaders = [][2]string{
{"Accept-Charset", "utf-8"}, {"Accept-Encoding", "identity"},
{"Access-Control-Request-Headers", "authorization, x-download"},
{"Access-Control-Request-Method", "POST"}, {"Connection", "close"},
{"Connection", "keep-alive"},
{"Content-Length", "4"}, {"Cookie", "sid=manual"}, {"Cookie2", "legacy=1"},
{"Date", "Fri, 09 Oct 2026 00:00:00 GMT"}, {"DNT", "1"},
{"Host", "download.example:1234"}, {"Origin", "https://download.example"},
{"Permissions-Policy", "camera=()"}, {"Referer", "https://download.example/file"},
{"TE", "trailers"}, {"Trailer", "X-Checksum"}, {"Set-Cookie", "sid=request"},
{"Set-Cookie2", "legacy=request"}, {"Via", "1.1 proxy"},
{"Proxy-Authorization", "Basic dGVzdA=="}, {"Proxy-", "custom"},
{"Sec-Fetch-Site", "same-origin"}, {"Sec-Fetch-Mode", "custom"},
{"Sec-CH-UA", `"Download client"`}, {"Sec-", "custom"},
{"X-HTTP-Method", "TRACE"}, {"X-HTTP-Method-Override", "CONNECT"},
{"X-Method-Override", "TRACK"}, {"Authorization", "Bearer token"},
{"User-Agent", "Gopeed extension"}, {"Content-Type", "application/json"},
{"Range", "bytes=0-1,3-4"}, {"X-Download-Token", "custom"},
{"Accept-Language", strings.Repeat("x", 129)},
}
func TestExtensionRequestsSendNodeHeaders(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, request *http.Request) {
body, err := io.ReadAll(request.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
name := request.URL.Query().Get("header")
value := request.Header.Get(name)
if name == "Host" {
value = request.Host
}
_ = json.NewEncoder(w).Encode([]string{value, string(body)})
}))
t.Cleanup(server.Close)
clients := []struct{ name, script string }{
{"fetch record", `const response = await fetch(url, {method: "POST", body: "test", headers: {[name]: value}}); return (await response.text()).trim();`},
{"fetch no-cors Headers", `const response = await fetch(url, {method: "POST", body: "test", mode: "no-cors", headers: new Headers([[name.toLowerCase(), value]])}); return (await response.text()).trim();`},
{"fetch Request clone", `const request = new Request(url, {method: "POST", body: "test", mode: "same-origin", headers: [[name.toUpperCase(), value]]}); const response = await fetch(request.clone()); return (await response.text()).trim();`},
}
for _, credentials := range []bool{false, true} {
clients = append(clients, struct{ name, script string }{
fmt.Sprintf("XHR credentials=%t", credentials),
fmt.Sprintf(`return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open("POST", url);
xhr.withCredentials = %t;
xhr.setRequestHeader(name.toLowerCase(), value);
xhr.onload = () => resolve(xhr.responseText.trim());
xhr.onerror = () => reject(new Error("XHR failed"));
xhr.send("test");
});`, credentials),
})
}
for _, header := range extensionRequestHeaders {
for _, client := range clients {
t.Run(header[0]+"/"+client.name, func(t *testing.T) {
runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close)
value, err := runtime.RunString(fmt.Sprintf(`(async () => {
const name = %q, value = %q;
const url = %q + "?header=" + encodeURIComponent(name);
%s
})()`, header[0], header[1], server.URL, client.script))
if err != nil {
t.Fatal(err)
}
want, _ := json.Marshal([]string{header[1], "test"})
if value != string(want) {
t.Fatalf("server received %#v, want %s", value, want)
}
})
}
}
}
func TestExtensionHeaderMutationsMatchNode(t *testing.T) {
headers := append([][2]string(nil), extensionRequestHeaders...)
// Node stores these in Headers/Request, but rejects them during fetch.
headers = append(headers, [2]string{"Expect", "100-continue"}, [2]string{"Keep-Alive", "timeout=5"}, [2]string{"Transfer-Encoding", "chunked"}, [2]string{"Upgrade", "websocket"})
data, _ := json.Marshal(headers)
runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close)
_, err := runtime.RunString(fmt.Sprintf(`(() => {
for (const mode of ["cors", "same-origin", "no-cors"]) {
for (const [name, value] of %s) {
const request = new Request("http://example.com", {mode, headers: {[name]: value}});
const headers = request.headers;
if (headers.get(name) !== value) throw new Error("Filtered init " + name + " in " + mode);
headers.delete(name.toUpperCase());
if (headers.has(name)) throw new Error("Cannot delete " + name);
headers.append(name, value);
if (headers.get(name) !== value) throw new Error("Cannot append " + name);
headers.set(name, value);
if (request.clone().headers.get(name) !== value) throw new Error("Cannot clone " + name);
}
}
return true;
})()`, data))
if err != nil {
t.Fatal(err)
}
}
func TestExtensionRequestsIgnoreUnsupportedTransportHeaders(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, request *http.Request) {
body, err := io.ReadAll(request.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
value := request.Header.Get(request.URL.Query().Get("header"))
_ = json.NewEncoder(w).Encode([]string{value, string(body)})
}))
t.Cleanup(server.Close)
for _, header := range [][2]string{
{"Expect", "100-continue"}, {"Keep-Alive", "timeout=5"},
{"Transfer-Encoding", "chunked"}, {"Upgrade", "websocket"},
{"Connection", "upgrade"}, {"Content-Length", "3"},
{"Content-Length", "5"}, {"Content-Length", "invalid"},
{"Content-Length", "-1"}, {"Content-Length", ""},
} {
for _, xhr := range []bool{false, true} {
t.Run(fmt.Sprintf("%s=%s/XHR=%t", header[0], header[1], xhr), func(t *testing.T) {
runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close)
value, err := runtime.RunString(fmt.Sprintf(`(async () => {
const name = %q, value = %q;
const url = %q + "?header=" + encodeURIComponent(name);
if (%t) return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open("POST", url);
xhr.setRequestHeader(name, value);
xhr.onload = () => resolve(xhr.responseText.trim());
xhr.onerror = () => reject(new Error("XHR failed"));
xhr.send("test");
});
const response = await fetch(url, {method: "POST", body: "test", headers: {[name]: value}});
return (await response.text()).trim();
})()`, header[0], header[1], server.URL, xhr))
if err != nil {
t.Fatal(err)
}
wireValue := ""
if header[0] == "Content-Length" {
wireValue = "4"
}
want, _ := json.Marshal([]string{wireValue, "test"})
if value != string(want) {
t.Fatalf("server received %#v, want %s", value, want)
}
})
}
}
}
func TestExtensionResponseExposesSetCookie(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Add("Set-Cookie", "sid=one; HttpOnly")
w.Header().Add("Set-Cookie", "token=two")
w.Header().Set("Set-Cookie2", "legacy=three")
_, _ = w.Write([]byte("ok"))
}))
t.Cleanup(server.Close)
runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close)
value, err := runtime.RunString(fmt.Sprintf(`(async () => {
const response = await fetch(%q);
const clone = response.clone();
await response.text(); await clone.text();
let immutable = false;
try { response.headers.set("Set-Cookie", "changed=1"); } catch (error) { immutable = error instanceof TypeError; }
const synthetic = new Response(null, {headers: {"Set-Cookie": "initial=1"}});
synthetic.headers.append("Set-Cookie", "appended=2");
return JSON.stringify({cookies: clone.headers.getSetCookie(), legacy: response.headers.get("set-cookie2"), immutable, synthetic: synthetic.headers.getSetCookie()});
})()`, server.URL))
if err != nil {
t.Fatal(err)
}
const want = `{"cookies":["sid=one; HttpOnly","token=two"],"legacy":"legacy=three","immutable":true,"synthetic":["initial=1","appended=2"]}`
if value != want {
t.Fatalf("unexpected response headers: got %#v want %s", value, want)
}
}
func TestExtensionHeadersRejectInvalidSyntax(t *testing.T) {
runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close)
_, err := runtime.RunString(`(() => {
for (const [name, value] of [["Bad Header", "ok"], ["", "ok"], ["X:Header", "ok"],
["X-Test", "a\nb"], ["X-Test", "a\rb"], ["X-Test", "\0"], ["X-Test", "\u0100"]]) {
for (const create of [() => new Headers([[name, value]]),
() => new Request("http://example.com", {mode: "no-cors", headers: {[name]: value}}),
() => { const xhr = new XMLHttpRequest(); xhr.open("GET", "http://example.com"); xhr.setRequestHeader(name, value); }]) {
let rejected = false;
try { create(); } catch (error) { rejected = error instanceof TypeError; }
if (!rejected) throw new Error("Accepted invalid HTTP header syntax");
}
}
return true;
})()`)
if err != nil {
t.Fatal(err)
}
}
@@ -105,6 +105,105 @@ func TestXMLHttpRequestUsesFetch(t *testing.T) {
} }
} }
func TestExtensionRequestsSendExplicitCookie(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, request *http.Request) {
_, _ = w.Write([]byte(request.Header.Get("Cookie")))
}))
t.Cleanup(server.Close)
type requestTest struct {
name string
script string
}
tests := []requestTest{
{"fetch record", `const response = await fetch(url, {headers: {Cookie: cookie}}); return response.text();`},
{"fetch Headers", `const response = await fetch(url, {headers: new Headers({cookie})}); return response.text();`},
{"fetch pairs", `const response = await fetch(url, {headers: [["cOoKiE", cookie]]}); return response.text();`},
{"fetch repeated Cookie", `const headers = new Headers([["Cookie", "sid=manual"], ["cookie", "token=provided"]]); if (headers.get("cookie") !== cookie) throw new Error("Incorrect Cookie separator"); const response = await fetch(url, {headers}); return response.text();`},
{"fetch Request clone", `const request = new Request(url, {headers: {Cookie: cookie}}); const response = await fetch(request.clone()); return response.text();`},
{"fetch Request set", `const request = new Request(url); request.headers.set("Cookie", cookie); const response = await fetch(request); return response.text();`},
{"fetch Request append", `const request = new Request(url); request.headers.append("Cookie", cookie); const response = await fetch(request); return response.text();`},
{"fetch no-cors mutations", `const request = new Request(url, {mode: "no-cors"}); request.headers.set("Cookie", "removed=1"); request.headers.delete("COOKIE"); if (request.headers.has("cookie")) throw new Error("Cookie was not deleted"); request.headers.append("Cookie", cookie); const response = await fetch(request.clone()); return response.text();`},
}
for _, mode := range []string{"cors", "same-origin", "no-cors"} {
for _, credentials := range []string{"omit", "same-origin", "include"} {
tests = append(tests, requestTest{
name: "fetch " + mode + " " + credentials,
script: fmt.Sprintf(`const response = await fetch(url, {headers: {Cookie: cookie}, mode: %q, credentials: %q}); return response.text();`, mode, credentials),
})
}
}
for _, withCredentials := range []bool{false, true} {
tests = append(tests, requestTest{
name: fmt.Sprintf("XHR withCredentials=%t", withCredentials),
script: fmt.Sprintf(`return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open("GET", url);
xhr.withCredentials = %t;
xhr.setRequestHeader("cOoKiE", cookie);
xhr.onload = () => resolve(xhr.responseText);
xhr.onerror = () => reject(new Error("XHR failed"));
xhr.send();
});`, withCredentials),
})
}
tests = append(tests, requestTest{"XHR repeated Cookie", `return new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open("GET", url);
xhr.setRequestHeader("Cookie", "sid=manual");
xhr.setRequestHeader("cookie", "token=provided");
xhr.onload = () => resolve(xhr.responseText);
xhr.onerror = () => reject(new Error("XHR failed"));
xhr.send();
});`})
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close)
const cookie = "sid=manual; token=provided"
value, err := runtime.RunString(fmt.Sprintf(`(async () => {
const url = %q;
const cookie = %q;
%s
})()`, server.URL, cookie, test.script))
if err != nil {
t.Fatal(err)
}
if value != cookie {
t.Fatalf("server received Cookie %q, want %q", value, cookie)
}
})
}
}
func TestFetchExplicitCookieWithCookieJar(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, request *http.Request) {
if request.URL.Path == "/seed" {
http.SetCookie(w, &http.Cookie{Name: "stored", Value: "jar", Path: "/"})
}
_, _ = w.Write([]byte(request.Header.Get("Cookie")))
}))
t.Cleanup(server.Close)
runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close)
value, err := runtime.RunString(fmt.Sprintf(`(async () => {
const url = %q;
await (await fetch(url + "/seed")).text();
const automatic = await (await fetch(url)).text();
const combined = await (await fetch(url, {headers: {Cookie: "manual=provided"}})).text();
const omitted = await (await fetch(url, {credentials: "omit", headers: {Cookie: "manual=provided"}})).text();
const empty = await (await fetch(url, {credentials: "omit"})).text();
return JSON.stringify({automatic, combined, omitted, empty});
})()`, server.URL))
if err != nil {
t.Fatal(err)
}
const want = `{"automatic":"stored=jar","combined":"manual=provided; stored=jar","omitted":"manual=provided","empty":""}`
if value != want {
t.Fatalf("unexpected CookieJar result: got %#v want %s", value, want)
}
}
func TestFetchResponseAndRequestCloneHaveIndependentBodies(t *testing.T) { func TestFetchResponseAndRequestCloneHaveIndependentBodies(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(response http.ResponseWriter, request *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(response http.ResponseWriter, request *http.Request) {
if request.Method == http.MethodPost { if request.Method == http.MethodPost {
+11 -56
View File
@@ -122,7 +122,7 @@
if (name === "set-cookie") { if (name === "set-cookie") {
for (const value of values) result.push([name, value]); for (const value of values) result.push([name, value]);
} else { } else {
result.push([name, values.join(", ")]); result.push([name, values.join(name === "cookie" ? "; " : ", ")]);
} }
} }
return result; return result;
@@ -158,49 +158,10 @@
return iterator; return iterator;
} }
function forbiddenRequestHeader(name, value) { function assertMutableHeaders(headers) {
if (name.startsWith("proxy-") || name.startsWith("sec-")) return true; // Match Node/Undici: HTTP syntax and immutability still apply, but browser
if ([ // forbidden-header and no-cors safelist rules do not apply to extensions.
"accept-charset", "accept-encoding", "access-control-request-headers",
"access-control-request-method", "connection", "content-length", "cookie",
"cookie2", "date", "dnt", "expect", "host", "keep-alive", "origin",
"permissions-policy", "referer", "te", "trailer", "transfer-encoding",
"set-cookie", "upgrade", "via",
].includes(name)) return true;
if (["x-http-method", "x-http-method-override", "x-method-override"].includes(name)) {
return String(value).split(",").some((method) => {
method = method.trim().toUpperCase();
return method === "CONNECT" || method === "TRACE" || method === "TRACK";
});
}
return false;
}
function noCORSSafelistedRequestHeader(name, value) {
if (value.length > 128) return false;
const hasUnsafeByte = /[\x00-\x08\x0A-\x1F\x7F"():<>?@\[\\\]{}]/.test(value);
if (name === "accept") return !hasUnsafeByte;
if (name === "accept-language" || name === "content-language") {
return /^[0-9A-Za-z *,\-.;=]*$/.test(value);
}
if (name === "content-type") {
if (hasUnsafeByte) return false;
const essence = value.split(";", 1)[0].trim().toLowerCase();
return essence === "application/x-www-form-urlencoded" ||
essence === "multipart/form-data" || essence === "text/plain";
}
if (name === "range") return /^bytes=[0-9]+-[0-9]*$/.test(value);
return false;
}
function blockedByGuard(headers, name, value) {
if (headers._guard === "immutable") throw new TypeError("Headers are immutable"); if (headers._guard === "immutable") throw new TypeError("Headers are immutable");
if (headers._guard === "request") return forbiddenRequestHeader(name, value);
if (headers._guard === "request-no-cors") {
return forbiddenRequestHeader(name, value) || !noCORSSafelistedRequestHeader(name, value);
}
if (headers._guard === "response") return name === "set-cookie" || name === "set-cookie2";
return false;
} }
class Headers { class Headers {
@@ -235,27 +196,21 @@
name = normalizeHeaderName(name); name = normalizeHeaderName(name);
value = normalizeHeaderValue(value); value = normalizeHeaderValue(value);
const values = this._values.get(name); const values = this._values.get(name);
const combinedValue = values && name !== "set-cookie" ? values.join(", ") + ", " + value : value; assertMutableHeaders(this);
if (blockedByGuard(this, name, combinedValue)) return;
if (values) values.push(value); if (values) values.push(value);
else this._values.set(name, [value]); else this._values.set(name, [value]);
} }
delete(name) { delete(name) {
name = normalizeHeaderName(name); name = normalizeHeaderName(name);
if (this._guard === "immutable") throw new TypeError("Headers are immutable"); assertMutableHeaders(this);
if (this._guard === "request-no-cors") {
if (forbiddenRequestHeader(name, "") || !["accept", "accept-language", "content-language", "content-type", "range"].includes(name)) return;
this._values.delete(name);
return;
}
if (blockedByGuard(this, name, "")) return;
this._values.delete(name); this._values.delete(name);
} }
get(name) { get(name) {
const values = this._values.get(normalizeHeaderName(name)); name = normalizeHeaderName(name);
return values ? values.join(", ") : null; const values = this._values.get(name);
return values ? values.join(name === "cookie" ? "; " : ", ") : null;
} }
getSetCookie() { getSetCookie() {
@@ -270,7 +225,7 @@
set(name, value) { set(name, value) {
name = normalizeHeaderName(name); name = normalizeHeaderName(name);
value = normalizeHeaderValue(value); value = normalizeHeaderValue(value);
if (blockedByGuard(this, name, value)) return; assertMutableHeaders(this);
this._values.set(name, [value]); this._values.set(name, [value]);
} }
@@ -484,7 +439,7 @@
} }
this._duplex = "half"; this._duplex = "half";
const headerInit = init.headers !== undefined ? init.headers : (source ? source.headers : undefined); const headerInit = init.headers !== undefined ? init.headers : (source ? source.headers : undefined);
this._headers = new Headers(headerInit, this._mode === "no-cors" ? "request-no-cors" : "request"); this._headers = new Headers(headerInit, "request");
let body = init.body !== undefined ? init.body : (source ? source._bodyInit : null); let body = init.body !== undefined ? init.body : (source ? source._bodyInit : null);
if (source && !bodyWasOverridden && source.body) body = transferReadableStream(source.body); if (source && !bodyWasOverridden && source.body) body = transferReadableStream(source.body);
const bodyIsStream = typeof ReadableStream !== "undefined" && body instanceof ReadableStream; const bodyIsStream = typeof ReadableStream !== "undefined" && body instanceof ReadableStream;
@@ -302,6 +302,23 @@ func (r *registry) Open(fingerprint string, proxyHandler func(*http.Request) (*u
return nil, err return nil, err
} }
for _, header := range metadata.Headers { for _, header := range metadata.Headers {
// Ignore transport controls unsupported by Node/Undici rather than
// applying a browser forbidden-header list or failing the whole request.
switch strings.ToLower(header[0]) {
case "expect", "keep-alive", "transfer-encoding", "upgrade", "content-length":
// Content-Length is generated from the buffered request body below.
continue
case "connection":
if value := strings.ToLower(header[1]); value != "close" && value != "keep-alive" {
continue
}
case "trailer":
// Fetch supplies a declaration, not a Go Request.Trailer payload.
// Keep its lower-case wire name so Go's writer does not omit it as
// the canonical Trailer field it normally generates itself.
httpRequest.Header["trailer"] = append(httpRequest.Header["trailer"], header[1])
continue
}
httpRequest.Header.Add(header[0], header[1]) httpRequest.Header.Add(header[0], header[1])
} }
if host := httpRequest.Header.Get("Host"); host != "" { if host := httpRequest.Header.Get("Host"); host != "" {
+26 -2
View File
@@ -9,7 +9,7 @@ The files under `fetch/` are copied from the official
Gopeed executes applicable worker-style Fetch API tests at the extension Gopeed executes applicable worker-style Fetch API tests at the extension
JavaScript boundary. The checked-in runner currently executes unchanged WPT JavaScript boundary. The checked-in runner currently executes unchanged WPT
tests for Headers, no-CORS header guards, Request construction and body tests for Headers syntax and iteration, Request construction and body
consumption, ReadableStream bodies, keepalive, Request errors, Response consumption, ReadableStream bodies, keepalive, Request errors, Response
construction and body consumption, immutable network response headers, and construction and body consumption, immutable network response headers, and
the Response static constructors. the Response static constructors.
@@ -19,10 +19,34 @@ a claim that Gopeed is a browser. Browser-only suites that require a document,
CORS origin enforcement or preflight, CSP, Mixed Content, Service Workers, CORS origin enforcement or preflight, CSP, Mixed Content, Service Workers,
browser HTTP cache, navigation, browser authentication UI, or WPT's browser HTTP cache, navigation, browser authentication UI, or WPT's
multi-origin server infrastructure are intentionally excluded. Gopeed also multi-origin server infrastructure are intentionally excluded. Gopeed also
keeps one extension-specific behavior: `redirect: "manual"` exposes the HTTP keeps extension-specific behavior: `redirect: "manual"` exposes the HTTP
redirect response so download extensions can inspect `Location`, instead of redirect response so download extensions can inspect `Location`, instead of
returning a browser `opaqueredirect` response. returning a browser `opaqueredirect` response.
Header permissions follow [Node/Undici's server-side Fetch API](https://github.com/nodejs/undici/blob/main/README.md#forbidden-and-safelisted-header-names). Gopeed does not
apply browser forbidden request/response header lists or no-CORS safelists.
Extensions may supply authentication, referrer, origin, host, proxy and fetch
metadata headers, and read `Set-Cookie` response headers. Network response
headers remain immutable; invalid header syntax still raises an error.
Repeated `Cookie` values are combined with `; `, as in Node/Undici.
The runner excludes the browser-only `headers-no-cors.any.js` and
`headers-forbidden-override.any.js` suites, the filtering assertions within
`request-headers.any.js`, and the forbidden-response-cookie assertion within
`header-setcookie.any.js`. Gopeed's `extension_headers_test.go` replaces those
restrictions with integration tests that check headers actually received by
an HTTP server through fetch and XMLHttpRequest, plus header mutations,
cloning and response cookies. The upstream test files remain unchanged.
Gopeed deliberately ignores transport controls that Node/Undici rejects:
`Expect`, `Keep-Alive`, `Transfer-Encoding`, `Upgrade`, and `Connection` values
other than `close` or `keep-alive`. `Content-Length` is generated from the
actual request body, ignoring a supplied length. Explicit `Host` and
`Sec-Fetch-Mode` values are preserved rather than replaced by defaults.
These transport behaviors are also covered by integration tests.
`credentials: "omit"` disables automatic CookieJar cookies but preserves
explicit headers; Gopeed retains its extension CookieJar support.
`wpt_harness.js` is a small Goja adapter maintained by Gopeed; it is not copied `wpt_harness.js` is a small Goja adapter maintained by Gopeed; it is not copied
from WPT. Test logic below `fetch/` is unchanged; the checked-in copy may only from WPT. Test logic below `fetch/` is unchanged; the checked-in copy may only
normalize a final newline. normalize a final newline.
+25 -17
View File
@@ -20,9 +20,6 @@ var wptFetchUtils string
//go:embed testdata/wpt/fetch/api/request/request-error.js //go:embed testdata/wpt/fetch/api/request/request-error.js
var wptRequestError string var wptRequestError string
//go:embed testdata/wpt/fetch/api/cors/resources/not-cors-safelisted.json
var wptNoCORSSafelistedHeaders []byte
//go:embed testdata/wpt/fetch/api/headers/*.any.js //go:embed testdata/wpt/fetch/api/headers/*.any.js
var wptHeaders embed.FS var wptHeaders embed.FS
@@ -30,31 +27,19 @@ var wptHeaders embed.FS
var wptRequestResponse embed.FS var wptRequestResponse embed.FS
func TestWPTHeaders(t *testing.T) { func TestWPTHeaders(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(response http.ResponseWriter, _ *http.Request) {
response.Header().Set("Content-Type", "application/json")
_, _ = response.Write(wptNoCORSSafelistedHeaders)
}))
t.Cleanup(server.Close)
tests := []string{ tests := []string{
"header-setcookie.any.js", "header-setcookie.any.js",
"headers-basic.any.js", "headers-basic.any.js",
"headers-casing.any.js", "headers-casing.any.js",
"headers-combine.any.js", "headers-combine.any.js",
"headers-errors.any.js", "headers-errors.any.js",
"headers-forbidden-override.any.js",
"headers-normalize.any.js", "headers-normalize.any.js",
"headers-no-cors.any.js",
"headers-record.any.js", "headers-record.any.js",
"headers-structure.any.js", "headers-structure.any.js",
} }
for _, name := range tests { for _, name := range tests {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
setup := "" runWPTFile(t, wptHeaders, path.Join("testdata/wpt/fetch/api/headers", name), "")
if name == "headers-no-cors.any.js" {
setup = fmt.Sprintf("globalThis.location = new URL(%q);", server.URL+"/fetch/api/headers/test.any.js")
}
runWPTFile(t, wptHeaders, path.Join("testdata/wpt/fetch/api/headers", name), setup)
}) })
} }
} }
@@ -110,7 +95,30 @@ func runWPTFile(t *testing.T, files embed.FS, name, setup string) {
} }
runtime := engine.NewEngine(nil) runtime := engine.NewEngine(nil)
t.Cleanup(runtime.Close) t.Cleanup(runtime.Close)
value, err := runtime.RunString(setup + "\n" + wptHarness + "\n" + wptFetchUtils + "\n" + wptRequestError + "\n" + string(source) + "\n__wptFinish();") adaptation := ""
if name == "testdata/wpt/fetch/api/request/request-headers.any.js" {
// Node-style header permissions are covered by extension integration tests.
// Keep upstream tests unchanged and exclude browser-only restrictions.
adaptation = `
const upstreamTest = globalThis.test;
globalThis.test = function (callback, name) {
if (name.startsWith('Adding invalid request header ') ||
name.startsWith('Adding invalid no-cors request header ') ||
name.startsWith('Check that request constructor is filtering headers ') ||
name.startsWith('Check that no-cors request constructor is filtering headers ')) return;
upstreamTest(callback, name);
};
`
} else if name == "testdata/wpt/fetch/api/headers/header-setcookie.any.js" {
adaptation = `
const upstreamTest = globalThis.test;
globalThis.test = function (callback, name) {
if (name === 'Set-Cookie is a forbidden response header') return;
upstreamTest(callback, name);
};
`
}
value, err := runtime.RunString(setup + "\n" + wptHarness + "\n" + wptFetchUtils + "\n" + wptRequestError + "\n" + adaptation + "\n" + string(source) + "\n__wptFinish();")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+17 -6
View File
@@ -11,6 +11,7 @@ import (
"strings" "strings"
"time" "time"
"github.com/GopeedLab/gopeed/internal/fetcher"
"github.com/GopeedLab/gopeed/internal/logger" "github.com/GopeedLab/gopeed/internal/logger"
"github.com/GopeedLab/gopeed/pkg/base" "github.com/GopeedLab/gopeed/pkg/base"
"github.com/GopeedLab/gopeed/pkg/download/engine" "github.com/GopeedLab/gopeed/pkg/download/engine"
@@ -733,6 +734,14 @@ type OnDoneContext struct {
// only some fields can be modified, such as request info. // only some fields can be modified, such as request info.
type ExtensionTask struct { type ExtensionTask struct {
*Task *Task
Meta *ExtensionTaskMeta `json:"meta"`
}
// ExtensionTaskMeta preserves metadata fields while exposing request operations
// at the existing meta.req path.
type ExtensionTaskMeta struct {
*fetcher.FetcherMeta
Req *ExtensionTaskRequest `json:"req"`
} }
// OnErrorExtensionTask adds error-recovery controls to ExtensionTask. // OnErrorExtensionTask adds error-recovery controls to ExtensionTask.
@@ -750,7 +759,14 @@ func cloneExtensionTask(task *Task) *Task {
} }
func newExtensionTask(task *Task) *ExtensionTask { func newExtensionTask(task *Task) *ExtensionTask {
return &ExtensionTask{Task: cloneExtensionTask(task)} view := cloneExtensionTask(task)
return &ExtensionTask{
Task: view,
Meta: &ExtensionTaskMeta{
FetcherMeta: view.Meta,
Req: &ExtensionTaskRequest{Request: view.Meta.Req, task: view},
},
}
} }
func newOnStartExtensionTask(task *Task) *ExtensionTask { func newOnStartExtensionTask(task *Task) *ExtensionTask {
@@ -768,11 +784,6 @@ func newOnDoneExtensionTask(task *Task) *Task {
return cloneExtensionTask(task) return cloneExtensionTask(task)
} }
// SetUrl replaces the task request URL.
func (t *ExtensionTask) SetUrl(url string) {
t.Meta.Req.URL = url
}
func (t *OnErrorExtensionTask) Continue() error { func (t *OnErrorExtensionTask) Continue() error {
return t.download.Continue(&TaskFilter{ return t.download.Continue(&TaskFilter{
IDs: []string{t.ID}, IDs: []string{t.ID},
@@ -0,0 +1,188 @@
package download
import (
"fmt"
"reflect"
"testing"
"github.com/GopeedLab/gopeed/internal/fetcher"
"github.com/GopeedLab/gopeed/pkg/base"
btProtocol "github.com/GopeedLab/gopeed/pkg/protocol/bt"
httpProtocol "github.com/GopeedLab/gopeed/pkg/protocol/http"
"github.com/dop251/goja"
)
func TestExtensionProtocolMethods(t *testing.T) {
for _, extra := range []any{nil, map[string]any{"header": map[string]any{"Keep": "yes"}}, &httpProtocol.ReqExtra{Header: map[string]string{"Keep": "yes"}}} {
vm := goja.New()
vm.SetFieldNameMapper(goja.TagFieldNameMapper("json", true))
req := &base.Request{Extra: extra}
task := &Task{Protocol: "http", Meta: &fetcher.FetcherMeta{Req: req, Opts: &base.Options{}}}
if err := vm.Set("task", newOnStartExtensionTask(task)); err != nil {
t.Fatal(err)
}
_, err := vm.RunString(`task.meta.req.setMethod('POST'); task.meta.req.setBody('payload'); task.meta.req.putHeader('Authorization','Bearer token'); task.meta.req.putHeader('Remove','remove'); task.meta.req.delHeader('REMOVE');`)
if err != nil {
t.Fatal(err)
}
if err := base.ParseReqExtra[httpProtocol.ReqExtra](req); err != nil {
t.Fatal(err)
}
got := req.Extra.(*httpProtocol.ReqExtra)
if got.Method != "POST" || got.Body != "payload" || got.Header["Authorization"] != "Bearer token" {
t.Fatalf("mutation not applied: %#v", got)
}
if _, ok := got.Header["Remove"]; ok {
t.Fatal("header deletion failed")
}
if extra != nil && got.Header["Keep"] != "yes" {
t.Fatal("existing header lost")
}
}
vm := goja.New()
vm.SetFieldNameMapper(goja.TagFieldNameMapper("json", true))
req := &base.Request{Extra: &btProtocol.ReqExtra{Trackers: []string{"old"}}}
task := &Task{Protocol: "bt", Meta: &fetcher.FetcherMeta{Req: req, Opts: &base.Options{}}}
if err := vm.Set("task", newOnStartExtensionTask(task)); err != nil {
t.Fatal(err)
}
if _, err := vm.RunString(`task.meta.req.setTrackers(['new','another']);`); err != nil {
t.Fatal(err)
}
got := req.Extra.(*btProtocol.ReqExtra).Trackers
if len(got) != 2 || got[0] != "new" || got[1] != "another" {
t.Fatalf("trackers = %#v", got)
}
}
func TestExtensionProtocolMethodsRejectInvalidMutation(t *testing.T) {
vm := goja.New()
vm.SetFieldNameMapper(goja.TagFieldNameMapper("json", true))
original := &httpProtocol.ReqExtra{Method: "POST", Header: map[string]string{"authorization": "old", "Keep": "yes"}}
req := &base.Request{Extra: original}
task := newOnStartExtensionTask(&Task{Protocol: "http", Meta: &fetcher.FetcherMeta{Req: req, Opts: &base.Options{}}})
if err := vm.Set("task", task); err != nil {
t.Fatal(err)
}
for _, script := range []string{`task.meta.req.setHeaders({Keep:null})`, `task.meta.req.setHeaders({Keep:123})`, `task.meta.req.setHeaders({a:'1',A:'2'})`} {
if _, err := vm.RunString(script); err == nil {
t.Fatalf("accepted invalid mutation: %s", script)
}
if req.Extra != original || original.Header["Keep"] != "yes" {
t.Fatal("invalid operation changed original extra")
}
}
if _, err := vm.RunString(`task.meta.req.putHeader('Authorization','new');`); err != nil {
t.Fatal(err)
}
got := req.Extra.(*httpProtocol.ReqExtra)
if len(got.Header) != 2 || got.Header["Authorization"] != "new" || original.Header["authorization"] != "old" {
t.Fatalf("header update = %#v", got.Header)
}
if _, err := vm.RunString(`task.meta.req.setHeaders({Only:'one'});`); err != nil {
t.Fatal(err)
}
got = req.Extra.(*httpProtocol.ReqExtra)
if got.Method != "POST" || len(got.Header) != 1 {
t.Fatalf("setHeaders = %#v", got)
}
req.Extra = map[string]any{"header": 123}
if _, err := vm.RunString(`task.meta.req.setBody('new')`); err == nil {
t.Fatal("invalid protocol extra accepted")
}
if req.Extra.(map[string]any)["header"] != 123 {
t.Fatal("conversion failure mutated extra")
}
}
func TestExtensionProtocolMethodsOtherProtocolsAreNoOp(t *testing.T) {
for _, protocol := range []string{"http", "bt", "hls"} {
for _, onError := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/onError=%v", protocol, onError), func(t *testing.T) {
vm := goja.New()
vm.SetFieldNameMapper(goja.TagFieldNameMapper("json", true))
// An invalid extra proves that unrelated methods skip conversion entirely.
original := map[string]any{"unchanged": true, "header": 123}
req := &base.Request{Extra: original}
task := &Task{Protocol: protocol, Meta: &fetcher.FetcherMeta{Req: req, Opts: &base.Options{}}}
var wrapper any = newOnStartExtensionTask(task)
if onError {
wrapper = newOnErrorExtensionTask(nil, task)
}
if err := vm.Set("task", wrapper); err != nil {
t.Fatal(err)
}
if protocol != "http" {
if _, err := vm.RunString(`task.meta.req.setMethod('POST');task.meta.req.setBody('body');task.meta.req.setHeaders({a:null});task.meta.req.putHeader('A','value');task.meta.req.delHeader('A');`); err != nil {
t.Fatal(err)
}
}
if protocol != "bt" {
if _, err := vm.RunString(`task.meta.req.setTrackers(['udp://example.com']);`); err != nil {
t.Fatal(err)
}
}
if !reflect.DeepEqual(req.Extra, original) {
t.Fatalf("extra changed: %#v", req.Extra)
}
original["same-object"] = true
if req.Extra.(map[string]any)["same-object"] != true {
t.Fatal("no-op replaced extra")
}
})
}
}
}
func TestExtensionOnErrorRequestWritesToOriginalTask(t *testing.T) {
vm := goja.New()
vm.SetFieldNameMapper(goja.TagFieldNameMapper("json", true))
req := &base.Request{URL: "https://example.com", Extra: &httpProtocol.ReqExtra{Header: map[string]string{"Keep": "yes"}}}
task := &Task{Protocol: "http", Meta: &fetcher.FetcherMeta{Req: req, Opts: &base.Options{}}}
if err := vm.Set("task", newOnErrorExtensionTask(nil, task)); err != nil {
t.Fatal(err)
}
if _, err := vm.RunString(`
if (typeof task.continue !== 'function') throw new Error('missing continue');
if (typeof task.putHeader !== 'undefined') throw new Error('task still exposes protocol methods');
if (typeof task.req !== 'undefined') throw new Error('legacy task.req');
const request = task.meta.req;
if (request.url !== 'https://example.com' || request.extra.header.Keep !== 'yes') throw new Error('request data missing');
if (!task.meta.opts) throw new Error('metadata fields missing');
request.setHeaders({Keep:'new'});
request.putHeader('Authorization','Bearer new');
request.setMethod('POST');
if (typeof task.setUrl !== 'undefined' || typeof task.extra !== 'undefined') throw new Error('legacy task API');
request.setUrl('https://example.com/new');
request.setLabels({keep:'yes',remove:'yes'});
request.putLabel('new','value');
request.delLabel('remove');
if (request.url !== 'https://example.com/new' || request.labels.new !== 'value' || request.extra.method !== 'POST' || request.extra.header.Authorization !== 'Bearer new') throw new Error('request data did not refresh');
`); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(req.Labels, map[string]string{"keep": "yes", "new": "value"}) {
t.Fatalf("labels = %#v", req.Labels)
}
got := req.Extra.(*httpProtocol.ReqExtra)
if got.Method != "POST" || got.Header["Keep"] != "new" || got.Header["Authorization"] != "Bearer new" || req.URL != "https://example.com/new" {
t.Fatalf("original request not updated: %#v, %#v", req, got)
}
}
func TestExtensionBTRequestInitializesNilExtra(t *testing.T) {
vm := goja.New()
vm.SetFieldNameMapper(goja.TagFieldNameMapper("json", true))
req := &base.Request{}
task := &Task{Protocol: "bt", Meta: &fetcher.FetcherMeta{Req: req, Opts: &base.Options{}}}
if err := vm.Set("task", newOnStartExtensionTask(task)); err != nil {
t.Fatal(err)
}
if _, err := vm.RunString(`task.meta.req.setTrackers(['udp://example.com']);`); err != nil {
t.Fatal(err)
}
got, ok := req.Extra.(*btProtocol.ReqExtra)
if !ok || !reflect.DeepEqual(got.Trackers, []string{"udp://example.com"}) {
t.Fatalf("extra = %#v", req.Extra)
}
}
+104
View File
@@ -0,0 +1,104 @@
package download
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/GopeedLab/gopeed/pkg/base"
httpProtocol "github.com/GopeedLab/gopeed/pkg/protocol/http"
)
func TestExtensionExtraMutationHTTP(t *testing.T) {
for _, resolved := range []bool{false, true} {
for _, invalid := range []bool{false, true} {
t.Run(fmt.Sprintf("resolved=%v/invalid=%v", resolved, invalid), func(t *testing.T) {
var updatedRequests atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("X-New") == "new" {
updatedRequests.Add(1)
if r.Header.Get("X-Old") != "" {
t.Error("extra mutation retained the old header")
}
}
http.ServeContent(w, r, "test.txt", time.Time{}, strings.NewReader("extra mutation payload"))
}))
defer server.Close()
d := NewDownloader(&DownloaderConfig{Storage: NewMemStorage(), StorageDir: t.TempDir()})
if err := d.Setup(); err != nil {
t.Fatal(err)
}
defer d.Clear()
extDir := t.TempDir()
manifest := `{"name":"http-methods","title":"HTTP methods test","version":"0.0.1","scripts":[{"event":"onStart","match":{"labels":["test"]},"entry":"index.js"}]}`
mutation := `await ctx.task.meta.req.setMethod('GET');
await ctx.task.meta.req.setBody('');
await ctx.task.meta.req.delHeader('x-old');
await ctx.task.meta.req.putHeader('X-New', 'new');`
if invalid {
mutation = `try { await ctx.task.meta.req.setHeaders({'X-New':123}); }
catch (err) { await ctx.task.meta.req.putLabel('rejected','true'); } ` + mutation
}
script := `gopeed.events.onStart(async ctx => { ` + mutation + ` });`
for name, content := range map[string]string{"manifest.json": manifest, "index.js": script} {
if err := os.WriteFile(filepath.Join(extDir, name), []byte(content), 0600); err != nil {
t.Fatal(err)
}
}
if _, err := d.InstallExtensionByFolder(extDir, false); err != nil {
t.Fatal(err)
}
done := make(chan error, 1)
d.Listener(func(event *Event) {
if event.Key == EventKeyFinally {
done <- event.Err
}
})
req := &base.Request{URL: server.URL + "/test.txt", Labels: map[string]string{"test": "true"}, Extra: &httpProtocol.ReqExtra{Header: map[string]string{"X-Old": "old"}}}
opts := &base.Options{Path: t.TempDir()}
var id string
var err error
if resolved {
rr, resolveErr := d.Resolve(req, opts)
if resolveErr != nil {
t.Fatal(resolveErr)
}
id, err = d.Create(rr.ID)
} else {
id, err = d.CreateDirect(req, opts)
}
if err != nil {
t.Fatal(err)
}
select {
case err = <-done:
case <-time.After(10 * time.Second):
t.Fatal("download timed out")
}
if invalid && d.GetTask(id).Meta.Req.Labels["rejected"] != "true" {
t.Fatal("invalid header was not rejected")
}
if err != nil {
t.Fatal(err)
}
got, ok := d.GetTask(id).Meta.Req.Extra.(*httpProtocol.ReqExtra)
if !ok || got.Header["X-New"] != "new" || len(got.Header) != 1 {
t.Fatalf("unexpected protocol extra: %#v", d.GetTask(id).Meta.Req.Extra)
}
if !resolved && updatedRequests.Load() == 0 {
t.Fatal("download did not use the extension headers")
}
data, err := os.ReadFile(filepath.Join(opts.Path, "test.txt"))
if err != nil || string(data) != "extra mutation payload" {
t.Fatalf("download content = %q, error = %v", data, err)
}
})
}
}
}
+140
View File
@@ -0,0 +1,140 @@
package download
import (
"encoding/json"
"fmt"
"net/http"
"strings"
"github.com/GopeedLab/gopeed/pkg/base"
btProtocol "github.com/GopeedLab/gopeed/pkg/protocol/bt"
httpProtocol "github.com/GopeedLab/gopeed/pkg/protocol/http"
"github.com/GopeedLab/gopeed/pkg/util"
"github.com/dop251/goja"
)
// ExtensionTaskRequest exposes request operations without exposing the backing task.
// It always reads the current request so a retained handle sees subsequent updates.
type ExtensionTaskRequest struct {
*base.Request
task *Task
}
// SetUrl replaces the task request URL.
func (r *ExtensionTaskRequest) SetUrl(url string) { r.task.Meta.Req.URL = url }
// SetLabels replaces all labels on the request.
func (r *ExtensionTaskRequest) SetLabels(labels map[string]string) {
r.Labels = labels
}
// PutLabel sets a label on the request.
func (r *ExtensionTaskRequest) PutLabel(key, value string) {
if r.Labels == nil {
r.Labels = make(map[string]string)
}
r.Labels[key] = value
}
// DelLabel deletes a label from the request.
func (r *ExtensionTaskRequest) DelLabel(key string) {
delete(r.Labels, key)
}
// mutateTaskExtra converts into an independent protocol value before mutation.
// Other protocols are a no-op. Failed conversion or mutation must not change the live request.
func mutateTaskExtra[E any](task *Task, protocol string, mutate func(*E) error) error {
if task.Protocol != protocol {
return nil
}
if task.Meta == nil || task.Meta.Req == nil {
return fmt.Errorf("task request is missing")
}
var extra E
if err := util.MapToStruct(task.Meta.Req.Extra, &extra); err != nil {
return err
}
if err := mutate(&extra); err != nil {
return err
}
task.Meta.Req.Extra = &extra
return nil
}
func (t *ExtensionTaskRequest) SetMethod(method string) error {
return mutateTaskExtra(t.task, "http", func(extra *httpProtocol.ReqExtra) error { extra.Method = method; return nil })
}
func (t *ExtensionTaskRequest) SetBody(body string) error {
return mutateTaskExtra(t.task, "http", func(extra *httpProtocol.ReqExtra) error { extra.Body = body; return nil })
}
// decodeHeaders omits undefined properties and rejects non-string values.
func decodeHeaders(value *goja.Object) (map[string]string, error) {
if value == nil {
return nil, fmt.Errorf("headers must be an object")
}
data, err := value.MarshalJSON()
if err != nil {
return nil, err
}
var input map[string]*string
if err := json.Unmarshal(data, &input); err != nil {
return nil, err
}
if input == nil {
return nil, fmt.Errorf("headers must be an object")
}
headers := make(map[string]string, len(input))
seen := make(map[string]bool, len(input))
for name, value := range input {
if value == nil {
return nil, fmt.Errorf("header %q must be a string", name)
}
lower := strings.ToLower(name)
if seen[lower] {
return nil, fmt.Errorf("duplicate header name %q", name)
}
seen[lower] = true
headers[http.CanonicalHeaderKey(name)] = *value
}
return headers, nil
}
func (t *ExtensionTaskRequest) SetHeaders(headers *goja.Object) error {
return mutateTaskExtra(t.task, "http", func(extra *httpProtocol.ReqExtra) error {
values, err := decodeHeaders(headers)
if err != nil {
return err
}
extra.Header = values
return nil
})
}
func (t *ExtensionTaskRequest) PutHeader(name, value string) error {
return mutateTaskExtra(t.task, "http", func(extra *httpProtocol.ReqExtra) error {
if extra.Header == nil {
extra.Header = make(map[string]string)
}
deleteHeader(extra.Header, name)
extra.Header[http.CanonicalHeaderKey(name)] = value
return nil
})
}
func (t *ExtensionTaskRequest) DelHeader(name string) error {
return mutateTaskExtra(t.task, "http", func(extra *httpProtocol.ReqExtra) error { deleteHeader(extra.Header, name); return nil })
}
func deleteHeader(headers map[string]string, name string) {
for key := range headers {
if strings.EqualFold(key, name) {
delete(headers, key)
}
}
}
func (t *ExtensionTaskRequest) SetTrackers(trackers []string) error {
return mutateTaskExtra(t.task, "bt", func(extra *btProtocol.ReqExtra) error { extra.Trackers = append([]string{}, trackers...); return nil })
}
+36 -3
View File
@@ -14,10 +14,12 @@ import (
"testing" "testing"
"time" "time"
"github.com/GopeedLab/gopeed/internal/fetcher"
"github.com/GopeedLab/gopeed/internal/logger" "github.com/GopeedLab/gopeed/internal/logger"
"github.com/GopeedLab/gopeed/pkg/base" "github.com/GopeedLab/gopeed/pkg/base"
gojaerror "github.com/GopeedLab/gopeed/pkg/download/engine/inject/error" gojaerror "github.com/GopeedLab/gopeed/pkg/download/engine/inject/error"
enginewebview "github.com/GopeedLab/gopeed/pkg/download/engine/webview" enginewebview "github.com/GopeedLab/gopeed/pkg/download/engine/webview"
httpProtocol "github.com/GopeedLab/gopeed/pkg/protocol/http"
"github.com/dop251/goja" "github.com/dop251/goja"
) )
@@ -38,13 +40,13 @@ func TestDownloader_InstallExtensionByFolder(t *testing.T) {
t.Fatal("resolve error") t.Fatal("resolve error")
} }
if req.Labels["replaced"] != "true" || req.Labels["modified"] != "true" { if req.Labels["replaced"] != "true" || req.Labels["modified"] != "true" {
t.Fatalf("request label methods did not update labels: %#v", req.Labels) t.Fatalf("declarative labels did not update: %#v", req.Labels)
} }
if _, ok := req.Labels["original"]; ok { if _, ok := req.Labels["original"]; ok {
t.Fatalf("setLabels did not replace existing labels: %#v", req.Labels) t.Fatalf("label assignment did not replace existing labels: %#v", req.Labels)
} }
if _, ok := req.Labels["removed"]; ok { if _, ok := req.Labels["removed"]; ok {
t.Fatalf("delLabel did not remove label: %#v", req.Labels) t.Fatalf("declarative label deletion failed: %#v", req.Labels)
} }
}) })
} }
@@ -1188,6 +1190,37 @@ func TestDownloader_UpgradeExtension(t *testing.T) {
}) })
} }
func TestDownloader_Extension_OnStartExtraMutation(t *testing.T) {
d := NewDownloader(&DownloaderConfig{Storage: NewMemStorage(), StorageDir: t.TempDir()})
if err := d.Setup(); err != nil {
t.Fatal(err)
}
defer d.Clear()
if _, err := d.InstallExtensionByFolder("./testdata/extensions/on_start", false); err != nil {
t.Fatal(err)
}
for _, extra := range []any{nil, &httpProtocol.ReqExtra{
Method: "POST", Body: "old body", Header: map[string]string{"Old": "value"},
}} {
req := &base.Request{URL: "https://github.com/test", Extra: extra}
task := &Task{Protocol: "http", Meta: &fetcher.FetcherMeta{Req: req, Opts: &base.Options{}}}
d.triggerOnStart(task)
if err := base.ParseReqExtra[httpProtocol.ReqExtra](req); err != nil {
t.Fatal(err)
}
got, ok := req.Extra.(*httpProtocol.ReqExtra)
if !ok || got.Header["X-Gopeed-Test"] != "on-start" {
t.Fatalf("setHeaders did not update the original request: %#v", req.Extra)
}
if len(got.Header) != 1 {
t.Fatalf("headers not replaced: %#v", got)
}
if original, ok := extra.(*httpProtocol.ReqExtra); ok && (got.Method != original.Method || got.Body != original.Body) {
t.Fatalf("setHeaders changed unrelated fields: %#v", got)
}
}
}
func TestDownloader_Extension_OnStart(t *testing.T) { func TestDownloader_Extension_OnStart(t *testing.T) {
downloadAndCheck := func(req *base.Request) { downloadAndCheck := func(req *base.Request) {
setupDownloader(func(downloader *Downloader) { setupDownloader(func(downloader *Downloader) {
+13
View File
@@ -23,6 +23,9 @@ type ResolveResult struct {
Res *base.Resource `json:"res"` Res *base.Resource `json:"res"`
} }
// Task represents a download unit.
// Options flow path:
// REST Request (model.CreateTask/ResolveTask) -> pkg/download Downloader -> Task.Meta.Opts (base.Options) -> Fetcher.Resolve/Start (e.g. internal/protocol/http/fetcher.go)
type Task struct { type Task struct {
ID string `json:"id"` ID string `json:"id"`
Protocol string `json:"protocol"` Protocol string `json:"protocol"`
@@ -32,6 +35,10 @@ type Task struct {
Progress *Progress `json:"progress"` Progress *Progress `json:"progress"`
CreatedAt time.Time `json:"createdAt"` CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"` UpdatedAt time.Time `json:"updatedAt"`
// DoneAt is the time when the task completed; nil if it has never
// finished. A done task never leaves the done state, so the value is
// written once and kept.
DoneAt *time.Time `json:"doneAt"`
producer production.Source producer production.Source
producerReceived int64 producerReceived int64
@@ -134,6 +141,12 @@ func (t *Task) MarshalJSON() ([]byte, error) {
func (t *Task) updateStatus(status base.Status) { func (t *Task) updateStatus(status base.Status) {
t.UpdatedAt = time.Now() t.UpdatedAt = time.Now()
t.Status = status t.Status = status
// A done task never leaves the done state, so there is no branch to clear
// DoneAt on other transitions.
if status == base.DownloadStatusDone {
doneAt := time.Now()
t.DoneAt = &doneAt
}
} }
func (t *Task) clone() *Task { func (t *Task) clone() *Task {
+6 -3
View File
@@ -1,7 +1,10 @@
gopeed.events.onResolve(async function (ctx) { gopeed.events.onResolve(async function (ctx) {
await ctx.req.setLabels({ replaced: "true", removed: "true" }); for (const method of ['setUrl', 'setLabels', 'putLabel', 'delLabel', 'setMethod', 'putHeader', 'setTrackers']) {
await ctx.req.putLabel("modified", "true"); if (typeof ctx.req[method] !== 'undefined') throw new Error(`onResolve exposes ${method}`);
await ctx.req.delLabel("removed"); }
ctx.req.labels = { replaced: "true", removed: "true" };
ctx.req.labels.modified = "true";
delete ctx.req.labels.removed;
ctx.res = { ctx.res = {
name: "test", name: "test",
files: Array(2).fill(true).map((_, i) => ({ files: Array(2).fill(true).map((_, i) => ({
@@ -132,7 +132,7 @@ gopeed.events.onError(async function (ctx) {
return; return;
} }
await ctx.task.setUrl(await createPayloadURL(source, false)); await ctx.task.meta.req.setUrl(await createPayloadURL(source, false));
await req.putLabel('started', 'true'); await req.putLabel('started', 'true');
await req.putLabel('rebuilt', 'true'); await req.putLabel('rebuilt', 'true');
await ctx.task.continue(); await ctx.task.continue();
+1 -1
View File
@@ -41,6 +41,6 @@ gopeed.events.onError(async function (ctx) {
} }
await req.putLabel("recovered", "true"); await req.putLabel("recovered", "true");
await ctx.task.setUrl(await gopeed.runtime.blob.createObjectURL(new Blob(["ok\n"], { type: "text/plain" }))); await ctx.task.meta.req.setUrl(await gopeed.runtime.blob.createObjectURL(new Blob(["ok\n"], { type: "text/plain" })));
await ctx.task.continue(); await ctx.task.continue();
}); });
+1 -1
View File
@@ -46,7 +46,7 @@ gopeed.events.onError(async function (ctx) {
} }
try { try {
await ctx.task.setUrl(await createRangeBlobUrl(req.rawUrl)); await ctx.task.meta.req.setUrl(await createRangeBlobUrl(req.rawUrl));
await req.putLabel('started', 'true'); await req.putLabel('started', 'true');
await req.putLabel('rebuilt', 'true'); await req.putLabel('rebuilt', 'true');
await ctx.task.continue(); await ctx.task.continue();
+1 -1
View File
@@ -1,6 +1,6 @@
gopeed.events.onError(async function (ctx) { gopeed.events.onError(async function (ctx) {
gopeed.logger.info("url", ctx.task.meta.req.url); gopeed.logger.info("url", ctx.task.meta.req.url);
gopeed.logger.info("error", ctx.error); gopeed.logger.info("error", ctx.error);
await ctx.task.setUrl("https://github.com"); await ctx.task.meta.req.setUrl("https://github.com");
await ctx.task.continue(); await ctx.task.continue();
}); });
+2 -1
View File
@@ -1,5 +1,6 @@
gopeed.events.onStart(async function (ctx) { gopeed.events.onStart(async function (ctx) {
gopeed.logger.info("url", ctx.task.meta.req.url); gopeed.logger.info("url", ctx.task.meta.req.url);
await ctx.task.setUrl("https://github.com"); await ctx.task.meta.req.setUrl("https://github.com");
await ctx.task.meta.req.setHeaders({ "X-Gopeed-Test": "on-start" });
await ctx.task.meta.req.putLabel('modified', 'true'); await ctx.task.meta.req.putLabel('modified', 'true');
}); });
+1
View File
@@ -21,6 +21,7 @@ type WebhookEvent string
const ( const (
WebhookEventDownloadDone WebhookEvent = "DOWNLOAD_DONE" WebhookEventDownloadDone WebhookEvent = "DOWNLOAD_DONE"
WebhookEventDownloadError WebhookEvent = "DOWNLOAD_ERROR" WebhookEventDownloadError WebhookEvent = "DOWNLOAD_ERROR"
WebhookEventDownloadPause WebhookEvent = "DOWNLOAD_PAUSE"
) )
// WebhookData is the data sent to webhook URLs // WebhookData is the data sent to webhook URLs
+43
View File
@@ -814,6 +814,49 @@ func TestWebhook_TestWebhookUrl_VerifyTestPayload(t *testing.T) {
}) })
} }
func TestWebhook_TriggerOnPause(t *testing.T) {
receivedData := make(chan *WebhookData, 1)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var data WebhookData
if err := json.NewDecoder(r.Body).Decode(&data); err != nil {
t.Errorf("Failed to decode data: %v", err)
return
}
receivedData <- &data
w.WriteHeader(http.StatusOK)
}))
defer server.Close()
setupWebhookTest(t, func(downloader *Downloader) {
cfg, _ := downloader.GetConfig()
cfg.Webhook = &base.WebhookConfig{
Enable: true,
URLs: []string{server.URL},
}
downloader.PutConfig(cfg)
task := NewTask()
task.Protocol = "http"
task.Meta = &mockFetcherMeta
downloader.triggerWebhooks(WebhookEventDownloadPause, task, nil)
select {
case data := <-receivedData:
if data.Event != WebhookEventDownloadPause {
t.Errorf("Expected event 'DOWNLOAD_PAUSE', got '%s'", data.Event)
}
if data.Payload == nil || data.Payload.Task == nil {
t.Error("Expected payload.task to be present")
} else if data.Payload.Task.ID != task.ID {
t.Errorf("Expected task ID '%s', got '%s'", task.ID, data.Payload.Task.ID)
}
case <-time.After(2 * time.Second):
t.Error("Timeout waiting for webhook")
}
})
}
func setupWebhookTest(t *testing.T, fn func(downloader *Downloader)) { func setupWebhookTest(t *testing.T, fn func(downloader *Downloader)) {
defaultDownloader.Setup() defaultDownloader.Setup()
defaultDownloader.cfg.StorageDir = ".test_storage" defaultDownloader.cfg.StorageDir = ".test_storage"
+61 -1
View File
@@ -12,6 +12,7 @@ import (
"github.com/GopeedLab/gopeed/pkg/base" "github.com/GopeedLab/gopeed/pkg/base"
"github.com/GopeedLab/gopeed/pkg/download" "github.com/GopeedLab/gopeed/pkg/download"
phttp "github.com/GopeedLab/gopeed/pkg/protocol/http"
"github.com/google/jsonschema-go/jsonschema" "github.com/google/jsonschema-go/jsonschema"
"github.com/modelcontextprotocol/go-sdk/mcp" "github.com/modelcontextprotocol/go-sdk/mcp"
) )
@@ -71,6 +72,7 @@ type httpTaskExtra struct {
AutoExtract *bool `json:"autoExtract,omitempty" jsonschema:"Automatically extract the downloaded archive"` AutoExtract *bool `json:"autoExtract,omitempty" jsonschema:"Automatically extract the downloaded archive"`
ArchivePassword string `json:"archivePassword,omitempty" jsonschema:"Password for an encrypted archive"` ArchivePassword string `json:"archivePassword,omitempty" jsonschema:"Password for an encrypted archive"`
DeleteAfterExtract bool `json:"deleteAfterExtract,omitempty" jsonschema:"Delete archive files after successful extraction"` DeleteAfterExtract bool `json:"deleteAfterExtract,omitempty" jsonschema:"Delete archive files after successful extraction"`
Checksum *checksumOption `json:"checksum,omitempty" jsonschema:"Optional checksum verification settings"`
} }
// downloadRequest is the MCP-facing form of base.Request. The REST model uses // downloadRequest is the MCP-facing form of base.Request. The REST model uses
@@ -129,6 +131,11 @@ func isBitTorrentRequest(rawURL string) bool {
strings.HasPrefix(upperURL, "DATA:APPLICATION/X-BITTORRENT;BASE64,") strings.HasPrefix(upperURL, "DATA:APPLICATION/X-BITTORRENT;BASE64,")
} }
type checksumOption struct {
Algorithm string `json:"algorithm" jsonschema:"Checksum algorithm: md5, sha1, or sha256"`
Expected string `json:"expected" jsonschema:"Expected hex-encoded checksum hash"`
}
// downloadOptions is the MCP-facing form of base.Options. Only HTTP currently // downloadOptions is the MCP-facing form of base.Options. Only HTTP currently
// defines protocol-specific task options, so Extra has one concrete schema. // defines protocol-specific task options, so Extra has one concrete schema.
type downloadOptions struct { type downloadOptions struct {
@@ -186,12 +193,31 @@ func (o *downloadOptions) baseOptions() *base.Options {
if o == nil { if o == nil {
return nil return nil
} }
var extra any
if o.Extra != nil {
var checksum *phttp.ChecksumOption
if o.Extra.Checksum != nil {
checksum = &phttp.ChecksumOption{
Algorithm: o.Extra.Checksum.Algorithm,
Expected: o.Extra.Checksum.Expected,
}
}
extra = &phttp.OptsExtra{
Connections: o.Extra.Connections,
AutoTorrent: o.Extra.AutoTorrent,
DeleteTorrentAfterDownload: o.Extra.DeleteTorrentAfterDownload,
AutoExtract: o.Extra.AutoExtract,
ArchivePassword: o.Extra.ArchivePassword,
DeleteAfterExtract: o.Extra.DeleteAfterExtract,
Checksum: checksum,
}
}
return &base.Options{ return &base.Options{
Name: o.Name, Name: o.Name,
Path: o.Path, Path: o.Path,
AsDefaultPath: o.AsDefaultPath, AsDefaultPath: o.AsDefaultPath,
SelectFiles: o.SelectFiles, SelectFiles: o.SelectFiles,
Extra: o.Extra, Extra: extra,
} }
} }
@@ -270,6 +296,13 @@ type getTaskStatsOutput struct {
Stats any `json:"stats"` Stats any `json:"stats"`
} }
type getDownloadSpeedInput struct{}
type getDownloadSpeedOutput struct {
TotalSpeed int64 `json:"totalSpeed"`
ActiveTaskCount int `json:"activeTaskCount"`
}
type taskActionOutput struct { type taskActionOutput struct {
ID string `json:"id"` ID string `json:"id"`
Success bool `json:"success"` Success bool `json:"success"`
@@ -416,6 +449,33 @@ func registerTools(server *mcp.Server, downloader *download.Downloader) {
return nil, &getTaskStatsOutput{Stats: stats}, nil return nil, &getTaskStatsOutput{Stats: stats}, nil
}) })
mcp.AddTool(server, &mcp.Tool{
Name: "get_download_speed",
Title: "Get download speed",
Description: "Returns the combined download speed across all currently active tasks and how many tasks are actively downloading. Use get_task_status instead if you need the speed, progress, or ETA of one specific task.",
Annotations: &mcp.ToolAnnotations{ReadOnlyHint: true, OpenWorldHint: &closedWorld},
}, func(_ context.Context, _ *mcp.CallToolRequest, _ *getDownloadSpeedInput) (*mcp.CallToolResult, *getDownloadSpeedOutput, error) {
allTasks := downloader.GetTasks()
var totalSpeed int64
activeCount := 0
for _, task := range allTasks {
status, err := downloader.RuntimeStatus(task.ID)
if err != nil {
continue
}
if status.Status == base.DownloadStatusRunning {
activeCount++
totalSpeed += status.Speed
}
}
return nil, &getDownloadSpeedOutput{
TotalSpeed: totalSpeed,
ActiveTaskCount: activeCount,
}, nil
})
registerTaskAction(server, downloader, "pause_task", "Pause task", "Pause one Gopeed task.", &nondestructive, &closedWorld, downloader.Pause) registerTaskAction(server, downloader, "pause_task", "Pause task", "Pause one Gopeed task.", &nondestructive, &closedWorld, downloader.Pause)
registerTaskAction(server, downloader, "continue_task", "Continue task", "Continue a paused or failed Gopeed task.", &nondestructive, &openWorld, downloader.Continue) registerTaskAction(server, downloader, "continue_task", "Continue task", "Continue a paused or failed Gopeed task.", &nondestructive, &openWorld, downloader.Continue)
+1
View File
@@ -64,6 +64,7 @@ func TestStreamableHTTPTools(t *testing.T) {
"continue_task", "continue_task",
"create_task", "create_task",
"delete_task", "delete_task",
"get_download_speed",
"get_task", "get_task",
"get_task_stats", "get_task_stats",
"get_task_status", "get_task_status",
+103
View File
@@ -0,0 +1,103 @@
package mcpserver
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/GopeedLab/gopeed/pkg/base"
"github.com/GopeedLab/gopeed/pkg/download"
"github.com/modelcontextprotocol/go-sdk/mcp"
)
func TestGetDownloadSpeedTool(t *testing.T) {
// Serve a test payload that takes a little bit of time or streams bytes
testServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Length", "1048576")
w.WriteHeader(http.StatusOK)
chunk := make([]byte, 8192)
for i := 0; i < 128; i++ {
_, _ = w.Write(chunk)
time.Sleep(10 * time.Millisecond)
}
}))
defer testServer.Close()
downloader := download.NewDownloader(&download.DownloaderConfig{
Storage: download.NewMemStorage(),
StorageDir: t.TempDir(),
})
if err := downloader.Setup(); err != nil {
t.Fatal(err)
}
defer downloader.Close()
httpServer := httptest.NewServer(NewHandler(downloader))
defer httpServer.Close()
client := mcp.NewClient(&mcp.Implementation{Name: "gopeed-test", Version: "1.0.0"}, nil)
session, err := client.Connect(context.Background(), &mcp.StreamableClientTransport{Endpoint: httpServer.URL}, nil)
if err != nil {
t.Fatal(err)
}
defer session.Close()
// 1. Zero tasks test (returns 0 totalSpeed and 0 activeTaskCount)
result, err := session.CallTool(context.Background(), &mcp.CallToolParams{
Name: "get_download_speed",
Arguments: map[string]any{},
})
if err != nil {
t.Fatal(err)
}
if result.IsError {
t.Fatalf("get_download_speed returned error: %+v", result.Content)
}
structured, ok := result.StructuredContent.(map[string]any)
if !ok {
t.Fatalf("structured content type = %T", result.StructuredContent)
}
if activeCount, ok := structured["activeTaskCount"].(float64); !ok || activeCount != 0 {
t.Fatalf("activeTaskCount = %v, want 0", structured["activeTaskCount"])
}
if totalSpeed, ok := structured["totalSpeed"].(float64); !ok || totalSpeed != 0 {
t.Fatalf("totalSpeed = %v, want 0", structured["totalSpeed"])
}
// 2. Start a real download task
taskID, err := downloader.CreateDirect(&base.Request{
URL: testServer.URL,
}, &base.Options{
Path: t.TempDir(),
})
if err != nil {
t.Fatal(err)
}
defer func() {
_ = downloader.Delete(&download.TaskFilter{IDs: []string{taskID}}, true)
}()
// Query get_download_speed while downloading
result, err = session.CallTool(context.Background(), &mcp.CallToolParams{
Name: "get_download_speed",
Arguments: map[string]any{},
})
if err != nil {
t.Fatal(err)
}
if result.IsError {
t.Fatalf("get_download_speed returned error: %+v", result.Content)
}
structured, ok = result.StructuredContent.(map[string]any)
if !ok {
t.Fatalf("structured content type = %T", result.StructuredContent)
}
if _, ok := structured["activeTaskCount"].(float64); !ok {
t.Fatalf("activeTaskCount missing or wrong type: %v", structured["activeTaskCount"])
}
if _, ok := structured["totalSpeed"].(float64); !ok {
t.Fatalf("totalSpeed missing or wrong type: %v", structured["totalSpeed"])
}
}
+74
View File
@@ -1,11 +1,83 @@
package http package http
import (
"encoding/hex"
"fmt"
"strings"
)
type ReqExtra struct { type ReqExtra struct {
Method string `json:"method"` Method string `json:"method"`
Header map[string]string `json:"header"` Header map[string]string `json:"header"`
Body string `json:"body"` Body string `json:"body"`
} }
// ChecksumOption specifies an expected checksum to verify against
// the completed download. Only supported for HTTP downloads.
type ChecksumOption struct {
Algorithm string `json:"algorithm"` // "md5" | "sha1" | "sha256"
Expected string `json:"expected"` // hex-encoded expected hash
}
// IsEmpty returns true when chk is nil, or when both Algorithm and Expected
// are blank after trimming whitespace.
func (chk *ChecksumOption) IsEmpty() bool {
if chk == nil {
return true
}
return strings.TrimSpace(chk.Algorithm) == "" && strings.TrimSpace(chk.Expected) == ""
}
// NormalizedAlgorithm returns the canonical lower-cased algorithm ("md5", "sha1", "sha256")
// and true if recognized, or ("", false) if unsupported.
func (chk *ChecksumOption) NormalizedAlgorithm() (string, bool) {
if chk == nil {
return "", false
}
switch strings.ToLower(strings.TrimSpace(chk.Algorithm)) {
case "md5":
return "md5", true
case "sha1", "sha-1":
return "sha1", true
case "sha256", "sha-256":
return "sha256", true
default:
return "", false
}
}
// Validate checks whether the checksum option is valid.
// Empty options return nil (no verification).
func (chk *ChecksumOption) Validate() error {
if chk.IsEmpty() {
return nil
}
algo, ok := chk.NormalizedAlgorithm()
if !ok {
return fmt.Errorf("unsupported checksum algorithm: %s", chk.Algorithm)
}
expected := strings.TrimSpace(chk.Expected)
if expected == "" {
return fmt.Errorf("checksum expected hash is required when algorithm is set")
}
var expectedLen int
switch algo {
case "md5":
expectedLen = 32
case "sha1":
expectedLen = 40
case "sha256":
expectedLen = 64
}
if len(expected) != expectedLen {
return fmt.Errorf("invalid checksum hash length for %s: expected %d characters, got %d", algo, expectedLen, len(expected))
}
if _, err := hex.DecodeString(expected); err != nil {
return fmt.Errorf("invalid checksum hash for %s: expected valid hexadecimal string", algo)
}
return nil
}
type OptsExtra struct { type OptsExtra struct {
Connections int `json:"connections"` Connections int `json:"connections"`
// AutoTorrent when task download complete, and it is a .torrent file, it will be auto create a new task for the torrent file // AutoTorrent when task download complete, and it is a .torrent file, it will be auto create a new task for the torrent file
@@ -21,6 +93,8 @@ type OptsExtra struct {
ArchivePassword string `json:"archivePassword"` ArchivePassword string `json:"archivePassword"`
// DeleteAfterExtract when true, deletes the archive file after successful extraction // DeleteAfterExtract when true, deletes the archive file after successful extraction
DeleteAfterExtract bool `json:"deleteAfterExtract"` DeleteAfterExtract bool `json:"deleteAfterExtract"`
// Checksum verifies the completed file against an expected hash after download
Checksum *ChecksumOption `json:"checksum,omitempty"`
} }
// Stats for download // Stats for download
+162
View File
@@ -0,0 +1,162 @@
package http
import (
"strings"
"testing"
)
func TestOptsExtra_Checksum(t *testing.T) {
extra := &OptsExtra{
Checksum: &ChecksumOption{
Algorithm: "sha256",
Expected: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
},
}
if extra.Checksum.Algorithm != "sha256" || extra.Checksum.Expected != "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" {
t.Fatalf("unexpected checksum: %+v", extra.Checksum)
}
}
func TestChecksumOption_Validate(t *testing.T) {
validMd5 := "0123456789abcdef0123456789abcdef"
validSha1 := "0123456789abcdef0123456789abcdef01234567"
validSha256 := "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
tests := []struct {
name string
option *ChecksumOption
wantErr bool
errSubstr string
}{
{
name: "nil option",
option: nil,
wantErr: false,
},
{
name: "empty option (empty strings)",
option: &ChecksumOption{Algorithm: "", Expected: ""},
wantErr: false,
},
{
name: "empty option (whitespace strings)",
option: &ChecksumOption{Algorithm: " ", Expected: " "},
wantErr: false,
},
{
name: "valid md5 lowercase",
option: &ChecksumOption{Algorithm: "md5", Expected: validMd5},
wantErr: false,
},
{
name: "valid md5 uppercase",
option: &ChecksumOption{Algorithm: "MD5", Expected: strings.ToUpper(validMd5)},
wantErr: false,
},
{
name: "valid sha1 lowercase",
option: &ChecksumOption{Algorithm: "sha1", Expected: validSha1},
wantErr: false,
},
{
name: "valid sha-1 alias",
option: &ChecksumOption{Algorithm: "SHA-1", Expected: strings.ToUpper(validSha1)},
wantErr: false,
},
{
name: "valid sha256 lowercase",
option: &ChecksumOption{Algorithm: "sha256", Expected: validSha256},
wantErr: false,
},
{
name: "valid sha-256 alias",
option: &ChecksumOption{Algorithm: "sha-256", Expected: validSha256},
wantErr: false,
},
{
name: "valid sha256 with whitespace around hash and algorithm",
option: &ChecksumOption{Algorithm: " sha256 ", Expected: " " + validSha256 + " "},
wantErr: false,
},
{
name: "unknown algorithm",
option: &ChecksumOption{Algorithm: "crc32", Expected: "12345678"},
wantErr: true,
errSubstr: "unsupported checksum algorithm",
},
{
name: "algorithm set but hash blank",
option: &ChecksumOption{Algorithm: "sha256", Expected: " "},
wantErr: true,
errSubstr: "checksum expected hash is required",
},
{
name: "sha256 with abc (short length)",
option: &ChecksumOption{Algorithm: "sha256", Expected: "abc"},
wantErr: true,
errSubstr: "expected 64 characters, got 3",
},
{
name: "sha256 with 64 x g (non-hex)",
option: &ChecksumOption{Algorithm: "sha256", Expected: strings.Repeat("g", 64)},
wantErr: true,
errSubstr: "expected valid hexadecimal string",
},
{
name: "sha256 with 63 x a (off-by-one short)",
option: &ChecksumOption{Algorithm: "sha256", Expected: strings.Repeat("a", 63)},
wantErr: true,
errSubstr: "expected 64 characters, got 63",
},
{
name: "sha256 with 65 x a (off-by-one long)",
option: &ChecksumOption{Algorithm: "sha256", Expected: strings.Repeat("a", 65)},
wantErr: true,
errSubstr: "expected 64 characters, got 65",
},
{
name: "md5 with 31 chars",
option: &ChecksumOption{Algorithm: "md5", Expected: strings.Repeat("a", 31)},
wantErr: true,
errSubstr: "expected 32 characters, got 31",
},
{
name: "md5 with 33 chars",
option: &ChecksumOption{Algorithm: "md5", Expected: strings.Repeat("a", 33)},
wantErr: true,
errSubstr: "expected 32 characters, got 33",
},
{
name: "md5 with sha256-length hash (64 chars)",
option: &ChecksumOption{Algorithm: "md5", Expected: validSha256},
wantErr: true,
errSubstr: "expected 32 characters, got 64",
},
{
name: "sha1 with 39 chars",
option: &ChecksumOption{Algorithm: "sha1", Expected: strings.Repeat("a", 39)},
wantErr: true,
errSubstr: "expected 40 characters, got 39",
},
{
name: "sha1 with 41 chars",
option: &ChecksumOption{Algorithm: "sha1", Expected: strings.Repeat("a", 41)},
wantErr: true,
errSubstr: "expected 40 characters, got 41",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := tt.option.Validate()
if (err != nil) != tt.wantErr {
t.Fatalf("Validate() error = %v, wantErr %v", err, tt.wantErr)
}
if tt.wantErr && tt.errSubstr != "" {
if err == nil || !strings.Contains(err.Error(), tt.errSubstr) {
t.Fatalf("Validate() error = %v, want substring %q", err, tt.errSubstr)
}
}
})
}
}
@@ -18,6 +18,7 @@
3B3967161E833CAA004F5970 /* AppFrameworkInfo.plist in Resources */ = {isa = PBXBuildFile; fileRef = 3B3967151E833CAA004F5970 /* AppFrameworkInfo.plist */; }; 3B3967161E833CAA004F5970 /* AppFrameworkInfo.plist in Resources */ = {isa = PBXBuildFile; fileRef = 3B3967151E833CAA004F5970 /* AppFrameworkInfo.plist */; };
3EF29839EA13551EB8144269 /* GopeedLiveActivityExtension.appex in Embed Foundation Extensions */ = {isa = PBXBuildFile; fileRef = 76ED41E0163DFA97F018D1AC /* GopeedLiveActivityExtension.appex */; settings = {ATTRIBUTES = (RemoveHeadersOnCopy, ); }; }; 3EF29839EA13551EB8144269 /* GopeedLiveActivityExtension.appex in Embed Foundation Extensions */ = {isa = PBXBuildFile; fileRef = 76ED41E0163DFA97F018D1AC /* GopeedLiveActivityExtension.appex */; settings = {ATTRIBUTES = (RemoveHeadersOnCopy, ); }; };
5BF69191A426106A38B62F9F /* GopeedContinuedProcessingManager.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3879B6DB5398A19281BA5F8F /* GopeedContinuedProcessingManager.swift */; }; 5BF69191A426106A38B62F9F /* GopeedContinuedProcessingManager.swift in Sources */ = {isa = PBXBuildFile; fileRef = 3879B6DB5398A19281BA5F8F /* GopeedContinuedProcessingManager.swift */; };
B30000000000000000000001 /* GopeedTaskEvent.swift in Sources */ = {isa = PBXBuildFile; fileRef = B30000000000000000000002 /* GopeedTaskEvent.swift */; };
6085D504396791024CEC496B /* Pods_RunnerTests.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = 8823E31DA135C410E362683E /* Pods_RunnerTests.framework */; }; 6085D504396791024CEC496B /* Pods_RunnerTests.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = 8823E31DA135C410E362683E /* Pods_RunnerTests.framework */; };
6243BE8F995F5284DBDD7DE3 /* GopeedLiveActivityWidget.swift in Sources */ = {isa = PBXBuildFile; fileRef = 35C9F8DE4B991051758F2BE1 /* GopeedLiveActivityWidget.swift */; }; 6243BE8F995F5284DBDD7DE3 /* GopeedLiveActivityWidget.swift in Sources */ = {isa = PBXBuildFile; fileRef = 35C9F8DE4B991051758F2BE1 /* GopeedLiveActivityWidget.swift */; };
74858FAF1ED2DC5600515810 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 74858FAE1ED2DC5600515810 /* AppDelegate.swift */; }; 74858FAF1ED2DC5600515810 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 74858FAE1ED2DC5600515810 /* AppDelegate.swift */; };
@@ -102,6 +103,7 @@
331C8081294A63A400263BE5 /* RunnerTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = RunnerTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; }; 331C8081294A63A400263BE5 /* RunnerTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = RunnerTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
35C9F8DE4B991051758F2BE1 /* GopeedLiveActivityWidget.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = GopeedLiveActivityWidget.swift; sourceTree = "<group>"; }; 35C9F8DE4B991051758F2BE1 /* GopeedLiveActivityWidget.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = GopeedLiveActivityWidget.swift; sourceTree = "<group>"; };
3879B6DB5398A19281BA5F8F /* GopeedContinuedProcessingManager.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = GopeedContinuedProcessingManager.swift; sourceTree = "<group>"; }; 3879B6DB5398A19281BA5F8F /* GopeedContinuedProcessingManager.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = GopeedContinuedProcessingManager.swift; sourceTree = "<group>"; };
B30000000000000000000002 /* GopeedTaskEvent.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = GopeedTaskEvent.swift; sourceTree = "<group>"; };
3B3967151E833CAA004F5970 /* AppFrameworkInfo.plist */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.plist.xml; name = AppFrameworkInfo.plist; path = Flutter/AppFrameworkInfo.plist; sourceTree = "<group>"; }; 3B3967151E833CAA004F5970 /* AppFrameworkInfo.plist */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.plist.xml; name = AppFrameworkInfo.plist; path = Flutter/AppFrameworkInfo.plist; sourceTree = "<group>"; };
52A13633B52EDC9D3F479DFD /* Info.plist */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = "<group>"; }; 52A13633B52EDC9D3F479DFD /* Info.plist */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = "<group>"; };
55356FCC7E573C3FEEC2DF4D /* Pods-RunnerTests.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-RunnerTests.release.xcconfig"; path = "Target Support Files/Pods-RunnerTests/Pods-RunnerTests.release.xcconfig"; sourceTree = "<group>"; }; 55356FCC7E573C3FEEC2DF4D /* Pods-RunnerTests.release.xcconfig */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = text.xcconfig; name = "Pods-RunnerTests.release.xcconfig"; path = "Target Support Files/Pods-RunnerTests/Pods-RunnerTests.release.xcconfig"; sourceTree = "<group>"; };
@@ -286,6 +288,7 @@
09AC32C4BF90EAC54D048584 /* GopeedDownloadAttributes.swift */, 09AC32C4BF90EAC54D048584 /* GopeedDownloadAttributes.swift */,
EB8C142A455C80367DDD68AE /* GopeedLiveActivityManager.swift */, EB8C142A455C80367DDD68AE /* GopeedLiveActivityManager.swift */,
3879B6DB5398A19281BA5F8F /* GopeedContinuedProcessingManager.swift */, 3879B6DB5398A19281BA5F8F /* GopeedContinuedProcessingManager.swift */,
B30000000000000000000002 /* GopeedTaskEvent.swift */,
); );
path = Runner; path = Runner;
sourceTree = "<group>"; sourceTree = "<group>";
@@ -607,6 +610,7 @@
1E29080079AAFD1CF2C71262 /* GopeedDownloadAttributes.swift in Sources */, 1E29080079AAFD1CF2C71262 /* GopeedDownloadAttributes.swift in Sources */,
032447D610CFAF8A75642212 /* GopeedLiveActivityManager.swift in Sources */, 032447D610CFAF8A75642212 /* GopeedLiveActivityManager.swift in Sources */,
5BF69191A426106A38B62F9F /* GopeedContinuedProcessingManager.swift in Sources */, 5BF69191A426106A38B62F9F /* GopeedContinuedProcessingManager.swift in Sources */,
B30000000000000000000001 /* GopeedTaskEvent.swift in Sources */,
); );
runOnlyForDeploymentPostprocessing = 0; runOnlyForDeploymentPostprocessing = 0;
}; };
@@ -22,8 +22,8 @@ final class GopeedContinuedProcessingManager: NSObject {
// MARK: - BGCPT state // MARK: - BGCPT state
private var registeredIdentifiers = Set<String>()
private var pendingTaskIDs = Set<String>() private var pendingTaskIDs = Set<String>()
private var foregroundStartTokens: [String: UUID] = [:]
private struct ExpiringTaskState { private struct ExpiringTaskState {
let identifier: String let identifier: String
let generation: UUID let generation: UUID
@@ -167,16 +167,11 @@ final class GopeedContinuedProcessingManager: NSObject {
// Pre-mark task.start so the Objective-C forwarder can // Pre-mark task.start so the Objective-C forwarder can
// immediately suppress the custom ActivityKit Live Activity. // immediately suppress the custom ActivityKit Live Activity.
if if
let data = payload.data(using: .utf8), let event = GopeedTaskEvent.decode(payload),
let json = try? JSONSerialization.jsonObject( event.type == .start
with: data
) as? [String: Any],
let type = json["type"] as? String,
type == "task.start",
let taskID = json["taskId"] as? String
{ {
setTaskOwnership( setTaskOwnership(
taskID: taskID, taskID: event.taskID,
handled: true handled: true
) )
} }
@@ -198,33 +193,19 @@ final class GopeedContinuedProcessingManager: NSObject {
return return
} }
guard guard let event = GopeedTaskEvent.decode(payload) else {
let data = payload.data(
using: .utf8
),
let json =
try? JSONSerialization
.jsonObject(
with: data
) as? [String: Any],
let type =
json["type"] as? String,
let taskID =
json["taskId"] as? String
else {
print( print(
"ContinuedProcessing: invalid event" "ContinuedProcessing: invalid event"
) )
return return
} }
let name = let taskID = event.taskID
json["name"] as? String let name = event.name
?? "Download"
switch type { switch event.type {
case "task.start": case .start:
progressRecoveryTokens.removeValue( progressRecoveryTokens.removeValue(
forKey: taskID forKey: taskID
) )
@@ -234,7 +215,7 @@ final class GopeedContinuedProcessingManager: NSObject {
name: name name: name
) )
case "task.progress": case .progress:
if let generation = if let generation =
taskGenerations[taskID] { taskGenerations[taskID] {
updateProgress( updateProgress(
@@ -249,7 +230,7 @@ final class GopeedContinuedProcessingManager: NSObject {
) )
} }
case "task.pause": case .pause:
if let expiring = if let expiring =
expiringTasks[taskID] { expiringTasks[taskID] {
finishExpirationCleanup( finishExpirationCleanup(
@@ -265,7 +246,7 @@ final class GopeedContinuedProcessingManager: NSObject {
) )
} }
case "task.done": case .done:
finishTask( finishTask(
taskID: taskID, taskID: taskID,
success: true, success: true,
@@ -273,7 +254,7 @@ final class GopeedContinuedProcessingManager: NSObject {
"Download complete" "Download complete"
) )
case "task.error": case .error:
finishTask( finishTask(
taskID: taskID, taskID: taskID,
success: false, success: false,
@@ -281,7 +262,7 @@ final class GopeedContinuedProcessingManager: NSObject {
"Download failed" "Download failed"
) )
case "task.delete": case .delete:
finishTask( finishTask(
taskID: taskID, taskID: taskID,
success: true, success: true,
@@ -289,12 +270,11 @@ final class GopeedContinuedProcessingManager: NSObject {
"Download removed" "Download removed"
) )
default: case .other:
break break
} }
} }
// MARK: - Recover active task after setting changes // MARK: - Recover active task after setting changes
private func recoverTaskFromProgress( private func recoverTaskFromProgress(
@@ -307,6 +287,7 @@ final class GopeedContinuedProcessingManager: NSObject {
!pendingTaskIDs.contains(taskID), !pendingTaskIDs.contains(taskID),
activeTasks[taskID] == nil, activeTasks[taskID] == nil,
expiringTasks[taskID] == nil, expiringTasks[taskID] == nil,
foregroundStartTokens[taskID] == nil,
progressRecoveryTokens[taskID] == nil progressRecoveryTokens[taskID] == nil
else { else {
return return
@@ -395,7 +376,8 @@ final class GopeedContinuedProcessingManager: NSObject {
guard guard
!pendingTaskIDs.contains(taskID), !pendingTaskIDs.contains(taskID),
activeTasks[taskID] == nil activeTasks[taskID] == nil,
foregroundStartTokens[taskID] == nil
else { else {
setTaskOwnership( setTaskOwnership(
taskID: taskID, taskID: taskID,
@@ -404,20 +386,57 @@ final class GopeedContinuedProcessingManager: NSObject {
return return
} }
// BGCPT should originate from a foreground user action. // BGCPT must originate from a foreground user action.
// This main-thread check happens only once per task start. // Never synchronously hop from workerQueue to the main queue:
let appIsActive: Bool // doing so can deadlock if the main thread is waiting on work
// that eventually needs workerQueue.
let foregroundToken = UUID()
foregroundStartTokens[taskID] =
foregroundToken
if Thread.isMainThread { DispatchQueue.main.async { [weak self] in
appIsActive = let appIsActive =
UIApplication.shared
.applicationState == .active
} else {
appIsActive =
DispatchQueue.main.sync {
UIApplication.shared UIApplication.shared
.applicationState == .active .applicationState == .active
guard let self else {
return
} }
self.workerQueue.async {
self.continueBeginTaskAfterForegroundCheck(
taskID: taskID,
name: name,
foregroundToken: foregroundToken,
appIsActive: appIsActive
)
}
}
}
private func continueBeginTaskAfterForegroundCheck(
taskID: String,
name: String,
foregroundToken: UUID,
appIsActive: Bool
) {
guard
foregroundStartTokens[taskID]
== foregroundToken
else {
return
}
foregroundStartTokens.removeValue(
forKey: taskID
)
guard currentEnabledSnapshot() else {
setTaskOwnership(
taskID: taskID,
handled: false
)
return
} }
guard appIsActive else { guard appIsActive else {
@@ -434,6 +453,20 @@ final class GopeedContinuedProcessingManager: NSObject {
return return
} }
// Re-check after the asynchronous main-thread hop. A terminal
// event, disable, or another valid start may have changed the
// lifecycle while the foreground state was being read.
guard
!pendingTaskIDs.contains(taskID),
activeTasks[taskID] == nil
else {
setTaskOwnership(
taskID: taskID,
handled: true
)
return
}
let generation = UUID() let generation = UUID()
taskGenerations[taskID] = generation taskGenerations[taskID] = generation
@@ -498,8 +531,6 @@ final class GopeedContinuedProcessingManager: NSObject {
return return
} }
registeredIdentifiers.insert(identifier)
let request = let request =
BGContinuedProcessingTaskRequest( BGContinuedProcessingTaskRequest(
identifier: identifier, identifier: identifier,
@@ -558,7 +589,6 @@ final class GopeedContinuedProcessingManager: NSObject {
} }
} }
// MARK: - System launched task // MARK: - System launched task
private func activateTask( private func activateTask(
@@ -901,6 +931,10 @@ final class GopeedContinuedProcessingManager: NSObject {
success: Bool, success: Bool,
finalSubtitle: String finalSubtitle: String
) { ) {
foregroundStartTokens.removeValue(
forKey: taskID
)
progressRecoveryTokens.removeValue( progressRecoveryTokens.removeValue(
forKey: taskID forKey: taskID
) )
@@ -1115,6 +1149,7 @@ final class GopeedContinuedProcessingManager: NSObject {
Set( Set(
handledTaskIDsSnapshot() handledTaskIDsSnapshot()
+ Array(taskIdentifiers.keys) + Array(taskIdentifiers.keys)
+ Array(foregroundStartTokens.keys)
+ Array(pendingTaskIDs) + Array(pendingTaskIDs)
+ Array(activeTasks.keys) + Array(activeTasks.keys)
+ Array(expiringTasks.keys) + Array(expiringTasks.keys)
@@ -1140,6 +1175,7 @@ final class GopeedContinuedProcessingManager: NSObject {
} }
activeTasks.removeAll() activeTasks.removeAll()
foregroundStartTokens.removeAll()
pendingTaskIDs.removeAll() pendingTaskIDs.removeAll()
expiringTasks.removeAll() expiringTasks.removeAll()
taskGenerations.removeAll() taskGenerations.removeAll()
@@ -248,26 +248,22 @@ final class GopeedLiveActivityManager: NSObject {
return return
} }
guard guard let event = GopeedTaskEvent.decode(payload) else {
let data = payload.data(using: .utf8),
let json = try? JSONSerialization.jsonObject(with: data)
as? [String: Any],
let type = json["type"] as? String,
let taskID = json["taskId"] as? String
else {
print("LiveActivity: invalid task event payload") print("LiveActivity: invalid task event payload")
return return
} }
let taskID = event.taskID
guard !isTaskSuppressed(taskID) else { guard !isTaskSuppressed(taskID) else {
return return
} }
let name = json["name"] as? String ?? "Download" let name = event.name
switch type { switch event.type {
case "task.start": case .start:
let generation = let generation =
startGeneration( startGeneration(
taskID: taskID taskID: taskID
@@ -282,7 +278,7 @@ final class GopeedLiveActivityManager: NSObject {
) )
} }
case "task.progress": case .progress:
guard guard
let generation = let generation =
generationForProgress( generationForProgress(
@@ -298,7 +294,7 @@ final class GopeedLiveActivityManager: NSObject {
generation: generation generation: generation
) )
case "task.pause": case .pause:
if let generation = if let generation =
invalidateGeneration( invalidateGeneration(
taskID: taskID taskID: taskID
@@ -312,7 +308,7 @@ final class GopeedLiveActivityManager: NSObject {
} }
} }
case "task.done": case .done:
if let generation = if let generation =
invalidateGeneration( invalidateGeneration(
taskID: taskID taskID: taskID
@@ -325,9 +321,9 @@ final class GopeedLiveActivityManager: NSObject {
} }
} }
case "task.error": case .error:
let error = let error =
json["error"] as? String event.error
?? "Download failed" ?? "Download failed"
if let generation = if let generation =
@@ -343,7 +339,7 @@ final class GopeedLiveActivityManager: NSObject {
} }
} }
case "task.delete": case .delete:
if let generation = if let generation =
invalidateGeneration( invalidateGeneration(
taskID: taskID taskID: taskID
@@ -357,7 +353,7 @@ final class GopeedLiveActivityManager: NSObject {
} }
} }
default: case .other:
break break
} }
} }
@@ -0,0 +1,69 @@
import Foundation
enum GopeedTaskEventType: Equatable {
case start
case progress
case pause
case done
case error
case delete
case other(String)
init(rawValue: String) {
switch rawValue {
case "task.start":
self = .start
case "task.progress":
self = .progress
case "task.pause":
self = .pause
case "task.done":
self = .done
case "task.error":
self = .error
case "task.delete":
self = .delete
default:
self = .other(rawValue)
}
}
}
struct GopeedTaskEvent {
let type: GopeedTaskEventType
let taskID: String
let name: String
let error: String?
private struct Payload: Decodable {
let type: String
let taskId: String
let name: String?
let error: String?
}
static func decode(
_ payload: String
) -> GopeedTaskEvent? {
guard
let data = payload.data(using: .utf8),
let decoded =
try? JSONDecoder().decode(
Payload.self,
from: data
)
else {
return nil
}
return GopeedTaskEvent(
type:
GopeedTaskEventType(
rawValue: decoded.type
),
taskID: decoded.taskId,
name: decoded.name ?? "Download",
error: decoded.error
)
}
}
+17
View File
@@ -17,6 +17,21 @@ class Options {
Map<String, dynamic> toJson() => _$OptionsToJson(this); Map<String, dynamic> toJson() => _$OptionsToJson(this);
} }
@JsonSerializable()
class ChecksumOption {
String algorithm;
String expected;
ChecksumOption({
this.algorithm = '',
this.expected = '',
});
factory ChecksumOption.fromJson(Map<String, dynamic> json) => _$ChecksumOptionFromJson(json);
Map<String, dynamic> toJson() => _$ChecksumOptionToJson(this);
}
@JsonSerializable() @JsonSerializable()
class OptsExtraHttp { class OptsExtraHttp {
int connections; int connections;
@@ -25,6 +40,7 @@ class OptsExtraHttp {
bool? autoExtract; bool? autoExtract;
String archivePassword; String archivePassword;
bool deleteAfterExtract; bool deleteAfterExtract;
ChecksumOption? checksum;
OptsExtraHttp({ OptsExtraHttp({
this.connections = 0, this.connections = 0,
@@ -33,6 +49,7 @@ class OptsExtraHttp {
this.autoExtract, this.autoExtract,
this.archivePassword = '', this.archivePassword = '',
this.deleteAfterExtract = false, this.deleteAfterExtract = false,
this.checksum,
}); });
factory OptsExtraHttp.fromJson(Map<String, dynamic> json) => _$OptsExtraHttpFromJson(json); factory OptsExtraHttp.fromJson(Map<String, dynamic> json) => _$OptsExtraHttpFromJson(json);
+14
View File
@@ -22,6 +22,16 @@ Map<String, dynamic> _$OptionsToJson(Options instance) => <String, dynamic>{
'extra': ?instance.extra, 'extra': ?instance.extra,
}; };
ChecksumOption _$ChecksumOptionFromJson(Map<String, dynamic> json) => ChecksumOption(
algorithm: json['algorithm'] as String? ?? '',
expected: json['expected'] as String? ?? '',
);
Map<String, dynamic> _$ChecksumOptionToJson(ChecksumOption instance) => <String, dynamic>{
'algorithm': instance.algorithm,
'expected': instance.expected,
};
OptsExtraHttp _$OptsExtraHttpFromJson(Map<String, dynamic> json) => OptsExtraHttp( OptsExtraHttp _$OptsExtraHttpFromJson(Map<String, dynamic> json) => OptsExtraHttp(
connections: (json['connections'] as num?)?.toInt() ?? 0, connections: (json['connections'] as num?)?.toInt() ?? 0,
autoTorrent: json['autoTorrent'] as bool?, autoTorrent: json['autoTorrent'] as bool?,
@@ -29,6 +39,9 @@ OptsExtraHttp _$OptsExtraHttpFromJson(Map<String, dynamic> json) => OptsExtraHtt
autoExtract: json['autoExtract'] as bool?, autoExtract: json['autoExtract'] as bool?,
archivePassword: json['archivePassword'] as String? ?? '', archivePassword: json['archivePassword'] as String? ?? '',
deleteAfterExtract: json['deleteAfterExtract'] as bool? ?? false, deleteAfterExtract: json['deleteAfterExtract'] as bool? ?? false,
checksum: json['checksum'] == null
? null
: ChecksumOption.fromJson(json['checksum'] as Map<String, dynamic>),
); );
Map<String, dynamic> _$OptsExtraHttpToJson(OptsExtraHttp instance) => <String, dynamic>{ Map<String, dynamic> _$OptsExtraHttpToJson(OptsExtraHttp instance) => <String, dynamic>{
@@ -38,4 +51,5 @@ Map<String, dynamic> _$OptsExtraHttpToJson(OptsExtraHttp instance) => <String, d
'autoExtract': ?instance.autoExtract, 'autoExtract': ?instance.autoExtract,
'archivePassword': instance.archivePassword, 'archivePassword': instance.archivePassword,
'deleteAfterExtract': instance.deleteAfterExtract, 'deleteAfterExtract': instance.deleteAfterExtract,
'checksum': ?instance.checksum?.toJson(),
}; };
+2
View File
@@ -35,6 +35,7 @@ class Task {
Progress progress; Progress progress;
DateTime createdAt; DateTime createdAt;
DateTime updatedAt; DateTime updatedAt;
DateTime? doneAt;
Task({ Task({
required this.id, required this.id,
@@ -45,6 +46,7 @@ class Task {
required this.progress, required this.progress,
required this.createdAt, required this.createdAt,
required this.updatedAt, required this.updatedAt,
this.doneAt,
}); });
factory Task.fromJson(Map<String, dynamic> json) => _$TaskFromJson(json); factory Task.fromJson(Map<String, dynamic> json) => _$TaskFromJson(json);
+2
View File
@@ -15,6 +15,7 @@ Task _$TaskFromJson(Map<String, dynamic> json) => Task(
progress: Progress.fromJson(json['progress'] as Map<String, dynamic>), progress: Progress.fromJson(json['progress'] as Map<String, dynamic>),
createdAt: DateTime.parse(json['createdAt'] as String), createdAt: DateTime.parse(json['createdAt'] as String),
updatedAt: DateTime.parse(json['updatedAt'] as String), updatedAt: DateTime.parse(json['updatedAt'] as String),
doneAt: json['doneAt'] == null ? null : DateTime.parse(json['doneAt'] as String),
)..protocol = $enumDecodeNullable(_$ProtocolEnumMap, json['protocol']); )..protocol = $enumDecodeNullable(_$ProtocolEnumMap, json['protocol']);
Map<String, dynamic> _$TaskToJson(Task instance) => <String, dynamic>{ Map<String, dynamic> _$TaskToJson(Task instance) => <String, dynamic>{
@@ -27,6 +28,7 @@ Map<String, dynamic> _$TaskToJson(Task instance) => <String, dynamic>{
'progress': instance.progress.toJson(), 'progress': instance.progress.toJson(),
'createdAt': instance.createdAt.toIso8601String(), 'createdAt': instance.createdAt.toIso8601String(),
'updatedAt': instance.updatedAt.toIso8601String(), 'updatedAt': instance.updatedAt.toIso8601String(),
'doneAt': ?instance.doneAt?.toIso8601String(),
}; };
const _$StatusEnumMap = { const _$StatusEnumMap = {
Loaded 50 of 93 files, more files were not shown because too many files have changed in this diff. Show more