mirror of
https://github.com/GopeedLab/gopeed.git
synced 2026-11-16 13:23:41 +00:00
feat: replace gblob with blob http transport (#1407)
This commit is contained in:
34 files changed
+5303
-3057
No files matched your search
@@ -1,285 +0,0 @@
|
||||
# gblob Protocol
|
||||
|
||||
## Background
|
||||
|
||||
Gopeed extensions can resolve a download target into something that is not a normal remote URL.
|
||||
Typical examples are:
|
||||
|
||||
- text generated by an extension
|
||||
- binary data assembled in JavaScript
|
||||
- a stream produced incrementally inside the extension runtime
|
||||
|
||||
Returning a plain `http` or `https` URL is not enough for these cases.
|
||||
The `gblob` protocol provides an internal object-URL style transport so extensions can hand generated content to the downloader in a browser-like way.
|
||||
|
||||
The extension-facing model is intentionally close to the browser:
|
||||
|
||||
- `URL.createObjectURL(new Blob(...))`
|
||||
- `URL.createObjectURL(readableStream)`
|
||||
- `URL.createObjectURL(openReadable)`
|
||||
- `URL.revokeObjectURL(url)`
|
||||
|
||||
The returned URL looks like:
|
||||
|
||||
```text
|
||||
gblob:<id>
|
||||
```
|
||||
|
||||
`gblob` is a process-local protocol. The source only exists inside the current Gopeed process.
|
||||
|
||||
## Design Goals
|
||||
|
||||
- make extension-generated data download as a normal task
|
||||
- support eager `Blob` data, one-shot `ReadableStream` data, and resumable opener functions
|
||||
- avoid large in-memory buffering by spooling to disk
|
||||
- keep pause/continue behavior intuitive
|
||||
- allow an extension to recreate a new `gblob` URL after source expiration
|
||||
|
||||
## High-Level Architecture
|
||||
|
||||
The implementation has two layers in this directory:
|
||||
|
||||
- [`registry.go`](./registry.go): process-local source registry and spool-file manager
|
||||
- [`fetcher.go`](./fetcher.go): downloader fetcher implementation for the `gblob` scheme
|
||||
|
||||
The end-to-end flow is:
|
||||
|
||||
1. JavaScript creates a `Blob`, `ReadableStream`, or opener-function object URL.
|
||||
2. Go registers a `gblob` source and assigns it a unique `gblob:<id>` URL.
|
||||
3. Source bytes are written into a local spool file.
|
||||
4. The downloader resolves the `gblob` URL through the `gblob` fetcher.
|
||||
5. The fetcher copies bytes from the spool file into the final download target.
|
||||
|
||||
## Registry
|
||||
|
||||
The registry owns all active `gblob` sources for the current process.
|
||||
|
||||
Each source tracks:
|
||||
|
||||
- source id and `gblob` URL
|
||||
- local spool-file path
|
||||
- source type: `blob` or `writable_stream`
|
||||
- written byte count
|
||||
- source state: `open`, `closed`, `aborted`, `failed`
|
||||
- revoke state
|
||||
- pin count
|
||||
- optional extension session reference
|
||||
|
||||
Key behaviors:
|
||||
|
||||
- `Blob` sources are written once and immediately closed.
|
||||
- `ReadableStream` sources are bridged into the internal writable-source pipeline.
|
||||
- opener-function sources are opened lazily and can be reopened from an offset.
|
||||
- readers wait on a per-source notification channel when no new bytes are available yet.
|
||||
- revocation only prevents future use; cleanup happens when the source is no longer pinned.
|
||||
|
||||
## Fetcher
|
||||
|
||||
The fetcher treats `gblob` as a local file-backed download source.
|
||||
|
||||
### Resolve
|
||||
|
||||
`Resolve()` looks up the source in the registry and builds a single-file `Resource`.
|
||||
|
||||
Behavior differences by source type:
|
||||
|
||||
- `Blob`: resolved size is the current written size
|
||||
- `ReadableStream` / opener function: resolved size is whatever the extension declared; if unknown, it stays `0` until download completion updates the final size
|
||||
|
||||
`Range` is `true` only when the extension provides a resumable opener and marks the resource as ranged.
|
||||
|
||||
### Start
|
||||
|
||||
`Start()` opens the final target file and reads from the source spool file.
|
||||
|
||||
There are two resume modes:
|
||||
|
||||
- same `gblob` id: continue from the current target-file size
|
||||
- different `gblob` id: truncate the target file and restart from byte `0`
|
||||
|
||||
The fetcher persists the last started source id so it can distinguish these two cases across pause/continue and task restore.
|
||||
|
||||
### Pause / Continue
|
||||
|
||||
`Pause()` stops the current copy loop by canceling the reader side.
|
||||
It does not abort the `gblob` source.
|
||||
|
||||
If the source id stays the same, `Continue()` resumes from the existing target-file offset.
|
||||
This is how normal in-process `Pause -> Continue` works.
|
||||
|
||||
### Source Replacement
|
||||
|
||||
If an extension replaces `req.url` with a new `gblob:<new-id>` before the next start, the fetcher detects the id change and restarts from scratch.
|
||||
|
||||
This behavior is important for recovery flows such as:
|
||||
|
||||
- original `gblob` source expired
|
||||
- extension recreates a new `gblob` URL in `onError`
|
||||
- task continues with the new source
|
||||
|
||||
## Lifecycle Notes
|
||||
|
||||
### In-process pause and continue
|
||||
|
||||
For the same `gblob` id:
|
||||
|
||||
- the partially downloaded output file is kept
|
||||
- the fetcher resumes from the current file size
|
||||
- opener-function sources can be reopened from the current offset
|
||||
|
||||
For non-reopenable one-shot `ReadableStream` sources:
|
||||
|
||||
- a normal in-process pause/continue only works while the same in-memory source is still alive
|
||||
- they are not suitable for cross-session or reconstructable range resume
|
||||
|
||||
### Process restart
|
||||
|
||||
`gblob` sources are not persisted across process restarts.
|
||||
An unfinished `gblob` task is restored without a live source behind its old URL.
|
||||
|
||||
The intended recovery pattern is:
|
||||
|
||||
1. user clicks continue
|
||||
2. the old `gblob` source fails
|
||||
3. extension `onError` runs
|
||||
4. extension recreates a new `gblob` URL from `req.rawUrl`
|
||||
5. extension updates `ctx.task.meta.req.url`
|
||||
6. extension calls `ctx.task.continue()`
|
||||
|
||||
Because the new URL uses a different `gblob` id, the fetcher restarts from byte `0`.
|
||||
|
||||
## Why `rawUrl` Exists
|
||||
|
||||
When an extension resolves a request into `gblob:<id>`, the original site URL would otherwise be lost.
|
||||
`Request.RawURL` keeps the original creation URL so lifecycle events can still reason about the original target.
|
||||
|
||||
Extension event URL matching prefers `req.rawUrl` when present.
|
||||
|
||||
This is especially useful for `onStart`, `onError`, and `onDone` on `gblob` tasks.
|
||||
|
||||
## Extension Usage
|
||||
|
||||
### 1. Create a `Blob` object URL
|
||||
|
||||
```js
|
||||
gopeed.events.onResolve(async function (ctx) {
|
||||
const blob = new Blob(["hello world"], { type: "text/plain" });
|
||||
const url = URL.createObjectURL(blob);
|
||||
|
||||
ctx.res = {
|
||||
name: "example",
|
||||
files: [
|
||||
{
|
||||
name: "hello.txt",
|
||||
size: 11,
|
||||
req: { url }
|
||||
}
|
||||
]
|
||||
};
|
||||
});
|
||||
```
|
||||
|
||||
### 2. Create a one-shot `ReadableStream` object URL
|
||||
|
||||
```js
|
||||
gopeed.events.onResolve(async function (ctx) {
|
||||
const encoder = new TextEncoder();
|
||||
const stream = new ReadableStream({
|
||||
start(controller) {
|
||||
controller.enqueue(encoder.encode("line 1\n"));
|
||||
controller.enqueue(encoder.encode("line 2\n"));
|
||||
controller.close();
|
||||
}
|
||||
});
|
||||
const url = URL.createObjectURL(stream);
|
||||
|
||||
ctx.res = {
|
||||
name: "stream-example",
|
||||
files: [
|
||||
{
|
||||
name: "stream.txt",
|
||||
req: { url }
|
||||
}
|
||||
]
|
||||
};
|
||||
});
|
||||
```
|
||||
|
||||
### 3. Create a resumable opener object URL
|
||||
|
||||
```js
|
||||
gopeed.events.onResolve(async function (ctx) {
|
||||
const target = "https://example.com/file.bin";
|
||||
|
||||
const openReadable = async (offset) => {
|
||||
const headers = {};
|
||||
if (offset > 0) {
|
||||
headers.Range = `bytes=${offset}-`;
|
||||
}
|
||||
const response = await fetch(target, { headers });
|
||||
if (response.status !== 200 || response.status !== 206) {
|
||||
throw new Error(`unexpected response status: ${response.status}, expected ${expectedStatus}`);
|
||||
}
|
||||
if (!response.body) {
|
||||
throw new Error("empty response body");
|
||||
}
|
||||
return response.body;
|
||||
};
|
||||
|
||||
ctx.res = {
|
||||
name: "range-example",
|
||||
range: true,
|
||||
files: [
|
||||
{
|
||||
name: "file.bin",
|
||||
req: { url: URL.createObjectURL(openReadable) }
|
||||
}
|
||||
]
|
||||
};
|
||||
});
|
||||
```
|
||||
|
||||
Important:
|
||||
|
||||
- opener functions must return a `ReadableStream`
|
||||
- do not return `response.body.getReader()`
|
||||
- use opener functions for resumable / range-aware sources
|
||||
- validate HTTP status codes explicitly:
|
||||
first request usually expects `200`, resumed range requests usually expect `206`
|
||||
|
||||
### 4. Recover in `onError`
|
||||
|
||||
```js
|
||||
gopeed.events.onError(async function (ctx) {
|
||||
const req = ctx.task.meta.req;
|
||||
req.url = URL.createObjectURL(new Blob(["recovered\n"]));
|
||||
ctx.task.continue();
|
||||
});
|
||||
```
|
||||
|
||||
This creates a new `gblob` id, so the downloader restarts from byte `0`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- `gblob` is not a network protocol. Range support only exists when the extension recreates bytes through a resumable opener.
|
||||
- unfinished `gblob` sources do not survive process restarts.
|
||||
- recovery after restart depends on extension logic recreating a new source.
|
||||
- the current restart behavior is full restart on source-id change, not byte-range recovery across recreated sources.
|
||||
- one-shot `ReadableStream` tasks with unknown total size report size `0` until completion updates the final size.
|
||||
|
||||
## Summary
|
||||
|
||||
`gblob` lets Gopeed extensions treat generated content as a first-class download source.
|
||||
The registry handles source lifetime and disk spooling.
|
||||
The fetcher turns those sources into normal download tasks with predictable pause/continue behavior.
|
||||
|
||||
The most important rule is simple:
|
||||
|
||||
- same `gblob` id means resume
|
||||
- different `gblob` id means restart
|
||||
|
||||
And for extension authors:
|
||||
|
||||
- use `ReadableStream` for one-shot streaming content
|
||||
- use opener functions for resumable content
|
||||
- opener functions must return `ReadableStream`, not a reader
|
||||
@@ -1,415 +0,0 @@
|
||||
package gblob
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
||||
"github.com/GopeedLab/gopeed/internal/controller"
|
||||
"github.com/GopeedLab/gopeed/internal/fetcher"
|
||||
"github.com/GopeedLab/gopeed/pkg/base"
|
||||
)
|
||||
|
||||
type config struct{}
|
||||
|
||||
type Fetcher struct {
|
||||
ctl *controller.Controller
|
||||
registry *Registry
|
||||
meta *fetcher.FetcherMeta
|
||||
doneCh chan error
|
||||
|
||||
mu sync.Mutex
|
||||
cancel context.CancelFunc
|
||||
runDone chan struct{}
|
||||
paused bool
|
||||
closed bool
|
||||
|
||||
downloaded atomic.Int64
|
||||
lastSourceID string
|
||||
}
|
||||
|
||||
type fetcherData struct {
|
||||
LastSourceID string `json:"lastSourceId"`
|
||||
}
|
||||
|
||||
func (f *Fetcher) Setup(ctl *controller.Controller) {
|
||||
f.ctl = ctl
|
||||
if f.meta == nil {
|
||||
f.meta = &fetcher.FetcherMeta{}
|
||||
}
|
||||
f.doneCh = make(chan error, 1)
|
||||
}
|
||||
|
||||
func (f *Fetcher) Resolve(req *base.Request, opts *base.Options) error {
|
||||
if f.registry == nil {
|
||||
return ErrSourceNotFound
|
||||
}
|
||||
src, err := f.registry.Get(req.URL)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
f.meta.Req = req
|
||||
f.meta.Opts = opts
|
||||
if f.meta.Opts == nil {
|
||||
f.meta.Opts = &base.Options{}
|
||||
}
|
||||
name := f.meta.Opts.Name
|
||||
if name == "" {
|
||||
name = src.ID
|
||||
}
|
||||
snapshot := src.Snapshot()
|
||||
size := snapshot.Written
|
||||
if snapshot.DeclaredSize > 0 {
|
||||
size = snapshot.DeclaredSize
|
||||
}
|
||||
if src.Type == SourceTypeWritableStream {
|
||||
// Writable streams may continue producing bytes after Resolve,
|
||||
// so treat the total size as unknown until the writer closes.
|
||||
if snapshot.DeclaredSize == 0 {
|
||||
size = 0
|
||||
}
|
||||
}
|
||||
f.meta.Res = &base.Resource{
|
||||
Size: size,
|
||||
Range: snapshot.Range,
|
||||
Files: []*base.FileInfo{
|
||||
{
|
||||
Name: name,
|
||||
Path: "",
|
||||
Size: size,
|
||||
Req: req,
|
||||
},
|
||||
},
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fetcher) Start() error {
|
||||
if f.registry == nil {
|
||||
return ErrSourceNotFound
|
||||
}
|
||||
currentSourceID, err := ParseURL(f.meta.Req.URL)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
f.mu.Lock()
|
||||
if f.runDone != nil {
|
||||
f.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
f.paused = false
|
||||
runDone := make(chan struct{})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
f.cancel = cancel
|
||||
f.runDone = runDone
|
||||
f.mu.Unlock()
|
||||
cleanupStartErr := func(err error) error {
|
||||
f.mu.Lock()
|
||||
if f.runDone == runDone {
|
||||
f.runDone = nil
|
||||
f.cancel = nil
|
||||
}
|
||||
f.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
|
||||
path := f.meta.SingleFilepath()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
target, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0644)
|
||||
if err != nil {
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
offset := int64(0)
|
||||
allowResumeAcrossSource := f.meta.Res != nil && f.meta.Res.Range
|
||||
if f.lastSourceID != "" && f.lastSourceID != currentSourceID && !allowResumeAcrossSource {
|
||||
if err := target.Truncate(0); err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
if _, err := target.Seek(0, io.SeekStart); err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
} else {
|
||||
stat, err := target.Stat()
|
||||
if err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
offset = stat.Size()
|
||||
if _, err := target.Seek(offset, io.SeekStart); err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
}
|
||||
f.lastSourceID = currentSourceID
|
||||
f.downloaded.Store(offset)
|
||||
|
||||
sourceOffset := offset
|
||||
src, err := f.registry.Get(f.meta.Req.URL)
|
||||
if err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
snapshot := src.Snapshot()
|
||||
if snapshot.Written == 0 && !snapshot.Started {
|
||||
if f.meta.Res != nil && f.meta.Res.Range && offset > 0 {
|
||||
if err := f.registry.Reopen(f.meta.Req.URL, offset); err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
sourceOffset = 0
|
||||
} else if err := f.registry.StartSource(f.meta.Req.URL); err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
src, err = f.registry.Get(f.meta.Req.URL)
|
||||
if err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
snapshot = src.Snapshot()
|
||||
}
|
||||
if f.meta.Res != nil && f.meta.Res.Range && (snapshot.State == SourceStateAborted || snapshot.State == SourceStateFailed) {
|
||||
if err := f.registry.Reopen(f.meta.Req.URL, offset); err != nil {
|
||||
target.Close()
|
||||
return cleanupStartErr(err)
|
||||
}
|
||||
sourceOffset = 0
|
||||
}
|
||||
|
||||
go f.copyLoop(ctx, target, offset, sourceOffset)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fetcher) copyLoop(ctx context.Context, target *os.File, targetOffset, sourceOffset int64) {
|
||||
var finalErr error
|
||||
|
||||
defer target.Close()
|
||||
defer func() {
|
||||
f.mu.Lock()
|
||||
runDone := f.runDone
|
||||
f.runDone = nil
|
||||
f.cancel = nil
|
||||
paused := f.paused
|
||||
closed := f.closed
|
||||
f.mu.Unlock()
|
||||
close(runDone)
|
||||
if closed {
|
||||
select {
|
||||
case f.doneCh <- nil:
|
||||
default:
|
||||
}
|
||||
return
|
||||
}
|
||||
if paused {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case f.doneCh <- finalErr:
|
||||
default:
|
||||
}
|
||||
}()
|
||||
|
||||
src, err := f.registry.Get(f.meta.Req.URL)
|
||||
if err != nil {
|
||||
finalErr = err
|
||||
return
|
||||
}
|
||||
sourceFile, err := os.Open(src.Path)
|
||||
if err != nil {
|
||||
finalErr = err
|
||||
return
|
||||
}
|
||||
defer sourceFile.Close()
|
||||
|
||||
buf := make([]byte, 32*1024)
|
||||
for {
|
||||
if err := f.registry.WaitForReadable(ctx, f.meta.Req.URL, sourceOffset); err != nil {
|
||||
if errors.Is(err, context.Canceled) {
|
||||
return
|
||||
}
|
||||
finalErr = err
|
||||
return
|
||||
}
|
||||
|
||||
snapshot, serr := f.registry.Get(f.meta.Req.URL)
|
||||
if serr != nil {
|
||||
finalErr = serr
|
||||
return
|
||||
}
|
||||
current := snapshot.Snapshot()
|
||||
for sourceOffset < current.Written {
|
||||
remain := current.Written - sourceOffset
|
||||
readSize := int64(len(buf))
|
||||
if remain < readSize {
|
||||
readSize = remain
|
||||
}
|
||||
n, err := sourceFile.ReadAt(buf[:readSize], sourceOffset)
|
||||
if err != nil && !errors.Is(err, io.EOF) {
|
||||
finalErr = err
|
||||
return
|
||||
}
|
||||
if n == 0 {
|
||||
break
|
||||
}
|
||||
if _, err := target.Write(buf[:n]); err != nil {
|
||||
finalErr = err
|
||||
return
|
||||
}
|
||||
sourceOffset += int64(n)
|
||||
targetOffset += int64(n)
|
||||
f.downloaded.Store(targetOffset)
|
||||
}
|
||||
|
||||
if current.State == SourceStateClosed && sourceOffset >= current.Written {
|
||||
if f.meta.Res != nil {
|
||||
if f.meta.Res.Size > 0 && targetOffset < f.meta.Res.Size {
|
||||
finalErr = io.ErrUnexpectedEOF
|
||||
return
|
||||
}
|
||||
if f.meta.Res.Size == 0 {
|
||||
f.meta.Res.Size = targetOffset
|
||||
}
|
||||
f.meta.Res.Files[0].Size = f.meta.Res.Size
|
||||
}
|
||||
finalErr = nil
|
||||
return
|
||||
}
|
||||
if current.State == SourceStateAborted || current.State == SourceStateFailed {
|
||||
err := current.Err
|
||||
if err == nil {
|
||||
err = ErrSourceAborted
|
||||
}
|
||||
finalErr = err
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (f *Fetcher) Patch(req *base.Request, opts *base.Options) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fetcher) Pause() error {
|
||||
f.mu.Lock()
|
||||
f.paused = true
|
||||
cancel := f.cancel
|
||||
runDone := f.runDone
|
||||
f.mu.Unlock()
|
||||
if cancel != nil {
|
||||
cancel()
|
||||
}
|
||||
if runDone != nil {
|
||||
<-runDone
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fetcher) Close() error {
|
||||
f.mu.Lock()
|
||||
f.closed = true
|
||||
cancel := f.cancel
|
||||
runDone := f.runDone
|
||||
f.mu.Unlock()
|
||||
if cancel != nil {
|
||||
cancel()
|
||||
}
|
||||
if runDone != nil {
|
||||
<-runDone
|
||||
}
|
||||
select {
|
||||
case f.doneCh <- nil:
|
||||
default:
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *Fetcher) Stats() any {
|
||||
return map[string]any{}
|
||||
}
|
||||
|
||||
func (f *Fetcher) Meta() *fetcher.FetcherMeta {
|
||||
return f.meta
|
||||
}
|
||||
|
||||
func (f *Fetcher) Progress() fetcher.Progress {
|
||||
return fetcher.Progress{f.downloaded.Load()}
|
||||
}
|
||||
|
||||
func (f *Fetcher) Wait() error {
|
||||
return <-f.doneCh
|
||||
}
|
||||
|
||||
type FetcherManager struct {
|
||||
registry *Registry
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) SetRegistry(registry *Registry) {
|
||||
fm.registry = registry
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) Name() string {
|
||||
return Scheme
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) Filters() []*fetcher.SchemeFilter {
|
||||
return []*fetcher.SchemeFilter{
|
||||
{
|
||||
Type: fetcher.FilterTypeUrl,
|
||||
Pattern: strings.ToUpper(Scheme),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) Build() fetcher.Fetcher {
|
||||
return &Fetcher{
|
||||
registry: fm.registry,
|
||||
}
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) ParseName(u string) string {
|
||||
id, err := ParseURL(u)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) AutoRename() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) DefaultConfig() any {
|
||||
return &config{}
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) Store(f fetcher.Fetcher) (data any, err error) {
|
||||
_f := f.(*Fetcher)
|
||||
return &fetcherData{
|
||||
LastSourceID: _f.lastSourceID,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) Restore() (v any, f func(meta *fetcher.FetcherMeta, v any) fetcher.Fetcher) {
|
||||
return &fetcherData{}, func(meta *fetcher.FetcherMeta, v any) fetcher.Fetcher {
|
||||
fd := v.(*fetcherData)
|
||||
return &Fetcher{
|
||||
registry: fm.registry,
|
||||
meta: meta,
|
||||
doneCh: make(chan error, 1),
|
||||
lastSourceID: fd.LastSourceID,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (fm *FetcherManager) Close() error {
|
||||
return nil
|
||||
}
|
||||
@@ -1,122 +0,0 @@
|
||||
package gblob
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/GopeedLab/gopeed/internal/controller"
|
||||
internalfetcher "github.com/GopeedLab/gopeed/internal/fetcher"
|
||||
"github.com/GopeedLab/gopeed/pkg/base"
|
||||
)
|
||||
|
||||
func TestFetcherClosedSourceBeforeExpectedSizeReturnsError(t *testing.T) {
|
||||
registry := NewRegistry(t.TempDir())
|
||||
url, err := registry.CreateWritableStream(&CreateWritableStreamOptions{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registry.Write(url, []byte("abc")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registry.CloseSource(url); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
downloadDir := t.TempDir()
|
||||
filePath := filepath.Join(downloadDir, "partial.bin")
|
||||
f := &Fetcher{
|
||||
registry: registry,
|
||||
meta: &internalfetcher.FetcherMeta{
|
||||
Req: &base.Request{URL: url},
|
||||
Res: &base.Resource{
|
||||
Size: 10,
|
||||
Range: true,
|
||||
Files: []*base.FileInfo{{Name: "partial.bin", Req: &base.Request{URL: url}}},
|
||||
},
|
||||
Opts: &base.Options{Path: downloadDir, Name: "partial.bin"},
|
||||
},
|
||||
}
|
||||
f.Setup(controller.NewController())
|
||||
|
||||
if err := f.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
select {
|
||||
case err := <-f.doneCh:
|
||||
if !errors.Is(err, io.ErrUnexpectedEOF) {
|
||||
t.Fatalf("expected unexpected EOF, got %v", err)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("timeout waiting for fetcher completion")
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(filePath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(data) != "abc" {
|
||||
t.Fatalf("unexpected partial file content: %q", string(data))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetcherClosedUnknownSizeSourceCompletesSuccessfully(t *testing.T) {
|
||||
registry := NewRegistry(t.TempDir())
|
||||
url, err := registry.CreateWritableStream(&CreateWritableStreamOptions{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registry.Write(url, []byte("abcdef")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registry.CloseSource(url); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
downloadDir := t.TempDir()
|
||||
filePath := filepath.Join(downloadDir, "stream.bin")
|
||||
f := &Fetcher{
|
||||
registry: registry,
|
||||
meta: &internalfetcher.FetcherMeta{
|
||||
Req: &base.Request{URL: url},
|
||||
Res: &base.Resource{
|
||||
Size: 0,
|
||||
Range: false,
|
||||
Files: []*base.FileInfo{{Name: "stream.bin", Req: &base.Request{URL: url}}},
|
||||
},
|
||||
Opts: &base.Options{Path: downloadDir, Name: "stream.bin"},
|
||||
},
|
||||
}
|
||||
f.Setup(controller.NewController())
|
||||
|
||||
if err := f.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
select {
|
||||
case err := <-f.doneCh:
|
||||
if err != nil {
|
||||
t.Fatalf("expected nil completion error, got %v", err)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("timeout waiting for fetcher completion")
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(filePath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(data) != "abcdef" {
|
||||
t.Fatalf("unexpected stream file content: %q", string(data))
|
||||
}
|
||||
if got := f.meta.Res.Size; got != int64(len(data)) {
|
||||
t.Fatalf("unexpected resolved size: got %d want %d", got, len(data))
|
||||
}
|
||||
if got := f.meta.Res.Files[0].Size; got != int64(len(data)) {
|
||||
t.Fatalf("unexpected resolved file size: got %d want %d", got, len(data))
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,46 +0,0 @@
|
||||
package gblob
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRegistry_RevokeOpenWritableStreamAbortsAndCleansUpAfterUnpin(t *testing.T) {
|
||||
registry := NewRegistry(t.TempDir())
|
||||
url, err := registry.CreateWritableStream(&CreateWritableStreamOptions{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registry.Pin(url); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
src, err := registry.Get(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if state := src.Snapshot().State; state != SourceStateOpen {
|
||||
t.Fatalf("expected initial open state, got %s", state)
|
||||
}
|
||||
|
||||
if err := registry.Revoke(url); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
src, err = registry.Get(url)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
snapshot := src.Snapshot()
|
||||
if snapshot.State != SourceStateAborted {
|
||||
t.Fatalf("expected revoked open source to become aborted, got %s", snapshot.State)
|
||||
}
|
||||
if !errors.Is(snapshot.Err, ErrSourceRevoked) {
|
||||
t.Fatalf("expected revoke error, got %v", snapshot.Err)
|
||||
}
|
||||
|
||||
registry.Unpin(url)
|
||||
if _, err := registry.Get(url); !errors.Is(err, ErrSourceNotFound) {
|
||||
t.Fatalf("expected source cleanup after unpin, got %v", err)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user