feat: replace gblob with blob http transport (#1407)

This commit is contained in:
Levi authored and GitHub committed 2026-07-14 20:05:33 +08:00
1 parent e2ac8ce64b
commit 9fb3758203
34 files changed
+5303 -3057

No files matched your search

-285
View File
@@ -1,285 +0,0 @@
# gblob Protocol
## Background
Gopeed extensions can resolve a download target into something that is not a normal remote URL.
Typical examples are:
- text generated by an extension
- binary data assembled in JavaScript
- a stream produced incrementally inside the extension runtime
Returning a plain `http` or `https` URL is not enough for these cases.
The `gblob` protocol provides an internal object-URL style transport so extensions can hand generated content to the downloader in a browser-like way.
The extension-facing model is intentionally close to the browser:
- `URL.createObjectURL(new Blob(...))`
- `URL.createObjectURL(readableStream)`
- `URL.createObjectURL(openReadable)`
- `URL.revokeObjectURL(url)`
The returned URL looks like:
```text
gblob:<id>
```
`gblob` is a process-local protocol. The source only exists inside the current Gopeed process.
## Design Goals
- make extension-generated data download as a normal task
- support eager `Blob` data, one-shot `ReadableStream` data, and resumable opener functions
- avoid large in-memory buffering by spooling to disk
- keep pause/continue behavior intuitive
- allow an extension to recreate a new `gblob` URL after source expiration
## High-Level Architecture
The implementation has two layers in this directory:
- [`registry.go`](./registry.go): process-local source registry and spool-file manager
- [`fetcher.go`](./fetcher.go): downloader fetcher implementation for the `gblob` scheme
The end-to-end flow is:
1. JavaScript creates a `Blob`, `ReadableStream`, or opener-function object URL.
2. Go registers a `gblob` source and assigns it a unique `gblob:<id>` URL.
3. Source bytes are written into a local spool file.
4. The downloader resolves the `gblob` URL through the `gblob` fetcher.
5. The fetcher copies bytes from the spool file into the final download target.
## Registry
The registry owns all active `gblob` sources for the current process.
Each source tracks:
- source id and `gblob` URL
- local spool-file path
- source type: `blob` or `writable_stream`
- written byte count
- source state: `open`, `closed`, `aborted`, `failed`
- revoke state
- pin count
- optional extension session reference
Key behaviors:
- `Blob` sources are written once and immediately closed.
- `ReadableStream` sources are bridged into the internal writable-source pipeline.
- opener-function sources are opened lazily and can be reopened from an offset.
- readers wait on a per-source notification channel when no new bytes are available yet.
- revocation only prevents future use; cleanup happens when the source is no longer pinned.
## Fetcher
The fetcher treats `gblob` as a local file-backed download source.
### Resolve
`Resolve()` looks up the source in the registry and builds a single-file `Resource`.
Behavior differences by source type:
- `Blob`: resolved size is the current written size
- `ReadableStream` / opener function: resolved size is whatever the extension declared; if unknown, it stays `0` until download completion updates the final size
`Range` is `true` only when the extension provides a resumable opener and marks the resource as ranged.
### Start
`Start()` opens the final target file and reads from the source spool file.
There are two resume modes:
- same `gblob` id: continue from the current target-file size
- different `gblob` id: truncate the target file and restart from byte `0`
The fetcher persists the last started source id so it can distinguish these two cases across pause/continue and task restore.
### Pause / Continue
`Pause()` stops the current copy loop by canceling the reader side.
It does not abort the `gblob` source.
If the source id stays the same, `Continue()` resumes from the existing target-file offset.
This is how normal in-process `Pause -> Continue` works.
### Source Replacement
If an extension replaces `req.url` with a new `gblob:<new-id>` before the next start, the fetcher detects the id change and restarts from scratch.
This behavior is important for recovery flows such as:
- original `gblob` source expired
- extension recreates a new `gblob` URL in `onError`
- task continues with the new source
## Lifecycle Notes
### In-process pause and continue
For the same `gblob` id:
- the partially downloaded output file is kept
- the fetcher resumes from the current file size
- opener-function sources can be reopened from the current offset
For non-reopenable one-shot `ReadableStream` sources:
- a normal in-process pause/continue only works while the same in-memory source is still alive
- they are not suitable for cross-session or reconstructable range resume
### Process restart
`gblob` sources are not persisted across process restarts.
An unfinished `gblob` task is restored without a live source behind its old URL.
The intended recovery pattern is:
1. user clicks continue
2. the old `gblob` source fails
3. extension `onError` runs
4. extension recreates a new `gblob` URL from `req.rawUrl`
5. extension updates `ctx.task.meta.req.url`
6. extension calls `ctx.task.continue()`
Because the new URL uses a different `gblob` id, the fetcher restarts from byte `0`.
## Why `rawUrl` Exists
When an extension resolves a request into `gblob:<id>`, the original site URL would otherwise be lost.
`Request.RawURL` keeps the original creation URL so lifecycle events can still reason about the original target.
Extension event URL matching prefers `req.rawUrl` when present.
This is especially useful for `onStart`, `onError`, and `onDone` on `gblob` tasks.
## Extension Usage
### 1. Create a `Blob` object URL
```js
gopeed.events.onResolve(async function (ctx) {
const blob = new Blob(["hello world"], { type: "text/plain" });
const url = URL.createObjectURL(blob);
ctx.res = {
name: "example",
files: [
{
name: "hello.txt",
size: 11,
req: { url }
}
]
};
});
```
### 2. Create a one-shot `ReadableStream` object URL
```js
gopeed.events.onResolve(async function (ctx) {
const encoder = new TextEncoder();
const stream = new ReadableStream({
start(controller) {
controller.enqueue(encoder.encode("line 1\n"));
controller.enqueue(encoder.encode("line 2\n"));
controller.close();
}
});
const url = URL.createObjectURL(stream);
ctx.res = {
name: "stream-example",
files: [
{
name: "stream.txt",
req: { url }
}
]
};
});
```
### 3. Create a resumable opener object URL
```js
gopeed.events.onResolve(async function (ctx) {
const target = "https://example.com/file.bin";
const openReadable = async (offset) => {
const headers = {};
if (offset > 0) {
headers.Range = `bytes=${offset}-`;
}
const response = await fetch(target, { headers });
if (response.status !== 200 || response.status !== 206) {
throw new Error(`unexpected response status: ${response.status}, expected ${expectedStatus}`);
}
if (!response.body) {
throw new Error("empty response body");
}
return response.body;
};
ctx.res = {
name: "range-example",
range: true,
files: [
{
name: "file.bin",
req: { url: URL.createObjectURL(openReadable) }
}
]
};
});
```
Important:
- opener functions must return a `ReadableStream`
- do not return `response.body.getReader()`
- use opener functions for resumable / range-aware sources
- validate HTTP status codes explicitly:
first request usually expects `200`, resumed range requests usually expect `206`
### 4. Recover in `onError`
```js
gopeed.events.onError(async function (ctx) {
const req = ctx.task.meta.req;
req.url = URL.createObjectURL(new Blob(["recovered\n"]));
ctx.task.continue();
});
```
This creates a new `gblob` id, so the downloader restarts from byte `0`.
## Limitations
- `gblob` is not a network protocol. Range support only exists when the extension recreates bytes through a resumable opener.
- unfinished `gblob` sources do not survive process restarts.
- recovery after restart depends on extension logic recreating a new source.
- the current restart behavior is full restart on source-id change, not byte-range recovery across recreated sources.
- one-shot `ReadableStream` tasks with unknown total size report size `0` until completion updates the final size.
## Summary
`gblob` lets Gopeed extensions treat generated content as a first-class download source.
The registry handles source lifetime and disk spooling.
The fetcher turns those sources into normal download tasks with predictable pause/continue behavior.
The most important rule is simple:
- same `gblob` id means resume
- different `gblob` id means restart
And for extension authors:
- use `ReadableStream` for one-shot streaming content
- use opener functions for resumable content
- opener functions must return `ReadableStream`, not a reader
-415
View File
@@ -1,415 +0,0 @@
package gblob
import (
"context"
"errors"
"io"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"github.com/GopeedLab/gopeed/internal/controller"
"github.com/GopeedLab/gopeed/internal/fetcher"
"github.com/GopeedLab/gopeed/pkg/base"
)
type config struct{}
type Fetcher struct {
ctl *controller.Controller
registry *Registry
meta *fetcher.FetcherMeta
doneCh chan error
mu sync.Mutex
cancel context.CancelFunc
runDone chan struct{}
paused bool
closed bool
downloaded atomic.Int64
lastSourceID string
}
type fetcherData struct {
LastSourceID string `json:"lastSourceId"`
}
func (f *Fetcher) Setup(ctl *controller.Controller) {
f.ctl = ctl
if f.meta == nil {
f.meta = &fetcher.FetcherMeta{}
}
f.doneCh = make(chan error, 1)
}
func (f *Fetcher) Resolve(req *base.Request, opts *base.Options) error {
if f.registry == nil {
return ErrSourceNotFound
}
src, err := f.registry.Get(req.URL)
if err != nil {
return err
}
f.meta.Req = req
f.meta.Opts = opts
if f.meta.Opts == nil {
f.meta.Opts = &base.Options{}
}
name := f.meta.Opts.Name
if name == "" {
name = src.ID
}
snapshot := src.Snapshot()
size := snapshot.Written
if snapshot.DeclaredSize > 0 {
size = snapshot.DeclaredSize
}
if src.Type == SourceTypeWritableStream {
// Writable streams may continue producing bytes after Resolve,
// so treat the total size as unknown until the writer closes.
if snapshot.DeclaredSize == 0 {
size = 0
}
}
f.meta.Res = &base.Resource{
Size: size,
Range: snapshot.Range,
Files: []*base.FileInfo{
{
Name: name,
Path: "",
Size: size,
Req: req,
},
},
}
return nil
}
func (f *Fetcher) Start() error {
if f.registry == nil {
return ErrSourceNotFound
}
currentSourceID, err := ParseURL(f.meta.Req.URL)
if err != nil {
return err
}
f.mu.Lock()
if f.runDone != nil {
f.mu.Unlock()
return nil
}
f.paused = false
runDone := make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
f.cancel = cancel
f.runDone = runDone
f.mu.Unlock()
cleanupStartErr := func(err error) error {
f.mu.Lock()
if f.runDone == runDone {
f.runDone = nil
f.cancel = nil
}
f.mu.Unlock()
return err
}
path := f.meta.SingleFilepath()
if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
return cleanupStartErr(err)
}
target, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return cleanupStartErr(err)
}
offset := int64(0)
allowResumeAcrossSource := f.meta.Res != nil && f.meta.Res.Range
if f.lastSourceID != "" && f.lastSourceID != currentSourceID && !allowResumeAcrossSource {
if err := target.Truncate(0); err != nil {
target.Close()
return cleanupStartErr(err)
}
if _, err := target.Seek(0, io.SeekStart); err != nil {
target.Close()
return cleanupStartErr(err)
}
} else {
stat, err := target.Stat()
if err != nil {
target.Close()
return cleanupStartErr(err)
}
offset = stat.Size()
if _, err := target.Seek(offset, io.SeekStart); err != nil {
target.Close()
return cleanupStartErr(err)
}
}
f.lastSourceID = currentSourceID
f.downloaded.Store(offset)
sourceOffset := offset
src, err := f.registry.Get(f.meta.Req.URL)
if err != nil {
target.Close()
return cleanupStartErr(err)
}
snapshot := src.Snapshot()
if snapshot.Written == 0 && !snapshot.Started {
if f.meta.Res != nil && f.meta.Res.Range && offset > 0 {
if err := f.registry.Reopen(f.meta.Req.URL, offset); err != nil {
target.Close()
return cleanupStartErr(err)
}
sourceOffset = 0
} else if err := f.registry.StartSource(f.meta.Req.URL); err != nil {
target.Close()
return cleanupStartErr(err)
}
src, err = f.registry.Get(f.meta.Req.URL)
if err != nil {
target.Close()
return cleanupStartErr(err)
}
snapshot = src.Snapshot()
}
if f.meta.Res != nil && f.meta.Res.Range && (snapshot.State == SourceStateAborted || snapshot.State == SourceStateFailed) {
if err := f.registry.Reopen(f.meta.Req.URL, offset); err != nil {
target.Close()
return cleanupStartErr(err)
}
sourceOffset = 0
}
go f.copyLoop(ctx, target, offset, sourceOffset)
return nil
}
func (f *Fetcher) copyLoop(ctx context.Context, target *os.File, targetOffset, sourceOffset int64) {
var finalErr error
defer target.Close()
defer func() {
f.mu.Lock()
runDone := f.runDone
f.runDone = nil
f.cancel = nil
paused := f.paused
closed := f.closed
f.mu.Unlock()
close(runDone)
if closed {
select {
case f.doneCh <- nil:
default:
}
return
}
if paused {
return
}
select {
case f.doneCh <- finalErr:
default:
}
}()
src, err := f.registry.Get(f.meta.Req.URL)
if err != nil {
finalErr = err
return
}
sourceFile, err := os.Open(src.Path)
if err != nil {
finalErr = err
return
}
defer sourceFile.Close()
buf := make([]byte, 32*1024)
for {
if err := f.registry.WaitForReadable(ctx, f.meta.Req.URL, sourceOffset); err != nil {
if errors.Is(err, context.Canceled) {
return
}
finalErr = err
return
}
snapshot, serr := f.registry.Get(f.meta.Req.URL)
if serr != nil {
finalErr = serr
return
}
current := snapshot.Snapshot()
for sourceOffset < current.Written {
remain := current.Written - sourceOffset
readSize := int64(len(buf))
if remain < readSize {
readSize = remain
}
n, err := sourceFile.ReadAt(buf[:readSize], sourceOffset)
if err != nil && !errors.Is(err, io.EOF) {
finalErr = err
return
}
if n == 0 {
break
}
if _, err := target.Write(buf[:n]); err != nil {
finalErr = err
return
}
sourceOffset += int64(n)
targetOffset += int64(n)
f.downloaded.Store(targetOffset)
}
if current.State == SourceStateClosed && sourceOffset >= current.Written {
if f.meta.Res != nil {
if f.meta.Res.Size > 0 && targetOffset < f.meta.Res.Size {
finalErr = io.ErrUnexpectedEOF
return
}
if f.meta.Res.Size == 0 {
f.meta.Res.Size = targetOffset
}
f.meta.Res.Files[0].Size = f.meta.Res.Size
}
finalErr = nil
return
}
if current.State == SourceStateAborted || current.State == SourceStateFailed {
err := current.Err
if err == nil {
err = ErrSourceAborted
}
finalErr = err
return
}
}
}
func (f *Fetcher) Patch(req *base.Request, opts *base.Options) error {
return nil
}
func (f *Fetcher) Pause() error {
f.mu.Lock()
f.paused = true
cancel := f.cancel
runDone := f.runDone
f.mu.Unlock()
if cancel != nil {
cancel()
}
if runDone != nil {
<-runDone
}
return nil
}
func (f *Fetcher) Close() error {
f.mu.Lock()
f.closed = true
cancel := f.cancel
runDone := f.runDone
f.mu.Unlock()
if cancel != nil {
cancel()
}
if runDone != nil {
<-runDone
}
select {
case f.doneCh <- nil:
default:
}
return nil
}
func (f *Fetcher) Stats() any {
return map[string]any{}
}
func (f *Fetcher) Meta() *fetcher.FetcherMeta {
return f.meta
}
func (f *Fetcher) Progress() fetcher.Progress {
return fetcher.Progress{f.downloaded.Load()}
}
func (f *Fetcher) Wait() error {
return <-f.doneCh
}
type FetcherManager struct {
registry *Registry
}
func (fm *FetcherManager) SetRegistry(registry *Registry) {
fm.registry = registry
}
func (fm *FetcherManager) Name() string {
return Scheme
}
func (fm *FetcherManager) Filters() []*fetcher.SchemeFilter {
return []*fetcher.SchemeFilter{
{
Type: fetcher.FilterTypeUrl,
Pattern: strings.ToUpper(Scheme),
},
}
}
func (fm *FetcherManager) Build() fetcher.Fetcher {
return &Fetcher{
registry: fm.registry,
}
}
func (fm *FetcherManager) ParseName(u string) string {
id, err := ParseURL(u)
if err != nil {
return ""
}
return id
}
func (fm *FetcherManager) AutoRename() bool {
return true
}
func (fm *FetcherManager) DefaultConfig() any {
return &config{}
}
func (fm *FetcherManager) Store(f fetcher.Fetcher) (data any, err error) {
_f := f.(*Fetcher)
return &fetcherData{
LastSourceID: _f.lastSourceID,
}, nil
}
func (fm *FetcherManager) Restore() (v any, f func(meta *fetcher.FetcherMeta, v any) fetcher.Fetcher) {
return &fetcherData{}, func(meta *fetcher.FetcherMeta, v any) fetcher.Fetcher {
fd := v.(*fetcherData)
return &Fetcher{
registry: fm.registry,
meta: meta,
doneCh: make(chan error, 1),
lastSourceID: fd.LastSourceID,
}
}
}
func (fm *FetcherManager) Close() error {
return nil
}
-122
View File
@@ -1,122 +0,0 @@
package gblob
import (
"errors"
"io"
"os"
"path/filepath"
"testing"
"time"
"github.com/GopeedLab/gopeed/internal/controller"
internalfetcher "github.com/GopeedLab/gopeed/internal/fetcher"
"github.com/GopeedLab/gopeed/pkg/base"
)
func TestFetcherClosedSourceBeforeExpectedSizeReturnsError(t *testing.T) {
registry := NewRegistry(t.TempDir())
url, err := registry.CreateWritableStream(&CreateWritableStreamOptions{})
if err != nil {
t.Fatal(err)
}
if err := registry.Write(url, []byte("abc")); err != nil {
t.Fatal(err)
}
if err := registry.CloseSource(url); err != nil {
t.Fatal(err)
}
downloadDir := t.TempDir()
filePath := filepath.Join(downloadDir, "partial.bin")
f := &Fetcher{
registry: registry,
meta: &internalfetcher.FetcherMeta{
Req: &base.Request{URL: url},
Res: &base.Resource{
Size: 10,
Range: true,
Files: []*base.FileInfo{{Name: "partial.bin", Req: &base.Request{URL: url}}},
},
Opts: &base.Options{Path: downloadDir, Name: "partial.bin"},
},
}
f.Setup(controller.NewController())
if err := f.Start(); err != nil {
t.Fatal(err)
}
select {
case err := <-f.doneCh:
if !errors.Is(err, io.ErrUnexpectedEOF) {
t.Fatalf("expected unexpected EOF, got %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("timeout waiting for fetcher completion")
}
data, err := os.ReadFile(filePath)
if err != nil {
t.Fatal(err)
}
if string(data) != "abc" {
t.Fatalf("unexpected partial file content: %q", string(data))
}
}
func TestFetcherClosedUnknownSizeSourceCompletesSuccessfully(t *testing.T) {
registry := NewRegistry(t.TempDir())
url, err := registry.CreateWritableStream(&CreateWritableStreamOptions{})
if err != nil {
t.Fatal(err)
}
if err := registry.Write(url, []byte("abcdef")); err != nil {
t.Fatal(err)
}
if err := registry.CloseSource(url); err != nil {
t.Fatal(err)
}
downloadDir := t.TempDir()
filePath := filepath.Join(downloadDir, "stream.bin")
f := &Fetcher{
registry: registry,
meta: &internalfetcher.FetcherMeta{
Req: &base.Request{URL: url},
Res: &base.Resource{
Size: 0,
Range: false,
Files: []*base.FileInfo{{Name: "stream.bin", Req: &base.Request{URL: url}}},
},
Opts: &base.Options{Path: downloadDir, Name: "stream.bin"},
},
}
f.Setup(controller.NewController())
if err := f.Start(); err != nil {
t.Fatal(err)
}
select {
case err := <-f.doneCh:
if err != nil {
t.Fatalf("expected nil completion error, got %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("timeout waiting for fetcher completion")
}
data, err := os.ReadFile(filePath)
if err != nil {
t.Fatal(err)
}
if string(data) != "abcdef" {
t.Fatalf("unexpected stream file content: %q", string(data))
}
if got := f.meta.Res.Size; got != int64(len(data)) {
t.Fatalf("unexpected resolved size: got %d want %d", got, len(data))
}
if got := f.meta.Res.Files[0].Size; got != int64(len(data)) {
t.Fatalf("unexpected resolved file size: got %d want %d", got, len(data))
}
}
File diff suppressed because it is too large. Load diff
-46
View File
@@ -1,46 +0,0 @@
package gblob
import (
"errors"
"testing"
)
func TestRegistry_RevokeOpenWritableStreamAbortsAndCleansUpAfterUnpin(t *testing.T) {
registry := NewRegistry(t.TempDir())
url, err := registry.CreateWritableStream(&CreateWritableStreamOptions{})
if err != nil {
t.Fatal(err)
}
if err := registry.Pin(url); err != nil {
t.Fatal(err)
}
src, err := registry.Get(url)
if err != nil {
t.Fatal(err)
}
if state := src.Snapshot().State; state != SourceStateOpen {
t.Fatalf("expected initial open state, got %s", state)
}
if err := registry.Revoke(url); err != nil {
t.Fatal(err)
}
src, err = registry.Get(url)
if err != nil {
t.Fatal(err)
}
snapshot := src.Snapshot()
if snapshot.State != SourceStateAborted {
t.Fatalf("expected revoked open source to become aborted, got %s", snapshot.State)
}
if !errors.Is(snapshot.Err, ErrSourceRevoked) {
t.Fatalf("expected revoke error, got %v", snapshot.Err)
}
registry.Unpin(url)
if _, err := registry.Get(url); !errors.Is(err, ErrSourceNotFound) {
t.Fatalf("expected source cleanup after unpin, got %v", err)
}
}