mirror of
https://github.com/GopeedLab/gopeed.git
synced 2026-11-12 11:24:08 +00:00
feat: add gblob scheme support for extension (#1358)
This commit is contained in:
18 files changed
+2664
-147
No files matched your search
@@ -0,0 +1,233 @@
|
||||
# gblob Protocol
|
||||
|
||||
## Background
|
||||
|
||||
Gopeed extensions can resolve a download target into something that is not a normal remote URL.
|
||||
Typical examples are:
|
||||
|
||||
- text generated by an extension
|
||||
- binary data assembled in JavaScript
|
||||
- a stream produced incrementally inside the extension runtime
|
||||
|
||||
Returning a plain `http` or `https` URL is not enough for these cases.
|
||||
The `gblob` protocol provides an internal object-URL style transport so extensions can hand generated content to the downloader in a browser-like way.
|
||||
|
||||
The extension-facing model is intentionally close to the browser:
|
||||
|
||||
- `URL.createObjectURL(new Blob(...))`
|
||||
- `URL.createObjectURL(writableStream)`
|
||||
- `URL.revokeObjectURL(url)`
|
||||
|
||||
The returned URL looks like:
|
||||
|
||||
```text
|
||||
gblob:<id>
|
||||
```
|
||||
|
||||
`gblob` is a process-local protocol. The source only exists inside the current Gopeed process.
|
||||
|
||||
## Design Goals
|
||||
|
||||
- make extension-generated data download as a normal task
|
||||
- support both eager `Blob` data and incremental `WritableStream` data
|
||||
- avoid large in-memory buffering by spooling to disk
|
||||
- keep pause/continue behavior intuitive
|
||||
- allow an extension to recreate a new `gblob` URL after source expiration
|
||||
|
||||
## High-Level Architecture
|
||||
|
||||
The implementation has two layers in this directory:
|
||||
|
||||
- [`registry.go`](./registry.go): process-local source registry and spool-file manager
|
||||
- [`fetcher.go`](./fetcher.go): downloader fetcher implementation for the `gblob` scheme
|
||||
|
||||
The end-to-end flow is:
|
||||
|
||||
1. JavaScript creates a `Blob` or `WritableStream` object URL.
|
||||
2. Go registers a `gblob` source and assigns it a unique `gblob:<id>` URL.
|
||||
3. Source bytes are written into a local spool file.
|
||||
4. The downloader resolves the `gblob` URL through the `gblob` fetcher.
|
||||
5. The fetcher copies bytes from the spool file into the final download target.
|
||||
|
||||
## Registry
|
||||
|
||||
The registry owns all active `gblob` sources for the current process.
|
||||
|
||||
Each source tracks:
|
||||
|
||||
- source id and `gblob` URL
|
||||
- local spool-file path
|
||||
- source type: `blob` or `writable_stream`
|
||||
- written byte count
|
||||
- source state: `open`, `closed`, `aborted`, `failed`
|
||||
- revoke state
|
||||
- pin count
|
||||
- optional extension session reference
|
||||
|
||||
Key behaviors:
|
||||
|
||||
- `Blob` sources are written once and immediately closed.
|
||||
- `WritableStream` sources stay open until the writer closes or aborts.
|
||||
- readers wait on a per-source notification channel when no new bytes are available yet.
|
||||
- revocation only prevents future use; cleanup happens when the source is no longer pinned.
|
||||
|
||||
## Fetcher
|
||||
|
||||
The fetcher treats `gblob` as a local file-backed download source.
|
||||
|
||||
### Resolve
|
||||
|
||||
`Resolve()` looks up the source in the registry and builds a single-file `Resource`.
|
||||
|
||||
Behavior differences by source type:
|
||||
|
||||
- `Blob`: resolved size is the current written size
|
||||
- `WritableStream`: resolved size is reported as `0` until the writer closes
|
||||
|
||||
`Range` is always `false`.
|
||||
|
||||
### Start
|
||||
|
||||
`Start()` opens the final target file and reads from the source spool file.
|
||||
|
||||
There are two resume modes:
|
||||
|
||||
- same `gblob` id: continue from the current target-file size
|
||||
- different `gblob` id: truncate the target file and restart from byte `0`
|
||||
|
||||
The fetcher persists the last started source id so it can distinguish these two cases across pause/continue and task restore.
|
||||
|
||||
### Pause / Continue
|
||||
|
||||
`Pause()` stops the current copy loop by canceling the reader side.
|
||||
It does not abort the `gblob` source.
|
||||
|
||||
If the source id stays the same, `Continue()` resumes from the existing target-file offset.
|
||||
This is how normal in-process `Pause -> Continue` works.
|
||||
|
||||
### Source Replacement
|
||||
|
||||
If an extension replaces `req.url` with a new `gblob:<new-id>` before the next start, the fetcher detects the id change and restarts from scratch.
|
||||
|
||||
This behavior is important for recovery flows such as:
|
||||
|
||||
- original `gblob` source expired
|
||||
- extension recreates a new `gblob` URL in `onError`
|
||||
- task continues with the new source
|
||||
|
||||
## Lifecycle Notes
|
||||
|
||||
### In-process pause and continue
|
||||
|
||||
For the same `gblob` id:
|
||||
|
||||
- the partially downloaded output file is kept
|
||||
- the fetcher resumes from the current file size
|
||||
- `WritableStream` producers may continue writing while the task is paused
|
||||
|
||||
### Process restart
|
||||
|
||||
`gblob` sources are not persisted across process restarts.
|
||||
An unfinished `gblob` task is restored without a live source behind its old URL.
|
||||
|
||||
The intended recovery pattern is:
|
||||
|
||||
1. user clicks continue
|
||||
2. the old `gblob` source fails
|
||||
3. extension `onError` runs
|
||||
4. extension recreates a new `gblob` URL from `req.rawUrl`
|
||||
5. extension updates `ctx.task.meta.req.url`
|
||||
6. extension calls `ctx.task.continue()`
|
||||
|
||||
Because the new URL uses a different `gblob` id, the fetcher restarts from byte `0`.
|
||||
|
||||
## Why `rawUrl` Exists
|
||||
|
||||
When an extension resolves a request into `gblob:<id>`, the original site URL would otherwise be lost.
|
||||
`Request.RawURL` keeps the original creation URL so lifecycle events can still reason about the original target.
|
||||
|
||||
Extension event URL matching prefers `req.rawUrl` when present.
|
||||
|
||||
This is especially useful for `onStart`, `onError`, and `onDone` on `gblob` tasks.
|
||||
|
||||
## Extension Usage
|
||||
|
||||
### 1. Create a `Blob` object URL
|
||||
|
||||
```js
|
||||
gopeed.events.onResolve(async function (ctx) {
|
||||
const blob = new Blob(["hello world"], { type: "text/plain" });
|
||||
const url = URL.createObjectURL(blob);
|
||||
|
||||
ctx.res = {
|
||||
name: "example",
|
||||
files: [
|
||||
{
|
||||
name: "hello.txt",
|
||||
size: 11,
|
||||
req: { url }
|
||||
}
|
||||
]
|
||||
};
|
||||
});
|
||||
```
|
||||
|
||||
### 2. Create a streaming object URL
|
||||
|
||||
```js
|
||||
gopeed.events.onResolve(async function (ctx) {
|
||||
const stream = new TransformStream();
|
||||
const writer = stream.writable.getWriter();
|
||||
const url = URL.createObjectURL(stream.writable);
|
||||
|
||||
(async () => {
|
||||
await writer.write(new TextEncoder().encode("line 1\n"));
|
||||
await writer.write(new TextEncoder().encode("line 2\n"));
|
||||
await writer.close();
|
||||
})();
|
||||
|
||||
ctx.res = {
|
||||
name: "stream-example",
|
||||
files: [
|
||||
{
|
||||
name: "stream.txt",
|
||||
req: { url }
|
||||
}
|
||||
]
|
||||
};
|
||||
});
|
||||
```
|
||||
|
||||
### 3. Recover in `onError`
|
||||
|
||||
```js
|
||||
gopeed.events.onError(async function (ctx) {
|
||||
const req = ctx.task.meta.req;
|
||||
if (!req.rawUrl) return;
|
||||
if (!req.url.startsWith("gblob:")) return;
|
||||
|
||||
req.url = URL.createObjectURL(new Blob(["recovered\n"]));
|
||||
ctx.task.continue();
|
||||
});
|
||||
```
|
||||
|
||||
This creates a new `gblob` id, so the downloader restarts from byte `0`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- `gblob` is not a network protocol and does not support range requests.
|
||||
- unfinished `gblob` sources do not survive process restarts.
|
||||
- recovery after restart depends on extension logic recreating a new source.
|
||||
- the current restart behavior is full restart on source-id change, not byte-range recovery across recreated sources.
|
||||
- `WritableStream` tasks with unknown total size report size `0` until the writer closes.
|
||||
|
||||
## Summary
|
||||
|
||||
`gblob` lets Gopeed extensions treat generated content as a first-class download source.
|
||||
The registry handles source lifetime and disk spooling.
|
||||
The fetcher turns those sources into normal download tasks with predictable pause/continue behavior.
|
||||
|
||||
The most important rule is simple:
|
||||
|
||||
- same `gblob` id means resume
|
||||
- different `gblob` id means restart
|
||||
Reference in new issue
Block a user