Compare commits

...
29 Commits
Author SHA1 Message Date
silverwindandbircni d8d4a4657d fix: scope the action identity to the step executing an action (#1359)
`github.action_repository`/`github.action_ref` and their env vars now belong to the step executing an action, like on GitHub: set for remote actions, empty for `run`, `docker://` and local steps. Before, a local action nested in a remote composite resolved `$/` against the remote action instead of the workflow's repository.

Updates actionslib to v1.5.1, which accepts `$/` as the repository root action.

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1359
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-29 17:41:27 +00:00
silverwind 5fd2f82735 test: fix race in TestTreeKillCancelBeforeCaptureKillsTree (#1360)
`TestTreeKillCancelBeforeCaptureKillsTree` fails intermittently on CI, e.g. https://gitea.com/gitea/runner/actions/runs/1155856.

Dash, the Ubuntu `/bin/sh`, points a background job's stdin at `/dev/null` before applying its redirections, so `cat <&0 &` read EOF and exited at once and the liveness check raced it. The pipe is now handed over on fd 3, which dash leaves alone.

Introduced in https://gitea.com/gitea/runner/pulls/1338

Reviewed-on: https://gitea.com/gitea/runner/pulls/1360
Reviewed-by: Zettat123 <[email protected]>
2026-10-29 16:46:53 +00:00
silverwind c6567296a0 fix: stop the docker proxy probe from leaking its container (#1357)
When the probe's deadline expired during `ContainerCreate`, dockerd still created the container but the probe never got its ID, leaving it in `Created` with its image's anonymous volumes. A slow daemon leaked one per job.

The probe now runs detached with its own timeout, so the create is never cancelled and the container is always removed. Callers still fall back after their own deadline, and the runner waits for pending probes before exiting.

Fixes: https://gitea.com/gitea/runner/issues/1353
Reviewed-on: https://gitea.com/gitea/runner/pulls/1357
Reviewed-by: Zettat123 <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-29 15:53:32 +00:00
Renovate Bot e3607b576e fix(deps): update module golang.org/x/net to v0.62.0 [security] (#1356)
This PR contains the following updates:

| Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| [golang.org/x/net](https://pkg.go.dev/golang.org/x/net) | [`v0.61.0` → `v0.62.0`](https://cs.opensource.google/go/x/net/+/refs/tags/v0.61.0...refs/tags/v0.62.0) | ![age](https://developer.mend.io/api/mc/badges/age/go/golang.org%2fx%2fnet/v0.62.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/golang.org%2fx%2fnet/v0.61.0/v0.62.0?slim=true) |

---

### Avoid quadratic complexity when coalescing text nodes in html in golang.org/x/net
[CVE-2026-78661](https://nvd.nist.gov/vuln/detail/CVE-2026-78661) / [GO-2026-6860](https://pkg.go.dev/vuln/GO-2026-6860)

<details>
<summary>More information</summary>

#### Details
When coalescing adjacent text nodes in addText and fosterParent, appending to an existing TextNode's Data via string concatenation (n.Data += text) reallocated and copied the accumulated string on every token. Inputs containing many short text runs separated by ignored end tags (such as "a</x>") caused O(n^2) allocations and byte copies.

Accumulate coalesced text in per-node byte slices during parsing and materialize Data strings once at the end of parse.

#### Severity
Unknown

#### References
- [https://go.dev/cl/849365](https://go.dev/cl/849365)
- [https://go.dev/issue/82224](https://go.dev/issue/82224)
- [https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs](https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6860) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Check all ancestor namespace transitions when rendering raw text in html in golang.org/x/net
[CVE-2026-97045](https://nvd.nist.gov/vuln/detail/CVE-2026-97045) / [GO-2026-6866](https://pkg.go.dev/vuln/GO-2026-6866)

<details>
<summary>More information</summary>

#### Details
In childTextNodesAreLiteral, walking up the tree previously stopped at the nearest foreign ancestor once an HTML integration point was found. However, that integration point may itself be a descendant of an element that was foster-parented into a foreign namespace (such as <mglyph> or <malignmark> fostered under <mtext>), causing a parser to resolve the subtree in a different foreign namespace upon re-parsing the rendered output.

Continue walking the full ancestor chain and verify that every transition from a non-empty namespace occurs at an HTML integration point before emitting literal text.

#### Severity
Unknown

#### References
- [https://go.dev/cl/855121](https://go.dev/cl/855121)
- [https://go.dev/issue/82622](https://go.dev/issue/82622)
- [https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs](https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6866) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xODEuNyIsInVwZGF0ZWRJblZlciI6IjQ0LjE4MS43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://gitea.com/gitea/runner/pulls/1356
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-10-28 06:33:58 +00:00
Lunny XiaoandZettat123 ffd80101a3 enhance(kubernetes): gzip copies into job containers over exec (#1355)
The Kubernetes backend copies actions and other files into the job container with `tar` over `pods/exec`, so the data goes through the API server, which on managed clusters often sits in other zones than the nodes (each copied byte then crosses zones twice). Actions are megabytes of JavaScript bundles that gzip about fivefold:

| Action | tar | gzip |
|---|---|---|
| actions/setup-go@v5 | 8.0 MB | 1.5 MB |
| actions/cache@v4 | 11.6 MB | 2.0 MB |
| actions/setup-java@v4 | 15.3 MB | 2.5 MB |
| docker/build-push-action@v6 | 12.9 MB | 3.0 MB |

The startup probe now also reports whether the job container has `gzip`. If it does, `CopyDir` and copies of 64 KiB or more are gzipped at `BestSpeed` and extracted with `gzip -dc | tar -xf -`. Smaller copies (run scripts, file command files) stay uncompressed, where gzip would save next to nothing and cost a process in the pod. Images without `gzip` keep the plain tar stream, so the image requirements do not change.

This is independent of #1354 and gets most of its traffic saving with no new moving parts.

---
*Written by Codet (AI coding agent) on behalf of @lunny.*

---------

Co-authored-by: Zettat123 <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1355
Reviewed-by: Zettat123 <[email protected]>
2026-10-28 06:00:51 +00:00
b712c91c2d test(e2e): cover host mode, reusable workflows, scheduling, reconnect and CLI flags (#1347)
Extends the e2e suite with scenarios for host labels, local `workflow_call`, label routing, `runner.capacity`, a dropped and restored connection to Gitea, and the daemon's `--once`, `--labels` and `shutdown_timeout` behaviour.

Stacked on https://gitea.com/gitea/runner/pulls/1346, so its commit shows here until that merges. The new scenarios pass locally against gitea/gitea:latest and main-nightly.

Assisted-by: Codet:codet
Co-authored-by: Zettat123 <[email protected]>
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1347
Reviewed-by: Zettat123 <[email protected]>
2026-10-28 04:32:50 +00:00
d77e216674 fix(exec): serve the v4 artifact API so upload-artifact v4 and later work (#1341)
`exec` forwarded the toolkit's ArtifactService calls to an instance it never registered, and its random runtime token held no backend IDs, so `actions/upload-artifact` and `actions/download-artifact` v4 and later failed.

The local artifact server now serves the v4 API, including the single Put Blob uploads docker/build-push-action uses. `exec` gives each workflow its own unsigned JWT runtime token and artifact run, so artifact names conflict and carry over between runs as on GitHub. Artifacts default to a temporary directory instead of the workspace that jobs copy.

Co-authored-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1341
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-10-28 03:52:53 +00:00
Lunny Xiao a76ccd7020 test(e2e): cover the register/daemon CLI and failing jobs (#1346)
The e2e suite started the runner in-process and never exercised failing jobs. This adds a scenario that builds the binary and drives `register` then `daemon` against the Gitea fixture, and one for failed, skipped, continue-on-error and timed-out jobs.

Verified locally against gitea/gitea:main-nightly.

Assisted-by: Codet:codet
Reviewed-on: https://gitea.com/gitea/runner/pulls/1346
Reviewed-by: Zettat123 <[email protected]>
2026-10-27 21:23:59 +00:00
3dbed63f55 fix: close the action header group before the action runs (#1345)
`uses:` steps and job hooks kept their "Run …" group open until they finished. Gitea doesn't nest groups, so the action's own first `::group::` closed the header and everything before it stayed hidden there, e.g. all of goreleaser's output in https://gitea.com/gitea/cli/actions/runs/1121069/jobs/1519275. The header group now closes right after its details, as in actions/runner.

The action header also matches actions/runner's otherwise now: it is titled by the `uses:` reference, lists the resolved inputs including `action.yml` defaults without empty ones, is shown for pre stages too, and post stages start with `Post job cleanup.`.

---------

Co-authored-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1345
Reviewed-by: silverwind <[email protected]>
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-10-26 17:42:16 +00:00
silverwindandbircni dafe8d8d0c fix: keep dry runs off the docker daemon and speed up the tests (#1349)
Without a docker daemon six `act/runner` tests failed instead of passing. A dry run still created and removed job networks, the job teardown dropped the dry-run flag along with every other context value, and the container mocks asked the daemon for the runner arch.

This also speeds up the tests and CI without dropping assertions. Most of the time went to git and docker round trips, which are now pinned, batched or coalesced, and the dind and e2e steps cache or overlap more of their setup. On CI, `act/runner` went from 77s to 55s and `test-dind` from 108s to 55s. Locally, `act/runner` without docker went from 53s to 7s.

Split out of https://gitea.com/gitea/runner/pulls/1348, which adds Windows CI on top. The go cache changes moved to https://gitea.com/gitea/runner/pulls/1350.

Written by Claude Code.

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1349
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-26 17:25:17 +00:00
Zettat123 8ae48ae033 fix: keep the shared action cache as extracted, re-fetch damaged trees (#1352)
The toolkit patch rewrote bundles in place in the shared action cache. Since https://gitea.com/gitea/runner/pulls/1316 trees are never refreshed, so a bundle left truncated failed every later job using it.

Patched bundles now only overwrite the job container's copy, so the runner no longer writes to an extracted tree.

Extraction records each file's size beside the tree, and a tree that no longer matches is fetched again, offline mode included, as a missing one always was. This also catches damage from outside the runner, such as storage losing a write, for a stat per file, a fraction of the copy each stage already makes. Trees from older runners have no record and download once more, except in offline mode, which keeps using them as before.

Generated with Claude Code.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1352
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: bircni <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-10-26 17:20:53 +00:00
Renovate Bot 8dd3776014 chore(deps): update dependencies (#1351)
Co-authored-by: Renovate Bot <[email protected]>
2026-10-26 07:06:33 +00:00
silverwind c6ca020281 ci: implement gitea's cache-seeder (#1350)
Like Gitea's CI, only a `cache-seeder` workflow on main saves the go caches, all other jobs only restore them. Caches are scoped per repository, so until now any pull request, forks included, could write a cache that other runs restore. The seeder compiles the tests without running them, so no test result is ever cached.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1350
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-24 17:04:51 +00:00
silverwind fe2a66dc25 fix: align runner behaviours (#1338)
Contains a number of small bug fixes where jobs behaved differently from GitHub or failed silently.

1. Hooks fail on a refused `::set-env::` or `::add-path::` and no longer inherit the last step's identity
1. Docker build, exec and wait errors are returned instead of swallowed, token-only registry credentials work, logs of `--tty` services decode
1. Env files split lines like GitHub on each OS, so a carriage return is kept on Linux and CRLF ends a line on Windows, empty heredoc names or delimiters are rejected
1. Post steps run after their pre step even when main was skipped, composites see undeclared inputs, `runs.args` wins without `with.args` being parsed
1. Docker actions skip evaluating defaults of inputs given in `with` and read `entryPoint` and `args` in any case, a lowercase `dockerfile` is found, undeclared inputs get GitHub's warning
1. A user-set `CI` is kept, `GITHUB_REPOSITORY_ID` and the other missing `GITHUB_*` variables are exported
1. `timeout-minutes` can read the step's own `env`, a step-level `ACTIONS_ALLOW_UNSECURE_COMMANDS: false` beats a job-level `true` while the runner's own opt-in still wins, `pwsh` quotes the script path
1. A step `env:` given as one expression such as `${{ fromJSON(matrix.env) }}` is applied, `hashFiles` fails with GitHub's messages on script errors and after 120 seconds, a job image without node still gets an empty hash
1. A step whose command files cannot be reset fails instead of importing stale ones, a timeout between steps fails the job instead of reporting success
1. One named volume can be mounted at several paths, host-mode PTY steps keep their last output when they fail, host scratch directories are removed when job setup fails
1. Cancelling a step right after it starts kills its whole process tree
1. Cache prefix restores pick the newest entry created within the same second, also on S3, cache commits with a wrong size are rejected
1. Artifact chunks are written at their offset, download URLs are escaped, listing a run without artifacts no longer crashes
1. Poll backoff reaches `fetch_interval_max`, proxy passwords are masked in more forms, secrets are also masked in step names and in their quoted form in the runner log, a failed final report is logged
1. On Windows hosts a differently cased unsecure-commands opt-in or proxy variable counts, and an early cancel no longer leaks a job object handle
1. Interactive `register` prompts for a missing token and stops on SIGTERM, `exec --list --detect-event` no longer panics without events, `config set` works on `cache.s3` keys

Reviewed-on: https://gitea.com/gitea/runner/pulls/1338
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-23 09:18:24 +00:00
Zettat123 c719a88f36 fix(k8s): retry the startup exec until the kubelet accepts it (#1337)
A new node can run pods before its kubelet has a serving certificate, so the first exec into the job container fails with `error dialing backend: remote error: tls: internal error` and the job fails during setup.

That exec only reads the arch and PATH, so it now retries for up to a minute unless the API server rejects it (4xx) or the command ran.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1337
Reviewed-by: bircni <[email protected]>
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-10-22 20:59:48 +00:00
silverwind d882311ec6 refactor: remove dead code and consolidate tests (#1335)
Removes dead code and redundant logic and consolidates tests, so there is less to read and maintain. No behavior change.

1. **act/runner**: inline single-use wrappers, compile regexes once instead of per call, embed the trampoline as a string, drop the unused legacy result of `parseActionCommand`
2. **act/container**: `sanitizeConfig` and `mergeContainerConfigs` stop returning the configs they already mutate in place, drop never-set option fields and the unused `user` parameter of `ExecWithCmdLine`
3. **act/common and cache server**: drop the unused `Trace`/`Tracef` loggers and the short SHA of `FindGitRevision`, `RegisterJob` releases through `RevokeJob`
4. **daemon, poller and reporter**: one shutdown wait for both poll modes, drop an unreachable fetch timeout branch, guard the reporter's last report time with its existing lock instead of an atomic
5. **build**: drop unused Makefile and release workflow variables, collapse the goreleaser ignore list to the same build targets
6. **tests**: fold near-duplicate tests into tables with shared setup

Throughout, hand-rolled logic gives way to `cmp.Or`, `strings.Cut`, `slices` and `min`/`max`.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1335
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-22 19:41:10 +00:00
silverwind 923a0bfb59 refactor: remove bolthold and gofakes3 deps (#1336)
1. Replace bolthold with direct bbolt access in the cache server. bolthold has not changed since March 2024 and kept `encoding/gob` in the binary although the store was already JSON. The on-disk layout stays the same, so existing caches keep working, and the index buckets bolthold left behind are dropped on the first sweep. Queries now run against an in-memory copy of the entries that is only re-read after another process wrote to the store, so lookups and sweeps no longer decode the whole store.

2. Replace gofakes3 with a small in-memory S3 fake in the one test that used it, which removes aws-sdk-go-v2, afero, mgo and x/tools from the module graph.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1336
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-22 16:28:49 +00:00
silverwind 6f0d4f1b48 refactor: migrate to connect-go v2 (#1325)
Migrates the runner to `connectrpc.com/connect/v2` by pinning actionslib v1.5.0, which carries https://gitea.com/gitea/actionslib/pulls/28. The wire behaviour towards Gitea is unchanged and works against Gitea servers on connect v1 and v2. The client keeps v1's unlimited response size, and the error codes and messages the runner checks stay the same.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1325
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-21 16:10:58 +00:00
silverwindandbircni 430f7bb2cf fix: handle workflow commands like GitHub's runner (#1333)
Commands the runner applies itself, like the `::set-output::` older actions still print, showed up raw in the job log. The reporter also re-parsed every line with one job-wide stop-commands and echo state, while act keeps one command handler per step, stage, composite child and hook, as GitHub does. A `::stop-commands::` left active in one of them kept the reporter suppressed for the others, so their `::add-mask::` values were never registered, and post steps logged command lines twice.

- act's command handler decides what reaches the log, as GitHub's runner does: applied commands only under `ACTIONS_STEP_DEBUG` or `::echo::on`, `::stop-commands::` and its resume line shown with long tokens masked
- Masks go straight to the reporter, which no longer parses command lines
- One parser follows GitHub's rules: leading whitespace, `##[...]` anywhere in a line, registered commands only
- Service logs honour only `::add-mask::`

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1333
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-21 16:03:40 +00:00
silverwind befb1b4bb9 fix: improve the actions toolkit patching (#1328)
Patch the scripts beside each node action entrypoint too, so code-split bundles like rust-cache v2.9.2, setup-java v6 and setup-deno v2 reach the v2 cache API. Entrypoints may point anywhere in the action's repository, as `gradle/actions/setup-gradle` does with `../dist`.

The `.LOCALHOST` match is now anchored on the `.GHE.COM` test before it, so another library's uppercase localhost check is never rewritten.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1328
Reviewed-by: Lunny Xiao <[email protected]>
2026-10-21 05:26:32 +00:00
silverwind 1a7f09224f perf: speed up job setup, action preparation and the cache server (#1327)
- Look up containers by name instead of listing all of them
- On Linux, cache the docker proxy discovery instead of starting a probe container per job
- Download a job's actions concurrently and share identical fetches between concurrent jobs
- Cache `runner.arch` per run and poll service health every 250ms
- Cache server: queue on an in-process lock instead of polling the database file lock, upload S3 objects up to 64MiB in one request and larger ones in 4 concurrent parts

| `exec`, median of 5 | Before | After |
|---|---:|---:|
| 1 step | 1.70s | 0.87s |
| 10 steps | 2.84s | 1.97s |
| Service + step | 3.78s | 2.18s |
| checkout + Node action | 3.64s | 2.52s |

Reviewed-on: https://gitea.com/gitea/runner/pulls/1327
Reviewed-by: Zettat123 <[email protected]>
2026-10-21 05:12:54 +00:00
bircni af2b3f6df5 fix(deps): update go toolchain directive to v1.27.3 (#1332)
Go 1.27.3 includes a crypto/x509 security fix that distrusts a set of intermediate certificates (https://github.com/golang/go/issues/82345), so release builds need the new toolchain.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1332
Reviewed-by: TheFox0x7 <[email protected]>
2026-10-20 19:31:44 +00:00
silverwindandbircni 785313cc21 build: drop the xgo release targets (#1329)
Releases are built by goreleaser with `CGO_ENABLED=0`, so the xgo `release*` Makefile targets and their gxz compression are unused. Gitea dropped xgo the same way in https://github.com/go-gitea/gitea/pull/37828.

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1329
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-20 16:21:47 +00:00
silverwind 41f1e08bec chore(deps): bump the pins Renovate does not update (#1330)
Renovate gets no release timestamps for Docker Hub images with more than 1000 tags (https://github.com/renovatebot/renovate/issues/46925) or for quay.io, so the 5-day minimum release age holds them back forever. The e2e workflows, the Kubernetes examples and the toolkit test pins are outside its managers, so these are bumped by hand.

The builder follows the current Alpine instead of the frozen 3.23.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1330
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-20 16:14:16 +00:00
silverwindandbircni 4158e05dae refactor: drop logrus and other direct dependencies (#1319)
After https://gitea.com/gitea/runner/pulls/1304, logrus stayed only to forward logs from docker/cli and moby/go-archive. The runner used a small part of those: volume parsing, the build context tar, the `ssh://` connection and registry credentials. Own code that matches them replaces them, so logrus leaves the module graph. docker/cli stays for its logrus-free `opts` and `connhelper/ssh` packages.

Other dependencies that the standard library or existing ones cover go too: mergo, httprouter (replaced by `http.ServeMux`), retry-go, go-isatty, go-selinux, container-device-interface, gotest.tools and go-cmp (replaced by testify). The binary shrinks by 5%.

Behaviour differences:

1. An unreadable file in a build context is skipped with a warning, where go-archive left a truncated tar.
2. Build contexts no longer carry `security.capability` xattrs.
3. The cache and artifact servers answer HEAD on GET routes, and no longer redirect trailing slashes or answer OPTIONS.

*Written by Claude.*

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1319
Reviewed-by: bircni <[email protected]>
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-18 18:57:08 +00:00
silverwind e356c2ed85 chore(config): update stale go-yaml wrapping comment (#1324)
https://github.com/yaml/go-yaml/pull/441 fixed the wrapping bug this comment waited for, so reword it. `WithLineWidth(-1)` stays, `config edit` re-encodes the whole file and wrapping would reflow every long value.

Gitea: https://github.com/go-gitea/gitea/pull/39844

This PR was written with Claude.
Reviewed-on: https://gitea.com/gitea/runner/pulls/1324
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-18 13:15:28 +00:00
silverwindandbircni ed940db4f0 fix: don't fail host steps over a leftover holding their output (#1323)
A successful host step whose background child kept its output open failed with `exec: WaitDelay expired before I/O complete`. Like GitHub's runner (`ProcessExitedHandler` in https://github.com/actions/runner/blob/main/src/Runner.Sdk/ProcessInvoker.cs), the step now succeeds after the drain timeout.

The leftover is killed only on Windows, as GitHub does there. A step whose timeout or cancellation hits during the drain still fails.

Fixes: https://gitea.com/gitea/runner/issues/1290
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1323
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-17 09:03:48 +00:00
Zettat123andsilverwind 755af87095 enhance(k8s): reach Kubernetes services by their id when it is a DNS subdomain (#1320)
On Kubernetes a service is reached by its id made a DNS label, so some service ids might become unreachable. For example, the `devstoreaccount1.azurite.local` service of Gitea's [MSSQL tests](https://github.com/go-gitea/gitea/blob/2fe74d3cdd5184a7e22e32e47772caa78d077489/.github/workflows/pull-db-tests.yml#L229) is only in the pod's hosts as `devstoreaccount1-azurite-local`, while the tests connect to the id as written. Container names must be DNS labels, but `hostAliases` accepts DNS subdomains, so an id that is one is now added there too, and to `no_proxy`, as on Docker where a service is reached by its id.

The warning for a service not reachable by its id now also covers ids that differ only in case, as Kubernetes hostnames are lowercase.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1320
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: bircni <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-10-17 04:09:17 +00:00
silverwind b9aa47d1a5 fix(deps): bump go-yaml to fix block and folded scalar round-trips (#1321)
Bumps go-yaml to its main branch and drops the workaround it makes obsolete, matching https://github.com/go-gitea/gitea/pull/39787.

- go-yaml main keeps block scalar indentation indicators (https://github.com/yaml/go-yaml/pull/414), so values with a leading newline no longer need quoting
- `config edit` no longer wraps lines, because go-yaml still wraps more-indented lines of folded scalars and turns a space into a newline

Reviewed-on: https://gitea.com/gitea/runner/pulls/1321
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-16 21:34:11 +00:00
217 changed files with 9874 additions and 9305 deletions

No files matched your search

+5
View File
@@ -45,8 +45,13 @@ cpu.out
/bin /bin
/dist /dist
/.tmp
/.git/hooks
/.git/logs
/.git/rr-cache
/.env /.env
/.runner /.runner
/.runner.lock
/config.yaml /config.yaml
/Dockerfile /Dockerfile
.DS_Store .DS_Store
+29
View File
@@ -0,0 +1,29 @@
name: go-setup
description: Set up go and restore its caches, which only the cache-seeder workflow saves
runs:
using: composite
steps:
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
id: setup-go
with:
go-version-file: go.mod
check-latest: true
cache: false
- if: ${{ github.workflow == 'cache-seeder' }}
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
- if: ${{ github.workflow != 'cache-seeder' }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
restore-keys: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-
+26
View File
@@ -0,0 +1,26 @@
name: cache-seeder
on:
workflow_dispatch:
push:
branches:
- main
paths:
- go.mod
- go.sum
- .golangci.yml
- Makefile
- .gitea/actions/go-setup/action.yml
- .gitea/workflows/cache-seeder.yml
jobs:
seed:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.gitea/actions/go-setup
- run: make deps-tools lint build
- name: compile the test binaries without running them
run: |
make test GOTEST_FLAGS='-race -exec true'
go test -tags e2e -exec true ./e2e/...
+2 -3
View File
@@ -11,7 +11,6 @@ on:
env: env:
DOCKER_ORG: gitea DOCKER_ORG: gitea
DOCKER_LATEST: nightly
jobs: jobs:
goreleaser: goreleaser:
@@ -36,7 +35,7 @@ jobs:
go-version-file: "go.mod" go-version-file: "go.mod"
check-latest: true check-latest: true
- name: goreleaser - name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7 uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with: with:
distribution: goreleaser-pro distribution: goreleaser-pro
args: release --nightly args: release --nightly
@@ -80,7 +79,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub - name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with: with:
username: ${{ secrets.DOCKER_USERNAME }} username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }} password: ${{ secrets.DOCKER_PASSWORD }}
+2 -3
View File
@@ -35,7 +35,7 @@ jobs:
passphrase: ${{ secrets.PASSPHRASE }} passphrase: ${{ secrets.PASSPHRASE }}
fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0 fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0
- name: goreleaser - name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7 uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with: with:
distribution: goreleaser-pro distribution: goreleaser-pro
args: release args: release
@@ -69,7 +69,6 @@ jobs:
image: catthehacker/ubuntu:act-latest image: catthehacker/ubuntu:act-latest
env: env:
DOCKER_ORG: gitea DOCKER_ORG: gitea
DOCKER_LATEST: latest
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
@@ -83,7 +82,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub - name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with: with:
username: ${{ secrets.DOCKER_USERNAME }} username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }} password: ${{ secrets.DOCKER_PASSWORD }}
+11 -15
View File
@@ -17,25 +17,24 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 - uses: ./.gitea/actions/go-setup
with:
go-version-file: 'go.mod'
check-latest: true
- name: prepare anonymous docker config - name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json" run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
# Pre-pull act/runner's two largest base images so a slow pull can't dominate `make test`; # Pre-pull act/runner's two largest base images so a slow pull can't dominate `make test`;
# the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host # the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host
# daemon retains them between runs, so this is usually a fast manifest re-check. # daemon retains them between runs, and their digest pins skip the registry when it has them.
- name: pre-pull test images - name: pre-pull test images
env: env:
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:d6aa754f16b3197301076f047b5def2f02ea1dbbc2ca920407d46d7ec7f87b20 # renovate: datasource=docker TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:55fb5f5a527ea2c5c4a95f963e48ef90aa38c7b346054fdc8ee9957a2b55d205 # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:df221db836e1754089190208cee7eeda94f233197056426eda74a43ab1abeac2 # renovate: datasource=docker TEST_SERVICE_IMAGE: nginx:alpine@sha256:a3554c0d0b48177b1413c26e01f695334428020d9df01243fbae3e11cfd17417 # renovate: datasource=docker
run: | run: |
for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do
for attempt in 1 2 3; do if [ "$(docker image inspect --format '{{.Os}}/{{.Architecture}}' "$image" 2>/dev/null)" != linux/amd64 ]; then
docker pull "$image" && break for attempt in 1 2 3; do
[ "$attempt" = 3 ] || sleep 5 docker pull "$image" && break
done [ "$attempt" = 3 ] || sleep 5
done
fi
docker tag "$image" "${image%@*}" docker tag "$image" "${image%@*}"
done done
- name: lint - name: lint
@@ -68,10 +67,7 @@ jobs:
- gitea/gitea:main-nightly - gitea/gitea:main-nightly
steps: steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 - uses: ./.gitea/actions/go-setup
with:
go-version-file: 'go.mod'
check-latest: true
- name: prepare anonymous docker config - name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json" run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
- name: e2e compatibility - name: e2e compatibility
-2
View File
@@ -44,8 +44,6 @@ linters:
- $all - $all
- "!**/act/common/logging/**" - "!**/act/common/logging/**"
deny: deny:
- pkg: github.com/sirupsen/logrus
desc: log through gitea.com/gitea/runner/act/common/logging
- pkg: log/slog - pkg: log/slog
desc: log through gitea.com/gitea/runner/act/common/logging desc: log through gitea.com/gitea/runner/act/common/logging
nolintlint: nolintlint:
-20
View File
@@ -28,38 +28,18 @@ builds:
ignore: ignore:
- goos: darwin - goos: darwin
goarch: arm goarch: arm
- goos: darwin
goarch: ppc64le
- goos: darwin - goos: darwin
goarch: s390x goarch: s390x
- goos: windows
goarch: ppc64le
- goos: windows - goos: windows
goarch: s390x goarch: s390x
- goos: windows - goos: windows
goarch: arm goarch: arm
goarm: "5"
- goos: windows
goarch: arm
goarm: "6"
- goos: windows
goarch: arm
goarm: "7"
- goos: windows - goos: windows
goarch: arm64 goarch: arm64
- goos: freebsd
goarch: ppc64le
- goos: freebsd - goos: freebsd
goarch: s390x goarch: s390x
- goos: freebsd - goos: freebsd
goarch: arm goarch: arm
goarm: "5"
- goos: freebsd
goarch: arm
goarm: "6"
- goos: freebsd
goarch: arm
goarm: "7"
- goos: freebsd - goos: freebsd
goarch: arm64 goarch: arm64
flags: flags:
+10 -8
View File
@@ -28,12 +28,12 @@ depending on the prefix:
- **No prefix?** Do not escape, and split multi-line values into one row each. - **No prefix?** Do not escape, and split multi-line values into one row each.
- **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the - **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the
encoded forms too. encoded forms too.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the reporter - Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the command
decodes exactly those two when folding a location into an annotation. handler folds an annotation's location into its decoded message and escapes the result again.
## Logging ## Logging
Log through `act/common/logging`; the linter rejects importing `log/slog` or logrus elsewhere. Log through `act/common/logging`; the linter rejects importing `log/slog` elsewhere.
A job's logger comes from `common.Logger(ctx)`; code with no job uses the package functions A job's logger comes from `common.Logger(ctx)`; code with no job uses the package functions
(`logging.Infof`, `logging.With`). Both take printf-style messages, and `With("key", value)` adds (`logging.Infof`, `logging.With`). Both take printf-style messages, and `With("key", value)` adds
fields. fields.
@@ -43,11 +43,8 @@ fields.
- The job logger's output can reach the job log workflow authors see, the process-wide logger's - The job logger's output can reach the job log workflow authors see, the process-wide logger's
stays on the runner host, so keep operator-only detail (host paths, internal addresses) off the stays on the runner host, so keep operator-only detail (host paths, internal addresses) off the
job logger. job logger.
- Libraries that log through logrus's or slog's global logger are forwarded to the process-wide - Libraries that log through slog's global logger are forwarded to the process-wide one, so they
one, so they follow its level and masking too. follow its level and masking too. Avoid dependencies that log through logrus, nothing forwards it.
- That forwarding only works while logrus's global logger is left as `logging` configures it. A
dependency that calls `logrus.SetOutput` or `SetLevel` breaks it, so check
`logging.LogrusBridgeIntact()` when adding or upgrading one.
- Tests read what was logged with `logtest.NewLogger()`, or `logtest.SetDefault(t)` for the - Tests read what was logged with `logtest.NewLogger()`, or `logtest.SetDefault(t)` for the
package functions. package functions.
@@ -62,5 +59,10 @@ Each lives in `internal/pkg/action/<name>` and is registered in `builtinActions`
It requires Docker and is excluded from `make test`. CI runs stable and nightly variants in It requires Docker and is excluded from `make test`. CI runs stable and nightly variants in
parallel. parallel.
The `cli_*` scenarios build the binary and drive `register` and `daemon` as a user would, including `--once`,
`--labels` and shutdown with a running job. `failure_paths` covers failed, skipped, tolerated and timed-out jobs.
`label_routing` and `capacity` cover scheduling, `reconnect` drops the connection to Gitea through a proxy, and
`host_mode` and `reusable_workflow` cover host labels and `workflow_call`.
The suite shares one Gitea and regular runner. Cache and ephemeral scenarios use isolated The suite shares one Gitea and regular runner. Cache and ephemeral scenarios use isolated
repository runners. A run path is `<workflow>@<ref>` and a log row is `<timestamp>Z <payload>`. repository runners. A run path is `<workflow>@<ref>` and a log row is `<timestamp>Z <payload>`.
+6 -4
View File
@@ -1,7 +1,7 @@
### BUILDER STAGE ### BUILDER STAGE
# #
# #
FROM golang:1.27-alpine3.23 AS builder FROM golang:1.27-alpine AS builder
# Do not remove `git` here, it is required for getting runner version when executing `make build` # Do not remove `git` here, it is required for getting runner version when executing `make build`
RUN apk add --no-cache make git RUN apk add --no-cache make git
@@ -12,12 +12,14 @@ ENV GOPROXY=${GOPROXY:-}
COPY . /opt/src/runner COPY . /opt/src/runner
WORKDIR /opt/src/runner WORKDIR /opt/src/runner
RUN make clean && make build RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
make clean && make build
### DIND VARIANT ### DIND VARIANT
# #
# #
FROM docker:29.7.2-dind AS dind FROM docker:29.9.1-dind AS dind
ARG VERSION=dev ARG VERSION=dev
@@ -39,7 +41,7 @@ ENTRYPOINT ["s6-svscan","/etc/s6"]
### DIND-ROOTLESS VARIANT ### DIND-ROOTLESS VARIANT
# #
# #
FROM docker:29.7.2-dind-rootless AS dind-rootless FROM docker:29.9.1-dind-rootless AS dind-rootless
ARG VERSION=dev ARG VERSION=dev
+6 -62
View File
@@ -1,31 +1,22 @@
DIST := dist DIST := dist
EXECUTABLE := gitea-runner EXECUTABLE := gitea-runner
DIST_DIRS := $(DIST)/binaries $(DIST)/release
GO ?= go GO ?= go
SHASUM ?= shasum -a 256
HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" ) HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" )
XGO_PACKAGE ?= src.techknowlogick.com/[email protected] # renovate: datasource=go
XGO_VERSION := go-1.27.x
GXZ_PACKAGE ?= github.com/ulikunitz/xz/cmd/[email protected] # renovate: datasource=go
PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.4-immutable@sha256:90405e3c6a86f33c5a6f8dbdfa8e02a98143a8ba4337b84bdaec5336d2ef008a # renovate: datasource=docker PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.9-immutable@sha256:87f42f37fa85b421664c30b8f5f9cfc23920057f5e5fcadf0b6f68d9edc6d415 # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:0e62e56b382ebf485bff1e51a05cbe27c708545a2e91425467fd31eb3e249521 # renovate: datasource=docker E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:8856b200466e9a5af9cc0d4de4ca96f73547833bcdddf9a567189ff772c4d372 # renovate: datasource=docker
SERVICE_IMAGE ?= nginx:1.31.4-alpine@sha256:db35bfc6b2951e7f8a72db5db120288c127ffaeeb4a6d4b95a26fead017d5913 # renovate: datasource=docker SERVICE_IMAGE ?= nginx:1.31.7-alpine@sha256:892bb2152464f0c5d8972593a5f5afaf8054552580b699d95f8ff4af08c2d33d # renovate: datasource=docker
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
E2E_K3S_IMAGE ?= rancher/k3s:v1.36.4-k3s1@sha256:edad48e12bf81c3a09ac1c05c0c0ffaaa22145980b989d6fae84543a76b83657 # renovate: datasource=docker E2E_K3S_IMAGE ?= rancher/k3s:v1.37.1-k3s1@sha256:ca7f37d993d82ef0dcdcfecb2e0e2618ea541dbaffc620c8cedebe01a82acd0d # renovate: datasource=docker
LINUX_ARCHS ?= linux/amd64,linux/arm64
DARWIN_ARCHS ?= darwin-12/amd64,darwin-12/arm64
WINDOWS_ARCHS ?= windows/amd64
GOFILES := $(shell find . -type f -name "*.go" -o -name "go.mod" ! -name "generated.*") GOFILES := $(shell find . -type f -name "*.go" -o -name "go.mod" ! -name "generated.*")
DOCKER_IMAGE ?= gitea/runner DOCKER_IMAGE ?= gitea/runner
DOCKER_TAG ?= nightly DOCKER_TAG ?= nightly
DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG) DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)
DOCKER_ROOTLESS_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)-dind-rootless
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]4.0 # renovate: datasource=go GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]5.0 # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.8.0 # renovate: datasource=go GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.9.0 # renovate: datasource=go
GOTEST_FLAGS ?= -race -timeout 20m -parallel 8 GOTEST_FLAGS ?= -race -timeout 20m -parallel 8
@@ -38,22 +29,16 @@ endif
ifeq ($(HAS_GO), GO) ifeq ($(HAS_GO), GO)
GOPATH ?= $(shell $(GO) env GOPATH) GOPATH ?= $(shell $(GO) env GOPATH)
export PATH := $(GOPATH)/bin:$(PATH) export PATH := $(GOPATH)/bin:$(PATH)
CGO_EXTRA_CFLAGS := -DSQLITE_MAX_VARIABLE_NUMBER=32766
CGO_CFLAGS ?= $(shell $(GO) env CGO_CFLAGS) $(CGO_EXTRA_CFLAGS)
endif endif
ifeq ($(OS), Windows_NT) ifeq ($(OS), Windows_NT)
GOFLAGS := -v -buildmode=exe GOFLAGS := -v -buildmode=exe
EXECUTABLE ?= $(EXECUTABLE).exe
GO_ENV_WINDOWS := set GOOS=windows&& GO_ENV_WINDOWS := set GOOS=windows&&
else ifeq ($(OS), Windows) else ifeq ($(OS), Windows)
GOFLAGS := -v -buildmode=exe GOFLAGS := -v -buildmode=exe
EXECUTABLE ?= $(EXECUTABLE).exe
GO_ENV_WINDOWS := set GOOS=windows&& GO_ENV_WINDOWS := set GOOS=windows&&
else else
GOFLAGS := -v GOFLAGS := -v
EXECUTABLE ?= $(EXECUTABLE)
GO_ENV_WINDOWS := GOOS=windows GO_ENV_WINDOWS := GOOS=windows
endif endif
@@ -108,8 +93,6 @@ fmt-check: ## check the Go code formatting
.PHONY: deps-tools .PHONY: deps-tools
deps-tools: ## install tool dependencies deps-tools: ## install tool dependencies
$(GO) install $(GOLANGCI_LINT_PACKAGE) & \ $(GO) install $(GOLANGCI_LINT_PACKAGE) & \
$(GO) install $(GXZ_PACKAGE) & \
$(GO) install $(XGO_PACKAGE) & \
$(GO) install $(GOVULNCHECK_PACKAGE) & \ $(GO) install $(GOVULNCHECK_PACKAGE) & \
wait wait
@@ -187,45 +170,6 @@ $(EXECUTABLE): $(GOFILES)
deps-backend: ## install backend dependencies deps-backend: ## install backend dependencies
$(GO) mod download $(GO) mod download
.PHONY: release
release: release-windows release-linux release-darwin release-copy release-compress release-check ## build release artifacts
$(DIST_DIRS):
mkdir -p $(DIST_DIRS)
.PHONY: release-windows
release-windows: | $(DIST_DIRS)
CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -buildmode exe -dest $(DIST)/binaries -tags 'netgo osusergo $(TAGS)' -ldflags '-s -w -linkmode external -extldflags "-static" $(LDFLAGS)' -targets '$(WINDOWS_ARCHS)' -out $(EXECUTABLE)-$(VERSION) .
ifeq ($(CI),true)
cp -r /build/* $(DIST)/binaries/
endif
.PHONY: release-linux
release-linux: | $(DIST_DIRS)
CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -dest $(DIST)/binaries -tags 'netgo osusergo $(TAGS)' -ldflags '-s -w -linkmode external -extldflags "-static" $(LDFLAGS)' -targets '$(LINUX_ARCHS)' -out $(EXECUTABLE)-$(VERSION) .
ifeq ($(CI),true)
cp -r /build/* $(DIST)/binaries/
endif
.PHONY: release-darwin
release-darwin: | $(DIST_DIRS)
CGO_CFLAGS="$(CGO_CFLAGS)" $(GO) run $(XGO_PACKAGE) -go $(XGO_VERSION) -dest $(DIST)/binaries -tags 'netgo osusergo $(TAGS)' -ldflags '-s -w $(LDFLAGS)' -targets '$(DARWIN_ARCHS)' -out $(EXECUTABLE)-$(VERSION) .
ifeq ($(CI),true)
cp -r /build/* $(DIST)/binaries/
endif
.PHONY: release-copy
release-copy: | $(DIST_DIRS)
cd $(DIST); for file in `find . -type f -name "*"`; do cp $${file} ./release/; done;
.PHONY: release-check
release-check: | $(DIST_DIRS)
cd $(DIST)/release/; for file in `find . -type f -name "*"`; do echo "checksumming $${file}" && $(SHASUM) `echo $${file} | sed 's/^..//'` > $${file}.sha256; done;
.PHONY: release-compress
release-compress: | $(DIST_DIRS)
cd $(DIST)/release/; for file in `find . -type f -name "*"`; do echo "compressing $${file}" && $(GO) run $(GXZ_PACKAGE) -k -9 $${file}; done;
.PHONY: docker .PHONY: docker
docker: ## build the docker image docker: ## build the docker image
if ! docker buildx version >/dev/null 2>&1; then \ if ! docker buildx version >/dev/null 2>&1; then \
+1 -1
View File
@@ -242,7 +242,7 @@ A job's `services` are started before its steps run. When a service's image or i
```yaml ```yaml
services: services:
postgres: postgres:
image: postgres:17 image: postgres:18
options: >- options: >-
--health-cmd pg_isready --health-cmd pg_isready
--health-interval 5s --health-interval 5s
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+19 -20
View File
@@ -15,8 +15,6 @@ import (
"strconv" "strconv"
"strings" "strings"
"time" "time"
"github.com/julienschmidt/httprouter"
) )
// The cache service v2 API. A client on this version talks twirp to // The cache service v2 API. A client on this version talks twirp to
@@ -44,16 +42,16 @@ const (
twirpUnauthenticated = "unauthenticated" twirpUnauthenticated = "unauthenticated"
) )
func (h *Handler) registerV2Routes(router *httprouter.Router) { func (h *Handler) registerV2Routes(router *http.ServeMux) {
router.POST(cacheServiceV2Path+"/CreateCacheEntry", h.bearerAuth(h.v2CreateCacheEntry)) router.HandleFunc("POST "+cacheServiceV2Path+"/CreateCacheEntry", h.bearerAuth(h.v2CreateCacheEntry))
router.POST(cacheServiceV2Path+"/FinalizeCacheEntryUpload", h.bearerAuth(h.v2FinalizeCacheEntryUpload)) router.HandleFunc("POST "+cacheServiceV2Path+"/FinalizeCacheEntryUpload", h.bearerAuth(h.v2FinalizeCacheEntryUpload))
router.POST(cacheServiceV2Path+"/GetCacheEntryDownloadURL", h.bearerAuth(h.v2GetCacheEntryDownloadURL)) router.HandleFunc("POST "+cacheServiceV2Path+"/GetCacheEntryDownloadURL", h.bearerAuth(h.v2GetCacheEntryDownloadURL))
router.PUT(blobPath+"/:id", h.signedAuth(blobUploadPurpose, h.v2UploadBlob)) router.HandleFunc("PUT "+blobPath+"/{id}", h.signedAuth(blobUploadPurpose, h.v2UploadBlob))
} }
// An entry that already exists is reported as not ok, which is how the client learns to skip // An entry that already exists is reported as not ok, which is how the client learns to skip
// the upload. // the upload.
func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context()) cred := credFromContext(r.Context())
req, err := decodeTwirpRequest[v2CreateRequest](r) req, err := decodeTwirpRequest[v2CreateRequest](r)
if err != nil { if err != nil {
@@ -102,15 +100,16 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request, _ h
return return
} }
now := time.Now().Unix() now := time.Now()
cache := &Cache{ cache := &Cache{
Repo: cred.Repo, Repo: cred.Repo,
Key: req.Key, Key: req.Key,
Version: req.Version, Version: req.Version,
Owner: owner, Owner: owner,
Size: -1, // the size is only known at finalize time Size: -1, // the size is only known at finalize time
CreatedAt: now, CreatedAt: now.Unix(),
UsedAt: now, CreatedAtNanos: now.Nanosecond(),
UsedAt: now.Unix(),
} }
if err := h.insertCache(db, cache); err != nil { if err := h.insertCache(db, cache); err != nil {
h.twirpError(w, r, twirpInternal, err) h.twirpError(w, r, twirpInternal, err)
@@ -123,7 +122,7 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request, _ h
}) })
} }
func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context()) cred := credFromContext(r.Context())
req, err := decodeTwirpRequest[v2FinalizeRequest](r) req, err := decodeTwirpRequest[v2FinalizeRequest](r)
if err != nil { if err != nil {
@@ -172,7 +171,7 @@ func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Requ
}) })
} }
func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Request) {
cred := credFromContext(r.Context()) cred := credFromContext(r.Context())
req, err := decodeTwirpRequest[v2DownloadRequest](r) req, err := decodeTwirpRequest[v2DownloadRequest](r)
if err != nil { if err != nil {
@@ -207,8 +206,8 @@ func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Requ
// The archive arrives over the subset of the Azure blob API the toolkit uses: a small // The archive arrives over the subset of the Azure blob API the toolkit uses: a small
// cache is a single PUT, a large one is staged as blocks that a final block list puts // cache is a single PUT, a large one is staged as blocks that a final block list puts
// in order. // in order.
func (h *Handler) v2UploadBlob(w http.ResponseWriter, r *http.Request, params httprouter.Params) { func (h *Handler) v2UploadBlob(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseUint(params.ByName("id"), 10, 64) id, err := strconv.ParseUint(r.PathValue("id"), 10, 64)
if err != nil { if err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err) h.responseJSON(w, r, http.StatusBadRequest, err)
return return
+9 -36
View File
@@ -19,8 +19,6 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
// v2Call posts a twirp request to the cache service and returns the decoded response.
// Field names are the proto ones, which is what the toolkit's client sends.
func v2Call(t *testing.T, handler *Handler, client *http.Client, method string, request any) map[string]any { func v2Call(t *testing.T, handler *Handler, client *http.Client, method string, request any) map[string]any {
t.Helper() t.Helper()
@@ -37,7 +35,6 @@ func v2Call(t *testing.T, handler *Handler, client *http.Client, method string,
return got return got
} }
// putBlob uploads to a signed URL and returns the status, so a test can assert a refusal.
func putBlob(t *testing.T, url string, content []byte) int { func putBlob(t *testing.T, url string, content []byte) int {
t.Helper() t.Helper()
@@ -47,8 +44,6 @@ func putBlob(t *testing.T, url string, content []byte) int {
require.NoError(t, err) require.NoError(t, err)
defer resp.Body.Close() defer resp.Body.Close()
if resp.StatusCode == http.StatusCreated { if resp.StatusCode == http.StatusCreated {
// The Azure SDK client dereferences this header without checking, so a blob upload that
// omits it panics the caller rather than failing it.
require.NotEmpty(t, resp.Header.Get("x-ms-request-id")) require.NotEmpty(t, resp.Header.Get("x-ms-request-id"))
} }
return resp.StatusCode return resp.StatusCode
@@ -67,8 +62,6 @@ func getURL(t *testing.T, url string) []byte {
return body return body
} }
// saveV2 runs the reserve/upload/finalize sequence and returns the finalize response along
// with the upload URL it used.
func saveV2(t *testing.T, handler *Handler, key, version string, content []byte) (finalized map[string]any, uploadURL string) { func saveV2(t *testing.T, handler *Handler, key, version string, content []byte) (finalized map[string]any, uploadURL string) {
t.Helper() t.Helper()
@@ -84,10 +77,8 @@ func saveV2(t *testing.T, handler *Handler, key, version string, content []byte)
}), uploadURL }), uploadURL
} }
// The whole round trip an actions/cache v2 client makes, plus the guarantees on the signed
// URLs it is handed: unsigned requests are refused, an upload URL cannot be replayed to read
// or to replace a finalized entry.
func TestCacheServiceV2RoundTrip(t *testing.T) { func TestCacheServiceV2RoundTrip(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
content := []byte("the cached archive") content := []byte("the cached archive")
@@ -98,10 +89,8 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
require.Equal(t, true, finalized["ok"]) require.Equal(t, true, finalized["ok"])
assert.NotEmpty(t, finalized["entry_id"]) assert.NotEmpty(t, finalized["entry_id"])
// The upload URL outlives the finalize call, so replaying it must not poison the entry,
// and it is an upload URL only: nothing reads a blob back through it.
assert.Equal(t, http.StatusBadRequest, putBlob(t, uploadURL, []byte("poisoned"))) assert.Equal(t, http.StatusBadRequest, putBlob(t, uploadURL, []byte("poisoned")))
resp, err := http.Get(uploadURL) //nolint:noctx // the URL is the server under test resp, err := http.Get(uploadURL)
require.NoError(t, err) require.NoError(t, err)
defer resp.Body.Close() defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode) assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode)
@@ -114,9 +103,8 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
assert.Equal(t, content, getURL(t, downloadURL)) assert.Equal(t, content, getURL(t, downloadURL))
} }
// A large archive is staged as blocks and only put in order by the final block list, so
// blocks that arrive out of order must still be assembled the way the client asked.
func TestCacheServiceV2BlockUpload(t *testing.T) { func TestCacheServiceV2BlockUpload(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "blocks", "version": "v1"}) created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "blocks", "version": "v1"})
@@ -130,7 +118,6 @@ func TestCacheServiceV2BlockUpload(t *testing.T) {
blocks[blockID] = []byte(part) blocks[blockID] = []byte(part)
order = append(order, blockID) order = append(order, blockID)
} }
// Upload in an order that is not the block list order.
for _, blockID := range []string{order[2], order[0], order[1]} { for _, blockID := range []string{order[2], order[0], order[1]} {
require.Equal(t, http.StatusCreated, putBlob(t, uploadURL+"&comp=block&blockid="+blockID, blocks[blockID])) require.Equal(t, http.StatusCreated, putBlob(t, uploadURL+"&comp=block&blockid="+blockID, blocks[blockID]))
} }
@@ -165,6 +152,7 @@ func TestCacheServiceV2BlockUpload(t *testing.T) {
} }
func TestCacheServiceV2Lookups(t *testing.T) { func TestCacheServiceV2Lookups(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
saved, _ := saveV2(t, handler, "deps-abc", "v1", []byte("x")) saved, _ := saveV2(t, handler, "deps-abc", "v1", []byte("x"))
require.Equal(t, true, saved["ok"]) require.Equal(t, true, saved["ok"])
@@ -174,8 +162,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"]) assert.Equal(t, false, got["ok"])
}) })
// The toolkit serialises with the proto field names; the camelCase spellings of the same
// proto JSON mapping are accepted alongside them.
for _, field := range []string{"restore_keys", "restoreKeys"} { for _, field := range []string{"restore_keys", "restoreKeys"} {
t.Run("restore keys match by prefix, spelled "+field, func(t *testing.T) { t.Run("restore keys match by prefix, spelled "+field, func(t *testing.T) {
got := v2Call(t, handler, testClient, "GetCacheEntryDownloadURL", map[string]any{ got := v2Call(t, handler, testClient, "GetCacheEntryDownloadURL", map[string]any{
@@ -191,9 +177,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, again["ok"]) assert.Equal(t, false, again["ok"])
}) })
// A key that is only a prefix of an existing one is a different entry, so the
// reservation check must be exact and not a restore-key prefix match, or the shorter
// key would be reported as existing and silently never saved.
t.Run("a prefix of an existing key is still reserved", func(t *testing.T) { t.Run("a prefix of an existing key is still reserved", func(t *testing.T) {
reserved := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "deps", "version": "v1"}) reserved := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "deps", "version": "v1"})
require.Equal(t, true, reserved["ok"]) require.Equal(t, true, reserved["ok"])
@@ -219,7 +202,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"]) assert.Equal(t, false, got["ok"])
}) })
// The size the client declares is what Commit validates the assembled archive against.
t.Run("finalizing with the wrong size is not ok", func(t *testing.T) { t.Run("finalizing with the wrong size is not ok", func(t *testing.T) {
created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "wrong-size", "version": "v1"}) created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "wrong-size", "version": "v1"})
require.Equal(t, http.StatusCreated, putBlob(t, created["signed_upload_url"].(string), []byte("four"))) require.Equal(t, http.StatusCreated, putBlob(t, created["signed_upload_url"].(string), []byte("four")))
@@ -230,8 +212,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"]) assert.Equal(t, false, got["ok"])
}) })
// Both API versions are served from one store, so an entry written through v2 is a hit for
// a v1 client asking for the same key and version.
t.Run("a v1 client sees an entry written through v2", func(t *testing.T) { t.Run("a v1 client sees an entry written through v2", func(t *testing.T) {
resp, err := testClient.Get(fmt.Sprintf("%s%s/cache?keys=deps-abc&version=v1", handler.ExternalURL(), apiPath)) resp, err := testClient.Get(fmt.Sprintf("%s%s/cache?keys=deps-abc&version=v1", handler.ExternalURL(), apiPath))
require.NoError(t, err) require.NoError(t, err)
@@ -244,7 +224,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.NotEmpty(t, got["archiveLocation"]) assert.NotEmpty(t, got["archiveLocation"])
}) })
// The cache of one repository must stay invisible to another, as it does for the v1 API.
t.Run("another repository sees nothing", func(t *testing.T) { t.Run("another repository sees nothing", func(t *testing.T) {
handler.RegisterJob("other-runtime-token", JobCredential{Repo: "other/repo"}) handler.RegisterJob("other-runtime-token", JobCredential{Repo: "other/repo"})
otherClient := &http.Client{Transport: &bearerTransport{token: "other-runtime-token"}} otherClient := &http.Client{Transport: &bearerTransport{token: "other-runtime-token"}}
@@ -255,6 +234,7 @@ func TestCacheServiceV2Lookups(t *testing.T) {
} }
func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) { func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
tests := []struct { tests := []struct {
@@ -291,6 +271,7 @@ func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
} }
func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) { func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
content := []byte("the cached archive") content := []byte("the cached archive")
@@ -321,20 +302,12 @@ func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) {
"another job's upload is not this job's to finalize") "another job's upload is not this job's to finalize")
base := handler.ExternalURL() + apiPath base := handler.ExternalURL() + apiPath
reserveV1, err := json.Marshal(&Request{Key: "v1-made", Version: "v1", Size: int64(len(content))}) madeByV1 := reserveCache(t, testClient, base, "v1-made", "v1", int64(len(content)))
require.NoError(t, err)
resp, err := testClient.Post(base+"/caches", "application/json", bytes.NewReader(reserveV1))
require.NoError(t, err)
var madeByV1 struct {
CacheID uint64 `json:"cacheId"`
}
require.NoError(t, json.UnmarshalRead(resp.Body, &madeByV1))
resp.Body.Close()
upload, err := http.NewRequestWithContext(t.Context(), http.MethodPatch, upload, err := http.NewRequestWithContext(t.Context(), http.MethodPatch,
fmt.Sprintf("%s/caches/%d", base, madeByV1.CacheID), bytes.NewReader(content)) fmt.Sprintf("%s/caches/%d", base, madeByV1), bytes.NewReader(content))
require.NoError(t, err) require.NoError(t, err)
upload.Header.Set("Content-Range", fmt.Sprintf("bytes 0-%d/*", len(content)-1)) upload.Header.Set("Content-Range", fmt.Sprintf("bytes 0-%d/*", len(content)-1))
resp, err = testClient.Do(upload) resp, err := testClient.Do(upload)
require.NoError(t, err) require.NoError(t, err)
resp.Body.Close() resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode) require.Equal(t, http.StatusOK, resp.StatusCode)
+16 -9
View File
@@ -4,6 +4,8 @@
package artifactcache package artifactcache
import "time"
type Request struct { type Request struct {
Key string `json:"key" ` Key string `json:"key" `
Version string `json:"version"` Version string `json:"version"`
@@ -11,13 +13,18 @@ type Request struct {
} }
type Cache struct { type Cache struct {
ID uint64 `json:"id" boltholdKey:"ID"` ID uint64 `json:"id"`
Repo string `json:"repo" boltholdIndex:"Repo"` Repo string `json:"repo"`
Key string `json:"key" boltholdIndex:"Key"` Key string `json:"key"`
Version string `json:"version" boltholdIndex:"Version"` Version string `json:"version"`
Size int64 `json:"cacheSize"` Size int64 `json:"cacheSize"`
Owner string `json:"owner"` // hashed bearer, so a job's own retry is not another job's reservation Owner string `json:"owner"` // hashed bearer, so a job's own retry is not another job's reservation
Complete bool `json:"complete" boltholdIndex:"Complete"` Complete bool `json:"complete"`
UsedAt int64 `json:"usedAt" boltholdIndex:"UsedAt"` UsedAt int64 `json:"usedAt"`
CreatedAt int64 `json:"createdAt" boltholdIndex:"CreatedAt"` CreatedAt int64 `json:"createdAt"`
CreatedAtNanos int `json:"createdAtNanos,omitzero"`
}
func (cache *Cache) createdAtNano() int64 {
return cache.CreatedAt*int64(time.Second) + int64(cache.CreatedAtNanos)
} }
+2 -8
View File
@@ -4,6 +4,7 @@
package artifactcache package artifactcache
import ( import (
"cmp"
"crypto/tls" "crypto/tls"
"errors" "errors"
"fmt" "fmt"
@@ -74,14 +75,7 @@ func (h *Handler) forwardOrNotFound(w http.ResponseWriter, r *http.Request) {
// A protocol switch loses the bearer or the agent, and an untrusted instance needs skipped verification. // A protocol switch loses the bearer or the agent, and an untrusted instance needs skipped verification.
func (h *Handler) mustProxy(r *http.Request, cred JobCredential, target *url.URL) bool { func (h *Handler) mustProxy(r *http.Request, cred JobCredential, target *url.URL) bool {
if cred.InsecureTLS && target.Scheme == "https" { return cred.InsecureTLS && target.Scheme == "https" || cmp.Or(r.Header.Get("X-Forwarded-Proto"), "http") != target.Scheme
return true
}
scheme := r.Header.Get("X-Forwarded-Proto")
if scheme == "" {
scheme = "http"
}
return scheme != target.Scheme
} }
func mergeQuery(target, request string) string { func mergeQuery(target, request string) string {
+1
View File
@@ -17,6 +17,7 @@ import (
) )
func TestFrontResultsService(t *testing.T) { func TestFrontResultsService(t *testing.T) {
t.Parallel()
var gotHost, gotPath, gotProto string var gotHost, gotPath, gotProto string
gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotHost, gotPath, gotProto = r.Host, r.URL.Path, r.Header.Get("X-Forwarded-Proto") gotHost, gotPath, gotProto = r.Host, r.URL.Path, r.Header.Get("X-Forwarded-Proto")
+32 -10
View File
@@ -24,6 +24,10 @@ import (
"strconv" "strconv"
"strings" "strings"
"time" "time"
"gitea.com/gitea/runner/act/common"
"golang.org/x/sync/errgroup"
) )
// S3Options configures the bucket that stores completed cache entries. // S3Options configures the bucket that stores completed cache entries.
@@ -47,7 +51,7 @@ type cacheStorage interface {
Remove(uint64) error Remove(uint64) error
} }
const s3PartSize = 5 << 30 // the largest part S3 accepts const s3PartSize = 64 << 20
// s3Storage stages uploads on disk and keeps completed entries and their metadata in a bucket. // s3Storage stages uploads on disk and keeps completed entries and their metadata in a bucket.
type s3Storage struct { type s3Storage struct {
@@ -69,6 +73,7 @@ func newS3Storage(staging *Storage, opts S3Options) (*s3Storage, error) {
} }
transport, _ := http.DefaultTransport.(*http.Transport) transport, _ := http.DefaultTransport.(*http.Transport)
transport = transport.Clone() transport = transport.Clone()
transport.MaxIdleConnsPerHost = 32 // 4 parts each for the 8 jobs a runner commonly runs
transport.ResponseHeaderTimeout = time.Minute // a stalled endpoint must not hold the store forever transport.ResponseHeaderTimeout = time.Minute // a stalled endpoint must not hold the store forever
pathStyle := !strings.HasSuffix(endpoint.Hostname(), ".amazonaws.com") pathStyle := !strings.HasSuffix(endpoint.Hostname(), ".amazonaws.com")
if opts.PathStyle != nil { if opts.PathStyle != nil {
@@ -209,26 +214,43 @@ func (s *s3Storage) Commit(id uint64, size int64) (int64, error) {
return written, nil return written, nil
} }
// upload always goes in parts, the only way S3 takes an object over 5 GiB. func (s *s3Storage) upload(ctx context.Context, key string, file io.ReaderAt, size int64) (uploadErr error) {
func (s *s3Storage) upload(ctx context.Context, key string, file io.ReaderAt, size int64) error { if size <= s3PartSize {
_, err := s.call(ctx, http.MethodPut, key, nil, io.NewSectionReader(file, 0, size), nil)
return err
}
var created struct { var created struct {
UploadID string `xml:"UploadId"` UploadID string `xml:"UploadId"`
} }
if _, err := s.call(ctx, http.MethodPost, key, url.Values{"uploads": {""}}, nil, &created); err != nil { if _, err := s.call(ctx, http.MethodPost, key, url.Values{"uploads": {""}}, nil, &created); err != nil {
return err return err
} }
defer func() {
if uploadErr != nil && created.UploadID != "" {
abortCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer cancel()
_, _ = s.call(abortCtx, http.MethodDelete, key, url.Values{"uploadId": {created.UploadID}}, nil, nil)
}
}()
var completion struct { var completion struct {
XMLName xml.Name `xml:"CompleteMultipartUpload"` XMLName xml.Name `xml:"CompleteMultipartUpload"`
Parts []s3Part `xml:"Part"` Parts []s3Part `xml:"Part"`
} }
for offset := int64(0); offset == 0 || offset < size; offset += s3PartSize { partSize := max(int64(s3PartSize), (size-1)/10000+1)
number, length := len(completion.Parts)+1, min(s3PartSize, size-offset) completion.Parts = make([]s3Part, (size-1)/partSize+1)
query := url.Values{"partNumber": {strconv.Itoa(number)}, "uploadId": {created.UploadID}} workers, workerCtx := errgroup.WithContext(ctx)
header, err := s.call(ctx, http.MethodPut, key, query, io.NewSectionReader(file, offset, length), nil) workers.SetLimit(4)
if err != nil { for part := range completion.Parts {
workers.Go(func() (err error) {
defer common.RecoverPanic(&err)
offset := int64(part) * partSize
header, err := s.call(workerCtx, http.MethodPut, key, url.Values{"partNumber": {strconv.Itoa(part + 1)}, "uploadId": {created.UploadID}}, io.NewSectionReader(file, offset, min(partSize, size-offset)), nil)
completion.Parts[part] = s3Part{part + 1, header.Get("ETag")}
return err return err
} })
completion.Parts = append(completion.Parts, s3Part{number, header.Get("ETag")}) }
if err := workers.Wait(); err != nil {
return err
} }
body, err := xml.Marshal(completion) body, err := xml.Marshal(completion)
if err != nil { if err != nil {
+102 -7
View File
@@ -4,40 +4,135 @@
package artifactcache package artifactcache
import ( import (
"encoding/xml"
"io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing" "testing"
"time" "time"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) { func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
backend := s3mem.New() t.Parallel()
require.NoError(t, backend.CreateBucket("cache")) var objects sync.Map
server := httptest.NewServer(gofakes3.New(backend).Server()) var uploads atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
key := strings.TrimPrefix(request.URL.Path, "/cache/")
switch {
case request.Method == http.MethodPut:
if strings.Contains(key, "/blobs/") {
uploads.Add(1)
}
body, err := io.ReadAll(request.Body)
assert.NoError(t, err)
objects.Store(key, body)
case request.URL.Query().Has("list-type"):
_, _ = io.WriteString(writer, "<ListBucketResult>")
objects.Range(func(stored, _ any) bool {
_, _ = io.WriteString(writer, "<Contents><Key>"+stored.(string)+"</Key></Contents>")
return true
})
_, _ = io.WriteString(writer, "</ListBucketResult>")
default:
if body, ok := objects.Load(key); ok {
_, _ = writer.Write(body.([]byte))
} else {
writer.WriteHeader(http.StatusNotFound)
}
}
}))
t.Cleanup(server.Close) t.Cleanup(server.Close)
opts := Options{S3: &S3Options{Endpoint: server.URL, Bucket: "cache", Prefix: "runners", AccessKey: "key", SecretKey: "secret"}} opts := Options{S3: &S3Options{Endpoint: server.URL, Bucket: "cache", Prefix: "runners", AccessKey: "key", SecretKey: "secret"}}
saver, restorer := startTestHandler(t, opts), startTestHandler(t, opts) saver, restorer := startTestHandler(t, opts), startTestHandler(t, opts)
entry := map[string]any{"key": "deps", "version": "v1"} entry := map[string]any{"key": "deps", "version": "v1"}
started := time.Now()
finalized, _ := saveV2(t, saver, "deps", "v1", []byte("archive")) finalized, _ := saveV2(t, saver, "deps", "v1", []byte("archive"))
require.Equal(t, true, finalized["ok"]) require.Equal(t, true, finalized["ok"])
assert.EqualValues(t, 1, uploads.Load())
id, err := strconv.ParseUint(finalized["entry_id"].(string), 10, 64)
require.NoError(t, err)
cache, err := saver.s3.fetch(t.Context(), id)
require.NoError(t, err)
assert.WithinRange(t, time.Unix(cache.CreatedAt, int64(cache.CreatedAtNanos)), started, time.Now())
require.NoError(t, restorer.importMetadata(t.Context())) require.NoError(t, restorer.importMetadata(t.Context()))
found := v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", entry) found := v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", entry)
require.Equal(t, true, found["ok"]) require.Equal(t, true, found["ok"])
assert.Equal(t, []byte("archive"), getURL(t, found["signed_download_url"].(string))) assert.Equal(t, []byte("archive"), getURL(t, found["signed_download_url"].(string)))
require.NoError(t, backend.ForceDeleteBucket("cache")) for _, cache := range []*Cache{
require.NoError(t, backend.CreateBucket("cache")) {ID: 2, Repo: testRepo, Key: "deps-old", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt},
{ID: 1, Repo: testRepo, Key: "deps-new", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt, CreatedAtNanos: 1},
} {
require.NoError(t, saver.s3.publish(cache))
objects.Store(saver.s3.blobKey(cache.ID), []byte(cache.Key))
}
require.NoError(t, restorer.importMetadata(t.Context()))
found = v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-", "version": "v1"})
require.Equal(t, true, found["ok"])
assert.Equal(t, "deps-new", found["matched_key"])
assert.Equal(t, []byte("deps-new"), getURL(t, found["signed_download_url"].(string)))
objects.Clear()
require.NoError(t, restorer.importMetadata(t.Context())) require.NoError(t, restorer.importMetadata(t.Context()))
assert.Equal(t, true, v2Call(t, restorer, testClient, "CreateCacheEntry", entry)["ok"]) assert.Equal(t, true, v2Call(t, restorer, testClient, "CreateCacheEntry", entry)["ok"])
} }
type s3TestTransport func(*http.Request) (*http.Response, error)
func (transport s3TestTransport) RoundTrip(req *http.Request) (*http.Response, error) {
return transport(req)
}
func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) {
t.Parallel()
for _, failingPart := range []string{"", "2"} {
var parts, aborts atomic.Int32
var completion struct {
Parts []s3Part `xml:"Part"`
}
storage := &s3Storage{bucketURL: httptest.NewRequest(http.MethodGet, "http://cache.local/bucket", nil).URL}
storage.client = &http.Client{Transport: s3TestTransport(func(req *http.Request) (*http.Response, error) {
query, response := req.URL.Query(), &http.Response{StatusCode: http.StatusOK, Header: make(http.Header), Body: http.NoBody}
switch {
case query.Has("uploads"):
response.Body = io.NopCloser(strings.NewReader("<InitiateMultipartUploadResult><UploadId>upload</UploadId></InitiateMultipartUploadResult>"))
case req.Method == http.MethodPut:
parts.Add(1)
response.Header.Set("ETag", "etag-"+query.Get("partNumber"))
if query.Get("partNumber") == failingPart {
response.StatusCode, response.Body = http.StatusBadRequest, io.NopCloser(strings.NewReader("<Error><Code>InvalidPart</Code></Error>"))
}
case req.Method == http.MethodPost:
require.NoError(t, xml.NewDecoder(req.Body).Decode(&completion))
response.Body = io.NopCloser(strings.NewReader("<CompleteMultipartUploadResult/>"))
case req.Method == http.MethodDelete:
aborts.Add(1)
}
return response, nil
})}
err := storage.upload(t.Context(), "blobs/1", strings.NewReader(""), 2*int64(s3PartSize)+1)
assert.EqualValues(t, 3, parts.Load())
if failingPart == "" {
require.NoError(t, err)
assert.Equal(t, []s3Part{{1, "etag-1"}, {2, "etag-2"}, {3, "etag-3"}}, completion.Parts)
assert.Zero(t, aborts.Load())
} else {
require.ErrorContains(t, err, "InvalidPart")
assert.EqualValues(t, 1, aborts.Load())
}
}
}
func TestS3StorageSignsLikeAWS(t *testing.T) { func TestS3StorageSignsLikeAWS(t *testing.T) {
t.Parallel()
storage := &s3Storage{opts: S3Options{Region: "us-east-1", AccessKey: "AKIAIOSFODNN7EXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}} storage := &s3Storage{opts: S3Options{Region: "us-east-1", AccessKey: "AKIAIOSFODNN7EXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}}
req := httptest.NewRequest(http.MethodGet, "https://examplebucket.s3.amazonaws.com/?max-keys=2&prefix=J", nil) req := httptest.NewRequest(http.MethodGet, "https://examplebucket.s3.amazonaws.com/?max-keys=2&prefix=J", nil)
req.Header.Set("X-Amz-Content-Sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855") req.Header.Set("X-Amz-Content-Sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
+5 -10
View File
@@ -32,13 +32,11 @@ func NewStorage(rootDir string) (*Storage, error) {
} }
func (s *Storage) Exist(id uint64) (bool, error) { func (s *Storage) Exist(id uint64) (bool, error) {
name := s.filename(id) _, err := os.Stat(s.filename(id))
if _, err := os.Stat(name); os.IsNotExist(err) { if os.IsNotExist(err) {
return false, nil return false, nil
} else if err != nil {
return false, err
} }
return true, nil return err == nil, err
} }
func (s *Storage) Write(id uint64, offset int64, reader io.Reader) error { func (s *Storage) Write(id uint64, offset int64, reader io.Reader) error {
@@ -92,9 +90,7 @@ func (s *Storage) OrderBlocks(id uint64, blockIDs []string) error {
} }
func (s *Storage) Commit(id uint64, size int64) (int64, error) { func (s *Storage) Commit(id uint64, size int64) (int64, error) {
defer func() { defer func() { _ = os.RemoveAll(s.tempDir(id)) }()
_ = os.RemoveAll(s.tempDir(id))
}()
name := s.filename(id) name := s.filename(id)
tempNames, err := s.tempNames(id) tempNames, err := s.tempNames(id)
@@ -153,8 +149,7 @@ func assemble(name string, tempNames []string) (int64, error) {
} }
func (s *Storage) Serve(w http.ResponseWriter, r *http.Request, id uint64) { func (s *Storage) Serve(w http.ResponseWriter, r *http.Request, id uint64) {
name := s.filename(id) http.ServeFile(w, r, s.filename(id))
http.ServeFile(w, r, name)
} }
// Remove deletes an entry's blob and any staged parts. It reports failure so the caller can // Remove deletes an entry's blob and any staged parts. It reports failure so the caller can
+37 -37
View File
@@ -13,14 +13,13 @@ import (
"io/fs" "io/fs"
"net" "net"
"net/http" "net/http"
"net/url"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"time" "time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"github.com/julienschmidt/httprouter"
) )
type FileContainerResourceURL struct { type FileContainerResourceURL struct {
@@ -67,39 +66,44 @@ func writeJSON(w http.ResponseWriter, value any) {
} }
} }
func uploads(router *httprouter.Router, baseDir string) { func uploads(router *http.ServeMux, baseDir string) {
router.POST("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.HandleFunc("POST /_apis/pipelines/workflows/{runId}/artifacts", func(w http.ResponseWriter, req *http.Request) {
runID := params.ByName("runId")
writeJSON(w, FileContainerResourceURL{ writeJSON(w, FileContainerResourceURL{
FileContainerResourceURL: fmt.Sprintf("http://%s/upload/%s", req.Host, runID), FileContainerResourceURL: fmt.Sprintf("http://%s/upload/%s", req.Host, req.PathValue("runId")),
}) })
}) })
router.PUT("/upload/:runId", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.HandleFunc("PUT /upload/{runId}", func(w http.ResponseWriter, req *http.Request) {
itemPath := req.URL.Query().Get("itemPath") itemPath := req.URL.Query().Get("itemPath")
runID := params.ByName("runId")
if req.Header.Get("Content-Encoding") == "gzip" { if req.Header.Get("Content-Encoding") == "gzip" {
itemPath += gzipExtension itemPath += gzipExtension
} }
safeRunPath := safeResolve(baseDir, runID) safePath := safeResolve(safeResolve(baseDir, req.PathValue("runId")), itemPath)
safePath := safeResolve(safeRunPath, itemPath)
if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil { if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil {
panic(err) panic(err)
} }
flags := os.O_CREATE | os.O_WRONLY | os.O_TRUNC var start int64
appendUpload := req.Header.Get("Content-Range") if contentRange := req.Header.Get("Content-Range"); contentRange != "" {
if appendUpload != "" && !strings.HasPrefix(appendUpload, "bytes 0-") { if _, err := fmt.Sscanf(contentRange, "bytes %d-", &start); err != nil || start < 0 {
flags = os.O_CREATE | os.O_WRONLY | os.O_APPEND http.Error(w, "invalid content range", http.StatusBadRequest)
return
}
}
flags := os.O_CREATE | os.O_WRONLY
if start == 0 {
flags |= os.O_TRUNC
} }
file, err := os.OpenFile(safePath, flags, 0o644) file, err := os.OpenFile(safePath, flags, 0o644)
if err != nil { if err != nil {
panic(err) panic(err)
} }
defer file.Close() defer file.Close()
if _, err := file.Seek(start, io.SeekStart); err != nil {
panic(err)
}
if req.Body == nil { if req.Body == nil {
panic(errors.New("no body given")) panic(errors.New("no body given"))
} }
@@ -114,21 +118,19 @@ func uploads(router *httprouter.Router, baseDir string) {
}) })
}) })
router.PATCH("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.HandleFunc("PATCH /_apis/pipelines/workflows/{runId}/artifacts", func(w http.ResponseWriter, _req *http.Request) {
writeJSON(w, ResponseMessage{ writeJSON(w, ResponseMessage{
Message: "success", Message: "success",
}) })
}) })
} }
func downloads(router *httprouter.Router, baseDir string) { func downloads(router *http.ServeMux, baseDir string) {
router.GET("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.HandleFunc("GET /_apis/pipelines/workflows/{runId}/artifacts", func(w http.ResponseWriter, req *http.Request) {
runID := params.ByName("runId") runID := req.PathValue("runId")
safePath := safeResolve(baseDir, runID) entries, err := os.ReadDir(safeResolve(baseDir, runID))
if err != nil && !os.IsNotExist(err) {
entries, err := os.ReadDir(safePath)
if err != nil {
panic(err) panic(err)
} }
@@ -146,13 +148,16 @@ func downloads(router *httprouter.Router, baseDir string) {
}) })
}) })
router.GET("/download/:container", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.HandleFunc("GET /download/{container}", func(w http.ResponseWriter, req *http.Request) {
container := params.ByName("container") container := req.PathValue("container")
itemPath := req.URL.Query().Get("itemPath") itemPath := req.URL.Query().Get("itemPath")
safePath := safeResolve(baseDir, filepath.Join(container, itemPath)) safePath := safeResolve(baseDir, filepath.Join(container, itemPath))
var files []ContainerItem var files []ContainerItem
err := filepath.WalkDir(safePath, func(path string, entry fs.DirEntry, err error) error { err := filepath.WalkDir(safePath, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if !entry.IsDir() { if !entry.IsDir() {
rel, err := filepath.Rel(safePath, path) rel, err := filepath.Rel(safePath, path)
if err != nil { if err != nil {
@@ -161,20 +166,16 @@ func downloads(router *httprouter.Router, baseDir string) {
// if it was upload as gzip // if it was upload as gzip
rel = strings.TrimSuffix(rel, gzipExtension) rel = strings.TrimSuffix(rel, gzipExtension)
path := filepath.Join(itemPath, rel)
rel = filepath.ToSlash(rel)
path = filepath.ToSlash(path)
files = append(files, ContainerItem{ files = append(files, ContainerItem{
Path: path, Path: filepath.ToSlash(filepath.Join(itemPath, rel)),
ItemType: "file", ItemType: "file",
ContentLocation: fmt.Sprintf("http://%s/artifact/%s/%s/%s", req.Host, container, itemPath, rel), ContentLocation: (&url.URL{Scheme: "http", Host: req.Host, Path: "/artifact/" + container + "/" + itemPath + "/" + filepath.ToSlash(rel)}).String(),
}) })
} }
return nil return nil
}) })
if err != nil { if err != nil && !os.IsNotExist(err) {
panic(err) panic(err)
} }
@@ -183,10 +184,8 @@ func downloads(router *httprouter.Router, baseDir string) {
}) })
}) })
router.GET("/artifact/*path", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.HandleFunc("GET /artifact/{path...}", func(w http.ResponseWriter, req *http.Request) {
path := params.ByName("path")[1:] safePath := safeResolve(baseDir, req.PathValue("path"))
safePath := safeResolve(baseDir, path)
file, err := os.Open(safePath) file, err := os.Open(safePath)
if err != nil { if err != nil {
@@ -214,11 +213,12 @@ func Serve(ctx context.Context, artifactPath, addr, port string) context.CancelF
return cancel return cancel
} }
router := httprouter.New() router := http.NewServeMux()
logger.Debugf("Artifacts base path '%s'", artifactPath) logger.Debugf("Artifacts base path '%s'", artifactPath)
uploads(router, artifactPath) uploads(router, artifactPath)
downloads(router, artifactPath) downloads(router, artifactPath)
registerV4(router, artifactPath)
server := &http.Server{ server := &http.Server{
Addr: net.JoinHostPort(addr, port), Addr: net.JoinHostPort(addr, port),
+113 -9
View File
@@ -19,16 +19,16 @@ import (
"testing" "testing"
"time" "time"
"github.com/julienschmidt/httprouter"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
func TestArtifactFlow(t *testing.T) { func TestArtifactFlow(t *testing.T) {
artifactPath := t.TempDir() artifactPath := t.TempDir()
router := httprouter.New() router := http.NewServeMux()
uploads(router, artifactPath) uploads(router, artifactPath)
downloads(router, artifactPath) downloads(router, artifactPath)
registerV4(router, artifactPath)
server := httptest.NewServer(router) server := httptest.NewServer(router)
defer server.Close() defer server.Close()
@@ -49,8 +49,17 @@ func TestArtifactFlow(t *testing.T) {
return resp.StatusCode, data return resp.StatusCode, data
} }
t.Run("no-artifacts", func(t *testing.T) {
status, data := request(t, http.MethodGet, baseURL+"/_apis/pipelines/workflows/0/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data))
var list NamedFileContainerResourceURLResponse
require.NoError(t, json.Unmarshal(data, &list))
require.Zero(t, list.Count)
require.Empty(t, list.Value)
})
t.Run("upload-and-download", func(t *testing.T) { t.Run("upload-and-download", func(t *testing.T) {
const runID, item, content = "1", "my-artifact/data.txt", "hello artifact\n" const runID, item, content = "1", "my-artifact/data#1%25.txt", "hello artifact\n"
status, data := request(t, http.MethodPost, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil) status, data := request(t, http.MethodPost, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data)) require.Equal(t, http.StatusOK, status, string(data))
@@ -82,13 +91,13 @@ func TestArtifactFlow(t *testing.T) {
require.NoError(t, json.Unmarshal(data, &items)) require.NoError(t, json.Unmarshal(data, &items))
require.Len(t, items.Value, 1) require.Len(t, items.Value, 1)
require.Equal(t, "file", items.Value[0].ItemType) require.Equal(t, "file", items.Value[0].ItemType)
require.Equal(t, "my-artifact/data.txt", items.Value[0].Path) require.Equal(t, item, items.Value[0].Path)
status, data = request(t, http.MethodGet, items.Value[0].ContentLocation, nil, nil) status, data = request(t, http.MethodGet, items.Value[0].ContentLocation, nil, nil)
require.Equal(t, http.StatusOK, status) require.Equal(t, http.StatusOK, status)
require.Equal(t, content, string(data)) require.Equal(t, content, string(data))
stored, err := os.ReadFile(filepath.Join(artifactPath, runID, "my-artifact", "data.txt")) stored, err := os.ReadFile(filepath.Join(artifactPath, runID, filepath.FromSlash(item)))
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, content, string(stored)) require.Equal(t, content, string(stored))
}) })
@@ -96,16 +105,25 @@ func TestArtifactFlow(t *testing.T) {
t.Run("content-range", func(t *testing.T) { t.Run("content-range", func(t *testing.T) {
const rawURL = "/upload/4?itemPath=chunks.txt" const rawURL = "/upload/4?itemPath=chunks.txt"
status, data := request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("first"), status, data := request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("first"),
http.Header{"Content-Range": []string{"bytes 0-4/11"}}) http.Header{"Content-Range": []string{"bytes 0-4/12"}})
require.Equal(t, http.StatusOK, status, string(data)) require.Equal(t, http.StatusOK, status, string(data))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"), for range 2 {
http.Header{"Content-Range": []string{"bytes 5-11/11"}}) status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"),
require.Equal(t, http.StatusOK, status, string(data)) http.Header{"Content-Range": []string{"bytes 5-11/12"}})
require.Equal(t, http.StatusOK, status, string(data))
}
stored, err := os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt")) stored, err := os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, "first-second", string(stored)) require.Equal(t, "first-second", string(stored))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader(""),
http.Header{"Content-Range": []string{"bytes 0--1/0"}})
require.Equal(t, http.StatusOK, status, string(data))
stored, err = os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err)
require.Empty(t, stored)
}) })
t.Run("gzip-roundtrip", func(t *testing.T) { t.Run("gzip-roundtrip", func(t *testing.T) {
@@ -157,6 +175,92 @@ func TestArtifactFlow(t *testing.T) {
require.Equal(t, http.StatusOK, status) require.Equal(t, http.StatusOK, status)
require.Equal(t, content, string(data)) require.Equal(t, content, string(data))
}) })
t.Run("v4", func(t *testing.T) {
const artifactRef = `"workflow_run_backend_id":"7","name":"50% art@v1"`
call := func(method, body string) (int, map[string]any) {
status, data := request(t, http.MethodPost, baseURL+artifactServicePath+"/"+method, strings.NewReader(body), nil)
var out map[string]any
require.NoError(t, json.Unmarshal(data, &out), string(data))
return status, out
}
put := func(rawURL, body string) int {
status, _ := request(t, http.MethodPut, rawURL, strings.NewReader(body), nil)
return status
}
download := func() (string, http.Header) {
status, signed := call("GetSignedArtifactURL", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, signed)
resp, err := client.Get(signed["signedUrl"].(string))
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
return string(body), resp.Header
}
status, created := call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, created)
upload := created["signedUploadUrl"].(string)
require.Equal(t, http.StatusCreated, put(upload+"&comp=block&blockid=Yg", "world"))
require.Equal(t, http.StatusCreated, put(upload+"&comp=block&blockid=YQ", "hello "))
for attempt := range 2 {
require.Equal(t, http.StatusCreated, put(upload+"&comp=blocklist", `<?xml version="1.0"?><BlockList><Latest>YQ</Latest><Latest>Yg</Latest></BlockList>`), "commit attempt %d", attempt)
}
digest := "sha256:b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"
status, _ = call("FinalizeArtifact", "{"+artifactRef+`,"size":"10"}`)
require.Equal(t, http.StatusBadRequest, status)
status, finalized := call("FinalizeArtifact", "{"+artifactRef+`,"size":"11","hash":"`+digest+`"}`)
require.Equal(t, http.StatusOK, status, finalized)
id := finalized["artifactId"].(string)
status, refinalized := call("FinalizeArtifact", "{"+artifactRef+`,"size":"10"}`)
require.Equal(t, http.StatusOK, status)
require.Equal(t, id, refinalized["artifactId"])
status, conflict := call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusConflict, status)
require.Equal(t, "already_exists", conflict["code"])
status, listed := call("ListArtifacts", `{"workflowRunBackendId":"7","idFilter":"`+id+`"}`)
require.Equal(t, http.StatusOK, status)
artifacts := listed["artifacts"].([]any)
require.Len(t, artifacts, 1)
artifact := artifacts[0].(map[string]any)
require.Regexp(t, `^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$`, artifact["createdAt"])
delete(artifact, "createdAt")
require.Equal(t, map[string]any{"workflowRunBackendId": "7", "name": "50% art@v1", "size": "11", "digest": digest, "databaseId": id}, artifact)
_, other := call("ListArtifacts", `{"workflow_run_backend_id":"8"}`)
require.Empty(t, other["artifacts"])
body, header := download()
require.Equal(t, "hello world", body)
require.Equal(t, "application/zip", header.Get("Content-Type"))
require.Equal(t, "attachment; filename*=UTF-8''50%25%20art%40v1.zip", header.Get("Content-Disposition"))
status, _ = call("DeleteArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status)
status, missing := call("GetSignedArtifactURL", "{"+artifactRef+"}")
require.Equal(t, http.StatusNotFound, status)
require.Equal(t, "not_found", missing["code"])
status, created = call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, created)
status, _ = request(t, http.MethodPut, created["signedUploadUrl"].(string), strings.NewReader("gz"), http.Header{"X-Ms-Blob-Content-Type": {"application/gzip"}})
require.Equal(t, http.StatusCreated, status)
status, _ = call("FinalizeArtifact", "{"+artifactRef+`,"size":"2"}`)
require.Equal(t, http.StatusOK, status)
body, header = download()
require.Equal(t, "gz", body)
require.Equal(t, "application/gzip", header.Get("Content-Type"))
require.Equal(t, "attachment; filename*=UTF-8''50%25%20art%40v1", header.Get("Content-Disposition"))
rerun := http.NewServeMux()
registerV4(rerun, artifactPath)
recorder := httptest.NewRecorder()
rerun.ServeHTTP(recorder, httptest.NewRequest(http.MethodPost, artifactServicePath+"/CreateArtifact", strings.NewReader("{"+artifactRef+"}")))
require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String())
})
} }
func TestSafeResolve(t *testing.T) { func TestSafeResolve(t *testing.T) {
+329
View File
@@ -0,0 +1,329 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import (
"cmp"
"crypto/sha256"
"encoding/hex"
"encoding/json/v2"
"encoding/xml"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"net/url"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"time"
)
type serverV4 struct { // the subset of Gitea's v4 artifact service that the toolkit uses, unauthenticated like the v3 routes beside it
dir string
mu sync.Mutex
lastID int64
created map[string]bool
}
const (
zipMimeType = "application/zip"
maxTwirpBytes = 1 << 20
)
func registerV4(router *http.ServeMux, baseDir string) {
server := &serverV4{dir: filepath.Join(baseDir, "v4"), lastID: time.Now().UnixMilli(), created: map[string]bool{}} // IDs stay unique across invocations sharing baseDir
router.HandleFunc("POST "+artifactServicePath+"/CreateArtifact", twirp(server.createArtifact))
router.HandleFunc("POST "+artifactServicePath+"/FinalizeArtifact", twirp(server.finalizeArtifact))
router.HandleFunc("POST "+artifactServicePath+"/ListArtifacts", twirp(server.listArtifacts))
router.HandleFunc("POST "+artifactServicePath+"/GetSignedArtifactURL", twirp(server.getSignedArtifactURL))
router.HandleFunc("POST "+artifactServicePath+"/DeleteArtifact", twirp(server.deleteArtifact))
router.HandleFunc("PUT "+artifactServicePath+"/UploadArtifact", server.uploadArtifact)
router.HandleFunc("GET "+artifactServicePath+"/DownloadArtifact", server.downloadArtifact)
}
type twirpError struct {
status int
code string
msg string
}
func (e *twirpError) Error() string { return e.msg }
func notFound(name string) error {
return &twirpError{http.StatusNotFound, "not_found", fmt.Sprintf("artifact %q not found", name)}
}
func badRequest(format string, args ...any) error {
return &twirpError{http.StatusBadRequest, "invalid_argument", fmt.Sprintf(format, args...)}
}
func twirp[Req, Resp any](handle func(*http.Request, Req) (Resp, error)) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var req Req
var resp Resp
err := json.UnmarshalRead(io.LimitReader(r.Body, maxTwirpBytes), &req, json.MatchCaseInsensitiveNames(true)) // also matches the toolkit's proto field names
if err != nil {
err = badRequest("malformed request: %v", err)
} else {
resp, err = handle(r, req)
}
w.Header().Set("Content-Type", "application/json")
if err != nil {
terr := &twirpError{http.StatusInternalServerError, "internal", err.Error()}
errors.As(err, &terr)
w.WriteHeader(terr.status)
_ = json.MarshalWrite(w, map[string]string{"code": terr.code, "msg": terr.msg})
return
}
_ = json.MarshalWrite(w, resp)
}
}
func fileName(name string) string { // safe on any filesystem for any client-chosen name, case-insensitive ones included
sum := sha256.Sum256([]byte(name))
return hex.EncodeToString(sum[:])
}
func (s *serverV4) runDir(runID string) string {
return filepath.Join(s.dir, fileName(runID))
}
func (s *serverV4) artifactDir(runID, name string) string {
return filepath.Join(s.runDir(runID), fileName(name))
}
func (s *serverV4) urlDir(r *http.Request) string {
return s.artifactDir(r.URL.Query().Get("run"), r.URL.Query().Get("name"))
}
func readMeta(dir string) (*monolithArtifact, error) {
data, err := os.ReadFile(filepath.Join(dir, "meta.json"))
if err != nil {
return nil, err
}
var meta monolithArtifact
return &meta, json.Unmarshal(data, &meta)
}
func writeMeta(dir string, meta *monolithArtifact) error {
data, err := json.Marshal(meta)
if err != nil {
return err
}
return os.WriteFile(filepath.Join(dir, "meta.json"), data, 0o644)
}
func blobURL(r *http.Request, endpoint, runID, name string) string {
return "http://" + r.Host + artifactServicePath + "/" + endpoint + "?" + url.Values{"run": {runID}, "name": {name}}.Encode()
}
func (s *serverV4) createArtifact(r *http.Request, req createArtifactRequest) (createArtifactResponse, error) { // an exec invocation re-runs its workflows, so it replaces earlier invocations' artifacts but conflicts with its own
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
s.mu.Lock()
defer s.mu.Unlock()
if s.created[dir] {
return createArtifactResponse{}, &twirpError{http.StatusConflict, "already_exists", "an artifact with this name already exists on the workflow run"}
}
if err := os.RemoveAll(dir); err != nil {
return createArtifactResponse{}, err
}
if err := os.MkdirAll(filepath.Join(dir, "blocks"), 0o755); err != nil {
return createArtifactResponse{}, err
}
s.lastID++
if err := writeMeta(dir, &monolithArtifact{
WorkflowRunBackendID: req.WorkflowRunBackendID,
DatabaseID: s.lastID,
Name: req.Name,
CreatedAt: time.Now().UTC().Truncate(time.Second),
}); err != nil {
return createArtifactResponse{}, err
}
s.created[dir] = true
return createArtifactResponse{Ok: true, SignedUploadURL: blobURL(r, "UploadArtifact", req.WorkflowRunBackendID, req.Name)}, nil
}
type blockList struct {
Blocks []string `xml:",any"` // Latest, Committed and Uncommitted all name a staged block
}
func (s *serverV4) uploadArtifact(w http.ResponseWriter, r *http.Request) {
dir := s.urlDir(r)
if meta, err := readMeta(dir); err != nil || meta.Digest != "" {
http.Error(w, "no artifact awaits an upload", http.StatusNotFound)
return
}
contentType := r.Header.Get("x-ms-blob-content-type")
var err error
switch comp := r.URL.Query().Get("comp"); comp {
case "block":
err = writeFile(filepath.Join(dir, "blocks", fileName(r.URL.Query().Get("blockid"))), r.Body)
case "blocklist":
err = commitBlob(dir, contentType, func(out io.Writer) error { return joinBlocks(out, dir, r.Body) })
case "": // Put Blob, which docker/build-push-action uses
err = commitBlob(dir, contentType, func(out io.Writer) error {
_, err := io.Copy(out, r.Body)
return err
})
default:
http.Error(w, "unsupported comp "+comp, http.StatusBadRequest)
return
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusCreated)
}
func writeFile(name string, body io.Reader) error {
file, err := os.Create(name)
if err != nil {
return err
}
_, err = io.Copy(file, body)
return errors.Join(err, file.Close())
}
func commitBlob(dir, contentType string, write func(io.Writer) error) error { // replaces the blob whole and keeps its content type, as Azure does
out, err := os.CreateTemp(dir, "blob")
if err != nil {
return err
}
if err = errors.Join(write(out), out.Close()); err == nil {
err = os.Rename(out.Name(), filepath.Join(dir, "blob"))
}
if err != nil {
return errors.Join(err, os.Remove(out.Name()))
}
return os.WriteFile(filepath.Join(dir, "content-type"), []byte(contentType), 0o644)
}
func joinBlocks(out io.Writer, dir string, body io.Reader) error { // keeps the blocks for finalizeArtifact, so that a retried commit can join them again
var list blockList
if err := xml.NewDecoder(io.LimitReader(body, maxTwirpBytes)).Decode(&list); err != nil {
return err
}
for _, id := range list.Blocks {
if err := appendBlock(out, filepath.Join(dir, "blocks", fileName(id))); err != nil {
return err
}
}
return nil
}
func appendBlock(out io.Writer, name string) error {
in, err := os.Open(name)
if err != nil {
return err
}
defer in.Close()
_, err = io.Copy(out, in)
return err
}
func (s *serverV4) finalizeArtifact(_ *http.Request, req finalizeArtifactRequest) (finalizeArtifactResponse, error) {
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
meta, err := readMeta(dir)
if err != nil {
return finalizeArtifactResponse{}, notFound(req.Name)
} else if meta.Digest != "" {
return finalizeArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, nil
}
blob, err := os.Open(filepath.Join(dir, "blob"))
if err != nil {
return finalizeArtifactResponse{}, badRequest("artifact %q has no upload", req.Name)
}
defer blob.Close()
hash := sha256.New()
size, err := io.Copy(hash, blob)
if err != nil {
return finalizeArtifactResponse{}, err
}
digest := "sha256:" + hex.EncodeToString(hash.Sum(nil))
if size != req.Size {
return finalizeArtifactResponse{}, badRequest("size %d does not match the %d bytes uploaded", req.Size, size)
}
if req.Hash != "" && req.Hash != digest {
return finalizeArtifactResponse{}, badRequest("hash %s does not match the uploaded %s", req.Hash, digest)
}
s.mu.Lock() // only after hashing, so finalizing one large artifact does not stall the others
defer s.mu.Unlock()
if current, err := readMeta(dir); err != nil || current.DatabaseID != meta.DatabaseID {
return finalizeArtifactResponse{}, notFound(req.Name)
}
meta.Size, meta.Digest = size, digest
if err := writeMeta(dir, meta); err != nil {
return finalizeArtifactResponse{}, err
}
return finalizeArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, os.RemoveAll(filepath.Join(dir, "blocks"))
}
func (s *serverV4) listArtifacts(_ *http.Request, req listArtifactsRequest) (listArtifactsResponse, error) {
run := s.runDir(req.WorkflowRunBackendID)
entries, err := os.ReadDir(run)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return listArtifactsResponse{}, err
}
var resp listArtifactsResponse
for _, entry := range entries {
meta, err := readMeta(filepath.Join(run, entry.Name()))
if err != nil || meta.Digest == "" ||
req.NameFilter != "" && req.NameFilter != meta.Name || req.IDFilter != 0 && req.IDFilter != meta.DatabaseID {
continue
}
resp.Artifacts = append(resp.Artifacts, *meta)
}
slices.SortFunc(resp.Artifacts, func(left, right monolithArtifact) int { return cmp.Compare(left.DatabaseID, right.DatabaseID) })
return resp, nil
}
func (s *serverV4) getSignedArtifactURL(r *http.Request, req getSignedArtifactURLRequest) (getSignedArtifactURLResponse, error) {
if meta, err := readMeta(s.artifactDir(req.WorkflowRunBackendID, req.Name)); err != nil || meta.Digest == "" {
return getSignedArtifactURLResponse{}, notFound(req.Name)
}
return getSignedArtifactURLResponse{SignedURL: blobURL(r, "DownloadArtifact", req.WorkflowRunBackendID, req.Name)}, nil
}
func (s *serverV4) downloadArtifact(w http.ResponseWriter, r *http.Request) {
dir := s.urlDir(r)
meta, err := readMeta(dir)
if err != nil || meta.Digest == "" {
http.Error(w, "artifact not found", http.StatusNotFound)
return
}
blob, err := os.Open(filepath.Join(dir, "blob"))
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer blob.Close()
data, _ := os.ReadFile(filepath.Join(dir, "content-type"))
contentType, filename := string(data), meta.Name
if !strings.Contains(contentType, "/") {
contentType = zipMimeType
}
if contentType == zipMimeType && !strings.HasSuffix(filename, ".zip") {
filename += ".zip"
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Disposition", "attachment; filename*=UTF-8''"+strings.ReplaceAll(url.QueryEscape(filename), "+", "%20")) // RFC 5987 attr-chars only, which the toolkit and mime.ParseMediaType both accept
http.ServeContent(w, r, "", meta.CreatedAt, blob)
}
func (s *serverV4) deleteArtifact(_ *http.Request, req deleteArtifactRequest) (deleteArtifactResponse, error) {
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
s.mu.Lock()
defer s.mu.Unlock()
meta, err := readMeta(dir)
if err != nil {
return deleteArtifactResponse{}, notFound(req.Name)
}
delete(s.created, dir)
return deleteArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, os.RemoveAll(dir)
}
+23
View File
@@ -0,0 +1,23 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import (
"crypto/rand"
"encoding/base64"
"encoding/json/v2"
"time"
)
func LocalRuntimeToken(run string, lifetime time.Duration) string { // an unsecured JWT with the claims the toolkit and buildx read, its run hashed for scp and its jti an unguessable bearer
now := time.Now()
claims, _ := json.Marshal(map[string]any{
"scp": "Actions.Results:" + fileName(run) + ":1",
"ac": `[{"Scope":"","Permission":3}]`,
"nbf": now.Unix(),
"exp": now.Add(lifetime).Unix(),
"jti": rand.Text(),
})
return base64.RawURLEncoding.EncodeToString([]byte(`{"alg":"none","typ":"JWT"}`)) + "." + base64.RawURLEncoding.EncodeToString(claims) + "."
}
+55
View File
@@ -0,0 +1,55 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import "time"
const artifactServicePath = "/twirp/github.actions.results.api.v1.ArtifactService"
type ( // the ArtifactService messages in the protobuf JSON mapping, which writes int64 as strings
createArtifactResponse struct {
Ok bool `json:"ok"`
SignedUploadURL string `json:"signedUploadUrl"`
}
finalizeArtifactRequest struct {
WorkflowRunBackendID string
Name string
Size int64 `json:",string"`
Hash string
}
finalizeArtifactResponse struct {
Ok bool `json:"ok"`
ArtifactID int64 `json:"artifactId,string"`
}
listArtifactsRequest struct {
WorkflowRunBackendID string
NameFilter string
IDFilter int64 `json:",string"`
}
listArtifactsResponse struct {
Artifacts []monolithArtifact `json:"artifacts"`
}
monolithArtifact struct {
WorkflowRunBackendID string `json:"workflowRunBackendId"` // the toolkit sends it back to GetSignedArtifactURL
DatabaseID int64 `json:"databaseId,string"`
Name string `json:"name"`
Size int64 `json:"size,string"`
CreatedAt time.Time `json:"createdAt"` // whole seconds, the toolkit rejects the trimmed fractions json writes
Digest string `json:"digest"` // empty until the upload is finalized
}
getSignedArtifactURLRequest struct {
WorkflowRunBackendID string
Name string
}
getSignedArtifactURLResponse struct {
SignedURL string `json:"signedUrl"`
}
createArtifactRequest = getSignedArtifactURLRequest
deleteArtifactRequest = getSignedArtifactURLRequest
deleteArtifactResponse = finalizeArtifactResponse
)
+2 -7
View File
@@ -14,13 +14,8 @@ const dryrunContextKeyVal = dryrunContextKey("dryrun")
// Dryrun returns true if the current context is dryrun // Dryrun returns true if the current context is dryrun
func Dryrun(ctx context.Context) bool { func Dryrun(ctx context.Context) bool {
val := ctx.Value(dryrunContextKeyVal) dryrun, _ := ctx.Value(dryrunContextKeyVal).(bool)
if val != nil { return dryrun
if dryrun, ok := val.(bool); ok {
return dryrun
}
}
return false
} }
// WithDryrun adds a value to the context for dryrun // WithDryrun adds a value to the context for dryrun
+19 -23
View File
@@ -21,8 +21,7 @@ type Conditional func(ctx context.Context) bool
// NewInfoExecutor is an executor that logs messages // NewInfoExecutor is an executor that logs messages
func NewInfoExecutor(format string, args ...any) Executor { func NewInfoExecutor(format string, args ...any) Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := Logger(ctx) Logger(ctx).Infof(format, args...)
logger.Infof(format, args...)
return nil return nil
} }
} }
@@ -30,8 +29,7 @@ func NewInfoExecutor(format string, args ...any) Executor {
// NewDebugExecutor is an executor that logs messages // NewDebugExecutor is an executor that logs messages
func NewDebugExecutor(format string, args ...any) Executor { func NewDebugExecutor(format string, args ...any) Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := Logger(ctx) Logger(ctx).Debugf(format, args...)
logger.Debugf(format, args...)
return nil return nil
} }
} }
@@ -39,9 +37,7 @@ func NewDebugExecutor(format string, args ...any) Executor {
// NewPipelineExecutor creates a new executor from a series of other executors // NewPipelineExecutor creates a new executor from a series of other executors
func NewPipelineExecutor(executors ...Executor) Executor { func NewPipelineExecutor(executors ...Executor) Executor {
if len(executors) == 0 { if len(executors) == 0 {
return func(ctx context.Context) error { return NewErrorExecutor(nil)
return nil
}
} }
var rtn Executor var rtn Executor
for _, executor := range executors { for _, executor := range executors {
@@ -61,6 +57,13 @@ func NewErrorExecutor(err error) Executor {
} }
} }
func RecoverPanic(err *error) { // defer directly in workers to keep stacks in host logs
if recovered := recover(); recovered != nil {
logging.Errorf("panic: %v\n%s", recovered, debug.Stack())
*err = fmt.Errorf("panic: %v", recovered)
}
}
// NewParallelExecutor creates a new executor from a parallel of other executors // NewParallelExecutor creates a new executor from a parallel of other executors
func NewParallelExecutor(parallel int, executors ...Executor) Executor { func NewParallelExecutor(parallel int, executors ...Executor) Executor {
if len(executors) == 0 { if len(executors) == 0 {
@@ -80,8 +83,8 @@ func NewParallelExecutor(parallel int, executors ...Executor) Executor {
logging.Infof("NewParallelExecutor: Creating %d workers for %d executors", parallel, len(executors)) logging.Infof("NewParallelExecutor: Creating %d workers for %d executors", parallel, len(executors))
for i := 0; i < parallel; i++ { for workerID := range parallel {
go func(workerID int, work <-chan Executor, errs chan<- error) { go func() {
logging.Debugf("Worker %d started", workerID) logging.Debugf("Worker %d started", workerID)
taskCount := 0 taskCount := 0
for executor := range work { for executor := range work {
@@ -91,17 +94,12 @@ func NewParallelExecutor(parallel int, executors ...Executor) Executor {
// goroutine which would leave the runner process hung. // goroutine which would leave the runner process hung.
// https://gitea.com/gitea/runner/issues/371 // https://gitea.com/gitea/runner/issues/371
errs <- func() (err error) { errs <- func() (err error) {
defer func() { defer RecoverPanic(&err)
if r := recover(); r != nil {
logging.Errorf("panic in executor: %v\n%s", r, debug.Stack())
err = fmt.Errorf("panic: %v", r)
}
}()
return executor(ctx) return executor(ctx)
}() }()
} }
logging.Debugf("Worker %d finished (%d tasks executed)", workerID, taskCount) logging.Debugf("Worker %d finished (%d tasks executed)", workerID, taskCount)
}(i, work, errs) }()
} }
for i := range executors { for i := range executors {
@@ -150,12 +148,7 @@ func (e Executor) If(conditional Conditional) Executor {
// IfNot only runs this executor if conditional is true // IfNot only runs this executor if conditional is true
func (e Executor) IfNot(conditional Conditional) Executor { func (e Executor) IfNot(conditional Conditional) Executor {
return func(ctx context.Context) error { return e.If(func(ctx context.Context) bool { return !conditional(ctx) })
if !conditional(ctx) {
return e(ctx)
}
return nil
}
} }
// IfBool only runs this executor if conditional is true // IfBool only runs this executor if conditional is true
@@ -171,7 +164,10 @@ func (e Executor) Finally(finally Executor) Executor {
err := e(ctx) err := e(ctx)
err2 := finally(ctx) err2 := finally(ctx)
if err2 != nil { if err2 != nil {
return fmt.Errorf("error occurred running finally: %v (original error: %v)", err2, err) if err != nil {
return fmt.Errorf("error occurred running finally: %w (original error: %w)", err2, err)
}
return fmt.Errorf("error occurred running finally: %w (original error: %v)", err2, err)
} }
return err return err
} }
+61 -102
View File
@@ -7,8 +7,7 @@ package common
import ( import (
"context" "context"
"errors" "errors"
"reflect" "fmt"
"strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
@@ -18,42 +17,25 @@ import (
) )
func TestNewWorkflow(t *testing.T) { func TestNewWorkflow(t *testing.T) {
assert := assert.New(t)
ctx := t.Context() ctx := t.Context()
require.NoError(t, NewPipelineExecutor()(ctx))
require.EqualError(t, NewErrorExecutor(errors.New("test error"))(ctx), "test error")
// empty
emptyWorkflow := NewPipelineExecutor()
assert.NoError(emptyWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
// error case
errorWorkflow := NewErrorExecutor(errors.New("test error"))
assert.Error(errorWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
// multiple success case
runcount := 0 runcount := 0
successWorkflow := NewPipelineExecutor( increment := func(_ctx context.Context) error {
func(ctx context.Context) error { runcount++
runcount++ return nil
return nil }
}, require.NoError(t, NewPipelineExecutor(increment, increment)(ctx))
func(ctx context.Context) error { assert.Equal(t, 2, runcount)
runcount++
return nil
})
assert.NoError(successWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(2, runcount)
} }
// concurrencyProbe returns an executor recording the peak number of concurrent copies. Copies
// block until wantActive are in flight so the peak is exact without sleeping, and later copies
// find the gate already open so the last one still finishes with no partner left.
func concurrencyProbe(wantActive int32) (exec Executor, count, maxActive *atomic.Int32) { func concurrencyProbe(wantActive int32) (exec Executor, count, maxActive *atomic.Int32) {
var counted, active, peak atomic.Int32 var counted, active, peak atomic.Int32
var once sync.Once var once sync.Once
reached := make(chan struct{}) reached := make(chan struct{})
return func(ctx context.Context) error { return func(_ctx context.Context) error {
counted.Add(1) counted.Add(1)
running := active.Add(1) running := active.Add(1)
for { for {
@@ -72,18 +54,14 @@ func concurrencyProbe(wantActive int32) (exec Executor, count, maxActive *atomic
} }
func TestNewParallelExecutor(t *testing.T) { func TestNewParallelExecutor(t *testing.T) {
ctx := t.Context() for name, parallel := range map[string]int{"parallel": 2, "below minimum": 0} {
t.Run(name, func(t *testing.T) {
exec, count, maxActive := concurrencyProbe(2) exec, count, maxActive := concurrencyProbe(int32(max(1, parallel)))
require.NoError(t, NewParallelExecutor(2, exec, exec, exec)(ctx)) require.NoError(t, NewParallelExecutor(parallel, exec, exec, exec)(t.Context()))
assert.Equal(t, int32(3), count.Load(), "should run all 3 executors") assert.Equal(t, int32(3), count.Load())
assert.Equal(t, int32(2), maxActive.Load(), "should run at most 2 executors in parallel") assert.Equal(t, int32(max(1, parallel)), maxActive.Load())
})
// parallelism below 1 falls back to a single worker }
exec, count, maxActive = concurrencyProbe(1)
require.NoError(t, NewParallelExecutor(0, exec, exec, exec)(ctx))
assert.Equal(t, int32(3), count.Load(), "should run all 3 executors")
assert.Equal(t, int32(1), maxActive.Load(), "should run at most 1 executor in parallel")
} }
func TestNewParallelExecutorEmpty(t *testing.T) { func TestNewParallelExecutorEmpty(t *testing.T) {
@@ -99,66 +77,49 @@ func TestNewParallelExecutorEmpty(t *testing.T) {
assert.ErrorIs(err, context.Canceled) assert.ErrorIs(err, context.Canceled)
} }
func TestNewParallelExecutorFailed(t *testing.T) { func TestNewParallelExecutorRunsAllTasksAfterFailureOrCancellation(t *testing.T) {
assert := assert.New(t) for name, tc := range map[string]struct {
canceled bool
ctx, cancel := context.WithCancel(t.Context()) parallel, tasks int
cancel() }{
"failure": {false, 2, 5},
count := 0 "canceled": {true, 2, 5},
errorWorkflow := NewPipelineExecutor(func(ctx context.Context) error { "canceled single task": {true, 1, 1},
count++ } {
return errors.New("fake error") t.Run(name, func(t *testing.T) {
}) ctx, cancel := context.WithCancel(t.Context())
err := NewParallelExecutor(1, errorWorkflow)(ctx) defer cancel()
assert.Equal(1, count) if tc.canceled {
assert.ErrorIs(context.Canceled, err) cancel()
} }
var calls atomic.Int32
func TestNewParallelExecutorCanceled(t *testing.T) { executors := make([]Executor, tc.tasks)
assert := assert.New(t) for i := range executors {
executors[i] = func(_ctx context.Context) error {
ctx, cancel := context.WithCancel(t.Context()) calls.Add(1)
cancel() if i == tc.tasks/2 {
return errors.New("fake error")
errExpected := errors.New("fake error") }
return nil
var count atomic.Int32 }
successWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count.Add(1)
return nil
})
errorWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count.Add(1)
return errExpected
})
err := NewParallelExecutor(3, errorWorkflow, successWorkflow, successWorkflow)(ctx)
assert.Equal(int32(3), count.Load())
assert.Error(errExpected, err) //nolint:testifylint // pre-existing issue from nektos/act
}
func TestNewParallelExecutorRunsRemainingAfterFailure(t *testing.T) {
var successCount atomic.Int32
executors := make([]Executor, 5)
for i := range executors {
executors[i] = func(ctx context.Context) error {
if i == 2 {
return errors.New("fake error")
} }
successCount.Add(1)
return nil
}
}
require.Error(t, NewParallelExecutor(2, executors...)(t.Context())) err := NewParallelExecutor(tc.parallel, executors...)(ctx)
assert.Equal(t, int32(4), successCount.Load(), "a failing executor must not stop the others") if tc.canceled {
require.ErrorIs(t, err, context.Canceled)
} else {
require.EqualError(t, err, "fake error")
}
assert.Equal(t, int32(tc.tasks), calls.Load())
})
}
} }
func TestExecutorConditionalsAndFinally(t *testing.T) { func TestExecutorConditionalsAndFinally(t *testing.T) {
ctx := t.Context() ctx := t.Context()
var calls []string var calls []string
record := func(name string) Executor { record := func(name string) Executor {
return func(ctx context.Context) error { return func(_ctx context.Context) error {
calls = append(calls, name) calls = append(calls, name)
return nil return nil
} }
@@ -170,19 +131,17 @@ func TestExecutorConditionalsAndFinally(t *testing.T) {
require.NoError(t, record("if-bool").IfBool(true)(ctx)) require.NoError(t, record("if-bool").IfBool(true)(ctx))
require.NoError(t, record("main").Finally(record("finally"))(ctx)) require.NoError(t, record("main").Finally(record("finally"))(ctx))
want := []string{"if-true", "if-not", "if-bool", "main", "finally"} assert.Equal(t, []string{"if-true", "if-not", "if-bool", "main", "finally"}, calls)
if !reflect.DeepEqual(calls, want) {
t.Fatalf("calls = %v, want %v", calls, want)
}
} }
func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) { func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
mainErr := errors.New("main failed")
finalErr := errors.New("cleanup failed") finalErr := errors.New("cleanup failed")
for _, mainErr := range []error{context.Canceled, nil} {
err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context()) err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context())
require.Error(t, err) require.EqualError(t, err, fmt.Sprintf("error occurred running finally: cleanup failed (original error: %v)", mainErr))
if !strings.Contains(err.Error(), "cleanup failed") || !strings.Contains(err.Error(), "main failed") { require.ErrorIs(t, err, finalErr)
t.Fatalf("finally error = %q, want both cleanup and original error", err) if mainErr != nil {
require.ErrorIs(t, err, mainErr)
}
} }
} }
+125 -14
View File
@@ -8,6 +8,7 @@ import (
"bytes" "bytes"
"compress/gzip" "compress/gzip"
"context" "context"
"encoding/json"
"errors" "errors"
"io" "io"
"math/rand/v2" "math/rand/v2"
@@ -16,14 +17,32 @@ import (
"net/url" "net/url"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"runtime" "runtime"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
) )
var actionFetches sync.Map
const treeSizes = ".sizes" // beside the tree rather than in it, which is copied into job containers as it is
type treeFile struct {
Name []byte // not a string, which JSON would turn into UTF-8, as git allows any bytes but NUL
Size int64
}
type actionFetch struct {
done chan struct{}
dir, sha string
err error
retry bool // the leader was cancelled or panicked, so followers fetch themselves
}
var archiveRetryDelay = func(retryAfter string) time.Duration { // as actions/runner backs off var archiveRetryDelay = func(retryAfter string) time.Duration { // as actions/runner backs off
if seconds, err := strconv.Atoi(retryAfter); err == nil { if seconds, err := strconv.Atoi(retryAfter); err == nil {
return min(max(time.Duration(seconds)*time.Second, 10*time.Second), 10*time.Minute) return min(max(time.Duration(seconds)*time.Second, 10*time.Second), 10*time.Minute)
@@ -32,6 +51,36 @@ var archiveRetryDelay = func(retryAfter string) time.Duration { // as actions/ru
} }
func FetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, string, error) { // input.Dir is one clone URL and ref's, holding a directory per commit func FetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, string, error) { // input.Dir is one clone URL and ref's, holding a directory per commit
if !reflect.ValueOf(input).Comparable() {
return fetchAction(ctx, input)
}
for ctx.Err() == nil {
call := &actionFetch{done: make(chan struct{}), retry: true}
previous, loaded := actionFetches.LoadOrStore(input, call)
if !loaded {
func() {
defer func() { actionFetches.Delete(input); close(call.done) }()
call.dir, call.sha, call.err = fetchAction(ctx, input)
call.retry = ctx.Err() != nil
}()
if call.retry {
continue
}
return call.dir, call.sha, call.err
}
call, _ = previous.(*actionFetch)
select {
case <-ctx.Done():
case <-call.done:
if !call.retry && ctx.Err() == nil {
return call.dir, call.sha, call.err
}
}
}
return "", "", ctx.Err()
}
func fetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, string, error) {
unlock, err := AcquireCloneLock(ctx, input.Dir) unlock, err := AcquireCloneLock(ctx, input.Dir)
if err != nil { if err != nil {
return "", "", err return "", "", err
@@ -45,9 +94,14 @@ func FetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, s
record := filepath.Join(input.Dir, "ref") record := filepath.Join(input.Dir, "ref")
data, _ := os.ReadFile(record) data, _ := os.ReadFile(record)
sha, etag, _ := strings.Cut(string(data), "\n") sha, etag, _ := strings.Cut(string(data), "\n")
if !isFullObjectID(sha) || !exists(tree(sha)) { switch {
case !isFullObjectID(sha):
sha, etag = "", "" sha, etag = "", ""
} else if input.OfflineMode || isFullObjectID(input.Ref) { case input.OfflineMode && exists(tree(sha)) && !exists(tree(sha)+treeSizes):
return tree(sha), sha, nil // from an older runner, with no sizes to check, so offline mode uses it as before
case !treeIntact(input.Dir, sha):
sha, etag = "", ""
case input.OfflineMode || isFullObjectID(input.Ref):
return tree(sha), sha, nil return tree(sha), sha, nil
} }
@@ -132,7 +186,7 @@ func downloadArchiveOnce(ctx context.Context, client *http.Client, archive, toke
newEtag := response.Header.Get("ETag") newEtag := response.Header.Get("ETag")
gz, err := gzip.NewReader(response.Body) gz, err := gzip.NewReader(response.Body)
if err == nil { if err == nil {
sha, err = extractTree(gz, dir, !strings.HasSuffix(newEtag, `-false"`)) // Gitea's mark of an unprefixed archive sha, err = extractTree(ctx, gz, dir, !strings.HasSuffix(newEtag, `-false"`)) // Gitea's mark of an unprefixed archive
} }
var netErr net.Error var netErr net.Error
if errors.Is(err, io.ErrUnexpectedEOF) || errors.As(err, &netErr) { if errors.Is(err, io.ErrUnexpectedEOF) || errors.As(err, &netErr) {
@@ -150,18 +204,18 @@ func exportClone(ctx context.Context, input NewGitCloneExecutorInput) (string, e
clone := input clone := input
clone.Dir = filepath.Join(input.Dir, "git") clone.Dir = filepath.Join(input.Dir, "git")
sha, _, err := resolveClone(ctx, clone) sha, _, err := resolveClone(ctx, clone)
if err != nil || exists(filepath.Join(input.Dir, sha)) { if err != nil || treeIntact(input.Dir, sha) {
return sha, err return sha, err
} }
archive, err := runGit(ctx, clone.Dir, nil, "archive", "--format=tar", sha) archive, err := runGit(ctx, clone.Dir, nil, "archive", "--format=tar", sha)
if err != nil { if err != nil {
return "", err return "", err
} }
_, err = extractTree(bytes.NewReader(archive), input.Dir, false) _, err = extractTree(ctx, bytes.NewReader(archive), input.Dir, false)
return sha, err return sha, err
} }
func extractTree(archive io.Reader, dir string, strip bool) (string, error) { // into dir/<commit of the pax header>, strip dropping the directory wrapping the files func extractTree(ctx context.Context, archive io.Reader, dir string, strip bool) (string, error) { // into dir/<commit of the pax header>, strip dropping the directory wrapping the files
temp, err := os.MkdirTemp(dir, "tmp-") temp, err := os.MkdirTemp(dir, "tmp-")
if err != nil { if err != nil {
return "", err return "", err
@@ -174,6 +228,7 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
defer root.Close() defer root.Close()
sha := "" sha := ""
sizes := map[string]int64{}
reader := tar.NewReader(archive) reader := tar.NewReader(archive)
for { for {
header, err := reader.Next() header, err := reader.Next()
@@ -184,7 +239,7 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
return "", err return "", err
} }
if header.Typeflag == tar.TypeXGlobalHeader { if header.Typeflag == tar.TypeXGlobalHeader {
if sha = header.PAXRecords["comment"]; isFullObjectID(sha) && exists(filepath.Join(dir, sha)) { if sha = header.PAXRecords["comment"]; isFullObjectID(sha) && treeIntact(dir, sha) {
return sha, nil return sha, nil
} }
continue continue
@@ -200,10 +255,10 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
case tar.TypeDir: case tar.TypeDir:
err = root.MkdirAll(name, 0o755) err = root.MkdirAll(name, 0o755)
case tar.TypeReg: case tar.TypeReg:
err = writeTreeFile(root, name, header.Mode, reader) sizes[name], err = writeTreeFile(root, name, header.Mode, reader)
case tar.TypeSymlink: case tar.TypeSymlink:
if err = root.Symlink(header.Linkname, name); err != nil && runtime.GOOS == "windows" { if err = root.Symlink(header.Linkname, name); err != nil && runtime.GOOS == "windows" {
err = writeTreeFile(root, name, 0, strings.NewReader(header.Linkname)) sizes[name], err = writeTreeFile(root, name, 0, strings.NewReader(header.Linkname))
} }
} }
if err != nil { if err != nil {
@@ -216,20 +271,76 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
if err := root.Close(); err != nil { if err := root.Close(); err != nil {
return "", err return "", err
} }
return sha, os.Rename(temp, filepath.Join(dir, sha)) return sha, installTree(ctx, dir, sha, temp, sizes)
} }
func writeTreeFile(root *os.Root, name string, mode int64, content io.Reader) error { var lockTree = AcquireCloneLock // replaced in tests to see installTree wait for a job holding the tree
// installTree moves temp into place as dir/sha, replacing a tree that failed treeIntact.
func installTree(ctx context.Context, dir, sha, temp string, sizes map[string]int64) error {
files := make([]treeFile, 0, len(sizes))
for name, size := range sizes {
files = append(files, treeFile{[]byte(name), size})
}
data, err := json.Marshal(files)
if err != nil {
return err
}
// Written before the swap: it describes the commit, not a copy of it, so it holds for whichever tree is left.
if err := os.WriteFile(filepath.Join(dir, sha+treeSizes), data, 0o600); err != nil { // a torn record fails treeIntact, so it needs no rename
return err
}
tree := filepath.Join(dir, sha)
if !exists(tree) {
return os.Rename(temp, tree)
}
stale, err := os.MkdirTemp(dir, "tmp-")
if err != nil {
return err
}
defer os.RemoveAll(stale) // after the unlock, as no job reads the old tree once it is moved
unlock, err := lockTree(ctx, tree) // jobs copying the old tree hold it
if err != nil {
return err
}
defer unlock()
if err := os.Rename(tree, filepath.Join(stale, sha)); err != nil {
return err
}
return os.Rename(temp, tree)
}
// treeIntact reports whether dir/sha still holds each file extracted into it at the size it was written.
// Only sizes are compared, not content: a stat per file is cheap enough for every job, hashing is not.
func treeIntact(dir, sha string) bool {
data, err := os.ReadFile(filepath.Join(dir, sha+treeSizes))
if err != nil || !exists(filepath.Join(dir, sha)) {
return false
}
var files []treeFile
if json.Unmarshal(data, &files) != nil {
return false
}
for _, file := range files {
info, err := os.Lstat(filepath.Join(dir, sha, filepath.FromSlash(string(file.Name))))
if err != nil || !info.Mode().IsRegular() || info.Size() != file.Size {
return false
}
}
return true
}
func writeTreeFile(root *os.Root, name string, mode int64, content io.Reader) (int64, error) {
perm := os.FileMode(0o644) perm := os.FileMode(0o644)
if mode&0o111 != 0 { if mode&0o111 != 0 {
perm = 0o755 perm = 0o755
} }
file, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm) file, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm)
if err != nil { if err != nil {
return err return 0, err
} }
_, err = io.Copy(file, content) n, err := io.Copy(file, content)
return errors.Join(err, file.Close()) return n, errors.Join(err, file.Close())
} }
func exists(name string) bool { func exists(name string) bool {
+197 -10
View File
@@ -6,6 +6,7 @@ package git
import ( import (
"archive/tar" "archive/tar"
"bytes" "bytes"
"context"
"encoding/base64" "encoding/base64"
"fmt" "fmt"
"net/http" "net/http"
@@ -14,11 +15,15 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"runtime"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"testing" "testing"
"time" "time"
"gitea.com/gitea/runner/act/common"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -37,6 +42,7 @@ func TestFetchAction(t *testing.T) {
require.NoError(t, gitCmd("-C", remote, "add", ".")) require.NoError(t, gitCmd("-C", remote, "add", "."))
require.NoError(t, gitCmd("-C", remote, "commit", "-m", "initial")) require.NoError(t, gitCmd("-C", remote, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", remote, "tag", "v1")) require.NoError(t, gitCmd("-C", remote, "tag", "v1"))
expectedSha := gitRevParse(t, remote, "main")
gitPath, err := exec.LookPath("git") gitPath, err := exec.LookPath("git")
require.NoError(t, err) require.NoError(t, err)
@@ -44,6 +50,7 @@ func TestFetchAction(t *testing.T) {
var mu sync.Mutex var mu sync.Mutex
var archiveRequests []http.Header var archiveRequests []http.Header
var mirror *httptest.Server var mirror *httptest.Server
archiveCache := map[string][]byte{}
requests, archives, failures := 0, "", 0 requests, archives, failures := 0, "", 0
handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
mu.Lock() mu.Lock()
@@ -80,7 +87,12 @@ func TestFetchAction(t *testing.T) {
if archives != "unprefixed" { if archives != "unprefixed" {
args = append(args, "--prefix=action/") args = append(args, "--prefix=action/")
} }
archive, _ := runGit(request.Context(), remote, nil, args...) archiveKey := strings.Join(args, "\x00")
archive, ok := archiveCache[archiveKey]
if !ok {
archive, _ = runGit(request.Context(), remote, nil, args...)
archiveCache[archiveKey] = archive
}
writer.Header().Set("ETag", etag) writer.Header().Set("ETag", etag)
_, _ = writer.Write(archive) _, _ = writer.Write(archive)
}) })
@@ -102,12 +114,80 @@ func TestFetchAction(t *testing.T) {
return NewGitCloneExecutorInput{URL: server.URL + "/org/action", Ref: ref, Dir: t.TempDir(), Token: "secret"} return NewGitCloneExecutorInput{URL: server.URL + "/org/action", Ref: ref, Dir: t.TempDir(), Token: "secret"}
} }
t.Run("shares identical fetches and isolates cancellation, panics and differing inputs", func(t *testing.T) {
for _, scenario := range []string{"shared", "leader cancelled", "leader panics", "different token"} {
t.Run(scenario, func(t *testing.T) {
serve("prefixed", 0)
input := newInput("main")
started, release := make(chan struct{}), make(chan struct{})
releaseFetches := sync.OnceFunc(func() { close(release) })
t.Cleanup(releaseFetches)
var once sync.Once
var panicked atomic.Bool
transport := actionFetchTransport(func(request *http.Request) (*http.Response, error) {
once.Do(func() { close(started) })
select {
case <-request.Context().Done():
return nil, request.Context().Err()
case <-release:
}
if scenario == "leader panics" && panicked.CompareAndSwap(false, true) {
panic("broken archive")
}
return http.DefaultTransport.RoundTrip(request)
})
input.Transport = &transport
leaderCtx, cancelLeader := context.WithCancel(t.Context())
defer cancelLeader()
waiting := &actionFetchContext{Context: t.Context(), waiting: make(chan struct{})}
followerInput := input
expectedRequests := 1
if scenario == "different token" {
followerInput.Token, expectedRequests = "other", 2
}
leaderDone, followerDone := make(chan error, 1), make(chan error, 1)
fetch := func(ctx context.Context, input NewGitCloneExecutorInput, done chan<- error) {
var err error
defer func() { done <- err }()
defer common.RecoverPanic(&err)
var tree, sha string
if tree, sha, err = FetchAction(ctx, input); err == nil {
assert.Equal(t, expectedSha, sha)
assert.Equal(t, filepath.Join(input.Dir, sha), tree)
}
}
go fetch(leaderCtx, input, leaderDone)
<-started
go fetch(waiting, followerInput, followerDone)
<-waiting.waiting
if scenario == "leader cancelled" {
cancelLeader()
require.ErrorIs(t, <-leaderDone, context.Canceled)
}
releaseFetches()
switch scenario {
case "leader cancelled":
case "leader panics":
require.ErrorContains(t, <-leaderDone, "panic: broken archive")
default:
require.NoError(t, <-leaderDone)
}
require.NoError(t, <-followerDone)
_, headers := served()
assert.Len(t, headers, expectedRequests)
if scenario == "different token" {
assert.NotEqual(t, headers[0].Get("Authorization"), headers[1].Get("Authorization"))
}
})
}
})
t.Run("downloads, revalidates and follows a moved tag", func(t *testing.T) { t.Run("downloads, revalidates and follows a moved tag", func(t *testing.T) {
serve("prefixed", 0) serve("prefixed", 0)
input := newInput("v1") input := newInput("v1")
tree, sha, err := FetchAction(t.Context(), input) tree, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "v1"), sha) assert.Equal(t, expectedSha, sha)
assert.Equal(t, filepath.Join(input.Dir, sha), tree) assert.Equal(t, filepath.Join(input.Dir, sha), tree)
assert.NoDirExists(t, filepath.Join(tree, ".git")) assert.NoDirExists(t, filepath.Join(tree, ".git"))
info, err := os.Stat(filepath.Join(tree, "run.sh")) info, err := os.Stat(filepath.Join(tree, "run.sh"))
@@ -123,9 +203,10 @@ func TestFetchAction(t *testing.T) {
require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved")) require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved"))
require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1")) require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1"))
expectedSha = gitRevParse(t, remote, "v1")
moved, movedSha, err := FetchAction(t.Context(), input) moved, movedSha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "v1"), movedSha) assert.Equal(t, expectedSha, movedSha)
assert.NotEqual(t, tree, moved) assert.NotEqual(t, tree, moved)
assert.FileExists(t, filepath.Join(tree, "action.yml")) assert.FileExists(t, filepath.Join(tree, "action.yml"))
@@ -133,10 +214,17 @@ func TestFetchAction(t *testing.T) {
_, offlineSha, err := FetchAction(t.Context(), input) _, offlineSha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, movedSha, offlineSha) assert.Equal(t, movedSha, offlineSha)
require.NoError(t, os.Truncate(filepath.Join(moved, "action.yml"), 4)) // fetched again, offline or not
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err)
require.NoError(t, os.Remove(moved+treeSizes)) // from an older runner, used as it is offline
reused, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, moved, reused)
requests, headers := served() requests, headers := served()
assert.Equal(t, 3, requests) assert.Equal(t, 4, requests)
require.Len(t, headers, 3) require.Len(t, headers, 4)
assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), headers[0].Get("Authorization")) assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), headers[0].Get("Authorization"))
assert.Empty(t, headers[0].Get("If-None-Match")) assert.Empty(t, headers[0].Get("If-None-Match"))
assert.Equal(t, fmt.Sprintf(`W/"%s-tar.gz-true"`, sha), headers[1].Get("If-None-Match")) assert.Equal(t, fmt.Sprintf(`W/"%s-tar.gz-true"`, sha), headers[1].Get("If-None-Match"))
@@ -145,7 +233,7 @@ func TestFetchAction(t *testing.T) {
t.Run("downloads a pinned commit once and rejects a short one", func(t *testing.T) { t.Run("downloads a pinned commit once and rejects a short one", func(t *testing.T) {
serve("prefixed", 0) serve("prefixed", 0)
sha := gitRevParse(t, remote, "main") sha := expectedSha
input := newInput(sha) input := newInput(sha)
for range 2 { for range 2 {
tree, got, err := FetchAction(t.Context(), input) tree, got, err := FetchAction(t.Context(), input)
@@ -160,12 +248,74 @@ func TestFetchAction(t *testing.T) {
assert.NoDirExists(t, filepath.Join(input.Dir, "git")) assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
}) })
t.Run("downloads again over a tree whose files changed size or that has no record, not under a job copying it", func(t *testing.T) {
sha := gitRevParse(t, remote, "main")
for _, mode := range []string{"prefixed", "none"} {
serve(mode, 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.WriteFile(filepath.Join(tree, "trampoline.js"), nil, 0o644)) // the runner's own, no damage
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err, mode)
for _, damage := range []func() error{
func() error { return os.Truncate(yml, 4) },
func() error { return os.Remove(tree + treeSizes) },
} {
require.NoError(t, damage(), mode)
_, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
body, err := os.ReadFile(yml)
require.NoError(t, err, mode)
assert.Equal(t, "name: action\n", string(body), mode)
}
_, headers := served()
assert.Len(t, headers, 3, mode)
}
serve("prefixed", 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.Truncate(yml, 4))
unlock, err := AcquireCloneLock(t.Context(), tree)
require.NoError(t, err)
release := sync.OnceFunc(unlock)
defer release()
waiting := make(chan struct{})
lockTree = func(ctx context.Context, dir string) (func(), error) {
close(waiting)
return AcquireCloneLock(ctx, dir)
}
t.Cleanup(func() { lockTree = AcquireCloneLock })
done := make(chan error, 1)
go func() {
_, _, err := FetchAction(t.Context(), input)
done <- err
}()
select {
case err := <-done:
t.Fatalf("FetchAction replaced the tree without waiting for the job holding it: %v", err)
case <-waiting:
}
info, err := os.Stat(yml)
require.NoError(t, err)
assert.Equal(t, int64(4), info.Size(), "the tree a job holds is left alone")
release()
require.NoError(t, <-done)
body, err := os.ReadFile(yml)
require.NoError(t, err)
assert.Equal(t, "name: action\n", string(body))
})
t.Run("retries a failing download and drops the token on a redirect to another origin", func(t *testing.T) { t.Run("retries a failing download and drops the token on a redirect to another origin", func(t *testing.T) {
serve("redirect", 2) serve("redirect", 2)
input := newInput("main") input := newInput("main")
_, sha, err := FetchAction(t.Context(), input) _, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "main"), sha) assert.Equal(t, expectedSha, sha)
assert.NoDirExists(t, filepath.Join(input.Dir, "git")) assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
_, headers := served() _, headers := served()
require.Len(t, headers, 4) require.Len(t, headers, 4)
@@ -178,7 +328,7 @@ func TestFetchAction(t *testing.T) {
serve(mode, 0) serve(mode, 0)
tree, sha, err := FetchAction(t.Context(), newInput("main")) tree, sha, err := FetchAction(t.Context(), newInput("main"))
require.NoError(t, err, mode) require.NoError(t, err, mode)
assert.Equal(t, gitRevParse(t, remote, "main"), sha, mode) assert.Equal(t, expectedSha, sha, mode)
assert.FileExists(t, filepath.Join(tree, "action.yml"), mode) assert.FileExists(t, filepath.Join(tree, "action.yml"), mode)
assert.NoDirExists(t, filepath.Join(tree, ".git"), mode) assert.NoDirExists(t, filepath.Join(tree, ".git"), mode)
_, headers := served() _, headers := served()
@@ -193,13 +343,30 @@ func TestFetchAction(t *testing.T) {
input.URL = "https://example.invalid/org/action" input.URL = "https://example.invalid/org/action"
_, sha, err := FetchAction(t.Context(), input) _, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "main"), sha) assert.Equal(t, expectedSha, sha)
_, headers := served() _, headers := served()
assert.Len(t, headers, 1) assert.Len(t, headers, 1)
assert.NoDirExists(t, filepath.Join(input.Dir, "git")) assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
}) })
} }
type actionFetchTransport func(*http.Request) (*http.Response, error)
func (transport actionFetchTransport) RoundTrip(request *http.Request) (*http.Response, error) {
return transport(request)
}
type actionFetchContext struct {
context.Context
waiting chan struct{}
once sync.Once
}
func (ctx *actionFetchContext) Done() <-chan struct{} {
ctx.once.Do(func() { close(ctx.waiting) })
return ctx.Context.Done()
}
func TestExtractTreeConfinesEntries(t *testing.T) { func TestExtractTreeConfinesEntries(t *testing.T) {
outside := t.TempDir() outside := t.TempDir()
for name, entries := range map[string][]*tar.Header{ for name, entries := range map[string][]*tar.Header{
@@ -216,10 +383,30 @@ func TestExtractTreeConfinesEntries(t *testing.T) {
require.NoError(t, writer.Close()) require.NoError(t, writer.Close())
dir := t.TempDir() dir := t.TempDir()
_, err := extractTree(&archive, dir, false) _, err := extractTree(t.Context(), &archive, dir, false)
require.Error(t, err) require.Error(t, err)
assert.NoFileExists(t, filepath.Join(dir, "escape")) assert.NoFileExists(t, filepath.Join(dir, "escape"))
assert.NoFileExists(t, filepath.Join(outside, "escape")) assert.NoFileExists(t, filepath.Join(outside, "escape"))
}) })
} }
} }
func TestTreeIntactFindsNamesThatAreNotUTF8(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("macOS and Windows store no name that is not UTF-8")
}
sha := strings.Repeat("a", 40)
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": sha}}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "caf\xe9.js", Typeflag: tar.TypeReg, Mode: 0o644, Size: 1, Format: tar.FormatGNU}))
_, err := writer.Write([]byte("x"))
require.NoError(t, err)
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err = extractTree(t.Context(), &archive, dir, false)
require.NoError(t, err)
require.FileExists(t, filepath.Join(dir, sha, "caf\xe9.js"))
assert.True(t, treeIntact(dir, sha))
}
+24 -9
View File
@@ -10,6 +10,7 @@ import (
"encoding/base64" "encoding/base64"
"errors" "errors"
"fmt" "fmt"
"io"
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
@@ -79,25 +80,25 @@ func (e *Error) Unwrap() error { return e.err }
func (e *Error) Commit() string { return e.commit } func (e *Error) Commit() string { return e.commit }
// FindGitRevision get the current git revision // FindGitRevision get the current git revision
func FindGitRevision(ctx context.Context, file string) (shortSHA, sha string, err error) { func FindGitRevision(ctx context.Context, file string) (string, error) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
sha, err = gitOutput(ctx, file, "rev-parse", "--verify", "HEAD^{commit}") sha, err := gitOutput(ctx, file, "rev-parse", "--verify", "HEAD^{commit}")
if err != nil { if err != nil {
logger.WithError(err).Error("path", file, "not located inside a git repository") logger.WithError(err).Error("path", file, "not located inside a git repository")
return "", "", err return "", err
} }
if len(sha) < 7 { if len(sha) < 7 {
return "", "", errors.New("HEAD could not be resolved") return "", errors.New("HEAD could not be resolved")
} }
logger.Debugf("Found revision: %s", sha) logger.Debugf("Found revision: %s", sha)
return sha[:7], sha, nil return sha, nil
} }
// FindGitRef get the current git ref // FindGitRef get the current git ref
func FindGitRef(ctx context.Context, file string) (string, error) { func FindGitRef(ctx context.Context, file string) (string, error) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
logger.Debugf("Loading revision from git directory") logger.Debugf("Loading revision from git directory")
_, revision, err := FindGitRevision(ctx, file) revision, err := FindGitRevision(ctx, file)
if err != nil { if err != nil {
return "", err return "", err
} }
@@ -320,9 +321,18 @@ func cachedRef(ref string) string {
func resolveRef(ctx context.Context, dir, ref string) (string, error) { func resolveRef(ctx context.Context, dir, ref string) (string, error) {
if !isFullObjectID(ref) { if !isFullObjectID(ref) {
for _, name := range []string{cachedRef(ref), "refs/tags/" + ref, "refs/remotes/origin/" + ref} { revisions := []string{cachedRef(ref) + "^{commit}", "refs/tags/" + ref + "^{commit}", "refs/remotes/origin/" + ref + "^{commit}"}
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", name+"^{commit}"); err == nil { if strings.ContainsAny(ref, "\x00\n") {
return sha, nil for _, revision := range revisions {
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", revision); err == nil {
return sha, nil
}
}
} else if output, err := runGitWithStdin(ctx, dir, nil, strings.NewReader(strings.Join(revisions, "\n")+"\n"), "cat-file", "--batch-check=%(objectname)"); err == nil {
for sha := range strings.SplitSeq(string(output), "\n") {
if isFullObjectID(sha) {
return sha, nil
}
} }
} }
} }
@@ -427,6 +437,10 @@ func ListFiles(ctx context.Context, dir string) ([]string, error) {
} }
func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, args ...string) ([]byte, error) { func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, args ...string) ([]byte, error) {
return runGitWithStdin(ctx, dir, input, nil, args...)
}
func runGitWithStdin(ctx context.Context, dir string, input *NewGitCloneExecutorInput, stdin io.Reader, args ...string) ([]byte, error) {
config := []string{"core.autocrlf=false", "core.eol=lf"} // keep action files LF config := []string{"core.autocrlf=false", "core.eol=lf"} // keep action files LF
if input != nil { if input != nil {
config = append(config, RemoteConfig(input.Token, input.InsecureSkipTLS)...) config = append(config, RemoteConfig(input.Token, input.InsecureSkipTLS)...)
@@ -438,6 +452,7 @@ func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, ar
args = append([]string{"-C", dir}, args...) args = append([]string{"-C", dir}, args...)
} }
cmd := exec.CommandContext(ctx, "git", args...) cmd := exec.CommandContext(ctx, "git", args...)
cmd.Stdin = stdin
cmd.Env = slices.DeleteFunc(os.Environ(), func(entry string) bool { cmd.Env = slices.DeleteFunc(os.Environ(), func(entry string) bool {
name, _, _ := strings.Cut(entry, "=") name, _, _ := strings.Cut(entry, "=")
return slices.Contains(LocalEnvVars, name) return slices.Contains(LocalEnvVars, name)
+120 -172
View File
@@ -29,6 +29,31 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
var gitFixtureDir string
func TestMain(m *testing.M) {
var err error
gitFixtureDir, err = os.MkdirTemp("", "runner-git-fixture-")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
code := 1
if err = gitCmd("init", "--quiet", "--template=", "--bare", "--initial-branch=main", filepath.Join(gitFixtureDir, "remote")); err == nil {
err = gitCmd("init", "--quiet", "--template=", "--initial-branch=main", filepath.Join(gitFixtureDir, "work"))
}
if err == nil {
code = m.Run()
} else {
fmt.Fprintln(os.Stderr, err)
}
if err := os.RemoveAll(gitFixtureDir); err != nil {
fmt.Fprintln(os.Stderr, err)
code = 1
}
os.Exit(code)
}
func TestFindGitSlug(t *testing.T) { func TestFindGitSlug(t *testing.T) {
for _, testcase := range []struct { for _, testcase := range []struct {
remoteURL string remoteURL string
@@ -67,42 +92,20 @@ func TestErrorWrapsCommitAndCause(t *testing.T) {
require.Equal(t, "abc123", err.Commit()) require.Equal(t, "abc123", err.Commit())
} }
func cleanGitHooks(dir string) error {
hooksDir := filepath.Join(dir, ".git", "hooks")
files, err := os.ReadDir(hooksDir)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
for _, f := range files {
if f.IsDir() {
continue
}
relName := filepath.Join(hooksDir, f.Name())
if err := os.Remove(relName); err != nil {
return err
}
}
return nil
}
func TestFindGitMetadataOfSHA256RepositoryUsesOrigin(t *testing.T) { func TestFindGitMetadataOfSHA256RepositoryUsesOrigin(t *testing.T) {
t.Parallel() t.Parallel()
basedir := t.TempDir() basedir := t.TempDir()
const remoteURL = "https://github.com/owner/repo.git" const remoteURL = "https://github.com/owner/repo.git"
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", basedir)) require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, cleanGitHooks(basedir)) require.NoError(t, os.RemoveAll(filepath.Join(basedir, ".git", "hooks")))
require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init")) require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init"))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL)) require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL))
require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git")) require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git"))
shortSHA, sha, err := FindGitRevision(t.Context(), basedir) sha, err := FindGitRevision(t.Context(), basedir)
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, gitRevParse(t, basedir, "HEAD"), sha) require.Equal(t, gitRevParse(t, basedir, "HEAD"), sha)
require.Len(t, sha, 64) require.Len(t, sha, 64)
require.Equal(t, sha[:7], shortSHA)
ref, err := FindGitRef(t.Context(), basedir) ref, err := FindGitRef(t.Context(), basedir)
require.NoError(t, err) require.NoError(t, err)
@@ -122,89 +125,65 @@ func TestGitFindRef(t *testing.T) {
basedir := t.TempDir() basedir := t.TempDir()
for name, tt := range map[string]struct { for name, tt := range map[string]struct {
Prepare func(t *testing.T, dir string) commands [][]string
Assert func(t *testing.T, ref string, err error) ref string
}{ }{
"new_repo": { "new_repo": {},
Prepare: func(t *testing.T, dir string) {},
Assert: func(t *testing.T, ref string, err error) {
require.Error(t, err)
},
},
"new_repo_with_commit": { "new_repo_with_commit": {
Prepare: func(t *testing.T, dir string) { commands: [][]string{{"commit", "--allow-empty", "-m", "msg"}},
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg")) ref: "refs/heads/master",
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/master", ref)
},
}, },
"current_head_is_tag": { "current_head_is_tag": {
Prepare: func(t *testing.T, dir string) { commands: [][]string{
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "commit msg")) {"commit", "--allow-empty", "-m", "commit msg"},
require.NoError(t, gitCmd("-C", dir, "tag", "v1.2.3")) {"tag", "v1.2.3"},
require.NoError(t, gitCmd("-C", dir, "checkout", "v1.2.3")) {"checkout", "v1.2.3"},
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/tags/v1.2.3", ref)
}, },
ref: "refs/tags/v1.2.3",
}, },
"current_head_is_same_as_tag": { "current_head_is_same_as_tag": {
Prepare: func(t *testing.T, dir string) { commands: [][]string{
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "1.4.2 release")) {"commit", "--allow-empty", "-m", "1.4.2 release"},
require.NoError(t, gitCmd("-C", dir, "tag", "v1.4.2")) {"tag", "v1.4.2"},
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/tags/v1.4.2", ref)
}, },
ref: "refs/tags/v1.4.2",
}, },
"current_head_is_not_tag": { "current_head_is_not_tag": {
Prepare: func(t *testing.T, dir string) { commands: [][]string{
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg")) {"commit", "--allow-empty", "-m", "msg"},
require.NoError(t, gitCmd("-C", dir, "tag", "v1.4.2")) {"tag", "v1.4.2"},
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg2")) {"commit", "--allow-empty", "-m", "msg2"},
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/master", ref)
}, },
ref: "refs/heads/master",
}, },
"current_head_is_another_branch": { "current_head_is_another_branch": {
Prepare: func(t *testing.T, dir string) { commands: [][]string{
require.NoError(t, gitCmd("-C", dir, "checkout", "-b", "mybranch")) {"checkout", "-b", "mybranch"},
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg")) {"commit", "--allow-empty", "-m", "msg"},
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/mybranch", ref)
}, },
ref: "refs/heads/mybranch",
}, },
} { } {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
dir := filepath.Join(basedir, name) dir := filepath.Join(basedir, name)
require.NoError(t, os.MkdirAll(dir, 0o755)) require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "init", "--initial-branch=master")) require.NoError(t, os.WriteFile(filepath.Join(dir, ".git", "HEAD"), []byte("ref: refs/heads/master\n"), 0o644))
require.NoError(t, cleanGitHooks(dir)) for _, command := range tt.commands {
tt.Prepare(t, dir) require.NoError(t, gitCmd(append([]string{"-C", dir}, command...)...))
}
ref, err := FindGitRef(t.Context(), dir) ref, err := FindGitRef(t.Context(), dir)
tt.Assert(t, ref, err) if tt.ref == "" {
require.Error(t, err)
} else {
require.NoError(t, err)
require.Equal(t, tt.ref, ref)
}
}) })
} }
} }
func TestGitCloneExecutor(t *testing.T) { func TestGitCloneExecutor(t *testing.T) {
// Build a local bare "remote" so this runs offline and fast. The cases below mirror remoteDir, workDir := newGitRemote(t)
// the tag/branch/sha/short-sha ref paths the executor handles, formerly exercised by
// cloning actions/checkout and anchore/scan-action over the network.
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, os.WriteFile(filepath.Join(workDir, "action.sh"), []byte("#!/bin/sh\necho hi\n"), 0o755)) require.NoError(t, os.WriteFile(filepath.Join(workDir, "action.sh"), []byte("#!/bin/sh\necho hi\n"), 0o755))
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh")) require.NoError(t, gitCmd("-C", workDir, "add", "action.sh"))
require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial")) require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial"))
@@ -212,14 +191,11 @@ func TestGitCloneExecutor(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v2")) require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v2"))
// A branch with a dash in the name (mirrors the historical scan-action@act-fails case).
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails")) require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit")) require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "act-fails")) require.NoError(t, gitCmd("-C", workDir, "push", "origin", "act-fails"))
out, err := exec.Command("git", "-C", workDir, "rev-parse", "main").Output() fullSha := gitRevParse(t, workDir, "main")
require.NoError(t, err)
fullSha := strings.TrimSpace(string(out))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:refs/heads/"+fullSha[:4])) require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:refs/heads/"+fullSha[:4]))
@@ -228,6 +204,7 @@ func TestGitCloneExecutor(t *testing.T) {
t.Setenv("GIT_CONFIG_GLOBAL", hostConfig) t.Setenv("GIT_CONFIG_GLOBAL", hostConfig)
t.Setenv("GIT_DIR", t.TempDir()) t.Setenv("GIT_DIR", t.TempDir())
var clones sync.WaitGroup
for name, tt := range map[string]struct { for name, tt := range map[string]struct {
Err error Err error
Ref string Ref string
@@ -255,43 +232,40 @@ func TestGitCloneExecutor(t *testing.T) {
Ref: fullSha[:4], Ref: fullSha[:4],
}, },
} { } {
t.Run(name, func(t *testing.T) { clones.Go(func() {
dir := t.TempDir() t.Run(name, func(t *testing.T) {
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{ dir := t.TempDir()
URL: "https://example.invalid/action", clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
Ref: tt.Ref, URL: "https://example.invalid/action",
Dir: dir, Ref: tt.Ref,
}) Dir: dir,
})
err := clone(t.Context()) err := clone(t.Context())
if tt.Err != nil { if tt.Err != nil {
assert.Error(t, err) //nolint:testifylint // pre-existing issue from nektos/act require.Error(t, err)
assert.Equal(t, tt.Err, err) assert.Equal(t, tt.Err, err)
assert.Equal(t, tt.Err, clone(t.Context()), "a retry on the cache keeps the error") assert.Equal(t, tt.Err, clone(t.Context()), "a retry on the cache keeps the error")
return return
} }
assert.Empty(t, err) //nolint:testifylint // pre-existing issue from nektos/act require.NoError(t, err)
marker := filepath.Join(dir, "marker") marker := filepath.Join(dir, "marker")
require.NoError(t, os.WriteFile(marker, nil, 0o644)) require.NoError(t, os.WriteFile(marker, nil, 0o644))
require.NoError(t, clone(t.Context())) require.NoError(t, clone(t.Context()))
assert.FileExists(t, marker, "an insteadOf rewrite must not evict the cache") assert.FileExists(t, marker, "an insteadOf rewrite must not evict the cache")
script, err := os.ReadFile(filepath.Join(dir, "action.sh")) script, err := os.ReadFile(filepath.Join(dir, "action.sh"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "#!/bin/sh\necho hi\n", string(script), "host core.autocrlf must not convert action files") assert.Equal(t, "#!/bin/sh\necho hi\n", string(script), "host core.autocrlf must not convert action files")
})
}) })
} }
clones.Wait()
} }
func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) { func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
t.Parallel() t.Parallel()
createRemote := func(message string) string { createRemote := func(message string) string {
remoteDir := t.TempDir() remoteDir, workDir := newGitRemote(t, message)
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
return remoteDir return remoteDir
@@ -341,13 +315,7 @@ func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
func TestGitCloneExecutorFollowsRemoteRefChanges(t *testing.T) { func TestGitCloneExecutorFollowsRemoteRefChanges(t *testing.T) {
t.Parallel() t.Parallel()
remoteDir := t.TempDir() remoteDir, workDir := newGitRemote(t, "initial", "feature-1")
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "feature-1"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
dir := t.TempDir() dir := t.TempDir()
@@ -380,16 +348,9 @@ func TestGitCloneExecutorFollowsRemoteRefChanges(t *testing.T) {
func TestGitCloneExecutorOfflineMode(t *testing.T) { func TestGitCloneExecutorOfflineMode(t *testing.T) {
t.Parallel() t.Parallel()
// Build a local "remote" with a single commit on main. remoteDir, workDir := newGitRemote(t, "initial")
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
// Prime the cache with an online clone of main.
cacheDir := t.TempDir() cacheDir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{ require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, URL: remoteDir,
@@ -398,8 +359,6 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
})(t.Context())) })(t.Context()))
t.Run("cached branch resolves without fetching", func(t *testing.T) { t.Run("cached branch resolves without fetching", func(t *testing.T) {
// Offline reuse of a cached branch must succeed even though ResolveRevision(input.Ref)
// finds no local refs/heads/<ref>.
err := NewGitCloneExecutor(NewGitCloneExecutorInput{ err := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, URL: remoteDir,
Ref: "main", Ref: "main",
@@ -414,7 +373,6 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
}) })
t.Run("unresolvable cached ref returns error", func(t *testing.T) { t.Run("unresolvable cached ref returns error", func(t *testing.T) {
// The ref was never cached; offline mode cannot resolve it and must return an error.
err := NewGitCloneExecutor(NewGitCloneExecutorInput{ err := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, URL: remoteDir,
Ref: "never-fetched", Ref: "never-fetched",
@@ -427,15 +385,7 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
func TestGitCloneExecutorShallow(t *testing.T) { func TestGitCloneExecutorShallow(t *testing.T) {
t.Parallel() t.Parallel()
// Build a local "remote" with several commits on main plus a tag, so a full clone would pull noticeably more history than a shallow one. remoteDir, workDir := newGitRemote(t, "c1", "c2", "c3")
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
for _, m := range []string{"c1", "c2", "c3"} {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", m))
}
require.NoError(t, gitCmd("-C", workDir, "tag", "v1")) require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1") unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
@@ -443,25 +393,17 @@ func TestGitCloneExecutorShallow(t *testing.T) {
shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") } shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") }
t.Run("branch is cloned shallowly", func(t *testing.T) { for name, ref := range map[string]string{"branch is cloned shallowly": "main", "tag is cloned shallowly": "v1"} {
dir := t.TempDir() t.Run(name, func(t *testing.T) {
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{ dir := t.TempDir()
URL: remoteDir, Ref: "main", Dir: dir, Depth: 1, require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
})(t.Context())) URL: remoteDir, Ref: ref, Dir: dir, Depth: 1,
assert.FileExists(t, shallowMarker(dir), "clone should be shallow") })(t.Context()))
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present") assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, "c3", gitHeadSubject(t, dir)) assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present")
}) assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
t.Run("tag is cloned shallowly", func(t *testing.T) { }
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: "v1", Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir))
assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string { unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string {
remote := filepath.Join(t.TempDir(), "remote.git") remote := filepath.Join(t.TempDir(), "remote.git")
@@ -541,12 +483,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) { func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
t.Parallel() t.Parallel()
remoteDir := t.TempDir() remoteDir, workDir := newGitRemote(t, "pinned")
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "pinned"))
pinned := gitRevParse(t, workDir, "HEAD") pinned := gitRevParse(t, workDir, "HEAD")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy")) require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy"))
require.NoError(t, gitCmd("-C", workDir, "tag", pinned)) require.NoError(t, gitCmd("-C", workDir, "tag", pinned))
@@ -564,6 +501,21 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
} }
} }
func newGitRemote(t *testing.T, messages ...string) (string, string) {
t.Helper()
remoteDir, workDir := t.TempDir(), t.TempDir()
require.NoError(t, os.CopyFS(remoteDir, os.DirFS(filepath.Join(gitFixtureDir, "remote"))))
require.NoError(t, os.CopyFS(workDir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
config, err := os.ReadFile(filepath.Join(workDir, ".git", "config"))
require.NoError(t, err)
require.NoError(t, os.WriteFile(filepath.Join(workDir, ".git", "config"),
fmt.Appendf(config, "\n[remote \"origin\"]\n\turl = %s\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n", strconv.Quote(remoteDir)), 0o644))
for _, message := range messages {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
}
return remoteDir, workDir
}
func gitRevParse(t *testing.T, dir, rev string) string { func gitRevParse(t *testing.T, dir, rev string) string {
t.Helper() t.Helper()
out, err := exec.Command("git", "-C", dir, "rev-parse", rev).Output() out, err := exec.Command("git", "-C", dir, "rev-parse", rev).Output()
@@ -591,9 +543,6 @@ func gitCmd(args ...string) error {
cmd := exec.Command("git", args...) cmd := exec.Command("git", args...)
cmd.Stdout = os.Stdout cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr cmd.Stderr = os.Stderr
// Inject a deterministic identity and ignore the host's global/system config so commits
// succeed regardless of the host having no user.name/user.email (e.g. CI, GITHUB_ACTIONS
// unset) or a global commit.gpgsign, and without mutating the developer's ~/.gitconfig.
cmd.Env = append(os.Environ(), cmd.Env = append(os.Environ(),
"GIT_AUTHOR_NAME=Unit Test", "GIT_AUTHOR_NAME=Unit Test",
"[email protected]", "[email protected]",
@@ -695,7 +644,6 @@ func TestAcquireCloneLock(t *testing.T) {
}) })
} }
// An unresponsive remote must not pin a job: the refresh has to be interruptible.
func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) { func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
t.Parallel() t.Parallel()
block := make(chan struct{}) block := make(chan struct{})
+3 -9
View File
@@ -14,13 +14,8 @@ const jobErrorContextKeyVal = jobErrorContextKey("job.error")
// JobError returns the job error for current context if any // JobError returns the job error for current context if any
func JobError(ctx context.Context) error { func JobError(ctx context.Context) error {
val := ctx.Value(jobErrorContextKeyVal) container, _ := ctx.Value(jobErrorContextKeyVal).(map[string]error)
if val != nil { return container["error"]
if container, ok := val.(map[string]error); ok {
return container["error"]
}
}
return nil
} }
func SetJobError(ctx context.Context, err error) { func SetJobError(ctx context.Context, err error) {
@@ -31,6 +26,5 @@ func SetJobError(ctx context.Context, err error) {
// WithJobErrorContainer adds a value to the context as a container for an error // WithJobErrorContainer adds a value to the context as a container for an error
func WithJobErrorContainer(ctx context.Context) context.Context { func WithJobErrorContainer(ctx context.Context) context.Context {
container := map[string]error{} return context.WithValue(ctx, jobErrorContextKeyVal, map[string]error{})
return context.WithValue(ctx, jobErrorContextKeyVal, container)
} }
+9 -21
View File
@@ -26,9 +26,7 @@ type lineWriter struct {
// NewLineWriter creates a new instance of a line writer // NewLineWriter creates a new instance of a line writer
func NewLineWriter(handlers ...LineHandler) io.Writer { func NewLineWriter(handlers ...LineHandler) io.Writer {
w := new(lineWriter) return &lineWriter{handlers: handlers}
w.handlers = handlers
return w
} }
// FlushWriter flushes w if it implements Flusher. It is a no-op otherwise, so // FlushWriter flushes w if it implements Flusher. It is a no-op otherwise, so
@@ -39,24 +37,15 @@ func FlushWriter(w io.Writer) {
} }
} }
func (lw *lineWriter) Write(p []byte) (n int, err error) { func (lw *lineWriter) Write(p []byte) (int, error) {
pBuf := bytes.NewBuffer(p) for line := range bytes.SplitAfterSeq(p, []byte{'\n'}) {
written := 0 _, _ = lw.buffer.Write(line)
for { if len(line) > 0 && line[len(line)-1] == '\n' {
line, err := pBuf.ReadString('\n') lw.handleLine(lw.buffer.String())
w, _ := lw.buffer.WriteString(line) lw.buffer.Reset()
written += w
if err != nil {
if err == io.EOF {
break
}
return written, err
} }
lw.handleLine(lw.buffer.String())
lw.buffer.Reset()
} }
return len(p), nil
return written, nil
} }
// Flush emits any buffered, not-yet-newline-terminated content as a final line. // Flush emits any buffered, not-yet-newline-terminated content as a final line.
@@ -72,8 +61,7 @@ func (lw *lineWriter) Flush() {
func (lw *lineWriter) handleLine(line string) { func (lw *lineWriter) handleLine(line string) {
for _, h := range lw.handlers { for _, h := range lw.handlers {
ok := h(line) if !h(line) {
if !ok {
break break
} }
} }
+12 -34
View File
@@ -9,24 +9,24 @@ import (
"testing" "testing"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
) )
func TestLineWriter(t *testing.T) { func TestLineWriterBuffersAndFlushesLines(t *testing.T) {
lines := make([]string, 0) var lines []string
lineHandler := func(s string) bool { lineWriter := NewLineWriter(func(s string) bool {
lines = append(lines, s) lines = append(lines, s)
return true return true
} })
lineWriter := NewLineWriter(lineHandler)
assert := assert.New(t) assert := assert.New(t)
write := func(s string) { write := func(s string) {
n, err := lineWriter.Write([]byte(s)) n, err := lineWriter.Write([]byte(s))
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act require.NoError(t, err)
assert.Equal(len(s), n, s) assert.Equal(len(s), n, s)
} }
write("")
write("hello") write("hello")
write(" ") write(" ")
write("world!!\nextra") write("world!!\nextra")
@@ -34,39 +34,17 @@ func TestLineWriter(t *testing.T) {
write(" line\n") write(" line\n")
write("no newline here...") write("no newline here...")
assert.Len(lines, 4) expected := []string{"hello world!!\n", "extra line\n", " and another\n", "last line\n"}
assert.Equal("hello world!!\n", lines[0]) assert.Equal(expected, lines)
assert.Equal("extra line\n", lines[1])
assert.Equal(" and another\n", lines[2])
assert.Equal("last line\n", lines[3])
}
func TestLineWriterFlush(t *testing.T) {
lines := make([]string, 0)
lineHandler := func(s string) bool {
lines = append(lines, s)
return true
}
lineWriter := NewLineWriter(lineHandler)
assert := assert.New(t)
_, err := lineWriter.Write([]byte("complete line\npartial line without newline"))
assert.NoError(err) //nolint:testifylint // pre-existing pattern from nektos/act
// Only the newline-terminated line is emitted before flushing.
assert.Equal([]string{"complete line\n"}, lines)
// Flushing emits the buffered, not-yet-terminated trailing line.
FlushWriter(lineWriter) FlushWriter(lineWriter)
assert.Equal([]string{"complete line\n", "partial line without newline"}, lines) expected = append(expected, "no newline here...")
assert.Equal(expected, lines)
// Flushing again is a no-op: nothing is buffered.
FlushWriter(lineWriter) FlushWriter(lineWriter)
assert.Len(lines, 2) assert.Equal(expected, lines)
} }
func TestFlushWriterIgnoresNonFlusher(t *testing.T) { func TestFlushWriterIgnoresNonFlusher(t *testing.T) {
// FlushWriter must be a safe no-op for writers that do not buffer lines.
assert.NotPanics(t, func() { FlushWriter(io.Discard) }) assert.NotPanics(t, func() { FlushWriter(io.Discard) })
} }
+2 -4
View File
@@ -34,10 +34,8 @@ const loggerHandlerKeyVal = loggerHandlerKey("logging.Handler")
// LoggerHandler returns the extra handler the job logger sends its records to, or nil. // LoggerHandler returns the extra handler the job logger sends its records to, or nil.
// It affects the job logger, not the global logger. // It affects the job logger, not the global logger.
func LoggerHandler(ctx context.Context) logging.Handler { func LoggerHandler(ctx context.Context) logging.Handler {
if h, ok := ctx.Value(loggerHandlerKeyVal).(logging.Handler); ok { h, _ := ctx.Value(loggerHandlerKeyVal).(logging.Handler)
return h return h
}
return nil
} }
// WithLoggerHandler adds the extra handler for the job logger to the context // WithLoggerHandler adds the extra handler for the job logger to the context
+2 -5
View File
@@ -9,6 +9,7 @@ import (
"fmt" "fmt"
"io" "io"
"log/slog" "log/slog"
"maps"
"reflect" "reflect"
"runtime" "runtime"
"slices" "slices"
@@ -94,11 +95,7 @@ func (h *consoleHandler) Handle(_ context.Context, r slog.Record) error {
} }
prefixFieldClashes(data, caller != nil) prefixFieldClashes(data, caller != nil)
keys := make([]string, 0, len(data)) keys := slices.Sorted(maps.Keys(data))
for k := range data {
keys = append(keys, k)
}
slices.Sort(keys)
var b bytes.Buffer var b bytes.Buffer
if terminal && !opts.DisableColors { if terminal && !opts.DisableColors {
+2 -10
View File
@@ -9,8 +9,6 @@ import (
"os" "os"
"runtime" "runtime"
"sync/atomic" "sync/atomic"
"github.com/sirupsen/logrus"
) )
// TextOptions describes how the process-wide Logger writes a record as text. // TextOptions describes how the process-wide Logger writes a record as text.
@@ -41,11 +39,8 @@ func Default() *Logger { return std.Load() }
// initial console Logger. // initial console Logger.
func SetDefault(l *Logger) { std.Store(l) } func SetDefault(l *Logger) { std.Store(l) }
// SetLevel sets the minimum level of the initial process-wide Logger and of what libraries log through logrus. // SetLevel sets the minimum level of the initial process-wide Logger.
func SetLevel(level Level) { func SetLevel(level Level) { stdState.level.Set(slog.Level(level)) }
stdState.level.Set(slog.Level(level))
logrus.SetLevel(toLogrusLevel(level))
}
// GetLevel returns the minimum level of the initial process-wide Logger. // GetLevel returns the minimum level of the initial process-wide Logger.
func GetLevel() Level { return Level(stdState.level.Level()) } func GetLevel() Level { return Level(stdState.level.Level()) }
@@ -72,19 +67,16 @@ func SetReportCaller(on bool) {
stdState.mu.Lock() stdState.mu.Lock()
defer stdState.mu.Unlock() defer stdState.mu.Unlock()
stdState.reportCaller = on stdState.reportCaller = on
logrus.SetReportCaller(on)
} }
func Enabled(level Level) bool { return Default().Enabled(level) } func Enabled(level Level) bool { return Default().Enabled(level) }
func With(args ...any) *Logger { return Default().With(args...) } func With(args ...any) *Logger { return Default().With(args...) }
func WithError(err error) *Logger { return Default().WithError(err) } func WithError(err error) *Logger { return Default().WithError(err) }
func Tracef(format string, args ...any) { Default().logf(LevelTrace, format, args) }
func Debugf(format string, args ...any) { Default().logf(LevelDebug, format, args) } func Debugf(format string, args ...any) { Default().logf(LevelDebug, format, args) }
func Infof(format string, args ...any) { Default().logf(LevelInfo, format, args) } func Infof(format string, args ...any) { Default().logf(LevelInfo, format, args) }
func Warnf(format string, args ...any) { Default().logf(LevelWarn, format, args) } func Warnf(format string, args ...any) { Default().logf(LevelWarn, format, args) }
func Errorf(format string, args ...any) { Default().logf(LevelError, format, args) } func Errorf(format string, args ...any) { Default().logf(LevelError, format, args) }
func Trace(args ...any) { Default().log(LevelTrace, args) }
func Debug(args ...any) { Default().log(LevelDebug, args) } func Debug(args ...any) { Default().log(LevelDebug, args) }
func Info(args ...any) { Default().log(LevelInfo, args) } func Info(args ...any) { Default().log(LevelInfo, args) }
func Warn(args ...any) { Default().log(LevelWarn, args) } func Warn(args ...any) { Default().log(LevelWarn, args) }
-1
View File
@@ -43,7 +43,6 @@ func (l *Logger) Enabled(level Level) bool {
return l.h.Enabled(l.ctx, level) return l.h.Enabled(l.ctx, level)
} }
func (l *Logger) Tracef(format string, args ...any) { l.logf(LevelTrace, format, args) }
func (l *Logger) Debugf(format string, args ...any) { l.logf(LevelDebug, format, args) } func (l *Logger) Debugf(format string, args ...any) { l.logf(LevelDebug, format, args) }
func (l *Logger) Infof(format string, args ...any) { l.logf(LevelInfo, format, args) } func (l *Logger) Infof(format string, args ...any) { l.logf(LevelInfo, format, args) }
func (l *Logger) Warnf(format string, args ...any) { l.logf(LevelWarn, format, args) } func (l *Logger) Warnf(format string, args ...any) { l.logf(LevelWarn, format, args) }
+3 -61
View File
@@ -13,7 +13,6 @@ import (
"gitea.com/gitea/runner/act/common/logging" "gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest" "gitea.com/gitea/runner/act/common/logging/logtest"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -39,6 +38,7 @@ func (c countingStringer) String() string { *c.n++; return "x" }
func TestLogger(t *testing.T) { func TestLogger(t *testing.T) {
h := logtest.NewWithLevel(logging.LevelTrace) h := logtest.NewWithLevel(logging.LevelTrace)
assert.Nil(t, h.LastRecord())
l := logging.New(h) l := logging.New(h)
l.With("job", "j1").WithError(errors.New("boom")).Warnf("step %d", 2) l.With("job", "j1").WithError(errors.New("boom")).Warnf("step %d", 2)
@@ -54,12 +54,9 @@ func TestLogger(t *testing.T) {
assert.Equal(t, "j1", job) assert.Equal(t, "j1", job)
errv, _ := recs[0].Attr("error") errv, _ := recs[0].Attr("error")
require.EqualError(t, errv.(error), "boom") require.EqualError(t, errv.(error), "boom")
assert.Equal(t, "a1 2b", recs[1].Message) // fmt.Sprint spacing assert.Equal(t, "a1 2b", recs[1].Message)
bad, _ := recs[2].Attr("!BADKEY") bad, _ := recs[2].Attr("!BADKEY")
assert.Equal(t, 7, bad) assert.Equal(t, 7, bad)
h.Reset()
assert.Nil(t, h.LastRecord())
} }
func TestLoggerSkipsFormattingWhenDisabled(t *testing.T) { func TestLoggerSkipsFormattingWhenDisabled(t *testing.T) {
@@ -118,10 +115,8 @@ func TestDefaultLevel(t *testing.T) {
func TestSlogHandler(t *testing.T) { func TestSlogHandler(t *testing.T) {
var buf strings.Builder var buf strings.Builder
slogger := slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug})) l := logging.New(logging.NewSlogHandler(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}))).With("job", "j1")
l := logging.New(logging.NewSlogHandler(slogger.Handler())).With("job", "j1")
l.Tracef("dropped")
assert.False(t, l.Enabled(logging.LevelTrace)) assert.False(t, l.Enabled(logging.LevelTrace))
l.WithError(errors.New("boom")).Errorf("fail %s", "x") l.WithError(errors.New("boom")).Errorf("fail %s", "x")
@@ -132,43 +127,6 @@ func TestSlogHandler(t *testing.T) {
assert.Contains(t, out, "error=boom") assert.Contains(t, out, "error=boom")
} }
func TestLogrusBridge(t *testing.T) {
prevLevel, prevDefault := logging.GetLevel(), logging.Default()
t.Cleanup(func() {
logging.SetLevel(prevLevel)
logging.SetReportCaller(false)
logging.SetDefault(prevDefault)
})
logging.SetLevel(logging.LevelInfo)
assert.False(t, logrus.IsLevelEnabled(logrus.DebugLevel))
logging.SetLevel(logging.LevelTrace)
logging.SetReportCaller(true)
h := logtest.NewWithLevel(logging.LevelTrace)
logging.SetDefault(logging.New(h))
logrus.WithField("lib", "docker").Warnf("from a library %d", 1)
_, _, line, _ := runtime.Caller(0)
logrus.Debug("below")
logrus.Trace("lowest")
recs := h.AllRecords()
require.Len(t, recs, 3)
assert.Equal(t, "from a library 1", recs[0].Message)
assert.Equal(t, logging.LevelWarn, recs[0].Level)
v, _ := recs[0].Attr("lib")
assert.Equal(t, "docker", v)
assert.Equal(t, logging.LevelDebug, recs[1].Level)
f := frameOf(t, recs[1].PC)
assert.True(t, strings.HasSuffix(f.Function, "TestLogrusBridge"), f.Function)
assert.Equal(t, line+1, f.Line)
assert.Equal(t, logging.LevelTrace, recs[2].Level)
h = logtest.New() // logrus passes Debug, the Logger does not
logging.SetDefault(logging.New(h))
logrus.Debug("dropped")
assert.Empty(t, h.AllRecords())
}
func TestSlogBridge(t *testing.T) { func TestSlogBridge(t *testing.T) {
h := logtest.SetDefault(t) h := logtest.SetDefault(t)
slog.With("lib", "actionslib").Warn("from a library", "n", 1) slog.With("lib", "actionslib").Warn("from a library", "n", 1)
@@ -185,19 +143,3 @@ func TestSlogBridge(t *testing.T) {
assert.EqualValues(t, 1, n) assert.EqualValues(t, 1, n)
assert.Equal(t, line-1, frameOf(t, recs[0].PC).Line) assert.Equal(t, line-1, frameOf(t, recs[0].PC).Line)
} }
func TestLogrusBridgeIntact(t *testing.T) {
assert.True(t, logging.LogrusBridgeIntact())
prev := logrus.StandardLogger().Out
t.Cleanup(func() { logrus.SetOutput(prev) })
logrus.SetOutput(&strings.Builder{}) // what a dependency doing this would do
assert.False(t, logging.LogrusBridgeIntact())
logrus.SetOutput(prev)
assert.True(t, logging.LogrusBridgeIntact())
prevLevel := logrus.GetLevel()
t.Cleanup(func() { logrus.SetLevel(prevLevel) })
logrus.SetLevel(logrus.PanicLevel)
assert.False(t, logging.LogrusBridgeIntact())
}
-102
View File
@@ -1,102 +0,0 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package logging
import (
"context"
"io"
"github.com/sirupsen/logrus"
)
// Libraries the runner depends on, such as docker/cli and moby/go-archive, log through
// logrus's global logger. Hand what they log to the process-wide Logger, so it follows the same
// level, format and secret masking as the rest of the output.
func init() {
logrus.SetOutput(io.Discard)
logrus.SetFormatter(discardFormatter{})
logrus.SetLevel(toLogrusLevel(GetLevel()))
logrus.AddHook(bridgeHook{})
}
type discardFormatter struct{}
func (discardFormatter) Format(*logrus.Entry) ([]byte, error) { return nil, nil }
type bridgeHook struct{}
func (bridgeHook) Levels() []logrus.Level { return logrus.AllLevels }
func (bridgeHook) Fire(e *logrus.Entry) error {
l := Default()
level := fromLogrusLevel(e.Level)
ctx := e.Context
if ctx == nil {
ctx = context.Background()
}
if !l.h.Enabled(ctx, level) {
return nil
}
attrs := make([]Attr, 0, len(e.Data))
for k, v := range e.Data {
attrs = append(attrs, Attr{Key: k, Value: v})
}
r := Record{Time: e.Time, Level: level, Message: e.Message, Attrs: attrs}
if e.Caller != nil {
r.PC = e.Caller.PC + 1 // Frame.PC is one below the return address Record.PC holds
}
return l.h.Handle(ctx, r)
}
func toLogrusLevel(l Level) logrus.Level {
switch {
case l <= LevelTrace:
return logrus.TraceLevel
case l <= LevelDebug:
return logrus.DebugLevel
case l <= LevelInfo:
return logrus.InfoLevel
case l <= LevelWarn:
return logrus.WarnLevel
default:
return logrus.ErrorLevel
}
}
func fromLogrusLevel(l logrus.Level) Level {
switch l {
case logrus.TraceLevel:
return LevelTrace
case logrus.DebugLevel:
return LevelDebug
case logrus.InfoLevel:
return LevelInfo
case logrus.WarnLevel:
return LevelWarn
default:
return LevelError
}
}
// LogrusBridgeIntact reports whether logrus's global logger is still set up as the bridge needs,
// discarding its own output and following the process-wide level. A dependency that calls
// logrus.SetOutput or SetLevel breaks that, and its lines then show up twice, or at the wrong level.
func LogrusBridgeIntact() bool {
std := logrus.StandardLogger()
if std.Out != io.Discard || std.GetLevel() != toLogrusLevel(GetLevel()) {
return false
}
for _, level := range logrus.AllLevels {
found := false
for _, h := range std.Hooks[level] {
if _, ok := h.(bridgeHook); ok {
found = true
}
}
if !found {
return false
}
}
return true
}
-6
View File
@@ -72,12 +72,6 @@ func (h *Handler) LastRecord() *logging.Record {
return &r return &r
} }
func (h *Handler) Reset() {
h.store.mu.Lock()
defer h.store.mu.Unlock()
h.store.records = nil
}
// SetDefault makes the process-wide Logger record into a new Handler until the test ends. // SetDefault makes the process-wide Logger record into a new Handler until the test ends.
func SetDefault(t testing.TB) *Handler { func SetDefault(t testing.TB) *Handler {
t.Helper() t.Helper()
+8 -12
View File
@@ -12,7 +12,7 @@ import (
type interfaceIP struct { type interfaceIP struct {
net.IP net.IP
net.Interface Name string
} }
// GetOutboundIP returns an outbound IP address of this machine. // GetOutboundIP returns an outbound IP address of this machine.
@@ -43,7 +43,7 @@ func GetOutboundIP() net.IP {
ip = v.IP ip = v.IP
} }
if ip.IsGlobalUnicast() { if ip.IsGlobalUnicast() {
ips = append(ips, interfaceIP{ip, i}) ips = append(ips, interfaceIP{ip, i.Name})
} }
} }
} }
@@ -59,29 +59,25 @@ func preferredIP(ips []interfaceIP) net.IP {
return nil return nil
} }
sort.Slice(ips, func(i, j int) bool { sort.Slice(ips, func(i, j int) bool {
ifi := ips[i].Interface first, second := ips[i], ips[j]
ifj := ips[j].Interface
// ethernet is preferred // ethernet is preferred
if vi, vj := strings.HasPrefix(ifi.Name, "e"), strings.HasPrefix(ifj.Name, "e"); vi != vj { if vi, vj := strings.HasPrefix(first.Name, "e"), strings.HasPrefix(second.Name, "e"); vi != vj {
return vi return vi
} }
ipi := ips[i].IP
ipj := ips[j].IP
// IPv4 is preferred // IPv4 is preferred
if vi, vj := ipi.To4() != nil, ipj.To4() != nil; vi != vj { if vi, vj := first.To4() != nil, second.To4() != nil; vi != vj {
return vi return vi
} }
// en0 is preferred to en1 // en0 is preferred to en1
if ifi.Name != ifj.Name { if first.Name != second.Name {
return ifi.Name < ifj.Name return first.Name < second.Name
} }
// fallback // fallback
return ipi.String() < ipj.String() return first.String() < second.String()
}) })
return ips[0].IP return ips[0].IP
} }
+244 -13
View File
@@ -7,20 +7,33 @@
package container package container
import ( import (
"bytes"
"context" "context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"maps"
"net"
"net/url"
"os"
"os/exec"
"os/user"
"path/filepath"
"runtime"
"strings"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"github.com/distribution/reference" "github.com/distribution/reference"
"github.com/docker/cli/cli/config"
"github.com/moby/moby/api/types/registry" "github.com/moby/moby/api/types/registry"
) )
func LoadDockerAuthConfig(ctx context.Context, image string) (registry.AuthConfig, error) { func LoadDockerAuthConfig(ctx context.Context, image string) (registry.AuthConfig, error) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
// config.LoadDefaultConfigFile panics on nil io.Writer when the config cfg, err := loadDockerConfig(ctx)
// file is malformed; use config.Load to route errors through the logger.
cfg, err := config.Load(config.Dir())
if err != nil { if err != nil {
logger.Warnf("Could not load docker config: %v", err) logger.Warnf("Could not load docker config: %v", err)
return registry.AuthConfig{}, err return registry.AuthConfig{}, err
@@ -34,32 +47,27 @@ func LoadDockerAuthConfig(ctx context.Context, image string) (registry.AuthConfi
} }
} }
authConfig, err := cfg.GetAuthConfig(registryKey) authConfig, err := cfg.authConfig(ctx, registryKey)
if err != nil { if err != nil {
logger.Warnf("Could not get auth config from docker config: %v", err) logger.Warnf("Could not get auth config from docker config: %v", err)
return registry.AuthConfig{}, err return registry.AuthConfig{}, err
} }
return registry.AuthConfig(authConfig), nil return authConfig, nil
} }
func LoadDockerAuthConfigs(ctx context.Context) map[string]registry.AuthConfig { func LoadDockerAuthConfigs(ctx context.Context) map[string]registry.AuthConfig {
logger := common.Logger(ctx) logger := common.Logger(ctx)
cfg, err := config.Load(config.Dir()) cfg, err := loadDockerConfig(ctx)
if err != nil { if err != nil {
logger.Warnf("Could not load docker config: %v", err) logger.Warnf("Could not load docker config: %v", err)
return nil return nil
} }
creds, err := cfg.GetAllCredentials() authConfigs, err := cfg.allAuthConfigs(ctx)
if err != nil { if err != nil {
logger.Warnf("Could not get docker auth configs: %v", err) logger.Warnf("Could not get docker auth configs: %v", err)
return nil return nil
} }
authConfigs := make(map[string]registry.AuthConfig, len(creds))
for k, v := range creds {
authConfigs[k] = registry.AuthConfig(v)
}
return authConfigs return authConfigs
} }
@@ -69,3 +77,226 @@ func registryAuthConfigKey(domainName string) string {
} }
return domainName return domainName
} }
type dockerConfig struct {
Auths map[string]registry.AuthConfig `json:"auths"`
CredsStore string `json:"credsStore"`
CredHelpers map[string]string `json:"credHelpers"`
env map[string]registry.AuthConfig // from DOCKER_AUTH_CONFIG, nil when unset or unusable
}
func loadDockerConfig(ctx context.Context) (*dockerConfig, error) {
dir := os.Getenv("DOCKER_CONFIG")
if dir == "" {
home, _ := os.UserHomeDir()
if home == "" && runtime.GOOS != "windows" {
if current, err := user.Current(); err == nil {
home = current.HomeDir
}
}
dir = filepath.Join(home, ".docker")
}
cfg := &dockerConfig{}
filename := filepath.Join(dir, "config.json")
file, err := os.Open(filename)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return nil, fmt.Errorf("loading config file: %w", err)
}
if err == nil {
defer file.Close()
if err := json.NewDecoder(file).Decode(cfg); err != nil && !errors.Is(err, io.EOF) {
return nil, fmt.Errorf("parsing config file (%s): %w", filename, err)
}
}
for address, auth := range cfg.Auths {
if auth.Auth != "" {
if auth.Username, auth.Password, err = decodeAuth(auth.Auth); err != nil {
return nil, fmt.Errorf("parsing config file (%s): %w", filename, err)
}
}
auth.Auth, auth.ServerAddress = "", address
cfg.Auths[address] = auth
}
if value := os.Getenv("DOCKER_AUTH_CONFIG"); value != "" {
if cfg.env, err = parseDockerAuthConfigEnv(value); err != nil {
common.Logger(ctx).Warnf("Failed to create credential store from DOCKER_AUTH_CONFIG: %v", err)
}
}
return cfg, nil
}
func decodeAuth(auth string) (string, string, error) {
decoded, err := base64.StdEncoding.DecodeString(auth)
if err != nil {
return "", "", err
}
username, password, ok := strings.Cut(string(decoded), ":")
if !ok || username == "" {
return "", "", errors.New("invalid auth configuration file")
}
return username, strings.Trim(password, "\x00"), nil
}
func parseDockerAuthConfigEnv(value string) (map[string]registry.AuthConfig, error) {
var env struct {
Auths map[string]struct {
Auth string `json:"auth"`
} `json:"auths"`
}
decoder := json.NewDecoder(strings.NewReader(value))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&env); err != nil && !errors.Is(err, io.EOF) {
return nil, err
}
if decoder.More() {
return nil, errors.New("DOCKER_AUTH_CONFIG does not support more than one JSON object")
}
auths := make(map[string]registry.AuthConfig, len(env.Auths))
for address, entry := range env.Auths {
if entry.Auth == "" {
return nil, fmt.Errorf("DOCKER_AUTH_CONFIG environment variable is missing key `auth` for %s", address)
}
username, password, err := decodeAuth(entry.Auth)
if err != nil {
return nil, err
}
auths[address] = registry.AuthConfig{Username: username, Password: password, ServerAddress: address}
}
return auths, nil
}
func (c *dockerConfig) authConfig(ctx context.Context, key string) (registry.AuthConfig, error) {
if auth, ok := c.env[key]; ok {
return auth, nil
}
helper := c.CredsStore
if named, ok := c.CredHelpers[key]; ok {
helper = named
}
if helper == "" {
return c.fileAuth(key), nil
}
auth := c.fileAuth(key)
stored, err := helperCredentials(ctx, helper, key)
if err != nil {
return auth, err
}
auth.Username, auth.Password, auth.IdentityToken, auth.ServerAddress = stored.Username, stored.Password, stored.IdentityToken, stored.ServerAddress
return auth, nil
}
func (c *dockerConfig) fileAuth(key string) registry.AuthConfig {
if auth, ok := c.Auths[key]; ok {
return auth
}
for address, auth := range c.Auths {
if registryHostname(address) == key {
return auth
}
}
return registry.AuthConfig{}
}
func (c *dockerConfig) allAuthConfigs(ctx context.Context) (map[string]registry.AuthConfig, error) {
var auths map[string]registry.AuthConfig
if c.CredsStore == "" {
auths = maps.Clone(c.Auths)
} else {
var err error
if auths, err = c.helperAuthConfigs(ctx); err != nil {
if c.env == nil {
return nil, err
}
common.Logger(ctx).Warnf("Could not list the credentials of docker-credential-%s: %v", c.CredsStore, err)
}
}
if auths == nil {
auths = map[string]registry.AuthConfig{}
}
maps.Copy(auths, c.env)
if c.CredsStore == "" {
c.Auths = maps.Clone(auths) // docker's file store holds the DOCKER_AUTH_CONFIG entries after listing, and credHelpers set to "" read it
}
for address := range c.CredHelpers {
auth, err := c.authConfig(ctx, registryAuthConfigKey(address))
if err != nil {
common.Logger(ctx).Warnf("Failed to get credentials for registry: %s: %v", address, err)
continue
}
auths[address] = auth
}
return auths, nil
}
func (c *dockerConfig) helperAuthConfigs(ctx context.Context) (map[string]registry.AuthConfig, error) {
out, err := runCredentialHelper(ctx, c.CredsStore, "list", "unused")
if err != nil {
return nil, fmt.Errorf("error listing credentials - err: %w, out: `%s`", err, strings.TrimSpace(string(out)))
}
var addresses map[string]string
if err := json.NewDecoder(bytes.NewReader(out)).Decode(&addresses); err != nil {
return nil, err
}
auths := make(map[string]registry.AuthConfig, len(addresses))
for address := range addresses {
stored, err := helperCredentials(ctx, c.CredsStore, address)
if err != nil {
return nil, err
}
auth := c.Auths[address]
auth.Username, auth.Password, auth.IdentityToken = stored.Username, stored.Password, stored.IdentityToken
if auth.ServerAddress == "" {
auth.ServerAddress = stored.ServerAddress
}
auths[address] = auth
}
return auths, nil
}
func helperCredentials(ctx context.Context, helper, address string) (registry.AuthConfig, error) {
out, err := runCredentialHelper(ctx, helper, "get", address)
if err != nil {
if strings.TrimSpace(string(out)) == "credentials not found in native keychain" {
return registry.AuthConfig{}, nil
}
return registry.AuthConfig{}, fmt.Errorf("error getting credentials - err: %w, out: `%s`", err, strings.TrimSpace(string(out)))
}
var stored struct{ Username, Secret string }
if err := json.NewDecoder(bytes.NewReader(out)).Decode(&stored); err != nil {
return registry.AuthConfig{}, err
}
if stored.Username == "<token>" {
return registry.AuthConfig{IdentityToken: stored.Secret, ServerAddress: address}, nil
}
return registry.AuthConfig{Username: stored.Username, Password: stored.Secret, ServerAddress: address}, nil
}
func runCredentialHelper(ctx context.Context, helper, action, input string) ([]byte, error) {
cmd := exec.CommandContext(ctx, "docker-credential-"+helper, action)
cmd.Stdin = strings.NewReader(input)
cmd.Stderr = os.Stderr
return cmd.Output()
}
func registryHostname(address string) string {
if _, rest, ok := strings.Cut(address, "://"); ok {
if u, err := url.Parse(address); err == nil && u.Hostname() != "" {
if u.Port() == "" {
return u.Hostname()
}
return net.JoinHostPort(u.Hostname(), u.Port())
}
if host, _, _ := strings.Cut(rest, "/"); host != "" {
if colon := strings.LastIndex(host, ":"); strings.Count(host, ":") > 1 && !strings.HasPrefix(host, "[") && isDigits(host[colon+1:]) {
return net.JoinHostPort(host[:colon], host[colon+1:])
}
return host
}
}
host, _, _ := strings.Cut(address, "/")
return host
}
func isDigits(s string) bool {
return s != "" && strings.Trim(s, "0123456789") == ""
}
+46
View File
@@ -0,0 +1,46 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !windows && !(WITHOUT_DOCKER || !(linux || darwin || netbsd))
package container
import (
"encoding/base64"
"os"
"path/filepath"
"testing"
"github.com/moby/moby/api/types/registry"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLoadDockerAuthConfigFromFileHelpersAndEnv(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "docker-credential-fake"), []byte("#!/bin/sh\nread -r server\n[ \"$1\" = get ] && [ \"$server\" = helper.example.com ] && exec echo '{\"Username\":\"helped\",\"Secret\":\"pw\"}'\necho 'credentials not found in native keychain'\nexit 1\n"), 0o755))
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
auth := func(credentials string) string { return base64.StdEncoding.EncodeToString([]byte(credentials)) }
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"auths": {"https://legacy.example.com/v1/": {"auth": "`+auth("legacy:pw")+`"}}, "credHelpers": {"helper.example.com": "fake", "gone.example.com": "fake", "env-legacy.example.com": ""}}`), 0o600))
t.Setenv("DOCKER_CONFIG", dir)
t.Setenv("DOCKER_AUTH_CONFIG", `{"auths": {"env.example.com": {"auth": "`+auth("env:pw")+`"}, "https://env-legacy.example.com/v1/": {"auth": "`+auth("env:pw")+`"}}}`)
for image, want := range map[string]registry.AuthConfig{
"legacy.example.com/app": {Username: "legacy", Password: "pw", ServerAddress: "https://legacy.example.com/v1/"},
"helper.example.com/app": {Username: "helped", Password: "pw", ServerAddress: "helper.example.com"},
"env.example.com/app": {Username: "env", Password: "pw", ServerAddress: "env.example.com"},
"gone.example.com/app": {},
} {
got, err := LoadDockerAuthConfig(t.Context(), image)
require.NoError(t, err)
assert.Equal(t, want, got, image)
}
assert.Equal(t, map[string]registry.AuthConfig{
"https://legacy.example.com/v1/": {Username: "legacy", Password: "pw", ServerAddress: "https://legacy.example.com/v1/"},
"helper.example.com": {Username: "helped", Password: "pw", ServerAddress: "helper.example.com"},
"gone.example.com": {},
"env.example.com": {Username: "env", Password: "pw", ServerAddress: "env.example.com"},
"https://env-legacy.example.com/v1/": {Username: "env", Password: "pw", ServerAddress: "https://env-legacy.example.com/v1/"},
"env-legacy.example.com": {Username: "env", Password: "pw", ServerAddress: "https://env-legacy.example.com/v1/"},
}, LoadDockerAuthConfigs(t.Context()))
}
+140 -14
View File
@@ -7,15 +7,18 @@
package container package container
import ( import (
"archive/tar"
"context" "context"
"io" "io"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime"
"strings"
"time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"github.com/moby/go-archive"
"github.com/moby/go-archive/compression"
"github.com/moby/moby/client" "github.com/moby/moby/client"
"github.com/moby/patternmatcher" "github.com/moby/patternmatcher"
"github.com/moby/patternmatcher/ignorefile" "github.com/moby/patternmatcher/ignorefile"
@@ -43,9 +46,8 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Building image from '%v'", input.ContextDir) logger.Debugf("Building image from '%v'", input.ContextDir)
tags := []string{input.ImageTag}
options := client.ImageBuildOptions{ options := client.ImageBuildOptions{
Tags: tags, Tags: []string{input.ImageTag},
Remove: true, Remove: true,
AuthConfigs: LoadDockerAuthConfigs(ctx), AuthConfigs: LoadDockerAuthConfigs(ctx),
Dockerfile: input.Dockerfile, Dockerfile: input.Dockerfile,
@@ -72,12 +74,10 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Creating image from context dir '%s' with tag '%s' and platform '%s'", input.ContextDir, input.ImageTag, input.Platform) logger.Debugf("Creating image from context dir '%s' with tag '%s' and platform '%s'", input.ContextDir, input.ImageTag, input.Platform)
resp, err := cli.ImageBuild(ctx, buildContext, options) resp, err := cli.ImageBuild(ctx, buildContext, options)
err = logDockerResponse(logger, resp.Body, err != nil)
if err != nil { if err != nil {
return err return err
} }
return nil return logDockerResponse(logger, resp.Body, false)
} }
} }
@@ -115,14 +115,140 @@ func createBuildContext(ctx context.Context, contextDir, relDockerfile string) (
includes = append(includes, ".dockerignore", relDockerfile) includes = append(includes, ".dockerignore", relDockerfile)
} }
buildCtx, err := archive.TarWithOptions(contextDir, &archive.TarOptions{ pm, err := patternmatcher.New(excludes)
Compression: compression.None,
ExcludePatterns: excludes,
IncludeFiles: includes,
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
reader, writer := io.Pipe()
return buildCtx, nil go func() { writer.CloseWithError(writeBuildContext(ctx, writer, contextDir, includes, pm)) }()
return reader, nil
} }
func writeBuildContext(ctx context.Context, w io.Writer, contextDir string, includes []string, pm *patternmatcher.PatternMatcher) error {
tw := tar.NewWriter(w)
seen := map[string]bool{}
hardlinks := map[uint64]string{}
buffer := make([]byte, 32*1024)
for _, include := range includes {
include = filepath.FromSlash(include)
var parents []parentMatch
root := strings.TrimSuffix(contextDir, string(filepath.Separator)) + string(filepath.Separator) + include // unlike filepath.Join, follows a symlinked context dir as docker build does
err := filepath.WalkDir(root, func(file string, entry fs.DirEntry, err error) error {
if err != nil {
return leaveOut(ctx, file, err)
}
rel, err := filepath.Rel(contextDir, file)
if err != nil || rel == "." {
return nil
}
if rel != include {
for len(parents) > 0 && !strings.HasPrefix(rel, parents[len(parents)-1].dir+string(filepath.Separator)) {
parents = parents[:len(parents)-1]
}
var parentInfo patternmatcher.MatchInfo
if len(parents) > 0 {
parentInfo = parents[len(parents)-1].info
}
skip, info, err := pm.MatchesUsingParentResults(rel, parentInfo)
if err != nil {
return err
}
if entry.IsDir() {
parents = append(parents, parentMatch{rel, info})
}
if skip {
if entry.IsDir() && !hasExceptionWithin(pm, rel) {
return filepath.SkipDir
}
return nil
}
}
if seen[rel] {
return nil
}
seen[rel] = true
return addBuildContextEntry(ctx, tw, file, rel, entry, hardlinks, buffer)
})
if err != nil {
return err
}
}
return tw.Close()
}
type parentMatch struct {
dir string
info patternmatcher.MatchInfo
}
func hasExceptionWithin(pm *patternmatcher.PatternMatcher, dir string) bool {
dirPrefix := dir + string(filepath.Separator)
for _, pattern := range pm.Patterns() {
if pattern.Exclusion() && strings.HasPrefix(pattern.String()+string(filepath.Separator), dirPrefix) {
return true
}
}
return false
}
func leaveOut(ctx context.Context, file string, err error) error {
common.Logger(ctx).Warnf("Leaving %s out of the build context: %v", file, err)
return nil
}
func addBuildContextEntry(ctx context.Context, tw *tar.Writer, file, rel string, entry fs.DirEntry, hardlinks map[uint64]string, buffer []byte) error {
info, err := entry.Info()
if err != nil {
return leaveOut(ctx, file, err)
}
if info.Mode()&fs.ModeSocket != 0 {
return nil // tar has no socket type
}
var link string
if info.Mode()&fs.ModeSymlink != 0 {
if link, err = os.Readlink(file); err != nil {
return leaveOut(ctx, file, err)
}
}
header, err := tar.FileInfoHeader(noNameLookups{info}, link)
if err != nil {
return err
}
header.Name = filepath.ToSlash(rel)
inode, linked := hardlinkInode(info)
if target, ok := hardlinks[inode]; linked && ok {
header.Typeflag, header.Linkname, header.Size = tar.TypeLink, target, 0
}
var content *os.File
if header.Typeflag == tar.TypeReg && header.Size > 0 {
if content, err = os.Open(file); err != nil {
return leaveOut(ctx, file, err)
}
defer content.Close()
}
if linked && header.Typeflag != tar.TypeLink {
hardlinks[inode] = header.Name
}
if info.IsDir() {
header.Name += "/"
}
header.Format = tar.FormatPAX
header.ModTime = header.ModTime.Truncate(time.Second)
header.AccessTime, header.ChangeTime = time.Time{}, time.Time{}
if runtime.GOOS == "windows" {
header.Mode = header.Mode&0o755 | 0o111 // Windows has no exec bit, so mark everything executable like docker build
}
if err := tw.WriteHeader(header); err != nil {
return err
}
if content != nil {
_, err = io.CopyBuffer(tw, struct{ io.Reader }{content}, buffer) // hides os.File's WriteTo, which would allocate a buffer per file
}
return err
}
type noNameLookups struct{ fs.FileInfo } // leaves Uname and Gname empty like docker build, so packing never waits on NSS
func (noNameLookups) Uname() (string, error) { return "", nil }
func (noNameLookups) Gname() (string, error) { return "", nil }
+71
View File
@@ -0,0 +1,71 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
package container
import (
"archive/tar"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
func TestDockerBuildReportsDaemonError(t *testing.T) {
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("API-Version", "1.47")
if strings.HasSuffix(request.URL.Path, "/build") {
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(writer, `{"message":"build rejected"}`)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
t.Setenv("DOCKER_CONFIG", t.TempDir())
t.Setenv("DOCKER_AUTH_CONFIG", "")
require.ErrorContains(t, NewDockerBuildExecutor(NewDockerBuildExecutorInput{
ImageTag: "action:test", Dockerfile: "Dockerfile", BuildContext: strings.NewReader(""),
})(t.Context()), "build rejected")
}
func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile(t *testing.T) {
dir := t.TempDir()
for name, content := range map[string]string{
".dockerignore": "*.log\nbuild\n!build/keep.txt\nDockerfile", "Dockerfile": "FROM scratch", "a.txt": "a",
"a.log": "log", "build/drop.txt": "drop", "build/keep.txt": "keep",
} {
require.NoError(t, os.MkdirAll(filepath.Join(dir, filepath.Dir(name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
}
require.NoError(t, os.Mkdir(filepath.Join(dir, "empty"), 0o755))
contexts := []string{dir}
if link := filepath.Join(t.TempDir(), "link"); os.Symlink(dir, link) == nil {
contexts = append(contexts, link+string(filepath.Separator))
}
for _, contextDir := range contexts {
buildContext, err := createBuildContext(t.Context(), contextDir, "Dockerfile")
require.NoError(t, err)
var names []string
for reader := tar.NewReader(buildContext); ; {
header, err := reader.Next()
if err == io.EOF {
break
}
require.NoError(t, err)
names = append(names, header.Name)
}
require.NoError(t, buildContext.Close())
require.Equal(t, []string{".dockerignore", "a.txt", "build/keep.txt", "empty/", "Dockerfile"}, names, contextDir)
}
}
+18
View File
@@ -0,0 +1,18 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd)
package container
import (
"io/fs"
"syscall"
)
func hardlinkInode(info fs.FileInfo) (uint64, bool) {
if stat, ok := info.Sys().(*syscall.Stat_t); ok && !info.IsDir() && stat.Nlink > 1 {
return stat.Ino, true
}
return 0, false
}
Loaded 50 of 217 files, more files were not shown because too many files have changed in this diff. Show more