Compare commits

...
52 Commits
Author SHA1 Message Date
silverwind 4f59e3e418 fix: support cache-mode and make low-trust runs cache read-only (#1412)
Low-trust runs like `pull_request_target` on the default branch could poison caches that `push` runs restore. As on GitHub, they now only restore unless `cache-mode` allows saving.

This and the Gitea change can be deployed in any order.

Refs:
https://gitea.com/gitea/actionslib/pulls/33
https://github.com/go-gitea/gitea/pull/40343

_Written by Claude (Opus 5.5)._
Reviewed-on: https://gitea.com/gitea/runner/pulls/1412
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-13 09:33:32 +00:00
clintlmartinandsilverwind c2046d597c fix: stop host teardown from deleting a workdir it never created (#1409)
Host jobs run in a per-job directory under `host.workdir_parent`, which teardown removes. Teardown also deleted `<container.workdir_parent>/<owner>/<repo>`, a path host jobs never create. With an empty parent and no repository that was the filesystem root, and any existing directory at that path was deleted although the job never used it.

This removes that delete instead of guarding it, so teardown only removes what the job created.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1409
Reviewed-by: bircni <[email protected]>
Co-authored-by: clintlmartin <[email protected]>
2026-11-13 06:50:44 +00:00
silverwind 645b17de72 fix: align common behaviour with GitHub's runner (#1413)
Closes differences from GitHub's runner that common workflows hit:

- `ACTIONS_STEP_DEBUG` set as a repository variable enables debug output as on GitHub, where a secret of the same name still takes precedence. `runner.debug` and `RUNNER_DEBUG` are now also set inside composite actions, which lost the secret.
- `github.env` and `github.path` hold the file-command paths instead of being empty, so `echo "A=1" >> "${{ github.env }}"` works.
- Remote composite actions only get a pre stage when they nest actions. That stage evaluated `with:` before earlier steps ran, so an input like `${{ fromJSON(steps.meta.outputs.json).v }}` failed the job.
- `ports` on a job container are published, they were ignored while service ports worked.
- Dockerfile action build output reaches the job log. It was logged at debug and dropped, so a failing `RUN` showed only the final error.
- Local Docker actions honor their `.dockerignore`, as `docker build` does.
- Cancelling or timing out a host-mode step sends SIGINT, then SIGTERM after 7.5 seconds and SIGKILL after 2.5 more, as GitHub's runner does, so `trap` handlers and shutdown hooks run.
- `.sh` job hooks fall back to `sh` where bash is missing, and `.ps1` hooks to `powershell` where pwsh is missing, as on GitHub.

_Written by Claude (Opus 5.5)._

Reviewed-on: https://gitea.com/gitea/runner/pulls/1413
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-12 15:28:47 +00:00
silverwind 0dfec377d2 fix: keep composite post steps in their own step scope (#1410)
Each composite step's post ran its earlier siblings' posts inside its own step-ID scope, so their log records listed siblings as ancestors, e.g. `outer.results.nodepost` instead of `outer.nodepost`. Every post now gets its own scope. Execution order and run-all-posts behavior are unchanged.

Prerequisite for https://gitea.com/gitea/runner/issues/809.

_Written by Claude (Opus 5.5)._

Reviewed-on: https://gitea.com/gitea/runner/pulls/1410
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-11 19:31:21 +00:00
silverwind b5305c6785 docs: add a launchd example for macOS (#1408)
Adds `examples/launchd` with a plist and the steps to run the runner on macOS, next to the existing systemd example.

The runner runs in the login session of the user that jobs should run as, which gives jobs the login keychain and the GUI. The README also covers the two problems that setup runs into: macOS blocks the runner from hosts on the Mac's own subnet, and Git's keychain helper can stop a job at a dialog.

_Written by Claude (Fable 5.2)._

Reviewed-on: https://gitea.com/gitea/runner/pulls/1408
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-10 17:17:35 +00:00
bircni a021d3139d enhance: make shared tool cache safe at any capacity (#1401)
Co-authored-by: bircni <[email protected]>
2026-11-10 14:25:50 +00:00
bircniandsilverwind 84c0fcd1fa fix!: remove the dbfs log workaround and require Gitea 1.26.2 (#1395)
Gitea 1.26.2 finalizes logs on an empty `UpdateLog` with `NoMore` (https://github.com/go-gitea/gitea/pull/37687), so the sentinel row is no longer needed. On older servers, the final log of most jobs would never be archived and stays in `dbfs_data`, so this raises the minimum Gitea version. It lives in `ver.MinGiteaVersion`, which the error for servers without `Declare` now names.

Fixes: https://gitea.com/gitea/runner/issues/953
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1395
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-10 11:35:16 +00:00
bircniandsilverwind c59caa9689 refactor!: remove the builtin actions (#1407)
Removes `builtin:checkout`, `builtin:upload-artifact` and `builtin:download-artifact`, along with the container helpers only they used.

Builtin actions are tied to the runner version, so users cannot pin them, and they diverge from the shareable actions ecosystem that tools like zizmor build on. Workflows should use `actions/checkout`, `actions/upload-artifact` and `actions/download-artifact`, or lightweight third-party actions such as https://github.com/taiki-e/checkout-action.

**Breaking:** steps with `uses: builtin:checkout`, released in v4.0.0, now fail and need to switch to one of the actions above. The artifact actions were never released.

Docs: https://gitea.com/gitea/docs/pulls/643
Fixes: https://gitea.com/gitea/runner/issues/1405
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1407
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-10 11:05:38 +00:00
Renovate Bot 85d007c47f chore(deps): update dependencies (#1406)
This PR contains the following updates:

| Package | Type | Update | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|---|---|
| docker.gitea.com/runner-images |  | digest | `657378e` → `b96ba46` |  |  |
| [github.com/docker/cli](https://github.com/docker/cli) | require | patch | `v29.9.2+incompatible` → `v29.9.3+incompatible` | ![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fdocker%2fcli/v29.9.3+incompatible?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fdocker%2fcli/v29.9.2+incompatible/v29.9.3+incompatible?slim=true) |
| [rancher/k3s](https://github.com/k3s-io/k3s) |  | patch | `v1.37.1-k3s1` → `v1.37.2-k3s1` | ![age](https://developer.mend.io/api/mc/badges/age/docker/rancher%2fk3s/v1.37.2?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/docker/rancher%2fk3s/v1.37.1/v1.37.2?slim=true) |

---

### Release Notes

<details>
<summary>docker/cli (github.com/docker/cli)</summary>

### [`v29.9.3+incompatible`](https://github.com/docker/cli/compare/v29.9.2...v29.9.3)

[Compare Source](https://github.com/docker/cli/compare/v29.9.2...v29.9.3)

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xOTkuMiIsInVwZGF0ZWRJblZlciI6IjQ0LjE5OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://gitea.com/gitea/runner/pulls/1406
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-11-09 04:31:50 +00:00
silverwindandbircni d34e4cf11a fix: stop a stalled docker daemon from hanging jobs and the e2e suite (#1404)
When the docker daemon stopped answering on https://gitea.com/gitea/runner/actions/runs/1204971, unbounded exec runs hung the e2e suite until the 20m `go test` panic, which skipped teardown. Exec runs and teardown are now bounded, and attached exec streams, including the runner's user ID probe, close with their context.

CLI daemons now stop with SIGTERM, so job volumes no longer leak.

_Written by Claude (Opus 5.5)._

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1404
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-07 08:07:41 +00:00
8e999cd039 fix: re-detect the cache address per job and prefer IPv4 (#1398)
The cache address was detected once at startup, and the interface fallback ranked ethernet names above IPv4. A runner that started before DHCPv4 finished could advertise a global IPv6 address to IPv4-only job networks until it restarted, failing every artifact and cache call.

IPv4 now ranks first, and the address is detected again for each job unless `cache.host` sets it. Detecting an IPv6 address logs a warning.

Fixes: https://gitea.com/gitea/runner/issues/1379
Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1398
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-07 06:41:25 +00:00
silverwind cd748053b0 docs: trim comments and docs from workspace emptying change (#1403)
Follow-up to https://gitea.com/gitea/runner/commit/79faf5e55df7a3b312fb4888b7405a58e01f567c. Reverts the post-task-script doc change, which is too much detail for that page. Also shortens the `DrainGracePeriod` comment and the other comments that commit added, since the code and log messages already say most of it.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1403
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-07 06:14:41 +00:00
bircni 9bb72bbe5c enhance: add ubuntu-26.04 to the default labels (#1396)
Newly registered runners get an `ubuntu-26.04` label alongside the existing defaults.

Ref: https://gitea.com/gitea/runner/issues/1334
Reviewed-on: https://gitea.com/gitea/runner/pulls/1396
Reviewed-by: Lunny Xiao <[email protected]>
2026-11-07 05:12:04 +00:00
79faf5e55d fix: empty root-owned bind workspaces for non-root runners (#1399)
A job running as root leaves root-owned files in the bind-mounted workspace. A non-root runner on Linux cannot remove them, so workspaces pile up with only a warning. Such a runner now empties the workspace as root from inside the job container, right before removing the container. Root runners and Docker Desktop already remove these files themselves and skip this.

The workspace is only emptied when it is exactly the runner's own mountpoint with no mount below it. Otherwise, or when the command fails, the existing removal and warning apply. Jobs of a reusable workflow share the workspace, so they are not covered and still end in the existing warning.

Fixes: https://gitea.com/gitea/runner/issues/879
Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1399
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-07 04:33:19 +00:00
bircni 993389d428 docs: trust a private registry CA in the dind images (#1397)
The bundled Docker daemon pulls job images, so it needs to trust a private registry's CA itself, which `runner.ca_file` doesn't cover. Documents the `certs.d` mount for both dind images.

Ref: https://gitea.com/gitea/runner/issues/1390
Reviewed-on: https://gitea.com/gitea/runner/pulls/1397
Reviewed-by: Lunny Xiao <[email protected]>
2026-11-06 21:36:06 +00:00
silverwind db76fd5159 enhance: improve hashFiles coercion and log grouping (#1394)
- Fix `hashFiles()` argument conversion for numbers, booleans, and null to match GitHub expression semantics.
- Fix rendering of mixed-case, indented, and prefixed legacy group commands while preserving recognized prefixes and escaping.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1394
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-06 20:03:39 +00:00
bircni c8b16f4792 enhance: add builtin:download-artifact (#1344)
Co-authored-by: bircni <[email protected]>
2026-11-06 03:32:59 +00:00
bircni 02446620df enhance: add builtin:upload-artifact (#1343)
Co-authored-by: bircni <[email protected]>
2026-11-06 03:17:49 +00:00
8c8950d9ef refactor(container): stream copies into the job and archives out of the host (#1340)
Adds `CopyTarStream` to the job environments. It extracts directories and files into the job as they are written, kept below the destination and owned by the job user along with any directories it creates, so built-in actions can put large content in place without buffering it. The host's `GetContainerArchive` now streams instead of building the archive in memory.

---------

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1340
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-06 02:22:40 +00:00
bircniandsilverwind b88d111cc5 feat: port the path patterns of @actions/glob (#1342)
Built-in actions that select files need to match what the JavaScript actions match. This ports @actions/glob 0.6.1 and the minimatch 3 subset it uses, as bundled in `actions/upload-artifact` v7. It only answers questions about paths; walking the files stays with the caller. Only malformed bracket expressions and some non-ASCII case folding on Windows can match differently.

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1342
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-06 01:13:07 +00:00
silverwind bb6d793cb9 fix: make container.sweep opt-in (#1389)
Sweeping removed every container a job started through its Docker proxy, which silently tore down services that jobs deploy to the runner's own Docker daemon. Deploying from a runner on the target host is common, so `container.sweep` now defaults to `false` and CI-only runners can opt in with `container.sweep: true`.

Fixes https://gitea.com/gitea/runner/issues/1385

Reviewed-on: https://gitea.com/gitea/runner/pulls/1389
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-05 13:50:46 +00:00
silverwind 95ef17d022 fix: prevent masking gaps in workflow command logs (#1384)
Mask partially escaped workflow command output and complete one-character values while preserving literal output and stream line endings.

Fixes: https://gitea.com/gitea/runner/issues/300
Reviewed-on: https://gitea.com/gitea/runner/pulls/1384
Reviewed-by: bircni <[email protected]>
2026-11-05 08:01:44 +00:00
Lunny Xiaoandsilverwind ee03687688 fix: upgrade Alpine packages in the runner images (#1388)
All images run `apk upgrade` before installing packages, so they pick up Alpine fixes their base images lag on. `docker:29.9.2-dind` shipped libexpat 2.9.0-r0 while Alpine 3.24 provides 2.10.0-r0 with the security fixes, and both `docker:29.9.3-dind` and `alpine:3.24` still ship apk-tools 3.0.8-r0 where 3.0.9-r0 is available. The dind bases are also bumped to `docker:29.9.3`, which drops several vulnerable bundled binaries' findings.

Verified by building the targets locally: libexpat is 2.10.0-r0 (was 2.9.0-r0), apk-tools is 3.0.9-r0 (was 3.0.8-r0), and Trivy reports no Alpine package findings on the dind images.

Assisted-by: Codet:claude
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1388
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
2026-11-05 07:45:27 +00:00
silverwind 30e600d8f6 test: speed up (#1386)
Reduce redundant fixture setup, Git subprocesses, and large data copies while preserving test coverage.

Local macOS measurements against main:

| Measurement | Before | After |
| --- | ---: | ---: |
| Git package time, median of 3 alternating runs | 4.09 s | 3.48 s (-15%) |
| Allocations across 18 action-bundle cases | 489 MiB | 251 MiB (-49%) |
| Traced Git starts per checkout table run | 234 | 175 |

Reviewed-on: https://gitea.com/gitea/runner/pulls/1386
Reviewed-by: bircni <[email protected]>
2026-11-05 07:33:17 +00:00
silverwind 664978c819 test: fix windows path test (#1387)
`TestCopyWorkflowFilesHandsNodeTheRunnerCAUnlessTheJobOrImageSetsIt` from https://gitea.com/gitea/runner/pulls/1366 fails on Windows. `HostEnvironment.GetActPath` turns backslashes into slashes there, so `NODE_EXTRA_CA_CERTS` gets the slash form while the test expects the raw temp dir path. The runner behaviour is correct, only the expectation is wrong.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1387
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-05 07:14:56 +00:00
aa56f46c89 enhance: trust a private CA with runner.ca_file and add container.volumes (#1366)
Adds `container.volumes`, mounted into every Docker job, service and action container. Sources are paths on the Docker daemon's host and need no `valid_volumes` entry, and workflow volumes may use them only with the same access.

Adds `runner.ca_file`, a private CA trusted for the Gitea instance besides the system's by the runner's connections and its artifact proxy. Runner-side git and `builtin:checkout` use it as their CA bundle for the instance, persisted with the checkout's credentials. Jobs get it at `/var/run/act/ca.crt` with `NODE_EXTRA_CA_CERTS` pointing there unless the job or its image sets it. Git in jobs, as run by `actions/checkout`, needs the CA in the image's bundle, for example through `container.volumes`.

Fixes: https://gitea.com/gitea/runner/issues/568
Fixes: https://gitea.com/gitea/runner/issues/671
Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1366
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-05 05:28:40 +00:00
silverwindandLunny Xiao 7a9fccb842 ci: sign release checksums with the imported GPG key (#1382)
The release job has imported the Teabot GPG key since https://gitea.com/gitea/runner/pulls/116, but `.goreleaser.yaml` never had a `signs:` block, so releases were never signed. Add it so `checksums.txt` gets a detached `checksums.txt.sig`, the same way https://gitea.com/gitea/cli signs its releases.

The nightly job imports the key as well since the `signs:` block applies to nightly builds, and the R2 publisher mirrors the signature to dl.gitea.com.

_Written by Claude (Opus 5.5)._

---------

Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1382
Reviewed-by: Lunny Xiao <[email protected]>
2026-11-05 04:28:22 +00:00
silverwindandLunny Xiao 7d6e1ff021 test: fix TestRunExecListUsesJobEventAndAllPlans hang on macOS (#1378)
`TestRunExecListUsesJobEventAndAllPlans` hangs until the test timeout on macOS.

Regression from https://gitea.com/gitea/runner/pulls/1338, which added a fifth `runExecList` call to the test to cover `exec --list --detect-event` without events. That took the captured output from 498 to 560 bytes. `captureStdout` swapped `os.Stdout` for a pipe it only read after the callback returned, and on macOS 27.2 an unread pipe holds 512 bytes, so the write blocked forever. Linux pipes hold 64 KiB, so was unaffected.

`printList` and `runExecList` now write to a passed writer, `exec --list` hands them `cmd.OutOrStdout()`, and the tests pass a buffer, so no test swaps `os.Stdout` anymore.

---------

Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1378
Reviewed-by: Lunny Xiao <[email protected]>
2026-11-05 04:22:01 +00:00
silverwind 319567213d docs: use latest image tags in examples (#1383)
The README's Docker example and the Docker and Kubernetes examples pointed at `nightly` images, while the README recommends `latest-dind`. Deployments copied from them should run the latest release, not a build of `main`.

None of the examples changed since v5.3.0, so they work as written with the current `latest` tags.

_Written by Claude (Opus 5.5)._

Reviewed-on: https://gitea.com/gitea/runner/pulls/1383
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 23:00:45 +00:00
silverwind 0093731295 chore(deps): update go.yaml.in/yaml/v4 to v4.0.0 (#1381)
Update go-yaml from the rc.6 pseudo-version to the stable v4.0.0 release, and actionslib to v1.5.2, which does the same in https://gitea.com/gitea/actionslib/pulls/32.

v4 accepts `:` as a mapping with an empty key, so the reusable workflow tests now use an unclosed `[` as their invalid workflow. Gitea counterpart: https://github.com/go-gitea/gitea/pull/40222.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1381
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 20:05:17 +00:00
silverwind 3e1b4b42c2 fix(cache): stop the cache server from running its volume out of space (#1377)
Committing a cache upload copied every part into a new archive before deleting the parts, so a 5 GB save needed 10 GB free. Failed writes and assemblies left their partial bytes until the next stale sweep, up to an hour later, and every other save on the runner failed meanwhile. Nothing made room for an upload, so once the cache had filled its volume down to the free space floor, larger saves kept failing.

- Assemble in place, deleting each part once appended.
- Drop partial parts and archives on failure.
- Before a part would cross the floor, evict the least recently used entries.
- Reject uploads to reservations the sweep already deleted, which left files nothing removed.

_Written by Claude (Opus 5.5)._

Reviewed-on: https://gitea.com/gitea/runner/pulls/1377
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 16:38:21 +00:00
silverwind dcba546cb3 fix: align process handling and fix Windows crashes (#1376)
The intermittent Windows test crashes came from step errors formatting a nil `rc.Run`. `fmt` recovers the nil dereference, but on CPUs with AMX the recovered fault overflows into the heap (https://github.com/golang/go/issues/81238). The errors now name the job, and the Windows test VM hides AMX and runs on UTC, so cached boots no longer jump back 8 hours mid-test.

Windows commands now start suspended and resume once they are in their job objects, so no child escapes them. That drops the process scan reading other processes' memory and the last `unsafe` code, which `depguard` now denies. Step processes outlive a crashed runner, and commands are no longer looked up in the runner's working directory, both as with GitHub's runner.

Like GitHub's runner, host jobs on Linux and macOS now end the processes carrying the job's `RUNNER_TRACKING_ID`, and image pulls and Dockerfile action builds are retried.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1376
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 16:32:22 +00:00
silverwind 142d9f8c88 fix: give each Dockerfile action image its own ID (#1372)
Identical Dockerfile builds in concurrent jobs share one image ID under their per-job tags. When a job ends and removes its tag, the daemon deletes the image while another job's build is about to tag it, failing that job with `No such image`. This also hits `TestRunEvent` in CI, where two parallel subtests build the same action.

Label each build with its tag, so every job gets its own image on top of the shared cached layers and removing it never touches another job's.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1372
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 07:08:45 +00:00
silverwindandbircni 54b2cb944c fix: give git-lfs the checkout token once (#1371)
`builtin:checkout` set the token both in its persisted credentials file and in the step's environment. git-lfs before 3.8.0 then sent `Authorization` twice on the LFS batch request, which nginx rejects with 400.

Write the credentials file once the checkout succeeded, since it only serves later steps. A failed checkout no longer persists credentials.

Users can also upgrade git-lfs to fix the root bug, but this fix is worth to keep regardless.

Fixes: https://gitea.com/gitea/runner/issues/1370
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1371
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 06:42:00 +00:00
Zettat123andsilverwind b65fd0f8f8 ci: run the Windows tests on a runner with KVM (#1369)
Run the Windows tests on `ubuntu-latest-kvm` as a PR job, with the installed Windows cached, minutes instead of ~1h40m. Aligned with https://gitea.com/gitea/gitea-mcp/pulls/321 and https://gitea.com/gitea/cli/pulls/79.

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1369
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-11-04 02:08:38 +00:00
silverwind f2fbd67962 enhance: support Docker on FreeBSD and restructure the README (#1367)
1. Restructures the README into installation, quickstart, deployment, configuration, jobs and operation, and shortens it to user-facing prose. Option details are left to `config.example.yaml` and `docs/`, existing heading anchors are kept where possible.
2. FreeBSD builds used the Docker stub, so `docker://` labels, services and Docker actions failed there. The Docker code works on FreeBSD against a remote daemon, so it is enabled, and host jobs report `runner.os` as `FreeBSD`.
3. `make build` on Windows now writes `gitea-runner.exe`, taking the suffix from `go env GOEXE`.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1367
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-03 02:42:25 +00:00
silverwind d6f5b68a0d enhance: test on Windows and fix what it found (#1348)
Adds a weekly and on-demand workflow that runs the cross-compiled tests on Windows Server 2025 Core in a [dockur/windows](https://github.com/dockur/windows) QEMU VM, and fixes the Windows bugs it found in host and UNC paths, env handling, shell quoting, post-task scripts, health checks, checkout, action symlinks, step summaries, Git client certificates and docker mode from a Windows host.

Host-mode steps now emit UTF-8, `bash` falls back to Git's bash instead of the WSL launcher, the runner can run as a native Windows service, steps can keep background helpers with `RUNNER_TRACKING_ID`, and Windows arm64 binaries are released.

Fixes: https://gitea.com/gitea/runner/issues/1339
Fixes: https://gitea.com/gitea/runner/issues/637
Fixes: https://gitea.com/gitea/runner/issues/1172
Fixes: https://gitea.com/gitea/runner/issues/347
Reviewed-on: https://gitea.com/gitea/runner/pulls/1348
Reviewed-by: bircni <[email protected]>
2026-11-03 01:26:10 +00:00
Renovate Bot 6e42621b4a chore(deps): update dependencies (#1365)
Co-authored-by: Renovate Bot <[email protected]>
2026-11-02 08:16:42 +00:00
adamaruandsilverwind 228effab64 fix(dind): double-fork in s6-notifyoncheck to eliminate zombie processes (#1364)
In `scripts/s6/docker/run`, `s6-notifyoncheck` runs without the `-d` flag:

```bash
exec s6-notifyoncheck -n 0 dockerd-entrypoint.sh
```

Without `-d`, `s6-notifyoncheck` performs a single fork to execute the background checking process. The parent immediately replaces itself via `exec` with `dockerd-entrypoint.sh` (which calls `exec rootlesskit` on `dind-rootless`).

Because only a single fork occurred, the background checking child remains parented to `rootlesskit`. When the check passes and the process terminates:
1. `rootlesskit` did not spawn this process and never reaps it via `waitpid()`.
2. Under standard Linux process semantics, `s6-svscan` (PID 1) cannot adopt or reap the child because its parent (`rootlesskit`) is still running.
3. The terminated process remains as a permanent zombie (`[s6-notifyoncheck] <defunct>`) for the container's entire lifespan.

Adding `-d` instructs `s6-notifyoncheck` to double-fork, detaching the background check so it is immediately adopted by `s6-svscan` (PID 1) and cleanly reaped upon exit.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1364
Reviewed-by: silverwind <[email protected]>
Co-authored-by: adamaru <[email protected]>
2026-11-01 07:20:20 +00:00
silverwindandbircni a46d3f3eda test(e2e): speed up and cover exec and Kubernetes modes (#1363)
Follow-up to https://gitea.com/gitea/runner/pulls/1362, whose crash e2e missed because it only drove the daemon on Docker.

- `dispatch`, `failure_paths` and `reusable_workflow` run through the daemon and `exec`, each on Docker and one shared k3s cluster.
- Fixed sleeps become gates the test releases, the job timeout uses a 15s runner deadline, k3s boots alongside Gitea, Gitea's data is on tmpfs.
- Outside a container, `make test-e2e` runs `go test` in a Linux container on its own network like CI, so local runs take the same paths.

Local wall time:

| | Before | After |
|---|---|---|
| Suite | 84s | 27–35s |
| `job_timeout` | 64s | 17s |
| `reconnect` | 22s | 4.5s |
| `capacity/serial` | 20s | 2s |
| `cli_shutdown_drains` | 14s | 2.3s |
| `kubernetes` | 31s | 21–23s |

_Written by Claude (Opus 5.5)._

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1363
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-31 10:47:35 +00:00
2e72cf902a enhance(kubernetes): fetch job pod copies from the runner over the pod network (#1354)
The Kubernetes backend copies actions and workspaces into job containers with `tar` over `pods/exec`, so every byte goes through the API server. On managed clusters that often sits in other zones than the nodes, so each byte can cross zones twice, and the copies load the API server.

With `kubernetes.transfer.mode: auto` or `direct`, an init container copies `gitea-runner` into the job pod, and the job container fetches directories and copies of 64 KiB or more gzipped from the runner over the pod network. `auto` falls back to the API server for a job pod that cannot reach the runner, `direct` fails the job.

Copies can hold secrets, so each is served once, over TLS 1.3 pinned to a certificate the runner generates, only to the job pod's IPs.

Copying setup-go@v5, cache@v4, setup-java@v4 and build-push-action@v6 sends 11.6 MB through the API server on main and 532 B with this PR.

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1354
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
2026-10-31 08:00:36 +00:00
silverwind f4741df90d fix(exec): load local reusable workflows from disk with --no-skip-checkout (#1362)
`gitea-runner exec --no-skip-checkout` panics with a nil pointer dereference on any job that calls a local reusable workflow (`uses: ./.gitea/workflows/x.yml`).

Fetching a local reusable workflow from the instance was gated on `NoSkipCheckout`, but it reads the repository and SHA from the task's preset context, which only the daemon has. `exec` always has the workflow on disk, so the fetch is now gated on the preset instead, and `--no-skip-checkout` only governs checkout again. Daemon behavior is unchanged.

The `exec` e2e test now also runs the existing reusable caller/callee pair with `--no-skip-checkout`.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1362
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-31 05:24:36 +00:00
261ab374ce enhance(cache): scope cache entries by ref like GitHub (#1361)
Cache entries were only scoped by repository, so an entry a pull request saved could be restored on the default branch, for example through a `restore-keys` prefix.

Entries now carry the ref of the job that saved them. As on GitHub ([Restrictions for accessing a cache](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)), a job restores entries of its own ref first, then of its pull request's base branch, then of the default branch. Pull request events are scoped by their pull request number.

Entries saved before this change are no longer restored and expire through `cache.retention`, so caches start cold once after the upgrade. Runners sharing a cache, and an external `cache-server`, should be upgraded together.

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1361
Reviewed-by: silverwind <[email protected]>
Co-authored-by: JorgeDLMV <[email protected]>
2026-10-30 22:34:45 +00:00
KenanZhuandsilverwind ca8e9c3c64 enhance: add GITEA_* aliases for built-in workflow environment variables (#1358)
Closes https://gitea.com/gitea/runner/issues/1317, which makes https://github.com/go-gitea/gitea/issues/39813 redundant as the runner sets these variables.

Every built-in `GITHUB_*` environment variable the runner sets is now also available as `GITEA_*` with the same value, so Gitea-native workflows don't have to rely on GitHub naming. The `GITEA_*` names mirror the `GITHUB_*` ones after their last write, so they stay equal and are unset together.

Workflow, job and runner `envs` values for these `GITEA_*` names are now overwritten, the same as for `GITHUB_*`.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1358
Reviewed-by: silverwind <[email protected]>
Co-authored-by: KenanZhu <[email protected]>
2026-10-30 21:21:02 +00:00
silverwindandbircni d8d4a4657d fix: scope the action identity to the step executing an action (#1359)
`github.action_repository`/`github.action_ref` and their env vars now belong to the step executing an action, like on GitHub: set for remote actions, empty for `run`, `docker://` and local steps. Before, a local action nested in a remote composite resolved `$/` against the remote action instead of the workflow's repository.

Updates actionslib to v1.5.1, which accepts `$/` as the repository root action.

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1359
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-29 17:41:27 +00:00
silverwind 5fd2f82735 test: fix race in TestTreeKillCancelBeforeCaptureKillsTree (#1360)
`TestTreeKillCancelBeforeCaptureKillsTree` fails intermittently on CI, e.g. https://gitea.com/gitea/runner/actions/runs/1155856.

Dash, the Ubuntu `/bin/sh`, points a background job's stdin at `/dev/null` before applying its redirections, so `cat <&0 &` read EOF and exited at once and the liveness check raced it. The pipe is now handed over on fd 3, which dash leaves alone.

Introduced in https://gitea.com/gitea/runner/pulls/1338

Reviewed-on: https://gitea.com/gitea/runner/pulls/1360
Reviewed-by: Zettat123 <[email protected]>
2026-10-29 16:46:53 +00:00
silverwind c6567296a0 fix: stop the docker proxy probe from leaking its container (#1357)
When the probe's deadline expired during `ContainerCreate`, dockerd still created the container but the probe never got its ID, leaving it in `Created` with its image's anonymous volumes. A slow daemon leaked one per job.

The probe now runs detached with its own timeout, so the create is never cancelled and the container is always removed. Callers still fall back after their own deadline, and the runner waits for pending probes before exiting.

Fixes: https://gitea.com/gitea/runner/issues/1353
Reviewed-on: https://gitea.com/gitea/runner/pulls/1357
Reviewed-by: Zettat123 <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-29 15:53:32 +00:00
Renovate Bot e3607b576e fix(deps): update module golang.org/x/net to v0.62.0 [security] (#1356)
This PR contains the following updates:

| Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| [golang.org/x/net](https://pkg.go.dev/golang.org/x/net) | [`v0.61.0` → `v0.62.0`](https://cs.opensource.google/go/x/net/+/refs/tags/v0.61.0...refs/tags/v0.62.0) | ![age](https://developer.mend.io/api/mc/badges/age/go/golang.org%2fx%2fnet/v0.62.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/golang.org%2fx%2fnet/v0.61.0/v0.62.0?slim=true) |

---

### Avoid quadratic complexity when coalescing text nodes in html in golang.org/x/net
[CVE-2026-78661](https://nvd.nist.gov/vuln/detail/CVE-2026-78661) / [GO-2026-6860](https://pkg.go.dev/vuln/GO-2026-6860)

<details>
<summary>More information</summary>

#### Details
When coalescing adjacent text nodes in addText and fosterParent, appending to an existing TextNode's Data via string concatenation (n.Data += text) reallocated and copied the accumulated string on every token. Inputs containing many short text runs separated by ignored end tags (such as "a</x>") caused O(n^2) allocations and byte copies.

Accumulate coalesced text in per-node byte slices during parsing and materialize Data strings once at the end of parse.

#### Severity
Unknown

#### References
- [https://go.dev/cl/849365](https://go.dev/cl/849365)
- [https://go.dev/issue/82224](https://go.dev/issue/82224)
- [https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs](https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6860) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Check all ancestor namespace transitions when rendering raw text in html in golang.org/x/net
[CVE-2026-97045](https://nvd.nist.gov/vuln/detail/CVE-2026-97045) / [GO-2026-6866](https://pkg.go.dev/vuln/GO-2026-6866)

<details>
<summary>More information</summary>

#### Details
In childTextNodesAreLiteral, walking up the tree previously stopped at the nearest foreign ancestor once an HTML integration point was found. However, that integration point may itself be a descendant of an element that was foster-parented into a foreign namespace (such as <mglyph> or <malignmark> fostered under <mtext>), causing a parser to resolve the subtree in a different foreign namespace upon re-parsing the rendered output.

Continue walking the full ancestor chain and verify that every transition from a non-empty namespace occurs at an HTML integration point before emitting literal text.

#### Severity
Unknown

#### References
- [https://go.dev/cl/855121](https://go.dev/cl/855121)
- [https://go.dev/issue/82622](https://go.dev/issue/82622)
- [https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs](https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6866) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xODEuNyIsInVwZGF0ZWRJblZlciI6IjQ0LjE4MS43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://gitea.com/gitea/runner/pulls/1356
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-10-28 06:33:58 +00:00
Lunny XiaoandZettat123 ffd80101a3 enhance(kubernetes): gzip copies into job containers over exec (#1355)
The Kubernetes backend copies actions and other files into the job container with `tar` over `pods/exec`, so the data goes through the API server, which on managed clusters often sits in other zones than the nodes (each copied byte then crosses zones twice). Actions are megabytes of JavaScript bundles that gzip about fivefold:

| Action | tar | gzip |
|---|---|---|
| actions/setup-go@v5 | 8.0 MB | 1.5 MB |
| actions/cache@v4 | 11.6 MB | 2.0 MB |
| actions/setup-java@v4 | 15.3 MB | 2.5 MB |
| docker/build-push-action@v6 | 12.9 MB | 3.0 MB |

The startup probe now also reports whether the job container has `gzip`. If it does, `CopyDir` and copies of 64 KiB or more are gzipped at `BestSpeed` and extracted with `gzip -dc | tar -xf -`. Smaller copies (run scripts, file command files) stay uncompressed, where gzip would save next to nothing and cost a process in the pod. Images without `gzip` keep the plain tar stream, so the image requirements do not change.

This is independent of #1354 and gets most of its traffic saving with no new moving parts.

---
*Written by Codet (AI coding agent) on behalf of @lunny.*

---------

Co-authored-by: Zettat123 <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1355
Reviewed-by: Zettat123 <[email protected]>
2026-10-28 06:00:51 +00:00
b712c91c2d test(e2e): cover host mode, reusable workflows, scheduling, reconnect and CLI flags (#1347)
Extends the e2e suite with scenarios for host labels, local `workflow_call`, label routing, `runner.capacity`, a dropped and restored connection to Gitea, and the daemon's `--once`, `--labels` and `shutdown_timeout` behaviour.

Stacked on https://gitea.com/gitea/runner/pulls/1346, so its commit shows here until that merges. The new scenarios pass locally against gitea/gitea:latest and main-nightly.

Assisted-by: Codet:codet
Co-authored-by: Zettat123 <[email protected]>
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1347
Reviewed-by: Zettat123 <[email protected]>
2026-10-28 04:32:50 +00:00
d77e216674 fix(exec): serve the v4 artifact API so upload-artifact v4 and later work (#1341)
`exec` forwarded the toolkit's ArtifactService calls to an instance it never registered, and its random runtime token held no backend IDs, so `actions/upload-artifact` and `actions/download-artifact` v4 and later failed.

The local artifact server now serves the v4 API, including the single Put Blob uploads docker/build-push-action uses. `exec` gives each workflow its own unsigned JWT runtime token and artifact run, so artifact names conflict and carry over between runs as on GitHub. Artifacts default to a temporary directory instead of the workspace that jobs copy.

Co-authored-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1341
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-10-28 03:52:53 +00:00
Lunny Xiao a76ccd7020 test(e2e): cover the register/daemon CLI and failing jobs (#1346)
The e2e suite started the runner in-process and never exercised failing jobs. This adds a scenario that builds the binary and drives `register` then `daemon` against the Gitea fixture, and one for failed, skipped, continue-on-error and timed-out jobs.

Verified locally against gitea/gitea:main-nightly.

Assisted-by: Codet:codet
Reviewed-on: https://gitea.com/gitea/runner/pulls/1346
Reviewed-by: Zettat123 <[email protected]>
2026-10-27 21:23:59 +00:00
228 changed files with 7908 additions and 3474 deletions

No files matched your search

+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7
with:
node-version: 24
node-version: 26
- run: make lint-pr-title
env:
PR_TITLE: ${{ github.event.pull_request.title }}
+8
View File
@@ -34,6 +34,13 @@ jobs:
with:
go-version-file: "go.mod"
check-latest: true
- name: Import GPG key
id: import_gpg
uses: crazy-max/ghaction-import-gpg@2dc316deee8e90f13e1a351ab510b4d5bc0c82cd # v7
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
passphrase: ${{ secrets.PASSPHRASE }}
fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0
- name: goreleaser
uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with:
@@ -47,6 +54,7 @@ jobs:
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
GORELEASER_FORCE_TOKEN: "gitea"
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GPG_FINGERPRINT: ${{ steps.import_gpg.outputs.fingerprint }}
release-image:
runs-on: ubuntu-latest
+27 -1
View File
@@ -25,7 +25,7 @@ jobs:
# daemon retains them between runs, and their digest pins skip the registry when it has them.
- name: pre-pull test images
env:
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:55fb5f5a527ea2c5c4a95f963e48ef90aa38c7b346054fdc8ee9957a2b55d205 # renovate: datasource=docker
TEST_JOB_IMAGE: node:26-bookworm-slim@sha256:bc0a2afd3b74ce12222fc8bab99271e66257f021078f5f4a5d687a77d280c1d9 # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:a3554c0d0b48177b1413c26e01f695334428020d9df01243fbae3e11cfd17417 # renovate: datasource=docker
run: |
for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do
@@ -57,6 +57,32 @@ jobs:
make coverage-report
cat .tmp/coverage.md >> "$GITHUB_STEP_SUMMARY"
windows:
name: test on windows
runs-on: ubuntu-latest-kvm
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.gitea/actions/go-setup
- name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
- name: windows vm cache key
id: windows-vm
run: echo "key=windows-vm-$(grep '^WINDOWS_TEST_IMAGE' Makefile | cat - scripts/test-windows.sh scripts/test-windows-setup.ps1 | sha256sum | cut -c1-16)" >> "$GITHUB_OUTPUT"
- uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/windows-vm
key: ${{ steps.windows-vm.outputs.key }}
- name: test on windows
id: test-windows
run: WINDOWS_TEST_CACHE="$HOME/.cache/windows-vm" make test-windows
env:
WINDOWS_TEST_TIMEOUT: 45
- if: always() && steps.test-windows.outputs.save-cache == 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/windows-vm
key: ${{ steps.windows-vm.outputs.key }}
e2e:
name: gitea compatibility (${{ matrix.gitea_image }})
runs-on: ubuntu-latest
+1
View File
@@ -1,4 +1,5 @@
/gitea-runner
/gitea-runner.exe
.env
!/act/runner/testdata/secrets/.env
.runner
+2
View File
@@ -39,6 +39,8 @@ linters:
deny:
- pkg: golang.org/x/exp
desc: it's experimental and unreliable
- pkg: unsafe
desc: use typed APIs instead
logging:
files:
- $all
+5 -2
View File
@@ -34,8 +34,6 @@ builds:
goarch: s390x
- goos: windows
goarch: arm
- goos: windows
goarch: arm64
- goos: freebsd
goarch: s390x
- goos: freebsd
@@ -82,6 +80,7 @@ builds:
publishers:
- name: cloudflare-r2
checksum: true
signature: true
extra_files:
- glob: ./**.xz
- glob: ./**.sha256
@@ -102,6 +101,10 @@ checksum:
extra_files:
- glob: ./**.xz
signs:
- artifacts: checksum
args: ["--batch", "-u", "{{ .Env.GPG_FINGERPRINT }}", "--output", "${signature}", "--detach-sign", "${artifact}"]
snapshot:
version_template: "{{ .Branch }}-devel"
+27 -11
View File
@@ -23,11 +23,14 @@ depending on the prefix:
covers both forms: it escapes `%` first, so a literal `%3B` becomes `%253B` that the extra
`##[…]` rules cannot match, and a raw `;` or `]` is never decoded. It is also what makes
multi-line work, `\n` becomes `%0A` and the UI turns it back into a line break.
- **Forwarding a command from step output?** Leave the payload alone, it arrived escaped and is
decoded once. Decoding here double-decodes and destroys multi-line.
- **Forwarding a command from step output?** Preserve its escaping unless masking changes it.
Decoding for display happens in the UI.
- **No prefix?** Do not escape, and split multi-line values into one row each.
- **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the
encoded forms too.
- **Interpolating a secret?** Masking runs after escaping. Recognised command payloads are
normalised before masking to cover partially escaped values. Their original escaping is
preserved when no replacement occurs. `AppendSecretMasker` also registers encoded forms for
other output. Complete nonblank one-character values are masked, while one-byte fragments
of multiline values are skipped.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the command
handler folds an annotation's location into its decoded message and escapes the result again.
@@ -48,16 +51,29 @@ fields.
- Tests read what was logged with `logtest.NewLogger()`, or `logtest.SetDefault(t)` for the
package functions.
## Built-in actions
Each lives in `internal/pkg/action/<name>` and is registered in `builtinActions` in
`act/runner/step_action_builtin.go`.
## End-to-end compatibility tests
`make test-e2e` runs the runner against `E2E_GITEA_IMAGE`. It defaults to the nightly image.
It requires Docker and is excluded from `make test`. CI runs stable and nightly variants in
parallel.
parallel. Outside a container it runs `go test` in a Linux container on its own network, as CI does,
so both take the same paths. `E2E_ARGS` passes extra flags, e.g. `make test-e2e E2E_ARGS='-run TestCompatibility/dispatch'`.
The suite shares one Gitea and regular runner. Cache and ephemeral scenarios use isolated
The `cli_*` scenarios build the binary and drive `register` and `daemon` as a user would, including `--once`,
`--labels` and shutdown with a running job. `failure_paths` covers failed, skipped and tolerated jobs, `job_timeout`
a job killed at the runner's deadline. `label_routing` and `capacity` cover scheduling, `reconnect` drops the
connection to Gitea through a proxy, and `host_mode` and `reusable_workflow` cover host labels and `workflow_call`.
`dispatch`, `failure_paths` and `reusable_workflow` run in every mode: through the daemon and `exec`, each on
Docker and Kubernetes.
The suite shares one Gitea, regular runner and k3s cluster. Cache and ephemeral scenarios use isolated
repository runners. A run path is `<workflow>@<ref>` and a log row is `<timestamp>Z <payload>`.
## Windows tests
`make test-windows` cross-compiles the test binaries, installs Windows Server 2025 Core in a
[dockur/windows](https://github.com/dockur/windows) QEMU VM on the docker host and runs them there.
The host must be amd64 and allow privileged containers. `WINDOWS_TEST_TIMEOUT` sets the minutes
allowed for installing Windows and running the tests, 165 by default. The VM uses KVM when the host
has `/dev/kvm`, otherwise Windows is emulated and much slower. Docker-backed tests skip in the VM
and the race detector is off, it needs cgo. `WINDOWS_TEST_CACHE` names a directory to keep the
installed Windows in, so later runs skip the install. CI runs this on every pull request.
+7 -5
View File
@@ -19,14 +19,15 @@ RUN --mount=type=cache,target=/go/pkg/mod \
### DIND VARIANT
#
#
FROM docker:29.9.1-dind AS dind
FROM docker:29.9.3-dind AS dind
ARG VERSION=dev
LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}"
RUN apk add --no-cache s6 bash git tzdata nftables
# upgrade first: the docker base image can lag behind Alpine security fixes
RUN apk upgrade --no-cache && apk add --no-cache s6 bash git tzdata nftables
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh
@@ -41,7 +42,7 @@ ENTRYPOINT ["s6-svscan","/etc/s6"]
### DIND-ROOTLESS VARIANT
#
#
FROM docker:29.9.1-dind-rootless AS dind-rootless
FROM docker:29.9.3-dind-rootless AS dind-rootless
ARG VERSION=dev
@@ -49,7 +50,8 @@ LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}"
USER root
RUN apk add --no-cache s6 bash git tzdata nftables
# upgrade first: the docker base image can lag behind Alpine security fixes
RUN apk upgrade --no-cache && apk add --no-cache s6 bash git tzdata nftables
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh
@@ -75,7 +77,7 @@ ARG VERSION=dev
LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}"
RUN apk add --no-cache tini bash git tzdata
RUN apk upgrade --no-cache && apk add --no-cache tini bash git tzdata
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh
+9 -4
View File
@@ -1,13 +1,14 @@
DIST := dist
EXECUTABLE := gitea-runner
GO ?= go
EXECUTABLE := gitea-runner$(shell $(GO) env GOEXE)
HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" )
PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.9-immutable@sha256:87f42f37fa85b421664c30b8f5f9cfc23920057f5e5fcadf0b6f68d9edc6d415 # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:8856b200466e9a5af9cc0d4de4ca96f73547833bcdddf9a567189ff772c4d372 # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:b96ba465021b1a4c6a1c9e79220c2b9bfad0f29b09dbb528bbfb35411443a3a4 # renovate: datasource=docker
SERVICE_IMAGE ?= nginx:1.31.7-alpine@sha256:892bb2152464f0c5d8972593a5f5afaf8054552580b699d95f8ff4af08c2d33d # renovate: datasource=docker
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
E2E_K3S_IMAGE ?= rancher/k3s:v1.37.1-k3s1@sha256:ca7f37d993d82ef0dcdcfecb2e0e2618ea541dbaffc620c8cedebe01a82acd0d # renovate: datasource=docker
E2E_K3S_IMAGE ?= rancher/k3s:v1.37.2-k3s1@sha256:21ea6e0fac4e3930c2ce8aa40ddd41a8b18cb3e9a0222c030dc005e7e88c4332 # renovate: datasource=docker
WINDOWS_TEST_IMAGE ?= dockurr/windows:6.06@sha256:16fd7c39a3e48bb946d29c59dc6015942e5b900a1fa5d7e5552053bfc48ee2ea # renovate: datasource=docker
GOFILES := $(shell find . -type f -name "*.go" -o -name "go.mod" ! -name "generated.*")
@@ -150,11 +151,15 @@ coverage-report: ## turn coverage.txt from `make test` into .tmp/coverage.md
test-dind: ## run the daemon-facing tests against the built dind image or podman (TARGET=dind|dind-rootless|podman)
@PODMAN_TEST_IMAGE=$(PODMAN_TEST_IMAGE) ./scripts/test-dind.sh $(TARGET)
.PHONY: test-windows
test-windows: ## run the tests on Windows Server Core in a QEMU VM on the docker host
@GO=$(GO) WINDOWS_TEST_IMAGE=$(WINDOWS_TEST_IMAGE) ./scripts/test-windows.sh
E2E_CONCURRENCY ?= 8
.PHONY: test-e2e
test-e2e: ## run Gitea compatibility tests against E2E_GITEA_IMAGE
@E2E_CONCURRENCY=$(E2E_CONCURRENCY) E2E_GITEA_IMAGE=$(E2E_GITEA_IMAGE) E2E_K3S_IMAGE=$(E2E_K3S_IMAGE) E2E_JOB_IMAGE=$(E2E_JOB_IMAGE) GO=$(GO) SERVICE_IMAGE=$(SERVICE_IMAGE) ./tools/test-e2e.sh
@E2E_CONCURRENCY=$(E2E_CONCURRENCY) E2E_GITEA_IMAGE=$(E2E_GITEA_IMAGE) E2E_K3S_IMAGE=$(E2E_K3S_IMAGE) E2E_JOB_IMAGE=$(E2E_JOB_IMAGE) GO=$(GO) SERVICE_IMAGE=$(SERVICE_IMAGE) ./tools/test-e2e.sh $(E2E_ARGS)
.PHONY: install
install: $(GOFILES) ## install the runner binary via `go install`
+142 -218
View File
File diff suppressed because it is too large. Load diff
-2
View File
@@ -6,7 +6,5 @@
//
// Inspired by https://github.com/sp-ricard-valverde/github-act-cache-server
//
// TODO: Authorization
// TODO: Restrictions for accessing a cache, see https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache
// TODO: Force deleting cache entries, see https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#force-deleting-cache-entries
package artifactcache
+128 -54
View File
@@ -59,6 +59,11 @@ type credKey struct{}
type JobCredential struct {
Repo string `json:"repo"`
Ref string `json:"ref"`
BaseRef string `json:"base_ref"`
DefaultRef string `json:"default_ref"`
CacheMode string `json:"cache_mode"` // empty allows everything
// Results is the instance whose artifact service this server forwards for the job, and
// InsecureTLS how the runner reaches it; see results.go.
Results string `json:"-"`
@@ -68,6 +73,10 @@ type JobCredential struct {
PublicURL string `json:"public_url"`
}
func (cred JobCredential) owns(cache *Cache) bool {
return cache.Repo == cred.Repo && cache.Ref == cred.Ref
}
// credEntry holds a registered job's credential along with an active
// registration count. RegisterJob is reference-counted so that if two tasks
// briefly share an ACTIONS_RUNTIME_TOKEN — e.g. a runner that retries a task
@@ -96,7 +105,10 @@ type Handler struct {
stopSync context.CancelFunc
syncing sync.WaitGroup
outboundIPMu sync.RWMutex
outboundIP string
detectOutboundIP func() net.IP // nil when the address is configured
instanceTransport http.RoundTripper
// internalSecret guards /_internal/{register,revoke}. When set, a remote
// runner can use these endpoints to pre-register per-job
@@ -116,7 +128,7 @@ type Handler struct {
policy Policy
// freeDisk is a field so tests can drive evictForFreeSpace without a full volume.
// freeDisk is a field so tests can drive makeRoom without a full volume.
freeDisk func(string) (uint64, error)
}
@@ -137,6 +149,7 @@ type Options struct {
Policy Policy
Logger *logging.Logger
InstanceTransport http.RoundTripper // reaches the instance artifact requests are forwarded to, nil for http.DefaultTransport
}
// StartHandler starts an embedded cache or forwards cache requests to Upstream.
@@ -148,6 +161,8 @@ func StartHandler(opts Options) (*Handler, error) {
policy: opts.Policy.withDefaults(),
freeDisk: disk.FreeBytes,
dbLock: make(chan struct{}, 1),
outboundIP: opts.OutboundIP,
instanceTransport: opts.InstanceTransport,
}
if logger == nil {
@@ -156,12 +171,11 @@ func StartHandler(opts Options) (*Handler, error) {
logger = logger.With("module", "artifactcache")
h.logger = logger
if opts.OutboundIP != "" {
h.outboundIP = opts.OutboundIP
} else if ip := common.GetOutboundIP(); ip == nil {
return nil, errors.New("unable to determine outbound IP address")
} else {
h.outboundIP = ip.String()
if opts.OutboundIP == "" {
h.detectOutboundIP = common.GetOutboundIP
if h.RefreshExternalURL() == "" {
logger.Warn("unable to determine an outbound IP address for the cache server yet, set cache.host to configure one")
}
}
if opts.Upstream != "" {
@@ -205,10 +219,10 @@ func StartHandler(opts Options) (*Handler, error) {
}
router := http.NewServeMux()
router.HandleFunc("GET "+apiPath+"/cache", h.bearerAuth(h.find))
router.HandleFunc("POST "+apiPath+"/caches", h.bearerAuth(h.reserve))
router.HandleFunc("PATCH "+apiPath+"/caches/{id}", h.bearerAuth(h.upload))
router.HandleFunc("POST "+apiPath+"/caches/{id}", h.bearerAuth(h.commit))
router.HandleFunc("GET "+apiPath+"/cache", h.cacheModeAuth(false, h.find))
router.HandleFunc("POST "+apiPath+"/caches", h.cacheModeAuth(true, h.reserve))
router.HandleFunc("PATCH "+apiPath+"/caches/{id}", h.cacheModeAuth(true, h.upload))
router.HandleFunc("POST "+apiPath+"/caches/{id}", h.cacheModeAuth(true, h.commit))
router.HandleFunc("POST "+apiPath+"/clean", h.bearerAuth(h.clean))
// Artifact GET is signed via query-string HMAC because @actions/cache
// does not attach Authorization when downloading archiveLocation.
@@ -293,11 +307,36 @@ func (h *Handler) cacheForwarder(upstream string) (http.Handler, error) {
}), nil
}
// ExternalURL is "" while no address is known.
func (h *Handler) ExternalURL() string {
// TODO: make the external url configurable if necessary
if h == nil {
return ""
}
h.outboundIPMu.RLock()
defer h.outboundIPMu.RUnlock()
if h.outboundIP == "" {
return ""
}
return "http://" + net.JoinHostPort(h.outboundIP, strconv.Itoa(h.port))
}
// RefreshExternalURL re-detects an unconfigured address, which can change after startup.
func (h *Handler) RefreshExternalURL() string {
if h != nil && h.detectOutboundIP != nil {
if ip := h.detectOutboundIP(); ip != nil {
h.outboundIPMu.Lock()
if address := ip.String(); address != h.outboundIP {
h.outboundIP = address
if ip.To4() == nil {
h.logger.Warnf("detected IPv6 address %s for the cache server, jobs on IPv4-only networks cannot reach it, set cache.host if they fail to", address)
}
}
h.outboundIPMu.Unlock()
}
}
return h.ExternalURL()
}
func (h *Handler) baseURL(cred JobCredential) string {
if base := strings.TrimRight(cred.PublicURL, "/"); base != "" {
return base
@@ -653,7 +692,7 @@ func (h *Handler) find(w http.ResponseWriter, r *http.Request) {
}
defer db.Close()
cache, err := h.lookupCache(db, cred.Repo, keys, version)
cache, err := h.lookupCache(db, cred, keys, version)
if err != nil {
h.responseJSON(w, r, 500, err)
return
@@ -669,11 +708,10 @@ func (h *Handler) find(w http.ResponseWriter, r *http.Request) {
})
}
// lookupCache returns the entry to restore for these keys, or (nil, nil) when there is none:
// either nothing matched, or the match had lost its blob to a prune, in which case the dangling
// entry is dropped on the way out.
func (h *Handler) lookupCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) {
cache, err := findCache(db, repo, keys, version)
// lookupCache returns the entry to restore or (nil, nil), dropping matches whose blob was pruned.
func (h *Handler) lookupCache(db *cacheDB, cred JobCredential, keys []string, version string) (*Cache, error) {
for {
cache, err := findCache(db, cred, keys, version)
if err != nil || cache == nil {
return nil, err
}
@@ -681,15 +719,17 @@ func (h *Handler) lookupCache(db *cacheDB, repo string, keys []string, version s
if err != nil {
return nil, err
}
if !ok {
h.deleteCache(db, cache)
return nil, nil //nolint:nilnil // absence is not an error here
}
if ok {
// Handing out a download URL counts as access, or eviction could drop the entry between
// this call and the GET that follows it.
h.touch(db, cache)
return cache, nil
}
if !h.deleteCache(db, cache) {
return nil, nil //nolint:nilnil // absence is not an error here
}
}
}
// POST /_apis/artifactcache/caches
func (h *Handler) reserve(w http.ResponseWriter, r *http.Request) {
@@ -700,7 +740,7 @@ func (h *Handler) reserve(w http.ResponseWriter, r *http.Request) {
return
}
cache := &Cache{Repo: cred.Repo, Key: api.Key, Version: api.Version, Size: api.Size}
cache := &Cache{Repo: cred.Repo, Ref: cred.Ref, Key: api.Key, Version: api.Version, Size: api.Size}
if cache.Size == 0 {
cache.Size = -1
}
@@ -749,7 +789,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
return
}
if cache.Repo != cred.Repo {
if !cred.owns(cache) {
h.responseJSON(w, r, 403, fmt.Errorf("cache %d: forbidden", id))
return
}
@@ -764,6 +804,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
h.responseJSON(w, r, 400, err)
return
}
h.makeRoom(r.ContentLength)
if err := h.storage.Write(cache.ID, start, r.Body); err != nil {
h.responseJSON(w, r, 500, err)
return
@@ -808,7 +849,7 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
return
}
if cache.Repo != cred.Repo {
if !cred.owns(cache) {
h.responseJSON(w, r, 403, fmt.Errorf("cache %d: forbidden", id))
return
}
@@ -866,11 +907,8 @@ func (h *Handler) commitCache(cache *Cache) error {
if h.s3 != nil {
return nil // lifecycle rules on the bucket bound its size
}
// A commit is the only thing that grows the store, so the only thing that can push the
// volume under the floor.
h.evictRepo(db, cache.Repo)
h.evictTotal(db)
h.evictForFreeSpace(db)
return nil
}
@@ -919,6 +957,30 @@ func (h *Handler) bearerAuth(handler http.HandlerFunc) http.HandlerFunc {
}
}
// cacheModeAuth is bearerAuth refusing what the job's cache-mode forbids, in the shapes the toolkit reports as a warning.
func (h *Handler) cacheModeAuth(write bool, handler http.HandlerFunc) http.HandlerFunc {
return h.bearerAuth(func(w http.ResponseWriter, r *http.Request) {
mode := credFromContext(r.Context()).CacheMode
access, forbidden := "read", "write-only"
if write {
access, forbidden = "write", "read"
}
if mode != "none" && mode != forbidden {
handler(w, r)
return
}
message := fmt.Sprintf("cache %s denied: the job's cache-mode is %s", access, mode)
switch {
case !strings.HasPrefix(r.URL.Path, cacheServiceV2Path):
h.responseJSON(w, r, http.StatusForbidden, map[string]any{"message": message})
case write:
h.responseJSON(w, r, http.StatusOK, map[string]any{"ok": false, "message": message})
default:
h.twirpError(w, r, twirpDenied, errors.New(message))
}
})
}
func (h *Handler) unauthorized(w http.ResponseWriter, r *http.Request, err error) {
if strings.HasPrefix(r.URL.Path, cacheServiceV2Path) {
h.twirpError(w, r, twirpUnauthenticated, err)
@@ -1071,27 +1133,37 @@ func (h *Handler) signedArtifactURL(cred JobCredential, cacheID uint64, exp time
return h.signedURL(cred, apiPath+"/artifacts", "", cacheID, exp)
}
// if not found, return (nil, nil) instead of an error.
func findCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) {
for _, prefix := range keys {
// if a key in the list matches exactly, don't return partial matches
exact, err := findExactCache(db, repo, prefix, version, true)
if err != nil {
return nil, err
// findCache tries every key in one ref before the next ref, and returns (nil, nil) on a miss.
func findCache(db *cacheDB, cred JobCredential, keys []string, version string) (*Cache, error) {
refs := []string{cred.Ref}
for _, ref := range []string{cred.BaseRef, cred.DefaultRef} {
if ref != "" && !slices.Contains(refs, ref) {
refs = append(refs, ref)
}
if exact != nil {
return exact, nil
}
caches, err := db.find(func(cache *Cache) bool {
return cache.Repo == repo && strings.HasPrefix(cache.Key, prefix) && cache.Version == version && cache.Complete
return cache.Repo == cred.Repo && cache.Version == version && cache.Complete && slices.Contains(refs, cache.Ref) &&
slices.ContainsFunc(keys, func(prefix string) bool { return strings.HasPrefix(cache.Key, prefix) })
})
if err != nil {
return nil, fmt.Errorf("find cache: %w", err)
}
if cache := latest(caches, (*Cache).createdAtNano); cache != nil {
for _, ref := range refs {
for _, prefix := range keys {
var exact, partial []*Cache
for _, cache := range caches {
if cache.Ref == ref && cache.Key == prefix {
exact = append(exact, cache)
} else if cache.Ref == ref && strings.HasPrefix(cache.Key, prefix) {
partial = append(partial, cache)
}
}
// if a key in the list matches exactly, don't return partial matches
if cache := cmp.Or(latest(exact, (*Cache).createdAtNano), latest(partial, (*Cache).createdAtNano)); cache != nil {
return cache, nil
}
}
}
return nil, nil //nolint:nilnil // pre-existing issue from nektos/act
}
@@ -1111,9 +1183,9 @@ func latest(caches []*Cache, stamp func(*Cache) int64) *Cache {
// A completed entry is the one to restore, sorted by when it was written. An incomplete one is a
// reservation being uploaded to, sorted by when it was last written to, because the upload route
// touches UsedAt on every part.
func findExactCache(db *cacheDB, repo, key, version string, complete bool) (*Cache, error) {
func findExactCache(db *cacheDB, cred JobCredential, key, version string, complete bool) (*Cache, error) {
caches, err := db.find(func(cache *Cache) bool {
return cache.Key == key && cache.Repo == repo && cache.Version == version && cache.Complete == complete
return cred.owns(cache) && cache.Key == key && cache.Version == version && cache.Complete == complete
})
if err != nil {
return nil, fmt.Errorf("find cache: %w", err)
@@ -1154,9 +1226,6 @@ func (h *Handler) touchCache(id uint64, requireIncomplete bool) error {
defer db.Close()
cache, err := db.get(id)
if errors.Is(err, errCacheNotFound) {
return nil
}
if err != nil {
return err
}
@@ -1241,23 +1310,29 @@ func (h *Handler) gcCache() {
h.evictExpired(db)
h.evictSuperseded(db)
h.evictOversized(db)
h.evictForFreeSpace(db)
_ = db.Close()
h.makeRoom(0)
}
// evictForFreeSpace bounds the volume itself, so it also covers bytes the cache never
// accounted for.
func (h *Handler) evictForFreeSpace(db *cacheDB) {
// makeRoom evicts the least recently used entries to keep need bytes above the floor, as GitHub saves new over old.
func (h *Handler) makeRoom(need int64) {
free, err := h.freeDisk(h.dir)
if err != nil {
h.logger.Debugf("free disk check: %v", err) // unsupported platform, treat as unavailable rather than full
return
}
if free >= uint64(h.policy.MinFreeDisk) {
shortfall := h.policy.MinFreeDisk + max(need, 0) - int64(free)
if h.s3 != nil || shortfall <= 0 {
return
}
db, err := h.openDB()
if err != nil {
return
}
defer db.Close()
caches := h.completedByUse(db)
total, shortfall := totalSize(caches), h.policy.MinFreeDisk-int64(free)
total := totalSize(caches)
if total <= shortfall {
// Say so, or shedding everything and still being short reads as the backstop working.
h.logger.Warnf("cache volume is %d MiB short of the free space floor with only %d MiB of cache on it; something else is filling it", shortfall/miB, total/miB)
@@ -1280,13 +1355,12 @@ func (h *Handler) evictExpired(db *cacheDB) {
h.sweep(db, func(cache *Cache) bool { return cache.UsedAt < expired })
}
// evictSuperseded removes entries a newer one with the same key and version replaced. The
// grouping includes Repo so two repos sharing a (key, version) do not evict each other.
// evictSuperseded removes entries a newer one with the same repo, ref, key and version replaced.
func (h *Handler) evictSuperseded(db *cacheDB) {
type entry struct{ repo, key, version string }
type entry struct{ repo, ref, key, version string }
groups := make(map[entry][]*Cache)
for _, cache := range h.caches(db, func(cache *Cache) bool { return cache.Complete }) {
group := entry{cache.Repo, cache.Key, cache.Version}
group := entry{cache.Repo, cache.Ref, cache.Key, cache.Version}
groups[group] = append(groups[group], cache)
}
for _, caches := range groups {
+132 -60
View File
@@ -11,6 +11,7 @@ import (
"errors"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
@@ -18,9 +19,12 @@ import (
"strconv"
"strings"
"testing"
"testing/iotest"
"testing/synctest"
"time"
"gitea.com/gitea/runner/act/common/logging/logtest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.etcd.io/bbolt"
@@ -49,7 +53,23 @@ func signArtifactURL(h *Handler, id int64) string {
func TestHandler_ExternalURL(t *testing.T) {
t.Parallel()
assert.Equal(t, "http://[2001:db8::1]:8080", (&Handler{outboundIP: "2001:db8::1", port: 8080}).ExternalURL())
assert.Empty(t, (*Handler)(nil).RefreshExternalURL())
assert.Equal(t, "http://cache.example:8080", (&Handler{outboundIP: "cache.example", port: 8080}).RefreshExternalURL())
logger, logs := logtest.NewLogger()
var ip net.IP
handler := &Handler{port: 8080, logger: logger, detectOutboundIP: func() net.IP { return ip }}
assert.Empty(t, handler.RefreshExternalURL())
ip = net.ParseIP("2001:db8::1")
assert.Equal(t, "http://[2001:db8::1]:8080", handler.RefreshExternalURL())
assert.Equal(t, "http://[2001:db8::1]:8080", handler.RefreshExternalURL())
ip = net.ParseIP("192.0.2.1")
assert.Equal(t, "http://[2001:db8::1]:8080", handler.ExternalURL())
assert.Equal(t, "http://192.0.2.1:8080", handler.RefreshExternalURL())
ip = nil
assert.Equal(t, "http://192.0.2.1:8080", handler.RefreshExternalURL())
require.Len(t, logs.AllRecords(), 1)
assert.Contains(t, logs.LastRecord().Message, "IPv6")
}
func TestHandler_Upstream(t *testing.T) {
@@ -673,7 +693,7 @@ func TestHandler_evictPolicy(t *testing.T) {
}
}
func TestHandler_evictForFreeSpace(t *testing.T) {
func TestHandler_makeRoom(t *testing.T) {
t.Parallel()
free := func(n int64) func(string) (uint64, error) {
return func(string) (uint64, error) { return uint64(n), nil }
@@ -682,14 +702,16 @@ func TestHandler_evictForFreeSpace(t *testing.T) {
for _, tc := range []struct {
name string
freeDisk func(string) (uint64, error)
upload int64
kept []string
}{
{"ample free space evicts nothing", free(defaultMinFreeDisk), []string{"oldest", "middle", "newest"}},
{"a small shortfall sheds one entry", free(defaultMinFreeDisk - 4*miB), []string{"middle", "newest"}},
{"a shortfall the cache cannot cover sheds all of it", free(0), nil},
{"ample free space evicts nothing", free(defaultMinFreeDisk), 0, []string{"oldest", "middle", "newest"}},
{"room for an upload sheds the oldest entry", free(defaultMinFreeDisk), 4 * miB, []string{"middle", "newest"}},
{"a shortfall the cache cannot cover sheds all of it", free(0), 0, nil},
{
"an unreadable volume is treated as unavailable, not as full",
func(string) (uint64, error) { return 0, errors.New("unsupported") },
0,
[]string{"oldest", "middle", "newest"},
},
} {
@@ -702,11 +724,7 @@ func TestHandler_evictForFreeSpace(t *testing.T) {
}
handler := newTestHandler(t, Policy{}, entries...)
handler.freeDisk = tc.freeDisk
db, err := handler.openDB()
require.NoError(t, err)
handler.evictForFreeSpace(db)
require.NoError(t, db.Close())
handler.makeRoom(tc.upload)
assert.ElementsMatch(t, tc.kept, keptKeys(t, handler, entries))
})
@@ -759,11 +777,55 @@ func TestHandler_PrefixUsesNewestReservationWithinSecond(t *testing.T) {
db, err := handler.openDB()
require.NoError(t, err)
defer db.Close()
cache, err := findCache(db, testRepo, []string{"deps-"}, "v")
cache, err := findCache(db, JobCredential{Repo: testRepo}, []string{"deps-"}, "v")
require.NoError(t, err)
require.Equal(t, newer, cache)
}
func TestHandler_RestoreRefs(t *testing.T) {
t.Parallel()
const main, release, pull = "refs/heads/main", "refs/heads/release", "refs/pull/1/head"
now := time.Now().Unix()
entry := func(ref, key string, age int64) *Cache {
return &Cache{Repo: testRepo, Ref: ref, Key: key, Version: "v", Complete: true, CreatedAt: now - 3600 + age, UsedAt: now}
}
exactOnMain := entry(main, "deps-exact", -1)
onRelease := entry(release, "deps-release", 0)
onMain := entry(main, "deps-main", 1)
onPull := entry(pull, "deps-pull", 2)
unscoped := entry("", "deps-unscoped", 3)
handler := newTestHandler(t, Policy{}, exactOnMain, onRelease, onMain, onPull, unscoped, entry("refs/pull/2/head", "deps-lost", 4))
for _, cache := range []*Cache{exactOnMain, onRelease, onMain, onPull, unscoped} {
writeBlob(t, handler, cache.ID)
}
db, err := handler.openDB()
require.NoError(t, err)
defer db.Close()
for _, tc := range []struct {
name string
cred JobCredential
keys []string
want *Cache
}{
{"the default branch never restores what a pull request wrote", JobCredential{Ref: main, DefaultRef: main}, []string{"deps-"}, onMain},
{"a pull request restores its own entry first", JobCredential{Ref: pull, BaseRef: release, DefaultRef: main}, []string{"deps-"}, onPull},
{"a pull request falls back past a lost blob to its base branch before the default branch", JobCredential{Ref: "refs/pull/2/head", BaseRef: release, DefaultRef: main}, []string{"deps-"}, onRelease},
{"a branch does not restore a sibling branch", JobCredential{Ref: "refs/heads/other", DefaultRef: main}, []string{"deps-"}, onMain},
{"a prefix in the own ref wins over an exact key in the default branch", JobCredential{Ref: release, DefaultRef: main}, []string{"deps-exact", "deps-"}, onRelease},
{"a job with a ref never restores entries saved without one", JobCredential{Ref: "refs/heads/other"}, []string{"deps-"}, nil},
{"a job without a ref only restores entries saved without one", JobCredential{}, []string{"deps-"}, unscoped},
} {
t.Run(tc.name, func(t *testing.T) {
tc.cred.Repo = testRepo
cache, err := handler.lookupCache(db, tc.cred, tc.keys, "v")
require.NoError(t, err)
assert.Equal(t, tc.want, cache)
})
}
}
func TestHandler_evictOnCommit(t *testing.T) {
t.Parallel()
full := &Cache{Repo: testRepo, Key: "full", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-time.Hour).Unix()}
@@ -808,6 +870,9 @@ func TestHandler_touchCacheSkipsFreshCompletedEntries(t *testing.T) {
func newTestHandler(t *testing.T, policy Policy, entries ...*Cache) *Handler {
t.Helper()
handler := startTestHandler(t, Options{Policy: policy})
if len(entries) == 0 {
return handler
}
db, err := handler.openDB()
require.NoError(t, err)
@@ -844,6 +909,13 @@ func keptKeys(t *testing.T, handler *Handler, entries []*Cache) []string {
return kept
}
func TestStorageDropsFailedWrites(t *testing.T) {
storage, err := NewStorage(t.TempDir())
require.NoError(t, err)
require.Error(t, storage.Write(1, 0, io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(io.ErrUnexpectedEOF))))
assert.NoFileExists(t, storage.tempName(1, 0))
}
func writeBlob(t *testing.T, handler *Handler, id uint64) {
t.Helper()
require.NoError(t, handler.storage.Write(id, 0, strings.NewReader("a")))
@@ -890,6 +962,10 @@ func TestHandler_RejectsBearer(t *testing.T) {
func TestHandler_UnregisterRevokes(t *testing.T) {
t.Parallel()
handler, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache")})
require.NoError(t, err)
defer handler.Close()
for _, tc := range []struct {
name string
token string
@@ -900,9 +976,6 @@ func TestHandler_UnregisterRevokes(t *testing.T) {
{"reference counted", "shared", "v", 2},
} {
t.Run(tc.name, func(t *testing.T) {
handler, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache")})
require.NoError(t, err)
defer handler.Close()
revokers := make([]func(), tc.registrations)
for i := range revokers {
revokers[i] = handler.RegisterJob(tc.token, JobCredential{Repo: testRepo})
@@ -930,7 +1003,7 @@ func TestHandler_UnregisterRevokes(t *testing.T) {
}
}
func TestHandler_CrossRepoIsolation(t *testing.T) {
func TestHandler_CrossRepoAndCacheModeIsolation(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
@@ -938,6 +1011,9 @@ func TestHandler_CrossRepoIsolation(t *testing.T) {
defer handler.Close()
handler.RegisterJob("token-a", JobCredential{Repo: "owner/repoA"})
handler.RegisterJob("token-b", JobCredential{Repo: "owner/repoB"})
handler.RegisterJob("token-a-pull", JobCredential{Repo: "owner/repoA", Ref: "refs/pull/1/head"})
handler.RegisterJob("token-a-read", JobCredential{Repo: "owner/repoA", CacheMode: "read"})
handler.RegisterJob("token-a-write-only", JobCredential{Repo: "owner/repoA", CacheMode: "write-only"})
base := handler.ExternalURL() + apiPath
key := "shared-key"
@@ -967,18 +1043,44 @@ func TestHandler_CrossRepoIsolation(t *testing.T) {
resp.Body.Close()
assert.Equal(t, http.StatusNoContent, resp.StatusCode)
resp, err = clientA.Get(fmt.Sprintf("%s/cache?keys=%s&version=%s", base, key, version))
clientRead := &http.Client{Transport: &bearerTransport{token: "token-a-read"}}
for _, client := range []*http.Client{clientA, clientRead} {
resp, err = client.Get(fmt.Sprintf("%s/cache?keys=%s&version=%s", base, key, version))
require.NoError(t, err)
resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
}
requireDenied := func(prefix string, do func() (*http.Response, error)) {
resp, err := do()
require.NoError(t, err)
defer resp.Body.Close()
denied := map[string]string{}
require.NoError(t, json.UnmarshalRead(resp.Body, &denied))
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
assert.True(t, strings.HasPrefix(denied["message"], prefix), denied["message"])
}
requireDenied("cache write denied:", func() (*http.Response, error) {
return clientRead.Post(base+"/caches", "application/json", strings.NewReader(`{"key":"new-key","version":"`+version+`"}`))
})
requireDenied("cache read denied:", func() (*http.Response, error) {
return (&http.Client{Transport: &bearerTransport{token: "token-a-write-only"}}).Get(fmt.Sprintf("%s/cache?keys=%s&version=%s", base, key, version))
})
for _, client := range []*http.Client{clientB, {Transport: &bearerTransport{token: "token-a-pull"}}, clientRead} {
req, err = http.NewRequest(http.MethodPatch, fmt.Sprintf("%s/caches/%d", base, id), bytes.NewReader([]byte("poison")))
require.NoError(t, err)
req.Header.Set("Content-Range", "bytes 0-5/*")
resp, err = clientB.Do(req)
resp, err = client.Do(req)
require.NoError(t, err)
resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
resp, err = client.Post(fmt.Sprintf("%s/caches/%d", base, id), "", nil)
require.NoError(t, err)
resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
}
}
func TestHandler_ArtifactSignature(t *testing.T) {
@@ -991,9 +1093,8 @@ func TestHandler_ArtifactSignature(t *testing.T) {
base := handler.ExternalURL() + apiPath
good := signArtifactURL(handler, 1)
other, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache2")})
require.NoError(t, err)
defer other.Close()
secret := bytes.Clone(handler.secret)
secret[0] ^= 1
for _, tc := range []struct {
name string
@@ -1002,7 +1103,7 @@ func TestHandler_ArtifactSignature(t *testing.T) {
{"missing signature", fmt.Sprintf("%s/artifacts/%d", base, 1)},
{"tampered signature", good[:len(good)-4] + "dead"},
{"expired signature", handler.signedArtifactURL(JobCredential{}, 1, time.Now().Add(-time.Second))},
{"signature from a different server", base + strings.SplitN(signArtifactURL(other, 1), apiPath, 2)[1]},
{"signature from a different server", base + strings.SplitN(signArtifactURL(&Handler{secret: secret}, 1), apiPath, 2)[1]},
} {
t.Run(tc.name, func(t *testing.T) {
resp, err := testClient.Get(tc.url)
@@ -1083,49 +1184,20 @@ func TestHandler_ArtifactSignatureDownload(t *testing.T) {
assert.Equal(t, content, body)
}
func TestHandler_GC_PerRepoDedup(t *testing.T) {
func TestHandler_GC_DedupPerRepoAndRef(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
defer handler.Close()
handler.RegisterJob("tok-a", JobCredential{Repo: "owner/repoA"})
handler.RegisterJob("tok-b", JobCredential{Repo: "owner/repoB"})
key := "shared-dedup-key"
version := "c19da02a2bd7e77277f1ac29ab45c09b7d46a4ee758284e26bb3045ad11d9d20"
db, err := handler.openDB()
require.NoError(t, err)
now := time.Now().Unix()
stale := time.Now().Add(-inUseGrace - time.Minute).Unix()
a := &Cache{Repo: "owner/repoA", Key: key, Version: version, Complete: true, CreatedAt: stale, UsedAt: stale, Size: 1}
b := &Cache{Repo: "owner/repoB", Key: key, Version: version, Complete: true, CreatedAt: now, UsedAt: now, Size: 1}
require.NoError(t, handler.insertCache(db, a))
require.NoError(t, handler.insertCache(db, b))
require.NoError(t, handler.storage.Write(a.ID, 0, strings.NewReader("a")))
_, err = handler.storage.Commit(a.ID, 1)
require.NoError(t, err)
require.NoError(t, handler.storage.Write(b.ID, 0, strings.NewReader("b")))
_, err = handler.storage.Commit(b.ID, 1)
require.NoError(t, err)
require.NoError(t, db.Close())
entries := []*Cache{
{Repo: "owner/repoA", Key: "shared", Version: "v", Complete: true, CreatedAt: stale, UsedAt: stale, Size: 1},
{Repo: "owner/repoB", Key: "shared", Version: "v", Complete: true, CreatedAt: now, UsedAt: now, Size: 1},
{Repo: "owner/repoA", Ref: "refs/pull/1/head", Key: "shared", Version: "v", Complete: true, CreatedAt: now, UsedAt: now, Size: 1},
}
handler := newTestHandler(t, Policy{}, entries...)
handler.gcAt = time.Time{}
handler.gcCache()
db, err = handler.openDB()
require.NoError(t, err)
defer db.Close()
after, err := db.find(func(cache *Cache) bool { return cache.Key == key && cache.Version == version })
require.NoError(t, err)
repos := make(map[string]bool)
for _, c := range after {
repos[c.Repo] = true
}
assert.True(t, repos["owner/repoA"], "repoA's cache must survive dedup against repoB")
assert.True(t, repos["owner/repoB"], "repoB's cache must survive dedup against repoA")
assert.Len(t, keptKeys(t, handler, entries), len(entries))
}
func TestHandler_InternalAPI_Disabled(t *testing.T) {
@@ -1190,10 +1262,10 @@ func TestHandler_InternalAPI_AuthAndUsage(t *testing.T) {
}
assert.Equal(t, http.StatusUnauthorized, probe("via-internal-api"))
assert.Equal(t, http.StatusOK, post("/_internal/register", secret, `{"token":"via-internal-api","repo":"owner/repo","public_url":"http://runner.example","results":"https://gitea.example","insecure_tls":true}`))
assert.Equal(t, http.StatusOK, post("/_internal/register", secret, `{"token":"via-internal-api","repo":"owner/repo","ref":"refs/pull/1/head","base_ref":"refs/heads/release","default_ref":"refs/heads/main","public_url":"http://runner.example","results":"https://gitea.example","insecure_tls":true}`))
cred, ok := handler.lookupCredential("via-internal-api")
require.True(t, ok)
assert.Equal(t, JobCredential{Repo: testRepo, PublicURL: "http://runner.example"}, cred)
assert.Equal(t, JobCredential{Repo: testRepo, Ref: "refs/pull/1/head", BaseRef: "refs/heads/release", DefaultRef: "refs/heads/main", PublicURL: "http://runner.example"}, cred)
assert.NotEqual(t, http.StatusUnauthorized, probe("via-internal-api"))
assert.Equal(t, http.StatusOK, post("/_internal/revoke", secret, `{"token":"via-internal-api"}`))
assert.Equal(t, http.StatusUnauthorized, probe("via-internal-api"))
+13 -8
View File
@@ -40,12 +40,13 @@ const (
twirpInternal = "internal"
twirpUnavailable = "unavailable"
twirpUnauthenticated = "unauthenticated"
twirpDenied = "permission_denied"
)
func (h *Handler) registerV2Routes(router *http.ServeMux) {
router.HandleFunc("POST "+cacheServiceV2Path+"/CreateCacheEntry", h.bearerAuth(h.v2CreateCacheEntry))
router.HandleFunc("POST "+cacheServiceV2Path+"/FinalizeCacheEntryUpload", h.bearerAuth(h.v2FinalizeCacheEntryUpload))
router.HandleFunc("POST "+cacheServiceV2Path+"/GetCacheEntryDownloadURL", h.bearerAuth(h.v2GetCacheEntryDownloadURL))
router.HandleFunc("POST "+cacheServiceV2Path+"/CreateCacheEntry", h.cacheModeAuth(true, h.v2CreateCacheEntry))
router.HandleFunc("POST "+cacheServiceV2Path+"/FinalizeCacheEntryUpload", h.cacheModeAuth(true, h.v2FinalizeCacheEntryUpload))
router.HandleFunc("POST "+cacheServiceV2Path+"/GetCacheEntryDownloadURL", h.cacheModeAuth(false, h.v2GetCacheEntryDownloadURL))
router.HandleFunc("PUT "+blobPath+"/{id}", h.signedAuth(blobUploadPurpose, h.v2UploadBlob))
}
@@ -73,7 +74,7 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
// An exact (key, version) match means the entry is already cached; the client then skips
// the upload. A prefix match must not count here, or a shorter key would be reported as
// existing and silently never saved.
if existing, err := findExactCache(db, cred.Repo, req.Key, req.Version, true); err != nil {
if existing, err := findExactCache(db, cred, req.Key, req.Version, true); err != nil {
h.twirpError(w, r, twirpInternal, err)
return
} else if existing != nil {
@@ -84,7 +85,7 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
// A second live reservation is finalized by whichever job calls last, against the other's upload.
owner := hashedToken(bearerToken(r))
if pending, err := findExactCache(db, cred.Repo, req.Key, req.Version, false); err != nil {
if pending, err := findExactCache(db, cred, req.Key, req.Version, false); err != nil {
h.twirpError(w, r, twirpInternal, err)
return
} else if pending != nil && pending.UsedAt > time.Now().Add(-uploadStallTimeout).Unix() {
@@ -103,6 +104,7 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
now := time.Now()
cache := &Cache{
Repo: cred.Repo,
Ref: cred.Ref,
Key: req.Key,
Version: req.Version,
Owner: owner,
@@ -137,13 +139,13 @@ func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Requ
}
defer db.Close()
cache, err := findExactCache(db, cred.Repo, req.Key, req.Version, false)
cache, err := findExactCache(db, cred, req.Key, req.Version, false)
if err == nil && cache != nil && cache.Owner != hashedToken(bearerToken(r)) {
cache = nil // not the reservation this job made, so not this job's to commit
}
if err == nil && cache == nil {
// A retry whose first response was lost finds the entry already committed.
cache, err = findExactCache(db, cred.Repo, req.Key, req.Version, true)
cache, err = findExactCache(db, cred, req.Key, req.Version, true)
}
if err != nil {
h.twirpError(w, r, twirpInternal, err)
@@ -186,7 +188,7 @@ func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Requ
}
defer db.Close()
cache, err := h.lookupCache(db, cred.Repo, req.keys(), req.Version)
cache, err := h.lookupCache(db, cred, req.keys(), req.Version)
if err != nil {
h.twirpError(w, r, twirpInternal, err)
return
@@ -217,6 +219,7 @@ func (h *Handler) v2UploadBlob(w http.ResponseWriter, r *http.Request) {
h.responseJSON(w, r, http.StatusBadRequest, err)
return
}
h.makeRoom(r.ContentLength)
query := r.URL.Query()
switch strings.ToLower(query.Get("comp")) {
@@ -265,6 +268,8 @@ func (h *Handler) twirpError(w http.ResponseWriter, r *http.Request, code string
status = http.StatusServiceUnavailable
case twirpUnauthenticated:
status = http.StatusUnauthorized
case twirpDenied:
status = http.StatusForbidden
}
h.responseJSON(w, r, status, map[string]any{"code": code, "msg": err.Error()})
}
+45
View File
@@ -14,6 +14,7 @@ import (
"strconv"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -84,6 +85,8 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
unsigned := fmt.Sprintf("%s%s/1", handler.ExternalURL(), blobPath)
assert.Equal(t, http.StatusUnauthorized, putBlob(t, unsigned, content))
unreserved := handler.signedURL(JobCredential{Repo: testRepo}, blobPath, blobUploadPurpose, 1, time.Now().Add(time.Minute))
assert.Equal(t, http.StatusBadRequest, putBlob(t, unreserved, content))
finalized, uploadURL := saveV2(t, handler, "deps-v1", "abc123", content)
require.Equal(t, true, finalized["ok"])
@@ -224,6 +227,30 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.NotEmpty(t, got["archiveLocation"])
})
t.Run("a read-only job only restores and a write-only one only saves", func(t *testing.T) {
handler.RegisterJob("read-only", JobCredential{Repo: testRepo, CacheMode: "read"})
readOnlyClient := &http.Client{Transport: &bearerTransport{token: "read-only"}}
assert.Equal(t, true, v2Call(t, handler, readOnlyClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-abc", "version": "v1"})["ok"])
for _, method := range []string{"CreateCacheEntry", "FinalizeCacheEntryUpload"} {
got := v2Call(t, handler, readOnlyClient, method, map[string]any{"key": "deps-new", "version": "v1", "size_bytes": 1})
assert.Equal(t, false, got["ok"], method)
assert.True(t, strings.HasPrefix(got["message"].(string), "cache write denied:"), method)
}
handler.RegisterJob("write-only", JobCredential{Repo: testRepo, CacheMode: "write-only"})
writeOnlyClient := &http.Client{Transport: &bearerTransport{token: "write-only"}}
assert.Equal(t, true, v2Call(t, handler, writeOnlyClient, "CreateCacheEntry", map[string]any{"key": "deps-write-only", "version": "v1"})["ok"])
resp, err := writeOnlyClient.Post(handler.ExternalURL()+cacheServiceV2Path+"/GetCacheEntryDownloadURL", "application/json", strings.NewReader(`{"key":"deps-abc","version":"v1"}`))
require.NoError(t, err)
defer resp.Body.Close()
got := map[string]string{}
require.NoError(t, json.UnmarshalRead(resp.Body, &got))
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
assert.Equal(t, twirpDenied, got["code"])
assert.True(t, strings.HasPrefix(got["msg"], "cache read denied:"), got["msg"])
})
t.Run("another repository sees nothing", func(t *testing.T) {
handler.RegisterJob("other-runtime-token", JobCredential{Repo: "other/repo"})
otherClient := &http.Client{Transport: &bearerTransport{token: "other-runtime-token"}}
@@ -231,6 +258,24 @@ func TestCacheServiceV2Lookups(t *testing.T) {
got := v2Call(t, handler, otherClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-abc", "version": "v1"})
assert.Equal(t, false, got["ok"])
})
t.Run("each ref saves and restores its own entry for the same key", func(t *testing.T) {
clients := map[string]*http.Client{"x": testClient}
for _, ref := range []string{"y", "z"} {
handler.RegisterJob(ref, JobCredential{Repo: testRepo, Ref: "refs/pull/" + ref + "/head"})
clients[ref] = &http.Client{Transport: &bearerTransport{token: ref}}
created := v2Call(t, handler, clients[ref], "CreateCacheEntry", map[string]any{"key": "deps-abc", "version": "v1"})
require.Equal(t, true, created["ok"])
require.Equal(t, http.StatusCreated, putBlob(t, created["signed_upload_url"].(string), []byte(ref)))
}
for _, ref := range []string{"y", "z"} {
require.Equal(t, true, v2Call(t, handler, clients[ref], "FinalizeCacheEntryUpload", map[string]any{"key": "deps-abc", "version": "v1", "size_bytes": 1})["ok"])
}
for content, client := range clients {
got := v2Call(t, handler, client, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-abc", "version": "v1"})
assert.Equal(t, content, string(getURL(t, got["signed_download_url"].(string))))
}
})
}
func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
+1
View File
@@ -15,6 +15,7 @@ type Request struct {
type Cache struct {
ID uint64 `json:"id"`
Repo string `json:"repo"`
Ref string `json:"ref,omitzero"`
Key string `json:"key"`
Version string `json:"version"`
Size int64 `json:"cacheSize"`
+1 -1
View File
@@ -67,7 +67,7 @@ func (h *Handler) forwardOrNotFound(w http.ResponseWriter, r *http.Request) {
h.twirpError(w, r, twirpInternal, fmt.Errorf("cache server cannot reach the artifact service at %s: %w", target, err))
},
}
if cred.InsecureTLS {
if proxy.Transport = h.instanceTransport; proxy.Transport == nil && cred.InsecureTLS { // the instance transport carries runner.insecure and the client cert
proxy.Transport = insecureTransport
}
proxy.ServeHTTP(w, r)
+17
View File
@@ -133,3 +133,20 @@ func TestFrontResultsService(t *testing.T) {
})
}
}
func TestFrontResultsServiceForwardsThroughTheInstanceTransport(t *testing.T) {
t.Parallel()
gitea := httptest.NewTLSServer(http.HandlerFunc(func(writer http.ResponseWriter, _request *http.Request) { _, _ = io.WriteString(writer, "{}") }))
defer gitea.Close()
handler, err := StartHandler(Options{Dir: t.TempDir(), OutboundIP: "127.0.0.1", InstanceTransport: gitea.Client().Transport})
require.NoError(t, err)
defer handler.Close()
defer handler.RegisterJob("token", JobCredential{Results: gitea.URL})()
req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, handler.ExternalURL()+artifactServicePath+"CreateArtifact", nil)
require.NoError(t, err)
resp, err := (&http.Client{Transport: &bearerTransport{token: "token"}}).Do(req)
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
}
+24 -26
View File
@@ -9,6 +9,7 @@ import (
"crypto/sha256"
"encoding/hex"
"encoding/json/v2"
"errors"
"fmt"
"io"
"net/http"
@@ -51,9 +52,10 @@ func (s *Storage) writeFile(name string, reader io.Reader) error {
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(file, reader)
if err = errors.Join(err, file.Close()); err != nil {
_ = os.Remove(name)
}
return err
}
@@ -102,50 +104,46 @@ func (s *Storage) Commit(id uint64, size int64) (int64, error) {
return 0, err
}
written, err := assemble(name, tempNames)
if err != nil {
return 0, err
if err == nil && size >= 0 && written != size { // old clients like actions/cache@v2 send no size
err = fmt.Errorf("broken file: %v != %v", written, size)
}
// If size is less than 0, it means the size is unknown.
// We can't check the size of the file, just skip the check.
// It happens when the request comes from old versions of actions, like `actions/cache@v2`.
if size >= 0 && written != size {
if err != nil {
_ = os.Remove(name)
return 0, fmt.Errorf("broken file: %v != %v", written, size)
return 0, err
}
return written, nil
}
// assemble concatenates the uploaded parts into name. A single part, which is what the v2 API
// produces below the client's block threshold, is already the whole archive and is moved.
// assemble appends the parts into the first, deleting each once copied, so it never needs twice the space.
func assemble(name string, tempNames []string) (int64, error) {
if len(tempNames) == 1 {
info, err := os.Stat(tempNames[0])
if err != nil {
if len(tempNames) == 0 {
return 0, errors.New("nothing was uploaded")
}
if err := os.Rename(tempNames[0], name); err != nil {
return 0, err
}
return info.Size(), os.Rename(tempNames[0], name)
}
file, err := os.Create(name)
file, err := os.OpenFile(name, os.O_WRONLY|os.O_APPEND, 0)
if err != nil {
return 0, err
}
defer file.Close()
var written int64
for _, v := range tempNames {
f, err := os.Open(v)
for _, v := range tempNames[1:] {
part, err := os.Open(v)
if err != nil {
return 0, err
}
n, err := io.Copy(file, f)
_ = f.Close()
_, err = io.Copy(file, part)
_ = part.Close()
if err != nil {
return 0, err
}
written += n
_ = os.Remove(v)
}
return written, nil
info, err := file.Stat()
if err != nil {
return 0, err
}
return info.Size(), file.Close()
}
func (s *Storage) Serve(w http.ResponseWriter, r *http.Request, id uint64) {
+1
View File
@@ -218,6 +218,7 @@ func Serve(ctx context.Context, artifactPath, addr, port string) context.CancelF
logger.Debugf("Artifacts base path '%s'", artifactPath)
uploads(router, artifactPath)
downloads(router, artifactPath)
registerV4(router, artifactPath)
server := &http.Server{
Addr: net.JoinHostPort(addr, port),
+88 -1
View File
@@ -28,6 +28,7 @@ func TestArtifactFlow(t *testing.T) {
router := http.NewServeMux()
uploads(router, artifactPath)
downloads(router, artifactPath)
registerV4(router, artifactPath)
server := httptest.NewServer(router)
defer server.Close()
@@ -174,6 +175,92 @@ func TestArtifactFlow(t *testing.T) {
require.Equal(t, http.StatusOK, status)
require.Equal(t, content, string(data))
})
t.Run("v4", func(t *testing.T) {
const artifactRef = `"workflow_run_backend_id":"7","name":"50% art@v1"`
call := func(method, body string) (int, map[string]any) {
status, data := request(t, http.MethodPost, baseURL+artifactServicePath+"/"+method, strings.NewReader(body), nil)
var out map[string]any
require.NoError(t, json.Unmarshal(data, &out), string(data))
return status, out
}
put := func(rawURL, body string) int {
status, _ := request(t, http.MethodPut, rawURL, strings.NewReader(body), nil)
return status
}
download := func() (string, http.Header) {
status, signed := call("GetSignedArtifactURL", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, signed)
resp, err := client.Get(signed["signedUrl"].(string))
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
return string(body), resp.Header
}
status, created := call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, created)
upload := created["signedUploadUrl"].(string)
require.Equal(t, http.StatusCreated, put(upload+"&comp=block&blockid=Yg", "world"))
require.Equal(t, http.StatusCreated, put(upload+"&comp=block&blockid=YQ", "hello "))
for attempt := range 2 {
require.Equal(t, http.StatusCreated, put(upload+"&comp=blocklist", `<?xml version="1.0"?><BlockList><Latest>YQ</Latest><Latest>Yg</Latest></BlockList>`), "commit attempt %d", attempt)
}
digest := "sha256:b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"
status, _ = call("FinalizeArtifact", "{"+artifactRef+`,"size":"10"}`)
require.Equal(t, http.StatusBadRequest, status)
status, finalized := call("FinalizeArtifact", "{"+artifactRef+`,"size":"11","hash":"`+digest+`"}`)
require.Equal(t, http.StatusOK, status, finalized)
id := finalized["artifactId"].(string)
status, refinalized := call("FinalizeArtifact", "{"+artifactRef+`,"size":"10"}`)
require.Equal(t, http.StatusOK, status)
require.Equal(t, id, refinalized["artifactId"])
status, conflict := call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusConflict, status)
require.Equal(t, "already_exists", conflict["code"])
status, listed := call("ListArtifacts", `{"workflowRunBackendId":"7","idFilter":"`+id+`"}`)
require.Equal(t, http.StatusOK, status)
artifacts := listed["artifacts"].([]any)
require.Len(t, artifacts, 1)
artifact := artifacts[0].(map[string]any)
require.Regexp(t, `^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$`, artifact["createdAt"])
delete(artifact, "createdAt")
require.Equal(t, map[string]any{"workflowRunBackendId": "7", "name": "50% art@v1", "size": "11", "digest": digest, "databaseId": id}, artifact)
_, other := call("ListArtifacts", `{"workflow_run_backend_id":"8"}`)
require.Empty(t, other["artifacts"])
body, header := download()
require.Equal(t, "hello world", body)
require.Equal(t, "application/zip", header.Get("Content-Type"))
require.Equal(t, "attachment; filename*=UTF-8''50%25%20art%40v1.zip", header.Get("Content-Disposition"))
status, _ = call("DeleteArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status)
status, missing := call("GetSignedArtifactURL", "{"+artifactRef+"}")
require.Equal(t, http.StatusNotFound, status)
require.Equal(t, "not_found", missing["code"])
status, created = call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, created)
status, _ = request(t, http.MethodPut, created["signedUploadUrl"].(string), strings.NewReader("gz"), http.Header{"X-Ms-Blob-Content-Type": {"application/gzip"}})
require.Equal(t, http.StatusCreated, status)
status, _ = call("FinalizeArtifact", "{"+artifactRef+`,"size":"2"}`)
require.Equal(t, http.StatusOK, status)
body, header = download()
require.Equal(t, "gz", body)
require.Equal(t, "application/gzip", header.Get("Content-Type"))
require.Equal(t, "attachment; filename*=UTF-8''50%25%20art%40v1", header.Get("Content-Disposition"))
rerun := http.NewServeMux()
registerV4(rerun, artifactPath)
recorder := httptest.NewRecorder()
rerun.ServeHTTP(recorder, httptest.NewRequest(http.MethodPost, artifactServicePath+"/CreateArtifact", strings.NewReader("{"+artifactRef+"}")))
require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String())
})
}
func TestSafeResolve(t *testing.T) {
@@ -194,7 +281,7 @@ func TestSafeResolve(t *testing.T) {
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
require.Equal(t, tc.want, safeResolve(baseDir, tc.input))
require.Equal(t, filepath.FromSlash(tc.want), safeResolve(baseDir, tc.input))
})
}
}
+329
View File
@@ -0,0 +1,329 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import (
"cmp"
"crypto/sha256"
"encoding/hex"
"encoding/json/v2"
"encoding/xml"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"net/url"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"time"
)
type serverV4 struct { // the subset of Gitea's v4 artifact service that the toolkit uses, unauthenticated like the v3 routes beside it
dir string
mu sync.Mutex
lastID int64
created map[string]bool
}
const (
zipMimeType = "application/zip"
maxTwirpBytes = 1 << 20
)
func registerV4(router *http.ServeMux, baseDir string) {
server := &serverV4{dir: filepath.Join(baseDir, "v4"), lastID: time.Now().UnixMilli(), created: map[string]bool{}} // IDs stay unique across invocations sharing baseDir
router.HandleFunc("POST "+artifactServicePath+"/CreateArtifact", twirp(server.createArtifact))
router.HandleFunc("POST "+artifactServicePath+"/FinalizeArtifact", twirp(server.finalizeArtifact))
router.HandleFunc("POST "+artifactServicePath+"/ListArtifacts", twirp(server.listArtifacts))
router.HandleFunc("POST "+artifactServicePath+"/GetSignedArtifactURL", twirp(server.getSignedArtifactURL))
router.HandleFunc("POST "+artifactServicePath+"/DeleteArtifact", twirp(server.deleteArtifact))
router.HandleFunc("PUT "+artifactServicePath+"/UploadArtifact", server.uploadArtifact)
router.HandleFunc("GET "+artifactServicePath+"/DownloadArtifact", server.downloadArtifact)
}
type twirpError struct {
status int
code string
msg string
}
func (e *twirpError) Error() string { return e.msg }
func notFound(name string) error {
return &twirpError{http.StatusNotFound, "not_found", fmt.Sprintf("artifact %q not found", name)}
}
func badRequest(format string, args ...any) error {
return &twirpError{http.StatusBadRequest, "invalid_argument", fmt.Sprintf(format, args...)}
}
func twirp[Req, Resp any](handle func(*http.Request, Req) (Resp, error)) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var req Req
var resp Resp
err := json.UnmarshalRead(io.LimitReader(r.Body, maxTwirpBytes), &req, json.MatchCaseInsensitiveNames(true)) // also matches the toolkit's proto field names
if err != nil {
err = badRequest("malformed request: %v", err)
} else {
resp, err = handle(r, req)
}
w.Header().Set("Content-Type", "application/json")
if err != nil {
terr := &twirpError{http.StatusInternalServerError, "internal", err.Error()}
errors.As(err, &terr)
w.WriteHeader(terr.status)
_ = json.MarshalWrite(w, map[string]string{"code": terr.code, "msg": terr.msg})
return
}
_ = json.MarshalWrite(w, resp)
}
}
func fileName(name string) string { // safe on any filesystem for any client-chosen name, case-insensitive ones included
sum := sha256.Sum256([]byte(name))
return hex.EncodeToString(sum[:])
}
func (s *serverV4) runDir(runID string) string {
return filepath.Join(s.dir, fileName(runID))
}
func (s *serverV4) artifactDir(runID, name string) string {
return filepath.Join(s.runDir(runID), fileName(name))
}
func (s *serverV4) urlDir(r *http.Request) string {
return s.artifactDir(r.URL.Query().Get("run"), r.URL.Query().Get("name"))
}
func readMeta(dir string) (*monolithArtifact, error) {
data, err := os.ReadFile(filepath.Join(dir, "meta.json"))
if err != nil {
return nil, err
}
var meta monolithArtifact
return &meta, json.Unmarshal(data, &meta)
}
func writeMeta(dir string, meta *monolithArtifact) error {
data, err := json.Marshal(meta)
if err != nil {
return err
}
return os.WriteFile(filepath.Join(dir, "meta.json"), data, 0o644)
}
func blobURL(r *http.Request, endpoint, runID, name string) string {
return "http://" + r.Host + artifactServicePath + "/" + endpoint + "?" + url.Values{"run": {runID}, "name": {name}}.Encode()
}
func (s *serverV4) createArtifact(r *http.Request, req createArtifactRequest) (createArtifactResponse, error) { // an exec invocation re-runs its workflows, so it replaces earlier invocations' artifacts but conflicts with its own
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
s.mu.Lock()
defer s.mu.Unlock()
if s.created[dir] {
return createArtifactResponse{}, &twirpError{http.StatusConflict, "already_exists", "an artifact with this name already exists on the workflow run"}
}
if err := os.RemoveAll(dir); err != nil {
return createArtifactResponse{}, err
}
if err := os.MkdirAll(filepath.Join(dir, "blocks"), 0o755); err != nil {
return createArtifactResponse{}, err
}
s.lastID++
if err := writeMeta(dir, &monolithArtifact{
WorkflowRunBackendID: req.WorkflowRunBackendID,
DatabaseID: s.lastID,
Name: req.Name,
CreatedAt: time.Now().UTC().Truncate(time.Second),
}); err != nil {
return createArtifactResponse{}, err
}
s.created[dir] = true
return createArtifactResponse{Ok: true, SignedUploadURL: blobURL(r, "UploadArtifact", req.WorkflowRunBackendID, req.Name)}, nil
}
type blockList struct {
Blocks []string `xml:",any"` // Latest, Committed and Uncommitted all name a staged block
}
func (s *serverV4) uploadArtifact(w http.ResponseWriter, r *http.Request) {
dir := s.urlDir(r)
if meta, err := readMeta(dir); err != nil || meta.Digest != "" {
http.Error(w, "no artifact awaits an upload", http.StatusNotFound)
return
}
contentType := r.Header.Get("x-ms-blob-content-type")
var err error
switch comp := r.URL.Query().Get("comp"); comp {
case "block":
err = writeFile(filepath.Join(dir, "blocks", fileName(r.URL.Query().Get("blockid"))), r.Body)
case "blocklist":
err = commitBlob(dir, contentType, func(out io.Writer) error { return joinBlocks(out, dir, r.Body) })
case "": // Put Blob, which docker/build-push-action uses
err = commitBlob(dir, contentType, func(out io.Writer) error {
_, err := io.Copy(out, r.Body)
return err
})
default:
http.Error(w, "unsupported comp "+comp, http.StatusBadRequest)
return
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusCreated)
}
func writeFile(name string, body io.Reader) error {
file, err := os.Create(name)
if err != nil {
return err
}
_, err = io.Copy(file, body)
return errors.Join(err, file.Close())
}
func commitBlob(dir, contentType string, write func(io.Writer) error) error { // replaces the blob whole and keeps its content type, as Azure does
out, err := os.CreateTemp(dir, "blob")
if err != nil {
return err
}
if err = errors.Join(write(out), out.Close()); err == nil {
err = os.Rename(out.Name(), filepath.Join(dir, "blob"))
}
if err != nil {
return errors.Join(err, os.Remove(out.Name()))
}
return os.WriteFile(filepath.Join(dir, "content-type"), []byte(contentType), 0o644)
}
func joinBlocks(out io.Writer, dir string, body io.Reader) error { // keeps the blocks for finalizeArtifact, so that a retried commit can join them again
var list blockList
if err := xml.NewDecoder(io.LimitReader(body, maxTwirpBytes)).Decode(&list); err != nil {
return err
}
for _, id := range list.Blocks {
if err := appendBlock(out, filepath.Join(dir, "blocks", fileName(id))); err != nil {
return err
}
}
return nil
}
func appendBlock(out io.Writer, name string) error {
in, err := os.Open(name)
if err != nil {
return err
}
defer in.Close()
_, err = io.Copy(out, in)
return err
}
func (s *serverV4) finalizeArtifact(_ *http.Request, req finalizeArtifactRequest) (finalizeArtifactResponse, error) {
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
meta, err := readMeta(dir)
if err != nil {
return finalizeArtifactResponse{}, notFound(req.Name)
} else if meta.Digest != "" {
return finalizeArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, nil
}
blob, err := os.Open(filepath.Join(dir, "blob"))
if err != nil {
return finalizeArtifactResponse{}, badRequest("artifact %q has no upload", req.Name)
}
defer blob.Close()
hash := sha256.New()
size, err := io.Copy(hash, blob)
if err != nil {
return finalizeArtifactResponse{}, err
}
digest := "sha256:" + hex.EncodeToString(hash.Sum(nil))
if size != req.Size {
return finalizeArtifactResponse{}, badRequest("size %d does not match the %d bytes uploaded", req.Size, size)
}
if req.Hash != "" && req.Hash != digest {
return finalizeArtifactResponse{}, badRequest("hash %s does not match the uploaded %s", req.Hash, digest)
}
s.mu.Lock() // only after hashing, so finalizing one large artifact does not stall the others
defer s.mu.Unlock()
if current, err := readMeta(dir); err != nil || current.DatabaseID != meta.DatabaseID {
return finalizeArtifactResponse{}, notFound(req.Name)
}
meta.Size, meta.Digest = size, digest
if err := writeMeta(dir, meta); err != nil {
return finalizeArtifactResponse{}, err
}
return finalizeArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, os.RemoveAll(filepath.Join(dir, "blocks"))
}
func (s *serverV4) listArtifacts(_ *http.Request, req listArtifactsRequest) (listArtifactsResponse, error) {
run := s.runDir(req.WorkflowRunBackendID)
entries, err := os.ReadDir(run)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return listArtifactsResponse{}, err
}
var resp listArtifactsResponse
for _, entry := range entries {
meta, err := readMeta(filepath.Join(run, entry.Name()))
if err != nil || meta.Digest == "" ||
req.NameFilter != "" && req.NameFilter != meta.Name || req.IDFilter != 0 && req.IDFilter != meta.DatabaseID {
continue
}
resp.Artifacts = append(resp.Artifacts, *meta)
}
slices.SortFunc(resp.Artifacts, func(left, right monolithArtifact) int { return cmp.Compare(left.DatabaseID, right.DatabaseID) })
return resp, nil
}
func (s *serverV4) getSignedArtifactURL(r *http.Request, req getSignedArtifactURLRequest) (getSignedArtifactURLResponse, error) {
if meta, err := readMeta(s.artifactDir(req.WorkflowRunBackendID, req.Name)); err != nil || meta.Digest == "" {
return getSignedArtifactURLResponse{}, notFound(req.Name)
}
return getSignedArtifactURLResponse{SignedURL: blobURL(r, "DownloadArtifact", req.WorkflowRunBackendID, req.Name)}, nil
}
func (s *serverV4) downloadArtifact(w http.ResponseWriter, r *http.Request) {
dir := s.urlDir(r)
meta, err := readMeta(dir)
if err != nil || meta.Digest == "" {
http.Error(w, "artifact not found", http.StatusNotFound)
return
}
blob, err := os.Open(filepath.Join(dir, "blob"))
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer blob.Close()
data, _ := os.ReadFile(filepath.Join(dir, "content-type"))
contentType, filename := string(data), meta.Name
if !strings.Contains(contentType, "/") {
contentType = zipMimeType
}
if contentType == zipMimeType && !strings.HasSuffix(filename, ".zip") {
filename += ".zip"
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Disposition", "attachment; filename*=UTF-8''"+strings.ReplaceAll(url.QueryEscape(filename), "+", "%20")) // RFC 5987 attr-chars only, which the toolkit and mime.ParseMediaType both accept
http.ServeContent(w, r, "", meta.CreatedAt, blob)
}
func (s *serverV4) deleteArtifact(_ *http.Request, req deleteArtifactRequest) (deleteArtifactResponse, error) {
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
s.mu.Lock()
defer s.mu.Unlock()
meta, err := readMeta(dir)
if err != nil {
return deleteArtifactResponse{}, notFound(req.Name)
}
delete(s.created, dir)
return deleteArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, os.RemoveAll(dir)
}
+23
View File
@@ -0,0 +1,23 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import (
"crypto/rand"
"encoding/base64"
"encoding/json/v2"
"time"
)
func LocalRuntimeToken(run string, lifetime time.Duration) string { // an unsecured JWT with the claims the toolkit and buildx read, its run hashed for scp and its jti an unguessable bearer
now := time.Now()
claims, _ := json.Marshal(map[string]any{
"scp": "Actions.Results:" + fileName(run) + ":1",
"ac": `[{"Scope":"","Permission":3}]`,
"nbf": now.Unix(),
"exp": now.Add(lifetime).Unix(),
"jti": rand.Text(),
})
return base64.RawURLEncoding.EncodeToString([]byte(`{"alg":"none","typ":"JWT"}`)) + "." + base64.RawURLEncoding.EncodeToString(claims) + "."
}
+55
View File
@@ -0,0 +1,55 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import "time"
const artifactServicePath = "/twirp/github.actions.results.api.v1.ArtifactService"
type ( // the ArtifactService messages in the protobuf JSON mapping, which writes int64 as strings
createArtifactResponse struct {
Ok bool `json:"ok"`
SignedUploadURL string `json:"signedUploadUrl"`
}
finalizeArtifactRequest struct {
WorkflowRunBackendID string
Name string
Size int64 `json:",string"`
Hash string
}
finalizeArtifactResponse struct {
Ok bool `json:"ok"`
ArtifactID int64 `json:"artifactId,string"`
}
listArtifactsRequest struct {
WorkflowRunBackendID string
NameFilter string
IDFilter int64 `json:",string"`
}
listArtifactsResponse struct {
Artifacts []monolithArtifact `json:"artifacts"`
}
monolithArtifact struct {
WorkflowRunBackendID string `json:"workflowRunBackendId"` // the toolkit sends it back to GetSignedArtifactURL
DatabaseID int64 `json:"databaseId,string"`
Name string `json:"name"`
Size int64 `json:"size,string"`
CreatedAt time.Time `json:"createdAt"` // whole seconds, the toolkit rejects the trimmed fractions json writes
Digest string `json:"digest"` // empty until the upload is finalized
}
getSignedArtifactURLRequest struct {
WorkflowRunBackendID string
Name string
}
getSignedArtifactURLResponse struct {
SignedURL string `json:"signedUrl"`
}
createArtifactRequest = getSignedArtifactURLRequest
deleteArtifactRequest = getSignedArtifactURLRequest
deleteArtifactResponse = finalizeArtifactResponse
)
+24
View File
@@ -7,7 +7,9 @@ package common
import (
"context"
"fmt"
"math/rand/v2"
"runtime/debug"
"time"
"gitea.com/gitea/runner/act/common/logging"
)
@@ -158,6 +160,28 @@ func (e Executor) IfBool(conditional bool) Executor {
})
}
// RetryBackoff is the wait between Retry's attempts, random between 1 and 10 seconds like GitHub's runner.
var RetryBackoff = func() time.Duration { return time.Second + rand.N(9*time.Second) }
// Retry runs the executor up to attempts times until it succeeds.
func (e Executor) Retry(attempts int) Executor {
return func(ctx context.Context) error {
for attempt := 1; ; attempt++ {
err := e(ctx)
if err == nil || attempt == attempts || ctx.Err() != nil {
return err
}
backoff := RetryBackoff()
Logger(ctx).Warnf("%v, retrying in %s", err, backoff.Round(time.Second))
select {
case <-ctx.Done():
return err
case <-time.After(backoff):
}
}
}
}
// Finally adds an executor to run after other executor
func (e Executor) Finally(finally Executor) Executor {
return func(ctx context.Context) error {
+26
View File
@@ -11,6 +11,7 @@ import (
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -134,6 +135,31 @@ func TestExecutorConditionalsAndFinally(t *testing.T) {
assert.Equal(t, []string{"if-true", "if-not", "if-bool", "main", "finally"}, calls)
}
func TestExecutorRetryStopsOnSuccessOrAfterAllAttempts(t *testing.T) {
backoff := RetryBackoff
RetryBackoff = func() time.Duration { return 0 }
t.Cleanup(func() { RetryBackoff = backoff })
transient := errors.New("transient")
for _, tc := range []struct {
failures, calls int
err error
}{
{failures: 2, calls: 3},
{failures: 5, calls: 3, err: transient},
} {
var calls int
err := Executor(func(context.Context) error {
calls++
if calls <= tc.failures {
return transient
}
return nil
}).Retry(3)(t.Context())
require.Equal(t, tc.err, err)
require.Equal(t, tc.calls, calls)
}
}
func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
finalErr := errors.New("cleanup failed")
for _, mainErr := range []error{context.Canceled, nil} {
+163 -3
View File
@@ -10,6 +10,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"math/rand/v2"
"net"
@@ -29,6 +30,10 @@ import (
var actionFetches sync.Map
var symlink = (*os.Root).Symlink
var symlinkFallback = runtime.GOOS == "windows"
const treeSizes = ".sizes" // beside the tree rather than in it, which is copied into job containers as it is
type treeFile struct {
@@ -229,6 +234,8 @@ func extractTree(ctx context.Context, archive io.Reader, dir string, strip bool)
sha := ""
sizes := map[string]int64{}
links := map[string]string{}
var deferred []string
reader := tar.NewReader(archive)
for {
header, err := reader.Next()
@@ -257,9 +264,8 @@ func extractTree(ctx context.Context, archive io.Reader, dir string, strip bool)
case tar.TypeReg:
sizes[name], err = writeTreeFile(root, name, header.Mode, reader)
case tar.TypeSymlink:
if err = root.Symlink(header.Linkname, name); err != nil && runtime.GOOS == "windows" {
sizes[name], err = writeTreeFile(root, name, 0, strings.NewReader(header.Linkname))
}
links[filepath.Clean(name)] = header.Linkname
deferred = append(deferred, filepath.Clean(name))
}
if err != nil {
return "", err
@@ -268,12 +274,166 @@ func extractTree(ctx context.Context, archive io.Reader, dir string, strip bool)
if !isFullObjectID(sha) {
return "", errors.New("the archive names no commit")
}
linksByParent := map[string][]string{}
if symlinkFallback {
for name := range links {
linksByParent[filepath.Dir(name)] = append(linksByParent[filepath.Dir(name)], filepath.Base(name))
}
}
for requireTarget := symlinkFallback; len(deferred) > 0; {
unresolved := deferred[:0]
for _, name := range deferred {
if err := ctx.Err(); err != nil {
return "", err
}
if requireTarget {
if _, err := root.Stat(filepath.Dir(name) + string(filepath.Separator) + links[name]); err != nil {
unresolved = append(unresolved, name)
continue
}
}
if err := symlink(root, links[name], name); err == nil || !symlinkFallback {
if err != nil {
return "", err
}
continue
}
if err := copyTreeLink(ctx, root, name, links, linksByParent, sizes); err != nil {
return "", fmt.Errorf("materialize symlink %q: %w", filepath.ToSlash(name), err)
}
}
if len(unresolved) == len(deferred) {
requireTarget = false
}
deferred = unresolved
}
if err := root.Close(); err != nil {
return "", err
}
return sha, installTree(ctx, dir, sha, temp, sizes)
}
func resolveTreeLink(name string, links map[string]string) (string, bool) {
if !filepath.IsLocal(name) {
return "", false
}
parts := strings.Split(name, string(filepath.Separator))
var resolved []string
for count := 0; len(parts) > 0; {
part := parts[0]
parts = parts[1:]
switch part {
case "", ".":
continue
case "..":
if len(resolved) == 0 {
return "", false
}
resolved = resolved[:len(resolved)-1]
continue
}
target, ok := links[filepath.Join(append(resolved, part)...)]
if !ok {
resolved = append(resolved, part)
continue
}
count++
if target == "" || os.IsPathSeparator(target[0]) || filepath.VolumeName(target) != "" || count > 255 {
return "", false
}
parts = append(strings.Split(filepath.FromSlash(target), string(filepath.Separator)), parts...)
}
return filepath.Join(append([]string{"."}, resolved...)...), true
}
var (
errTreeLinkTarget = errors.New("unmaterializable symlink target")
errTreeLinkCycle = fmt.Errorf("cyclic %w", errTreeLinkTarget)
)
func copyTreeLink(ctx context.Context, root *os.Root, name string, links map[string]string, linksByParent map[string][]string, sizes map[string]int64) error {
writeLinkText := func(destination, target string) error { // as git writes links it cannot create
if err := root.RemoveAll(destination); err != nil {
return err
}
for written := range sizes {
if written == destination || strings.HasPrefix(written, destination+string(filepath.Separator)) {
delete(sizes, written)
}
}
size, err := writeTreeFile(root, destination, 0o644, strings.NewReader(target))
sizes[destination] = size
return err
}
active := map[string]bool{}
var copyTarget func(string, string) error
copyTarget = func(source, destination string) error {
if err := ctx.Err(); err != nil {
return err
}
source, ok := resolveTreeLink(source, links)
if !ok {
return errTreeLinkTarget
}
file, err := root.Open(source)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return errTreeLinkTarget
}
return err
}
defer file.Close()
info, err := file.Stat()
if err != nil {
return err
}
if info.Mode().IsRegular() {
sizes[destination], err = writeTreeFile(root, destination, int64(info.Mode().Perm()), file)
return err
}
if !info.IsDir() {
return errTreeLinkTarget
}
if active[source] {
return errTreeLinkCycle
}
active[source] = true
defer delete(active, source)
entries, err := file.ReadDir(-1)
if err != nil {
return err
}
children := map[string]bool{}
for _, entry := range entries {
children[entry.Name()] = true
}
for _, child := range linksByParent[source] {
children[child] = true
}
if err := root.MkdirAll(destination, info.Mode().Perm()); err != nil {
return err
}
for child := range children {
childSource, childDestination := filepath.Join(source, child), filepath.Join(destination, child)
err := copyTarget(childSource, childDestination)
if target, isLink := links[childSource]; isLink && errors.Is(err, errTreeLinkTarget) && !errors.Is(err, errTreeLinkCycle) {
err = writeLinkText(childDestination, target)
}
if err != nil {
return err
}
}
return nil
}
if err := copyTarget(name, name); !errors.Is(err, errTreeLinkTarget) {
return err
}
if err := ctx.Err(); err != nil {
return err
}
return writeLinkText(name, links[name])
}
var lockTree = AcquireCloneLock // replaced in tests to see installTree wait for a job holding the tree
// installTree moves temp into place as dir/sha, replacing a tree that failed treeIntact.
+165 -4
View File
@@ -8,6 +8,7 @@ import (
"bytes"
"context"
"encoding/base64"
"errors"
"fmt"
"net/http"
"net/http/cgi"
@@ -35,11 +36,12 @@ func TestFetchAction(t *testing.T) {
root := t.TempDir()
remote := filepath.Join(root, "org", "action")
require.NoError(t, gitCmd("init", "--initial-branch=main", remote))
require.NoError(t, os.CopyFS(remote, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, os.WriteFile(filepath.Join(remote, "action.yml"), []byte("name: action\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(remote, "run.sh"), []byte("#!/bin/sh\n"), 0o755))
require.NoError(t, os.Symlink("action.yml", filepath.Join(remote, "link")))
require.NoError(t, gitCmd("-C", remote, "add", "."))
require.NoError(t, gitCmd("-C", remote, "update-index", "--chmod=+x", "run.sh"))
require.NoError(t, gitCmd("-C", remote, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", remote, "tag", "v1"))
expectedSha := gitRevParse(t, remote, "main")
@@ -51,6 +53,7 @@ func TestFetchAction(t *testing.T) {
var archiveRequests []http.Header
var mirror *httptest.Server
archiveCache := map[string][]byte{}
revisions := map[string]string{}
requests, archives, failures := 0, "", 0
handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
mu.Lock()
@@ -77,7 +80,14 @@ func TestFetchAction(t *testing.T) {
http.Redirect(writer, request, mirror.URL+request.URL.Path, http.StatusFound)
return
}
sha, _ := gitOutput(request.Context(), remote, "rev-parse", strings.TrimSuffix(ref, ".tar.gz"))
sha, ok := revisions[ref]
if !ok {
var err error
sha, err = gitOutput(request.Context(), remote, "rev-parse", strings.TrimSuffix(ref, ".tar.gz"))
if err == nil {
revisions[ref] = sha
}
}
etag := fmt.Sprintf(`W/"%s-tar.gz-%t"`, sha, archives != "unprefixed")
if request.Header.Get("If-None-Match") == etag {
writer.WriteHeader(http.StatusNotModified)
@@ -192,7 +202,9 @@ func TestFetchAction(t *testing.T) {
assert.NoDirExists(t, filepath.Join(tree, ".git"))
info, err := os.Stat(filepath.Join(tree, "run.sh"))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o755), info.Mode().Perm())
if runtime.GOOS != "windows" {
assert.EqualValues(t, 0o755, info.Mode().Perm())
}
link, err := os.Readlink(filepath.Join(tree, "link"))
require.NoError(t, err)
assert.Equal(t, "action.yml", link)
@@ -204,6 +216,9 @@ func TestFetchAction(t *testing.T) {
require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved"))
require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1"))
expectedSha = gitRevParse(t, remote, "v1")
mu.Lock()
clear(revisions)
mu.Unlock()
moved, movedSha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, expectedSha, movedSha)
@@ -249,7 +264,7 @@ func TestFetchAction(t *testing.T) {
})
t.Run("downloads again over a tree whose files changed size or that has no record, not under a job copying it", func(t *testing.T) {
sha := gitRevParse(t, remote, "main")
sha := expectedSha
for _, mode := range []string{"prefixed", "none"} {
serve(mode, 0)
input := newInput(sha)
@@ -391,6 +406,79 @@ func TestExtractTreeConfinesEntries(t *testing.T) {
}
}
func TestExtractTreeCopiesDeferredLinks(t *testing.T) {
originalSymlink, originalFallback := symlink, symlinkFallback
symlink = func(_root *os.Root, _target, _name string) error { return errors.New("symlink privilege unavailable") }
symlinkFallback = true
t.Cleanup(func() { symlink, symlinkFallback = originalSymlink, originalFallback })
for _, test := range []struct {
name string
links [][2]string
files []string
text bool
}{
{name: "file chain", links: [][2]string{{"index.js", "entry.js"}, {"entry.js", "dist/index.js"}}, files: []string{"index.js", "entry.js"}},
{name: "directory chain", links: [][2]string{{"lib", "bundle"}, {"bundle", "dist"}, {"dist/entry.js", "index.js"}}, files: []string{"lib/index.js", "lib/entry.js", "lib/nested/index.js", "bundle/index.js", "bundle/entry.js", "bundle/nested/index.js"}},
{name: "directory in target path", links: [][2]string{{"index.js", "bundle/nested/index.js"}, {"bundle", "dist"}}, files: []string{"index.js", "bundle/nested/index.js"}},
{name: "parent after directory link", links: [][2]string{{"index.js", "bundle/../index.js"}, {"bundle", "dist/nested"}}, files: []string{"index.js", "bundle/index.js"}},
{name: "outside", links: [][2]string{{"index.js", "../outside.js"}}, text: true},
{name: "absolute", links: [][2]string{{"index.js", t.TempDir()}}, text: true},
{name: "dangling", links: [][2]string{{"index.js", "missing.js"}}, text: true},
{name: "dangling child of directory link", links: [][2]string{{"lib", "dist"}, {"dist/optional", "missing.js"}}, files: []string{"lib/index.js", "lib/nested/index.js"}},
{name: "cycle", links: [][2]string{{"index.js", "entry.js"}, {"entry.js", "index.js"}}, text: true},
{name: "directory cycle", links: [][2]string{{"dist/loop", "."}}, text: true},
} {
t.Run(test.name, func(t *testing.T) {
sha := strings.Repeat("a", 40)
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": sha}}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "dist", Typeflag: tar.TypeDir, Mode: 0o755}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "dist/nested", Typeflag: tar.TypeDir, Mode: 0o755}))
for _, link := range test.links {
require.NoError(t, writer.WriteHeader(&tar.Header{Name: link[0], Typeflag: tar.TypeSymlink, Linkname: link[1]}))
}
content := "console.log('action');\n"
for _, name := range []string{"dist/index.js", "dist/nested/index.js"} {
require.NoError(t, writer.WriteHeader(&tar.Header{Name: name, Typeflag: tar.TypeReg, Mode: 0o755, Size: int64(len(content))}))
_, err := writer.Write([]byte(content))
require.NoError(t, err)
}
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err := extractTree(t.Context(), &archive, dir, false)
require.NoError(t, err)
assert.True(t, treeIntact(dir, sha))
if test.text {
for _, link := range test.links {
data, err := os.ReadFile(filepath.Join(dir, sha, link[0]))
require.NoError(t, err)
assert.Equal(t, link[1], string(data))
info, err := os.Lstat(filepath.Join(dir, sha, link[0]))
require.NoError(t, err)
assert.True(t, info.Mode().IsRegular())
}
}
for _, name := range test.files {
data, err := os.ReadFile(filepath.Join(dir, sha, name))
require.NoError(t, err)
assert.Equal(t, content, string(data))
info, err := os.Lstat(filepath.Join(dir, sha, name))
require.NoError(t, err)
assert.True(t, info.Mode().IsRegular())
if runtime.GOOS != "windows" {
assert.Equal(t, os.FileMode(0o755), info.Mode().Perm())
}
require.NoError(t, os.WriteFile(filepath.Join(dir, sha, name), nil, 0o644))
assert.False(t, treeIntact(dir, sha), name)
require.NoError(t, os.WriteFile(filepath.Join(dir, sha, name), []byte(content), 0o755))
}
})
}
}
func TestTreeIntactFindsNamesThatAreNotUTF8(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("macOS and Windows store no name that is not UTF-8")
@@ -410,3 +498,76 @@ func TestTreeIntactFindsNamesThatAreNotUTF8(t *testing.T) {
require.FileExists(t, filepath.Join(dir, sha, "caf\xe9.js"))
assert.True(t, treeIntact(dir, sha))
}
func TestExtractTreeCreatesLinksInPlatformOrder(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("requires symlink privileges")
}
original, originalFallback := symlink, symlinkFallback
var created []string
symlink = func(root *os.Root, target, name string) error {
created = append(created, name)
return root.Symlink(target, name)
}
t.Cleanup(func() { symlink, symlinkFallback = original, originalFallback })
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": strings.Repeat("a", 40)}}))
for _, entry := range []*tar.Header{
{Name: "a", Typeflag: tar.TypeSymlink, Linkname: "b"},
{Name: "b", Typeflag: tar.TypeSymlink, Linkname: "dist"},
{Name: "dangling", Typeflag: tar.TypeSymlink, Linkname: "missing"},
{Name: "dist", Typeflag: tar.TypeDir, Mode: 0o755},
} {
require.NoError(t, writer.WriteHeader(entry))
}
require.NoError(t, writer.Close())
for _, test := range []struct {
name string
fallback bool
want []string
}{
{name: "native", want: []string{"a", "b", "dangling"}},
{name: "fallback", fallback: true, want: []string{"b", "a", "dangling"}},
} {
t.Run(test.name, func(t *testing.T) {
symlinkFallback = test.fallback
created = nil
_, err := extractTree(t.Context(), bytes.NewReader(archive.Bytes()), t.TempDir(), false)
require.NoError(t, err)
assert.Equal(t, test.want, created)
})
}
}
func TestCopyTreeLinkCancellationBetweenEntries(t *testing.T) {
root, err := os.OpenRoot(t.TempDir())
require.NoError(t, err)
defer root.Close()
require.NoError(t, root.Mkdir("source", 0o755))
for _, name := range []string{"first", "second"} {
_, err := writeTreeFile(root, filepath.Join("source", name), 0o644, strings.NewReader("content"))
require.NoError(t, err)
}
sizes := map[string]int64{}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
observed := &treeCopyContext{Context: ctx, check: func() {
if len(sizes) == 1 {
cancel()
}
}}
require.ErrorIs(t, copyTreeLink(observed, root, "copy", map[string]string{"copy": "source"}, nil, sizes), context.Canceled)
assert.Len(t, sizes, 1)
}
type treeCopyContext struct {
context.Context
check func()
}
func (ctx *treeCopyContext) Err() error {
ctx.check()
return ctx.Context.Err()
}
+36 -7
View File
@@ -5,6 +5,7 @@
package git
import (
"bytes"
"context"
"crypto/sha256"
"encoding/base64"
@@ -27,6 +28,7 @@ import (
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/internal/pkg/lock"
"gitea.com/gitea/runner/internal/pkg/process"
)
var (
@@ -183,6 +185,7 @@ type NewGitCloneExecutorInput struct {
// For Gitea
InsecureSkipTLS bool
ClientCertFile, ClientKeyFile string // PEM, absolute as git runs in other directories
CAFile string // PEM, absolute, replaces git's CA bundle
Transport http.RoundTripper // for FetchAction's archive downloads, nil for http.DefaultTransport
}
@@ -387,6 +390,21 @@ func RemoteConfig(token string, insecureSkipTLS bool) []string {
return config
}
// SSLConfig uses OpenSSL on Windows, as Schannel takes no PEM key and ignores sslCAInfo.
func SSLConfig(prefix, certFile, keyFile, caFile, executionOS string) []string {
var config []string
if certFile != "" {
config = append(config, prefix+"sslCert="+certFile, prefix+"sslKey="+keyFile)
}
if caFile != "" {
config = append(config, prefix+"sslCAInfo="+caFile)
}
if config != nil && strings.EqualFold(executionOS, "windows") {
config = append(config, prefix+"sslBackend=openssl")
}
return config
}
// ConfigEnv applies `-c` style entries through the environment, keeping secrets out of argv.
func ConfigEnv(config ...string) map[string]string {
env := map[string]string{"GIT_CONFIG_COUNT": strconv.Itoa(len(config))}
@@ -444,9 +462,7 @@ func runGitWithStdin(ctx context.Context, dir string, input *NewGitCloneExecutor
config := []string{"core.autocrlf=false", "core.eol=lf"} // keep action files LF
if input != nil {
config = append(config, RemoteConfig(input.Token, input.InsecureSkipTLS)...)
if input.ClientCertFile != "" {
config = append(config, "http.sslCert="+input.ClientCertFile, "http.sslKey="+input.ClientKeyFile)
}
config = append(config, SSLConfig("http.", input.ClientCertFile, input.ClientKeyFile, input.CAFile, runtime.GOOS)...)
}
if dir != "" {
args = append([]string{"-C", dir}, args...)
@@ -461,15 +477,28 @@ func runGitWithStdin(ctx context.Context, dir string, input *NewGitCloneExecutor
cmd.Env = append(cmd.Env, key+"="+value)
}
cmd.Env = append(cmd.Env, "GIT_TERMINAL_PROMPT=0")
if runtime.GOOS != "windows" {
var treeKill *process.TreeKill
if runtime.GOOS == "windows" {
treeKill = process.NewTreeKill(cmd) // a cancelled git leaves its remote helper holding the repository otherwise
} else {
cmd.Cancel = func() error { return cmd.Process.Signal(os.Interrupt) } // lets git remove its lock files
}
cmd.WaitDelay = time.Second
var stdout bytes.Buffer
var stderr strings.Builder
cmd.Stdout = &stdout
cmd.Stderr = &stderr
output, err := cmd.Output()
err := cmd.Start()
if err == nil {
if treeKill != nil {
if killer, captureErr := treeKill.Capture(cmd.Process); captureErr == nil {
defer killer.Close()
}
}
err = cmd.Wait()
}
if err != nil {
return output, fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String()))
return stdout.Bytes(), fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String()))
}
return output, nil
return stdout.Bytes(), nil
}
+46 -35
View File
@@ -54,6 +54,22 @@ func TestMain(m *testing.M) {
os.Exit(code)
}
func TestSSLConfig(t *testing.T) {
for _, testcase := range []struct {
prefix, certFile, caFile, executionOS string
expected []string
}{
{"http.https://example.com/.", "cert.pem", "ca.pem", "Windows", []string{"http.https://example.com/.sslCert=cert.pem", "http.https://example.com/.sslKey=key.pem", "http.https://example.com/.sslCAInfo=ca.pem", "http.https://example.com/.sslBackend=openssl"}},
{"http.", "", "ca.pem", "windows", []string{"http.sslCAInfo=ca.pem", "http.sslBackend=openssl"}},
{"http.", "cert.pem", "ca.pem", "Linux", []string{"http.sslCert=cert.pem", "http.sslKey=key.pem", "http.sslCAInfo=ca.pem"}},
{"http.", "", "", "Windows", nil},
} {
t.Run(testcase.prefix+testcase.certFile+testcase.caFile+testcase.executionOS, func(t *testing.T) {
assert.Equal(t, testcase.expected, SSLConfig(testcase.prefix, testcase.certFile, "key.pem", testcase.caFile, testcase.executionOS))
})
}
}
func TestFindGitSlug(t *testing.T) {
for _, testcase := range []struct {
remoteURL string
@@ -96,8 +112,7 @@ func TestFindGitMetadataOfSHA256RepositoryUsesOrigin(t *testing.T) {
t.Parallel()
basedir := t.TempDir()
const remoteURL = "https://github.com/owner/repo.git"
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, os.RemoveAll(filepath.Join(basedir, ".git", "hooks")))
require.NoError(t, gitCmd("init", "--template=", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init"))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL))
require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git"))
@@ -185,22 +200,20 @@ func TestGitFindRef(t *testing.T) {
func TestGitCloneExecutor(t *testing.T) {
remoteDir, workDir := newGitRemote(t)
require.NoError(t, os.WriteFile(filepath.Join(workDir, "action.sh"), []byte("#!/bin/sh\necho hi\n"), 0o755))
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh"))
require.NoError(t, os.WriteFile(filepath.Join(workDir, ".gitattributes"), []byte("*.sh text\n"), 0o644))
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh", ".gitattributes"))
require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "tag", "v2"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v2"))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "act-fails"))
fullSha := gitRevParse(t, workDir, "main")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:refs/heads/"+fullSha[:4]))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", "v2", "act-fails", "main:refs/heads/"+fullSha[:4]))
hostConfig := filepath.Join(t.TempDir(), "gitconfig")
require.NoError(t, os.WriteFile(hostConfig, []byte("[core]\n\tautocrlf = true\n[url \""+remoteDir+"\"]\n\tinsteadOf = https://example.invalid/action\n"), 0o644))
require.NoError(t, os.WriteFile(hostConfig, []byte("[core]\n\tautocrlf = true\n\teol = crlf\n[url "+strconv.Quote(remoteDir)+"]\n\tinsteadOf = https://example.invalid/action\n"), 0o644))
t.Setenv("GIT_CONFIG_GLOBAL", hostConfig)
t.Setenv("GIT_DIR", t.TempDir())
@@ -265,10 +278,10 @@ func TestGitCloneExecutor(t *testing.T) {
func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
t.Parallel()
createRemote := func(message string) string {
remoteDir, workDir := newGitRemote(t, message)
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
return remoteDir
dir := t.TempDir()
require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", message))
return dir
}
oldRemoteDir := createRemote("old-action")
@@ -388,31 +401,30 @@ func TestGitCloneExecutorShallow(t *testing.T) {
remoteDir, workDir := newGitRemote(t, "c1", "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v1"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", "v1"))
shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") }
for name, ref := range map[string]string{"branch is cloned shallowly": "main", "tag is cloned shallowly": "v1"} {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: ref, Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present")
assert.Equal(t, "c3", gitHeadSubject(t, dir))
assert.Equal(t, []string{"c3"}, gitSubjects(t, dir), "only the tip commit should be present")
})
}
unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string {
remote := filepath.Join(t.TempDir(), "remote.git")
require.NoError(t, gitCmd("clone", "--bare", remoteDir, remote))
require.NoError(t, gitCmd("-C", remote, "config", "uploadpack.allowAnySHA1InWant", strconv.FormatBool(allowUnadvertised)))
require.NoError(t, gitCmd("clone", "--bare", "--template=", "--config", "uploadpack.allowAnySHA1InWant="+strconv.FormatBool(allowUnadvertised), remoteDir, remote))
return remote
}
t.Run("commit hash resolves when the remote accepts a direct shallow fetch", func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: unadvertisedRemote(t, false), Ref: unadvertisedSHA, Dir: dir, Depth: 1,
@@ -421,6 +433,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
})
t.Run("commit hash is fetched shallowly, moved to another hash, then reused without reaching the remote", func(t *testing.T) {
t.Parallel()
remote := unadvertisedRemote(t, true)
dir := t.TempDir()
cloneAt := func(ref string) error {
@@ -431,7 +444,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
require.NoError(t, cloneAt(unadvertisedSHA))
assert.FileExists(t, shallowMarker(dir))
assert.Equal(t, 1, gitRevCount(t, dir))
assert.Len(t, gitSubjects(t, dir), 1)
assert.Equal(t, unadvertisedSHA, gitRevParse(t, dir, "HEAD"))
olderSHA := gitRevParse(t, workDir, "HEAD~2")
@@ -444,6 +457,8 @@ func TestGitCloneExecutorShallow(t *testing.T) {
})
t.Run("moving hexadecimal-named branch updates while staying shallow", func(t *testing.T) {
t.Parallel()
remoteDir, workDir := newGitRemote(t, "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef"))
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
@@ -456,19 +471,17 @@ func TestGitCloneExecutorShallow(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef"))
require.NoError(t, clone(t.Context()))
assert.Equal(t, "c4", gitHeadSubject(t, dir), "reused shallow clone should update to the new tip")
assert.Equal(t, []string{"c4"}, gitSubjects(t, dir), "reused shallow clone should update to the new tip")
assert.FileExists(t, shallowMarker(dir), "repo should remain shallow after update")
assert.Equal(t, 1, gitRevCount(t, dir))
})
t.Run("sha256 full hash is cloned shallowly, reused without the remote, and its 40-character prefix is short", func(t *testing.T) {
t.Parallel()
remote := filepath.Join(t.TempDir(), "remote.git")
require.NoError(t, gitCmd("init", "--bare", "--object-format=sha256", "--initial-branch=main", remote))
work := t.TempDir()
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", work))
require.NoError(t, gitCmd("-C", work, "remote", "add", "origin", remote))
require.NoError(t, gitCmd("init", "--template=", "--object-format=sha256", "--initial-branch=main", work))
require.NoError(t, gitCmd("-C", work, "commit", "--allow-empty", "-m", "pinned"))
require.NoError(t, gitCmd("-C", work, "push", "origin", "main"))
require.NoError(t, gitCmd("clone", "--bare", "--template=", work, remote))
sha := gitRevParse(t, work, "HEAD")
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{URL: remote, Ref: sha, Dir: dir, Depth: 1})
@@ -487,11 +500,11 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
pinned := gitRevParse(t, workDir, "HEAD")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy"))
require.NoError(t, gitCmd("-C", workDir, "tag", pinned))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", pinned))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", pinned))
for name, depth := range map[string]int{"shallow": 1, "full clone": 0} {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: pinned, Dir: dir, Depth: depth,
@@ -523,13 +536,11 @@ func gitRevParse(t *testing.T, dir, rev string) string {
return strings.TrimSpace(string(out))
}
func gitRevCount(t *testing.T, dir string) int {
func gitSubjects(t *testing.T, dir string) []string {
t.Helper()
out, err := exec.Command("git", "-C", dir, "rev-list", "--count", "HEAD").Output()
out, err := exec.Command("git", "-C", dir, "log", "--format=%s").Output()
require.NoError(t, err)
n, err := strconv.Atoi(strings.TrimSpace(string(out)))
require.NoError(t, err)
return n
return strings.Split(strings.TrimSpace(string(out)), "\n")
}
func gitHeadSubject(t *testing.T, dir string) string {
@@ -663,7 +674,7 @@ func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
})
dir := filepath.Join(t.TempDir(), "cached-action")
require.NoError(t, gitCmd("init", dir))
require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "remote", "add", "origin", server.URL))
ctx, cancel := context.WithCancel(t.Context())
@@ -676,8 +687,8 @@ func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
select {
case <-reached:
case <-time.After(10 * time.Second):
t.Fatal("the executor never reached the remote")
case err := <-done:
t.Fatalf("the executor finished before reaching the remote: %v", err)
}
assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), authorization)
cancel()
+20
View File
@@ -0,0 +1,20 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package common
import "strings"
// LookupFold finds name in values, ignoring case where the environment does, as GitHub's inputs always do.
func LookupFold[V any](values map[string]V, name string, caseInsensitive bool) (V, bool) {
if value, ok := values[name]; ok || !caseInsensitive {
return value, ok
}
for key, value := range values {
if strings.EqualFold(key, name) {
return value, true
}
}
var zero V
return zero, false
}
+8 -4
View File
@@ -6,6 +6,7 @@ package common
import (
"net"
"slices"
"sort"
"strings"
)
@@ -55,19 +56,22 @@ func GetOutboundIP() net.IP {
}
func preferredIP(ips []interfaceIP) net.IP {
ips = slices.DeleteFunc(ips, func(ip interfaceIP) bool {
return len(ip.Name) == 15 && strings.HasPrefix(ip.Name, "br-") && strings.Trim(ip.Name[3:], "0123456789abcdef") == "" // a Docker network's bridge, removed with the network
})
if len(ips) == 0 {
return nil
}
sort.Slice(ips, func(i, j int) bool {
first, second := ips[i], ips[j]
// ethernet is preferred
if vi, vj := strings.HasPrefix(first.Name, "e"), strings.HasPrefix(second.Name, "e"); vi != vj {
// IPv4 is preferred, as job networks are IPv4-only by default
if vi, vj := first.To4() != nil, second.To4() != nil; vi != vj {
return vi
}
// IPv4 is preferred
if vi, vj := first.To4() != nil, second.To4() != nil; vi != vj {
// ethernet is preferred
if vi, vj := strings.HasPrefix(first.Name, "e"), strings.HasPrefix(second.Name, "e"); vi != vj {
return vi
}
+2 -1
View File
@@ -13,8 +13,9 @@ import (
func TestPreferredIP(t *testing.T) {
assert.Nil(t, preferredIP(nil))
assert.Equal(t, net.ParseIP("2001:db8::1"), preferredIP([]interfaceIP{{IP: net.ParseIP("2001:db8::1")}}))
assert.Equal(t, net.ParseIP("2001:db8::1"), preferredIP([]interfaceIP{
assert.Equal(t, net.ParseIP("192.0.2.1"), preferredIP([]interfaceIP{
{IP: net.ParseIP("192.0.2.1"), Name: "wlan0"},
{IP: net.ParseIP("2001:db8::1"), Name: "eth0"},
{IP: net.ParseIP("172.18.0.1"), Name: "br-0123456789ab"},
}))
}
+9
View File
@@ -10,9 +10,11 @@ import (
"fmt"
"io"
"path"
"runtime"
"strings"
"sync"
"sync/atomic"
"time"
"gitea.com/gitea/runner/act/common"
@@ -20,6 +22,11 @@ import (
"github.com/moby/moby/api/types/container"
)
var hostOS = runtime.GOOS // replaced in tests
// DrainGracePeriod caps the wait for container output to drain so the log tail is not truncated
const DrainGracePeriod = 2 * time.Second
// ExitCodeError reports a non-zero process exit code from a container command.
type ExitCodeError int
@@ -145,6 +152,8 @@ type NewDockerBuildExecutorInput struct {
ContextDir string
Dockerfile string
BuildContext io.Reader
Dockerignore []byte // applied to BuildContext as the docker CLI applies it
Stdout io.Writer
ImageTag string
Platform string
BuildArgs map[string]*string
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2021 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !windows && !(WITHOUT_DOCKER || !(linux || darwin || netbsd))
//go:build !windows && !(WITHOUT_DOCKER || !(linux || darwin || netbsd || freebsd))
package container
+100 -6
View File
@@ -2,16 +2,19 @@
// Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
import (
"archive/tar"
"bytes"
"context"
"fmt"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"runtime"
"strings"
@@ -48,6 +51,7 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
options := client.ImageBuildOptions{
Tags: []string{input.ImageTag},
Labels: map[string]string{"com.gitea.runner.image": input.ImageTag}, // own image ID per tag, so removing one never deletes another job's identical build
Remove: true,
AuthConfigs: LoadDockerAuthConfigs(ctx),
Dockerfile: input.Dockerfile,
@@ -62,7 +66,7 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
}
var buildContext io.ReadCloser
if input.BuildContext != nil {
buildContext = io.NopCloser(input.BuildContext)
buildContext, err = filterBuildContext(input.BuildContext, input.Dockerignore, input.Dockerfile)
} else {
buildContext, err = createBuildContext(ctx, input.ContextDir, input.Dockerfile)
}
@@ -77,10 +81,97 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
if err != nil {
return err
}
return logDockerResponse(logger, resp.Body, false)
err = logDockerResponse(logger, resp.Body, func(format string, args ...any) { fmt.Fprintf(input.Stdout, format, args...) })
common.FlushWriter(input.Stdout)
return err
}
}
func filterBuildContext(buildContext io.Reader, dockerignore []byte, relDockerfile string) (io.ReadCloser, error) {
excludes, err := ignorefile.ReadAll(bytes.NewReader(dockerignore))
if err != nil || len(excludes) == 0 {
return io.NopCloser(buildContext), err
}
pm, err := patternmatcher.New(excludes)
if err != nil {
return nil, err
}
reader, writer := io.Pipe()
go func() {
writer.CloseWithError(writeFilteredBuildContext(writer, buildContext, pm, path.Clean(filepath.ToSlash(relDockerfile))))
}()
return reader, nil
}
func writeFilteredBuildContext(w io.Writer, buildContext io.Reader, pm *patternmatcher.PatternMatcher, relDockerfile string) error {
spool, err := os.CreateTemp("", "gitea-runner-build-") // excluded content, for a kept hardlink to it
if err != nil {
return err
}
defer os.Remove(spool.Name())
defer spool.Close()
reader := tar.NewReader(buildContext)
writer := tar.NewWriter(w)
buffer := make([]byte, 32*1024)
excluded := map[string]*excludedBuildContextEntry{}
for {
header, err := reader.Next()
if err == io.EOF {
return writer.Close()
}
if err != nil {
return err
}
name := path.Clean(header.Name)
var target *excludedBuildContextEntry
if header.Typeflag == tar.TypeLink {
target = excluded[path.Clean(header.Linkname)]
}
if name != ".dockerignore" && name != relDockerfile {
skip, err := pm.MatchesOrParentMatches(name)
if err != nil {
return err
}
if skip {
if target != nil {
excluded[name] = target
} else if header.Typeflag != tar.TypeDir {
offset, err := spool.Seek(0, io.SeekCurrent)
if err != nil {
return err
}
excluded[name] = &excludedBuildContextEntry{header: header, offset: offset}
if _, err := io.CopyBuffer(spool, reader, buffer); err != nil {
return err
}
}
continue
}
}
var content io.Reader = reader
if target != nil && target.keptName != "" {
header.Linkname = target.keptName
} else if target != nil {
header.Typeflag, header.Linkname, header.Size = target.header.Typeflag, target.header.Linkname, target.header.Size
content = io.NewSectionReader(spool, target.offset, header.Size)
target.keptName = header.Name
}
if err := writer.WriteHeader(header); err != nil {
return err
}
if _, err := io.CopyBuffer(writer, content, buffer); err != nil {
return err
}
}
}
type excludedBuildContextEntry struct {
header *tar.Header
offset int64
keptName string
}
func createBuildContext(ctx context.Context, contextDir, relDockerfile string) (io.ReadCloser, error) {
common.Logger(ctx).Debugf("Creating archive for build context dir '%s' with relative dockerfile '%s'", contextDir, relDockerfile)
@@ -125,6 +216,10 @@ func createBuildContext(ctx context.Context, contextDir, relDockerfile string) (
}
func writeBuildContext(ctx context.Context, w io.Writer, contextDir string, includes []string, pm *patternmatcher.PatternMatcher) error {
contextDir, err := filepath.EvalSymlinks(contextDir)
if err != nil {
return err
}
tw := tar.NewWriter(w)
seen := map[string]bool{}
hardlinks := map[uint64]string{}
@@ -132,8 +227,7 @@ func writeBuildContext(ctx context.Context, w io.Writer, contextDir string, incl
for _, include := range includes {
include = filepath.FromSlash(include)
var parents []parentMatch
root := strings.TrimSuffix(contextDir, string(filepath.Separator)) + string(filepath.Separator) + include // unlike filepath.Join, follows a symlinked context dir as docker build does
err := filepath.WalkDir(root, func(file string, entry fs.DirEntry, err error) error {
err := filepath.WalkDir(filepath.Join(contextDir, include), func(file string, entry fs.DirEntry, err error) error {
if err != nil {
return leaveOut(ctx, file, err)
}
@@ -210,7 +304,7 @@ func addBuildContextEntry(ctx context.Context, tw *tar.Writer, file, rel string,
return leaveOut(ctx, file, err)
}
}
header, err := tar.FileInfoHeader(noNameLookups{info}, link)
header, err := tar.FileInfoHeader(noNameLookups{info}, filepath.ToSlash(link))
if err != nil {
return err
}
+63 -5
View File
@@ -1,12 +1,14 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
import (
"archive/tar"
"bytes"
"fmt"
"io"
"net/http"
"net/http/httptest"
@@ -18,10 +20,12 @@ import (
"github.com/stretchr/testify/require"
)
func TestDockerBuildReportsDaemonError(t *testing.T) {
func TestDockerBuildLabelsImageWithItsTagAndReportsDaemonError(t *testing.T) {
var labels string
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("API-Version", "1.47")
if strings.HasSuffix(request.URL.Path, "/build") {
labels = request.URL.Query().Get("labels")
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(writer, `{"message":"build rejected"}`)
@@ -33,24 +37,25 @@ func TestDockerBuildReportsDaemonError(t *testing.T) {
t.Setenv("DOCKER_AUTH_CONFIG", "")
require.ErrorContains(t, NewDockerBuildExecutor(NewDockerBuildExecutorInput{
ImageTag: "action:test", Dockerfile: "Dockerfile", BuildContext: strings.NewReader(""),
ImageTag: "action:test", Dockerfile: "Dockerfile", BuildContext: strings.NewReader(""), Stdout: io.Discard,
})(t.Context()), "build rejected")
require.JSONEq(t, `{"com.gitea.runner.image":"action:test"}`, labels)
}
func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.Mkdir(filepath.Join(dir, "build"), 0o755))
for name, content := range map[string]string{
".dockerignore": "*.log\nbuild\n!build/keep.txt\nDockerfile", "Dockerfile": "FROM scratch", "a.txt": "a",
"a.log": "log", "build/drop.txt": "drop", "build/keep.txt": "keep",
} {
require.NoError(t, os.MkdirAll(filepath.Join(dir, filepath.Dir(name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
}
require.NoError(t, os.Mkdir(filepath.Join(dir, "empty"), 0o755))
contexts := []string{dir}
if link := filepath.Join(t.TempDir(), "link"); os.Symlink(dir, link) == nil {
contexts = append(contexts, link+string(filepath.Separator))
contexts = append(contexts, link, link+string(filepath.Separator))
}
for _, contextDir := range contexts {
@@ -69,3 +74,56 @@ func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile
require.Equal(t, []string{".dockerignore", "a.txt", "build/keep.txt", "empty/", "Dockerfile"}, names, contextDir)
}
}
func TestFilterBuildContextAppliesDockerignoreAndMaterializesHardlinksToExcludedFiles(t *testing.T) {
const dockerignore = "*.log\nbuild\n!build/keep.txt\nActionfile\n.dockerignore\n"
for _, prefix := range []string{"", "./"} {
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
for _, entry := range []struct {
name, body, link string
typeflag byte
}{
{"Actionfile", "FROM scratch", "", tar.TypeReg},
{"output.log", "log", "", tar.TypeReg},
{"build/", "", "", tar.TypeDir},
{"build/drop.txt", "drop", "", tar.TypeReg},
{"build/keep.txt", "keep", "", tar.TypeReg},
{"entrypoint", "", prefix + "output.log", tar.TypeLink},
{"other-entrypoint", "", prefix + "output.log", tar.TypeLink},
{".dockerignore", dockerignore, "", tar.TypeReg},
} {
require.NoError(t, writer.WriteHeader(&tar.Header{Name: prefix + entry.name, Linkname: entry.link, Typeflag: entry.typeflag, Size: int64(len(entry.body)), Mode: 0o644}))
_, err := io.WriteString(writer, entry.body)
require.NoError(t, err)
}
require.NoError(t, writer.Close())
unfiltered, err := filterBuildContext(bytes.NewReader(archive.Bytes()), nil, "Actionfile")
require.NoError(t, err)
passedThrough, err := io.ReadAll(unfiltered)
require.NoError(t, err)
require.Equal(t, archive.Bytes(), passedThrough)
filtered, err := filterBuildContext(bytes.NewReader(archive.Bytes()), []byte(dockerignore), "Actionfile")
require.NoError(t, err)
var entries []string
for reader := tar.NewReader(filtered); ; {
header, err := reader.Next()
if err == io.EOF {
break
}
require.NoError(t, err)
body, err := io.ReadAll(reader)
require.NoError(t, err)
entries = append(entries, fmt.Sprintf("%s %c %s %s", strings.TrimPrefix(header.Name, prefix), header.Typeflag, strings.TrimPrefix(header.Linkname, prefix), body))
}
require.Equal(t, []string{
"Actionfile 0 FROM scratch",
"build/keep.txt 0 keep",
"entrypoint 0 log",
"other-entrypoint 1 entrypoint ",
".dockerignore 0 " + dockerignore,
}, entries, prefix)
}
}
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd)
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd || freebsd)
package container
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd)
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd || freebsd)
package container
+9 -5
View File
@@ -2,7 +2,7 @@
// Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
// This file is based on https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts.go with:
// * appended with license information
@@ -351,7 +351,7 @@ type containerConfig struct {
// parse parses the args for the specified command and generates a Config,
// a HostConfig and returns them with the specified command.
// If the specified args are not valid, it will return an error.
func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*containerConfig, error) { //nolint:gocyclo,unparam // verbatim copy from docker/cli
func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*containerConfig, error) { //nolint:gocyclo // verbatim copy from docker/cli
var (
attachStdin = copts.attach.Get("stdin")
attachStdout = copts.attach.Get("stdout")
@@ -724,6 +724,10 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
}, nil
}
func isUserDefinedNetwork(mode container.NetworkMode) bool { // Docker's helper uses the host OS instead of the daemon's.
return !mode.IsDefault() && mode != network.NetworkBridge && mode != network.NetworkHost && !mode.IsNone() && !mode.IsContainer()
}
// parseNetworkOpts converts --network advanced options to endpoint-specs, and combines
// them with the old --network-alias and --links. If returns an error if conflicting options
// are found.
@@ -752,7 +756,7 @@ func parseNetworkOpts(copts *containerOptions) (map[string]*network.EndpointSett
}
for i, n := range copts.netMode.Value() {
if container.NetworkMode(n.Target).IsUserDefined() {
if isUserDefinedNetwork(container.NetworkMode(n.Target)) {
hasUserDefined = true
} else {
hasNonUserDefined = true
@@ -814,7 +818,7 @@ func applyContainerOptions(n *opts.NetworkAttachmentOpts, copts *containerOption
}
// For a user-defined network, "--link" is an endpoint option, it creates an alias. But,
// for the default bridge it defines a legacy-link.
if container.NetworkMode(n.Target).IsUserDefined() && copts.links.Len() > 0 {
if isUserDefinedNetwork(container.NetworkMode(n.Target)) && copts.links.Len() > 0 {
n.Links = slices.Clone(copts.links.GetSlice())
}
if copts.ipv4Address != nil {
@@ -840,7 +844,7 @@ func parseNetworkAttachmentOpt(ep opts.NetworkAttachmentOpts) (*network.Endpoint
if strings.TrimSpace(ep.Target) == "" {
return nil, errors.New("no name set for network")
}
if !container.NetworkMode(ep.Target).IsUserDefined() {
if !isUserDefinedNetwork(container.NetworkMode(ep.Target)) {
if len(ep.Aliases) > 0 {
return nil, errors.New("network-scoped aliases are only supported for user-defined networks")
}
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
+1 -1
View File
@@ -38,7 +38,7 @@ func loadScratchImage(t *testing.T, platform string) string {
response, err := cli.ImageLoad(t.Context(), &archive)
require.NoError(t, err)
defer response.Close()
require.NoError(t, logDockerResponse(common.Logger(t.Context()), response, false))
require.NoError(t, logDockerResponse(common.Logger(t.Context()), response, common.Logger(t.Context()).Debugf))
t.Cleanup(func() {
_, _ = cli.ImageRemove(context.WithoutCancel(t.Context()), tag, mobyclient.ImageRemoveOptions{Force: true})
})
+11 -26
View File
@@ -2,12 +2,13 @@
// Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
import (
"bufio"
"cmp"
"encoding/json/v2"
"errors"
"io"
@@ -26,10 +27,7 @@ type dockerMessage struct {
Progress string `json:"progress"`
}
func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isError bool) error {
if dockerResponse == nil {
return nil
}
func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, log func(string, ...any)) error {
defer dockerResponse.Close()
scanner := bufio.NewScanner(dockerResponse)
@@ -38,41 +36,28 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
line := scanner.Bytes()
var msg dockerMessage
if err := json.Unmarshal(line, &msg); err != nil {
writeLog(logger, false, "Unable to unmarshal line [%s] ==> %v", string(line), err)
logger.Debugf("Unable to unmarshal line [%s] ==> %v", string(line), err)
continue
}
if msg.Error != "" {
writeLog(logger, isError, "%s", msg.Error)
return errors.New(msg.Error)
}
if msg.ErrorDetail.Message != "" {
writeLog(logger, isError, "%s", msg.ErrorDetail.Message)
return errors.New(msg.ErrorDetail.Message)
if message := cmp.Or(msg.Error, msg.ErrorDetail.Message); message != "" {
log("%s", message)
return errors.New(message)
}
switch {
case msg.Status != "":
if msg.Progress != "" {
writeLog(logger, isError, "%s :: %s :: %s\n", msg.Status, msg.ID, msg.Progress)
log("%s :: %s :: %s\n", msg.Status, msg.ID, msg.Progress)
} else {
writeLog(logger, isError, "%s :: %s\n", msg.Status, msg.ID)
log("%s :: %s\n", msg.Status, msg.ID)
}
case msg.Stream != "":
writeLog(logger, isError, "%s", msg.Stream)
log("%s", msg.Stream)
default:
writeLog(logger, false, "Unable to handle line: %s", string(line))
logger.Debugf("Unable to handle line: %s", string(line))
}
}
return scanner.Err()
}
func writeLog(logger *logging.Logger, isError bool, format string, args ...any) {
if isError {
logger.Errorf(format, args...)
} else {
logger.Debugf(format, args...)
}
}
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2023 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2025 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
+64 -28
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
@@ -41,6 +41,8 @@ const (
maxCreateBody = 8 << 20
dockerProxyProbeTimeout = 5 * time.Second
dockerProxyProbeWorkerTimeout = time.Minute
dockerProxyProbeTarget = "/gitea-runner-probe"
)
var (
@@ -48,7 +50,7 @@ var (
rawStreamPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers/[^/]+/attach|exec/[^/]+/start)$`)
)
var dockerProxyDirectories, dockerProxyProbes sync.Map
var dockerProxyProbes sync.Map
type dockerProxyDirectory struct {
dir, daemonDir string
@@ -68,7 +70,7 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
}
probeCtx, cancel := context.WithTimeout(ctx, dockerProxyProbeTimeout)
defer cancel()
cli, err := GetDockerClient(probeCtx)
cli, err := client.New(client.FromEnv) // negotiates on first request, so a cached probe costs no round trip
if err != nil {
return nil
}
@@ -83,53 +85,43 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
}
workdir, _ := os.Getwd()
key := [3]string{cli.DaemonHost(), workdir, os.TempDir()}
cached, _ := dockerProxyDirectories.Load(key)
directory, seen := cached.(dockerProxyDirectory)
seen = seen && os.SameFile(directory.socket, socket) // daemon replacement invalidates cached paths
if !seen {
directory, seen, err = probeDockerProxyDirectory(probeCtx, cli, key, socket)
}
probe, err := probeDockerProxyDirectory(probeCtx, cli, key, socket)
if err != nil {
common.Logger(ctx).Infof("docker proxy probe failed, jobs get the daemon socket directly: %v", err)
return nil
}
if !seen {
if !probe.seen {
common.Logger(ctx).Infof("the docker daemon cannot reach the runner's temporary or working directory, jobs get the daemon socket directly")
return nil
}
if ctx.Err() != nil {
return nil
}
proxy, err := StartDockerProxy(daemonSocket, directory.dir, job)
proxy, err := StartDockerProxy(daemonSocket, probe.directory.dir, job)
if err != nil {
dockerProxyDirectories.Delete(key)
dockerProxyProbes.CompareAndDelete(key, probe)
common.Logger(ctx).Warnf("docker proxy not started, the job gets the daemon socket directly: %v", err)
return nil
}
dockerProxyDirectories.Store(key, directory)
proxy.Socket = directory.daemonDir + strings.TrimPrefix(proxy.Socket, directory.dir)
proxy.Socket = probe.directory.daemonDir + strings.TrimPrefix(proxy.Socket, probe.directory.dir)
return proxy
}
func probeDockerProxyDirectory(ctx context.Context, cli client.APIClient, key [3]string, socket os.FileInfo) (dockerProxyDirectory, bool, error) {
func probeDockerProxyDirectory(ctx context.Context, cli client.APIClient, key [3]string, socket os.FileInfo) (*dockerProxyProbe, error) {
for {
probe := &dockerProxyProbe{directory: dockerProxyDirectory{socket: socket}, done: make(chan struct{})}
actual, loaded := dockerProxyProbes.LoadOrStore(key, probe)
if loaded {
probe, _ = actual.(*dockerProxyProbe)
if !os.SameFile(probe.directory.socket, socket) {
if !os.SameFile(probe.directory.socket, socket) { // daemon replacement invalidates cached paths
dockerProxyProbes.CompareAndDelete(key, probe)
continue
}
select {
case <-ctx.Done():
return dockerProxyDirectory{}, false, ctx.Err()
case <-probe.done:
}
if ctx.Err() == nil && (errors.Is(probe.err, context.Canceled) || errors.Is(probe.err, context.DeadlineExceeded)) {
continue
}
} else {
go func() { // outlives callers' deadlines so a slow daemon still gets its result cached
defer cli.Close()
ctx, cancel := context.WithTimeout(context.Background(), dockerProxyProbeWorkerTimeout)
defer cancel()
probe.directory.dir, probe.directory.daemonDir = runnerContainerWorkdir(ctx, cli)
probe.seen = probe.directory.daemonDir != ""
if !probe.seen {
@@ -138,10 +130,24 @@ func probeDockerProxyDirectory(ctx context.Context, cli client.APIClient, key [3
probe.seen, probe.err = daemonSeesDir(ctx, cli, probe.directory.dir, probe.directory.daemonDir)
}
}
if probe.seen {
dockerProxyProbes.LoadOrStore(key, probe)
} else {
dockerProxyProbes.CompareAndDelete(key, probe)
close(probe.done)
}
return probe.directory, probe.seen, probe.err
close(probe.done)
removeStaleProbeContainers(cli)
}()
}
select {
case <-ctx.Done():
return &dockerProxyProbe{}, ctx.Err()
case <-probe.done:
}
if ctx.Err() == nil && errors.Is(probe.err, context.DeadlineExceeded) {
continue
}
return probe, probe.err
}
}
@@ -200,7 +206,7 @@ func daemonSeesDir(ctx context.Context, cli client.APIClient, dir, daemonDir str
created, err = cli.ContainerCreate(ctx, client.ContainerCreateOptions{
Config: &container.Config{Image: image.ID, Cmd: []string{"true"}},
HostConfig: &container.HostConfig{Mounts: []mount.Mount{
{Type: mount.TypeBind, Source: daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}, // not the marker itself, podman creates a missing bind source where docker rejects it
{Type: mount.TypeBind, Source: daemonDir, Target: dockerProxyProbeTarget, ReadOnly: true}, // not the marker itself, podman creates a missing bind source where docker rejects it
}},
})
if !cerrdefs.IsNotFound(err) {
@@ -213,7 +219,7 @@ func daemonSeesDir(ctx context.Context, cli client.APIClient, dir, daemonDir str
if err != nil {
return false, err
}
seen, err := containerSeesMarker(ctx, cli, dir, created.ID, "/gitea-runner-probe")
seen, err := containerSeesMarker(ctx, cli, dir, created.ID, dockerProxyProbeTarget)
cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), dockerProxyProbeTimeout)
defer cancel()
if _, removeErr := cli.ContainerRemove(cleanupCtx, created.ID, client.ContainerRemoveOptions{Force: true, RemoveVolumes: true}); removeErr != nil {
@@ -222,6 +228,36 @@ func daemonSeesDir(ctx context.Context, cli client.APIClient, dir, daemonDir str
return seen, err
}
// removeStaleProbeContainers removes probe containers left by abandoned creates once any probe in flight when listed has ended.
func removeStaleProbeContainers(cli client.APIClient) {
ctx, cancel := context.WithTimeout(context.Background(), dockerProxyProbeTimeout)
defer cancel()
listed, err := cli.ContainerList(ctx, client.ContainerListOptions{
All: true,
Filters: make(client.Filters).Add("status", string(container.StateCreated)).Add("volume", dockerProxyProbeTarget),
})
if err != nil {
common.Logger(ctx).Warnf("listing stale docker proxy probe containers failed: %v", err)
return
}
listed.Items = slices.DeleteFunc(listed.Items, func(stale container.Summary) bool {
return stale.State != container.StateCreated || !slices.ContainsFunc(stale.Mounts, func(mountPoint container.MountPoint) bool {
return mountPoint.Type == mount.TypeBind && !mountPoint.RW && mountPoint.Destination == dockerProxyProbeTarget
})
})
if len(listed.Items) == 0 {
return
}
time.Sleep(dockerProxyProbeWorkerTimeout + dockerProxyProbeTimeout)
for _, stale := range listed.Items {
ctx, cancel := context.WithTimeout(context.Background(), dockerProxyProbeTimeout)
if err := (&containerReference{cli: cli, id: stale.ID}).remove()(ctx); err != nil {
common.Logger(ctx).Warnf("removing stale docker proxy probe container failed: %v", err)
}
cancel()
}
}
// StartDockerProxy serves a job's docker socket in dir, labelling what the job creates through it.
func StartDockerProxy(daemonSocket, dir, job string) (*DockerProxy, error) {
info, err := os.Stat(daemonSocket)
+127 -54
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container
@@ -94,6 +94,8 @@ func TestDockerProxy(t *testing.T) {
return
}
_, _ = w.Write([]byte(`[{"Id":"probe-image"}]`))
case strings.HasSuffix(r.URL.Path, "/containers/json"):
_, _ = w.Write([]byte(`[]`))
case strings.HasSuffix(r.URL.Path, "/json"):
http.NotFound(w, r)
case strings.HasSuffix(r.URL.Path, "/archive"):
@@ -235,7 +237,8 @@ func TestDockerProxy(t *testing.T) {
t.Skip("automatic proxy discovery requires Linux")
}
t.Setenv("DOCKER_HOST", "unix://"+daemonSocket)
t.Setenv("TMPDIR", shortTempDir(t))
tempDir := shortTempDir(t)
t.Setenv("TMPDIR", tempDir)
require.Nil(t, NewDockerProxy(t.Context(), "failed"))
for _, job := range []string{"first", "second"} {
proxy := NewDockerProxy(t.Context(), job)
@@ -243,12 +246,10 @@ func TestDockerProxy(t *testing.T) {
require.NoError(t, proxy.Close(context.Background()))
}
assert.EqualValues(t, 2, probes.Load())
dockerProxyDirectories.Range(func(key, value any) bool {
directory, _ := value.(dockerProxyDirectory)
directory.socket, _ = os.Stat(t.TempDir())
dockerProxyDirectories.Store(key, directory)
return true
})
workdir, err := os.Getwd()
require.NoError(t, err)
cached, _ := dockerProxyProbes.Load([3]string{"unix://" + daemonSocket, workdir, tempDir})
cached.(*dockerProxyProbe).directory.socket, _ = os.Stat(t.TempDir())
proxy := NewDockerProxy(t.Context(), "replaced daemon")
require.NotNil(t, proxy)
require.NoError(t, proxy.Close(context.Background()))
@@ -476,10 +477,27 @@ func BenchmarkDockerProxy(b *testing.B) {
type probeClient struct {
mobyclient.APIClient
create func(mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error)
create func(context.Context, mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error)
remove func(context.Context, string, mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error)
stat func(string) error
mounts []container.MountPoint
containers []container.Summary
}
func (c *probeClient) Info(_ctx context.Context, _opts mobyclient.InfoOptions) (mobyclient.SystemInfoResult, error) {
return mobyclient.SystemInfoResult{}, nil
}
func (c *probeClient) Close() error {
return nil
}
func (c *probeClient) ContainerList(context.Context, mobyclient.ContainerListOptions) (mobyclient.ContainerListResult, error) {
return mobyclient.ContainerListResult{Items: c.containers}, nil
}
func (c *probeClient) ContainerKill(context.Context, string, mobyclient.ContainerKillOptions) (mobyclient.ContainerKillResult, error) {
return mobyclient.ContainerKillResult{}, cerrdefs.ErrConflict
}
func (c *probeClient) ContainerInspect(context.Context, string, mobyclient.ContainerInspectOptions) (mobyclient.ContainerInspectResult, error) {
@@ -490,8 +508,8 @@ func (c *probeClient) ImageList(context.Context, mobyclient.ImageListOptions) (m
return mobyclient.ImageListResult{Items: []image.Summary{{ID: "removed-image"}, {ID: "probe-image"}}}, nil
}
func (c *probeClient) ContainerCreate(_ context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
return c.create(opts)
func (c *probeClient) ContainerCreate(ctx context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
return c.create(ctx, opts)
}
func (c *probeClient) ContainerRemove(ctx context.Context, id string, opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
@@ -518,7 +536,7 @@ func TestDockerProxyDirectoryProbes(t *testing.T) {
var calls, removals atomic.Int32
visible := false
cli := &probeClient{
create: func(_opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
create: func(_ctx context.Context, _opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls.Add(1)
<-release
return mobyclient.ContainerCreateResult{ID: "probe"}, createErr
@@ -538,23 +556,18 @@ func TestDockerProxyDirectoryProbes(t *testing.T) {
results := make(chan result, 8)
for range cap(results) {
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
results <- result{seen, err}
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
results <- result{probe.seen, err}
}()
}
synctest.Wait()
assert.EqualValues(t, 1, calls.Load())
ctx, cancel := context.WithCancel(t.Context())
cancel()
_, seen, err := probeDockerProxyDirectory(ctx, cli, key, socket)
require.ErrorIs(t, err, context.Canceled)
assert.False(t, seen)
replacement, err := os.Stat(t.TempDir())
require.NoError(t, err)
replaced := make(chan result, 1)
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, replacement)
replaced <- result{seen, err}
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, replacement)
replaced <- result{probe.seen, err}
}()
synctest.Wait()
assert.EqualValues(t, 2, calls.Load())
@@ -574,11 +587,11 @@ func TestDockerProxyDirectoryProbes(t *testing.T) {
assert.Zero(t, removals.Load())
}
createErr, visible = nil, true
directory, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
require.NoError(t, err)
assert.True(t, seen)
assert.Equal(t, dir, directory.dir)
assert.Equal(t, dir, directory.daemonDir)
assert.True(t, probe.seen)
assert.Equal(t, dir, probe.directory.dir)
assert.Equal(t, dir, probe.directory.daemonDir)
assert.EqualValues(t, 3, calls.Load())
entries, err := os.ReadDir(dir)
require.NoError(t, err)
@@ -586,7 +599,7 @@ func TestDockerProxyDirectoryProbes(t *testing.T) {
})
})
}
t.Run("cancelled probe does not cancel another caller", func(t *testing.T) {
t.Run("cancelled caller leaves the probe to finish for the others", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
@@ -595,49 +608,105 @@ func TestDockerProxyDirectoryProbes(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
release := make(chan struct{})
calls := 0
var calls, removals atomic.Int32
cli := &probeClient{
create: func(_opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls++
if calls == 1 {
create: func(ctx context.Context, _opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls.Add(1)
<-release
return mobyclient.ContainerCreateResult{}, ctx.Err()
} else {
return mobyclient.ContainerCreateResult{ID: "probe"}, nil
return mobyclient.ContainerCreateResult{ID: "probe"}, ctx.Err()
},
stat: func(_path string) error { return nil },
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removals.Add(1)
return mobyclient.ContainerRemoveResult{}, nil
},
}
leader, follower, late := make(chan result, 1), make(chan result, 1), make(chan result, 1)
go func() {
probe, err := probeDockerProxyDirectory(ctx, cli, key, socket)
leader <- result{probe.seen, err}
}()
synctest.Wait()
go func() {
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
follower <- result{probe.seen, err}
}()
synctest.Wait()
cancel()
leaderResult := <-leader
require.ErrorIs(t, leaderResult.err, context.Canceled)
assert.False(t, leaderResult.seen)
go func() {
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
late <- result{probe.seen, err}
}()
synctest.Wait()
close(release)
for _, results := range []chan result{follower, late} {
result := <-results
require.NoError(t, result.err)
assert.True(t, result.seen)
}
assert.EqualValues(t, 1, calls.Load())
assert.EqualValues(t, 1, removals.Load())
_, cached := dockerProxyProbes.Load(key)
assert.True(t, cached)
})
})
t.Run("caller joining a probe that ran out of time retries", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
var calls atomic.Int32
cli := &probeClient{
create: func(ctx context.Context, _opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
if calls.Add(1) == 1 {
<-ctx.Done()
return mobyclient.ContainerCreateResult{}, ctx.Err()
}
return mobyclient.ContainerCreateResult{ID: "probe"}, nil
},
stat: func(_path string) error { return nil },
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
return mobyclient.ContainerRemoveResult{}, nil
},
}
leader, follower := make(chan result, 1), make(chan result, 1)
go func() {
_, seen, err := probeDockerProxyDirectory(ctx, cli, key, socket)
leader <- result{seen, err}
}()
synctest.Wait()
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
follower <- result{seen, err}
}()
synctest.Wait()
assert.Equal(t, 1, calls)
cancel()
close(release)
leaderResult, followerResult := <-leader, <-follower
require.ErrorIs(t, leaderResult.err, context.Canceled)
assert.False(t, leaderResult.seen)
require.NoError(t, followerResult.err)
assert.True(t, followerResult.seen)
assert.Equal(t, 2, calls)
probe, err := probeDockerProxyDirectory(t.Context(), cli, [3]string{"daemon", dir, dir}, socket)
require.NoError(t, err)
assert.True(t, probe.seen)
assert.EqualValues(t, 2, calls.Load())
})
})
}
func TestRemoveStaleProbeContainers(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
probeMount := container.MountPoint{Type: mount.TypeBind, Destination: "/gitea-runner-probe"}
writableMount := container.MountPoint{Type: mount.TypeBind, Destination: "/gitea-runner-probe", RW: true}
var removed []string
cli := &probeClient{
containers: []container.Summary{
{ID: "abandoned", State: container.StateCreated, Mounts: []container.MountPoint{probeMount}},
{ID: "foreign", State: container.StateCreated, Mounts: []container.MountPoint{writableMount}},
{ID: "started", State: container.StateExited, Mounts: []container.MountPoint{probeMount}},
},
remove: func(_ctx context.Context, id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removed = append(removed, id)
return mobyclient.ContainerRemoveResult{}, nil
},
}
start := time.Now()
removeStaleProbeContainers(cli)
assert.Equal(t, []string{"abandoned"}, removed)
assert.GreaterOrEqual(t, time.Since(start), dockerProxyProbeWorkerTimeout+dockerProxyProbeTimeout)
})
}
func TestDaemonSeesDir(t *testing.T) {
dir := t.TempDir()
markers := make(map[string]bool)
for _, testCase := range []struct {
name string
daemonDir string
@@ -656,7 +725,7 @@ func TestDaemonSeesDir(t *testing.T) {
removed := false
var attemptedImages []string
cli := &probeClient{
create: func(opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
create: func(_ctx context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
assert.Equal(t, []mount.Mount{{Type: mount.TypeBind, Source: testCase.daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}}, opts.HostConfig.Mounts)
attemptedImages = append(attemptedImages, opts.Config.Image)
if opts.Config.Image == "removed-image" {
@@ -692,6 +761,7 @@ func TestDaemonSeesDir(t *testing.T) {
assert.Equal(t, []string{"removed-image", "probe-image"}, attemptedImages)
assert.Equal(t, testCase.seen, seen)
assert.Equal(t, testCase.removed, removed)
entries, err := os.ReadDir(dir)
require.NoError(t, err)
assert.Empty(t, entries)
@@ -700,6 +770,9 @@ func TestDaemonSeesDir(t *testing.T) {
}
func TestRunnerContainerWorkdir(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("a runner container is Linux")
}
workdir := t.TempDir()
t.Chdir(workdir)
volume := container.MountPoint{Type: mount.TypeVolume, Source: "/daemon/_data", Destination: filepath.Dir(workdir)}
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd)
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd || freebsd)
package container
Loaded 50 of 228 files, more files were not shown because too many files have changed in this diff. Show more