Compare commits

...
38 Commits
Author SHA1 Message Date
silverwind bb6d793cb9 fix: make container.sweep opt-in (#1389)
Sweeping removed every container a job started through its Docker proxy, which silently tore down services that jobs deploy to the runner's own Docker daemon. Deploying from a runner on the target host is common, so `container.sweep` now defaults to `false` and CI-only runners can opt in with `container.sweep: true`.

Fixes https://gitea.com/gitea/runner/issues/1385

Reviewed-on: https://gitea.com/gitea/runner/pulls/1389
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-05 13:50:46 +00:00
silverwind 95ef17d022 fix: prevent masking gaps in workflow command logs (#1384)
Mask partially escaped workflow command output and complete one-character values while preserving literal output and stream line endings.

Fixes: https://gitea.com/gitea/runner/issues/300
Reviewed-on: https://gitea.com/gitea/runner/pulls/1384
Reviewed-by: bircni <[email protected]>
2026-11-05 08:01:44 +00:00
Lunny Xiaoandsilverwind ee03687688 fix: upgrade Alpine packages in the runner images (#1388)
All images run `apk upgrade` before installing packages, so they pick up Alpine fixes their base images lag on. `docker:29.9.2-dind` shipped libexpat 2.9.0-r0 while Alpine 3.24 provides 2.10.0-r0 with the security fixes, and both `docker:29.9.3-dind` and `alpine:3.24` still ship apk-tools 3.0.8-r0 where 3.0.9-r0 is available. The dind bases are also bumped to `docker:29.9.3`, which drops several vulnerable bundled binaries' findings.

Verified by building the targets locally: libexpat is 2.10.0-r0 (was 2.9.0-r0), apk-tools is 3.0.9-r0 (was 3.0.8-r0), and Trivy reports no Alpine package findings on the dind images.

Assisted-by: Codet:claude
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1388
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
2026-11-05 07:45:27 +00:00
silverwind 30e600d8f6 test: speed up (#1386)
Reduce redundant fixture setup, Git subprocesses, and large data copies while preserving test coverage.

Local macOS measurements against main:

| Measurement | Before | After |
| --- | ---: | ---: |
| Git package time, median of 3 alternating runs | 4.09 s | 3.48 s (-15%) |
| Allocations across 18 action-bundle cases | 489 MiB | 251 MiB (-49%) |
| Traced Git starts per checkout table run | 234 | 175 |

Reviewed-on: https://gitea.com/gitea/runner/pulls/1386
Reviewed-by: bircni <[email protected]>
2026-11-05 07:33:17 +00:00
silverwind 664978c819 test: fix windows path test (#1387)
`TestCopyWorkflowFilesHandsNodeTheRunnerCAUnlessTheJobOrImageSetsIt` from https://gitea.com/gitea/runner/pulls/1366 fails on Windows. `HostEnvironment.GetActPath` turns backslashes into slashes there, so `NODE_EXTRA_CA_CERTS` gets the slash form while the test expects the raw temp dir path. The runner behaviour is correct, only the expectation is wrong.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1387
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-05 07:14:56 +00:00
aa56f46c89 enhance: trust a private CA with runner.ca_file and add container.volumes (#1366)
Adds `container.volumes`, mounted into every Docker job, service and action container. Sources are paths on the Docker daemon's host and need no `valid_volumes` entry, and workflow volumes may use them only with the same access.

Adds `runner.ca_file`, a private CA trusted for the Gitea instance besides the system's by the runner's connections and its artifact proxy. Runner-side git and `builtin:checkout` use it as their CA bundle for the instance, persisted with the checkout's credentials. Jobs get it at `/var/run/act/ca.crt` with `NODE_EXTRA_CA_CERTS` pointing there unless the job or its image sets it. Git in jobs, as run by `actions/checkout`, needs the CA in the image's bundle, for example through `container.volumes`.

Fixes: https://gitea.com/gitea/runner/issues/568
Fixes: https://gitea.com/gitea/runner/issues/671
Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1366
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-05 05:28:40 +00:00
silverwindandLunny Xiao 7a9fccb842 ci: sign release checksums with the imported GPG key (#1382)
The release job has imported the Teabot GPG key since https://gitea.com/gitea/runner/pulls/116, but `.goreleaser.yaml` never had a `signs:` block, so releases were never signed. Add it so `checksums.txt` gets a detached `checksums.txt.sig`, the same way https://gitea.com/gitea/cli signs its releases.

The nightly job imports the key as well since the `signs:` block applies to nightly builds, and the R2 publisher mirrors the signature to dl.gitea.com.

_Written by Claude (Opus 5.5)._

---------

Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1382
Reviewed-by: Lunny Xiao <[email protected]>
2026-11-05 04:28:22 +00:00
silverwindandLunny Xiao 7d6e1ff021 test: fix TestRunExecListUsesJobEventAndAllPlans hang on macOS (#1378)
`TestRunExecListUsesJobEventAndAllPlans` hangs until the test timeout on macOS.

Regression from https://gitea.com/gitea/runner/pulls/1338, which added a fifth `runExecList` call to the test to cover `exec --list --detect-event` without events. That took the captured output from 498 to 560 bytes. `captureStdout` swapped `os.Stdout` for a pipe it only read after the callback returned, and on macOS 27.2 an unread pipe holds 512 bytes, so the write blocked forever. Linux pipes hold 64 KiB, so was unaffected.

`printList` and `runExecList` now write to a passed writer, `exec --list` hands them `cmd.OutOrStdout()`, and the tests pass a buffer, so no test swaps `os.Stdout` anymore.

---------

Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1378
Reviewed-by: Lunny Xiao <[email protected]>
2026-11-05 04:22:01 +00:00
silverwind 319567213d docs: use latest image tags in examples (#1383)
The README's Docker example and the Docker and Kubernetes examples pointed at `nightly` images, while the README recommends `latest-dind`. Deployments copied from them should run the latest release, not a build of `main`.

None of the examples changed since v5.3.0, so they work as written with the current `latest` tags.

_Written by Claude (Opus 5.5)._

Reviewed-on: https://gitea.com/gitea/runner/pulls/1383
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 23:00:45 +00:00
silverwind 0093731295 chore(deps): update go.yaml.in/yaml/v4 to v4.0.0 (#1381)
Update go-yaml from the rc.6 pseudo-version to the stable v4.0.0 release, and actionslib to v1.5.2, which does the same in https://gitea.com/gitea/actionslib/pulls/32.

v4 accepts `:` as a mapping with an empty key, so the reusable workflow tests now use an unclosed `[` as their invalid workflow. Gitea counterpart: https://github.com/go-gitea/gitea/pull/40222.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1381
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 20:05:17 +00:00
silverwind 3e1b4b42c2 fix(cache): stop the cache server from running its volume out of space (#1377)
Committing a cache upload copied every part into a new archive before deleting the parts, so a 5 GB save needed 10 GB free. Failed writes and assemblies left their partial bytes until the next stale sweep, up to an hour later, and every other save on the runner failed meanwhile. Nothing made room for an upload, so once the cache had filled its volume down to the free space floor, larger saves kept failing.

- Assemble in place, deleting each part once appended.
- Drop partial parts and archives on failure.
- Before a part would cross the floor, evict the least recently used entries.
- Reject uploads to reservations the sweep already deleted, which left files nothing removed.

_Written by Claude (Opus 5.5)._

Reviewed-on: https://gitea.com/gitea/runner/pulls/1377
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 16:38:21 +00:00
silverwind dcba546cb3 fix: align process handling and fix Windows crashes (#1376)
The intermittent Windows test crashes came from step errors formatting a nil `rc.Run`. `fmt` recovers the nil dereference, but on CPUs with AMX the recovered fault overflows into the heap (https://github.com/golang/go/issues/81238). The errors now name the job, and the Windows test VM hides AMX and runs on UTC, so cached boots no longer jump back 8 hours mid-test.

Windows commands now start suspended and resume once they are in their job objects, so no child escapes them. That drops the process scan reading other processes' memory and the last `unsafe` code, which `depguard` now denies. Step processes outlive a crashed runner, and commands are no longer looked up in the runner's working directory, both as with GitHub's runner.

Like GitHub's runner, host jobs on Linux and macOS now end the processes carrying the job's `RUNNER_TRACKING_ID`, and image pulls and Dockerfile action builds are retried.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1376
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 16:32:22 +00:00
silverwind 142d9f8c88 fix: give each Dockerfile action image its own ID (#1372)
Identical Dockerfile builds in concurrent jobs share one image ID under their per-job tags. When a job ends and removes its tag, the daemon deletes the image while another job's build is about to tag it, failing that job with `No such image`. This also hits `TestRunEvent` in CI, where two parallel subtests build the same action.

Label each build with its tag, so every job gets its own image on top of the shared cached layers and removing it never touches another job's.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1372
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 07:08:45 +00:00
silverwindandbircni 54b2cb944c fix: give git-lfs the checkout token once (#1371)
`builtin:checkout` set the token both in its persisted credentials file and in the step's environment. git-lfs before 3.8.0 then sent `Authorization` twice on the LFS batch request, which nginx rejects with 400.

Write the credentials file once the checkout succeeded, since it only serves later steps. A failed checkout no longer persists credentials.

Users can also upgrade git-lfs to fix the root bug, but this fix is worth to keep regardless.

Fixes: https://gitea.com/gitea/runner/issues/1370
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1371
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-04 06:42:00 +00:00
Zettat123andsilverwind b65fd0f8f8 ci: run the Windows tests on a runner with KVM (#1369)
Run the Windows tests on `ubuntu-latest-kvm` as a PR job, with the installed Windows cached, minutes instead of ~1h40m. Aligned with https://gitea.com/gitea/gitea-mcp/pulls/321 and https://gitea.com/gitea/cli/pulls/79.

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1369
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-11-04 02:08:38 +00:00
silverwind f2fbd67962 enhance: support Docker on FreeBSD and restructure the README (#1367)
1. Restructures the README into installation, quickstart, deployment, configuration, jobs and operation, and shortens it to user-facing prose. Option details are left to `config.example.yaml` and `docs/`, existing heading anchors are kept where possible.
2. FreeBSD builds used the Docker stub, so `docker://` labels, services and Docker actions failed there. The Docker code works on FreeBSD against a remote daemon, so it is enabled, and host jobs report `runner.os` as `FreeBSD`.
3. `make build` on Windows now writes `gitea-runner.exe`, taking the suffix from `go env GOEXE`.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1367
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-03 02:42:25 +00:00
silverwind d6f5b68a0d enhance: test on Windows and fix what it found (#1348)
Adds a weekly and on-demand workflow that runs the cross-compiled tests on Windows Server 2025 Core in a [dockur/windows](https://github.com/dockur/windows) QEMU VM, and fixes the Windows bugs it found in host and UNC paths, env handling, shell quoting, post-task scripts, health checks, checkout, action symlinks, step summaries, Git client certificates and docker mode from a Windows host.

Host-mode steps now emit UTF-8, `bash` falls back to Git's bash instead of the WSL launcher, the runner can run as a native Windows service, steps can keep background helpers with `RUNNER_TRACKING_ID`, and Windows arm64 binaries are released.

Fixes: https://gitea.com/gitea/runner/issues/1339
Fixes: https://gitea.com/gitea/runner/issues/637
Fixes: https://gitea.com/gitea/runner/issues/1172
Fixes: https://gitea.com/gitea/runner/issues/347
Reviewed-on: https://gitea.com/gitea/runner/pulls/1348
Reviewed-by: bircni <[email protected]>
2026-11-03 01:26:10 +00:00
Renovate Bot 6e42621b4a chore(deps): update dependencies (#1365)
Co-authored-by: Renovate Bot <[email protected]>
2026-11-02 08:16:42 +00:00
adamaruandsilverwind 228effab64 fix(dind): double-fork in s6-notifyoncheck to eliminate zombie processes (#1364)
In `scripts/s6/docker/run`, `s6-notifyoncheck` runs without the `-d` flag:

```bash
exec s6-notifyoncheck -n 0 dockerd-entrypoint.sh
```

Without `-d`, `s6-notifyoncheck` performs a single fork to execute the background checking process. The parent immediately replaces itself via `exec` with `dockerd-entrypoint.sh` (which calls `exec rootlesskit` on `dind-rootless`).

Because only a single fork occurred, the background checking child remains parented to `rootlesskit`. When the check passes and the process terminates:
1. `rootlesskit` did not spawn this process and never reaps it via `waitpid()`.
2. Under standard Linux process semantics, `s6-svscan` (PID 1) cannot adopt or reap the child because its parent (`rootlesskit`) is still running.
3. The terminated process remains as a permanent zombie (`[s6-notifyoncheck] <defunct>`) for the container's entire lifespan.

Adding `-d` instructs `s6-notifyoncheck` to double-fork, detaching the background check so it is immediately adopted by `s6-svscan` (PID 1) and cleanly reaped upon exit.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1364
Reviewed-by: silverwind <[email protected]>
Co-authored-by: adamaru <[email protected]>
2026-11-01 07:20:20 +00:00
silverwindandbircni a46d3f3eda test(e2e): speed up and cover exec and Kubernetes modes (#1363)
Follow-up to https://gitea.com/gitea/runner/pulls/1362, whose crash e2e missed because it only drove the daemon on Docker.

- `dispatch`, `failure_paths` and `reusable_workflow` run through the daemon and `exec`, each on Docker and one shared k3s cluster.
- Fixed sleeps become gates the test releases, the job timeout uses a 15s runner deadline, k3s boots alongside Gitea, Gitea's data is on tmpfs.
- Outside a container, `make test-e2e` runs `go test` in a Linux container on its own network like CI, so local runs take the same paths.

Local wall time:

| | Before | After |
|---|---|---|
| Suite | 84s | 27–35s |
| `job_timeout` | 64s | 17s |
| `reconnect` | 22s | 4.5s |
| `capacity/serial` | 20s | 2s |
| `cli_shutdown_drains` | 14s | 2.3s |
| `kubernetes` | 31s | 21–23s |

_Written by Claude (Opus 5.5)._

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1363
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-31 10:47:35 +00:00
2e72cf902a enhance(kubernetes): fetch job pod copies from the runner over the pod network (#1354)
The Kubernetes backend copies actions and workspaces into job containers with `tar` over `pods/exec`, so every byte goes through the API server. On managed clusters that often sits in other zones than the nodes, so each byte can cross zones twice, and the copies load the API server.

With `kubernetes.transfer.mode: auto` or `direct`, an init container copies `gitea-runner` into the job pod, and the job container fetches directories and copies of 64 KiB or more gzipped from the runner over the pod network. `auto` falls back to the API server for a job pod that cannot reach the runner, `direct` fails the job.

Copies can hold secrets, so each is served once, over TLS 1.3 pinned to a certificate the runner generates, only to the job pod's IPs.

Copying setup-go@v5, cache@v4, setup-java@v4 and build-push-action@v6 sends 11.6 MB through the API server on main and 532 B with this PR.

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1354
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
2026-10-31 08:00:36 +00:00
silverwind f4741df90d fix(exec): load local reusable workflows from disk with --no-skip-checkout (#1362)
`gitea-runner exec --no-skip-checkout` panics with a nil pointer dereference on any job that calls a local reusable workflow (`uses: ./.gitea/workflows/x.yml`).

Fetching a local reusable workflow from the instance was gated on `NoSkipCheckout`, but it reads the repository and SHA from the task's preset context, which only the daemon has. `exec` always has the workflow on disk, so the fetch is now gated on the preset instead, and `--no-skip-checkout` only governs checkout again. Daemon behavior is unchanged.

The `exec` e2e test now also runs the existing reusable caller/callee pair with `--no-skip-checkout`.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1362
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-31 05:24:36 +00:00
261ab374ce enhance(cache): scope cache entries by ref like GitHub (#1361)
Cache entries were only scoped by repository, so an entry a pull request saved could be restored on the default branch, for example through a `restore-keys` prefix.

Entries now carry the ref of the job that saved them. As on GitHub ([Restrictions for accessing a cache](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)), a job restores entries of its own ref first, then of its pull request's base branch, then of the default branch. Pull request events are scoped by their pull request number.

Entries saved before this change are no longer restored and expire through `cache.retention`, so caches start cold once after the upgrade. Runners sharing a cache, and an external `cache-server`, should be upgraded together.

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1361
Reviewed-by: silverwind <[email protected]>
Co-authored-by: JorgeDLMV <[email protected]>
2026-10-30 22:34:45 +00:00
KenanZhuandsilverwind ca8e9c3c64 enhance: add GITEA_* aliases for built-in workflow environment variables (#1358)
Closes https://gitea.com/gitea/runner/issues/1317, which makes https://github.com/go-gitea/gitea/issues/39813 redundant as the runner sets these variables.

Every built-in `GITHUB_*` environment variable the runner sets is now also available as `GITEA_*` with the same value, so Gitea-native workflows don't have to rely on GitHub naming. The `GITEA_*` names mirror the `GITHUB_*` ones after their last write, so they stay equal and are unset together.

Workflow, job and runner `envs` values for these `GITEA_*` names are now overwritten, the same as for `GITHUB_*`.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1358
Reviewed-by: silverwind <[email protected]>
Co-authored-by: KenanZhu <[email protected]>
2026-10-30 21:21:02 +00:00
silverwindandbircni d8d4a4657d fix: scope the action identity to the step executing an action (#1359)
`github.action_repository`/`github.action_ref` and their env vars now belong to the step executing an action, like on GitHub: set for remote actions, empty for `run`, `docker://` and local steps. Before, a local action nested in a remote composite resolved `$/` against the remote action instead of the workflow's repository.

Updates actionslib to v1.5.1, which accepts `$/` as the repository root action.

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1359
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-29 17:41:27 +00:00
silverwind 5fd2f82735 test: fix race in TestTreeKillCancelBeforeCaptureKillsTree (#1360)
`TestTreeKillCancelBeforeCaptureKillsTree` fails intermittently on CI, e.g. https://gitea.com/gitea/runner/actions/runs/1155856.

Dash, the Ubuntu `/bin/sh`, points a background job's stdin at `/dev/null` before applying its redirections, so `cat <&0 &` read EOF and exited at once and the liveness check raced it. The pipe is now handed over on fd 3, which dash leaves alone.

Introduced in https://gitea.com/gitea/runner/pulls/1338

Reviewed-on: https://gitea.com/gitea/runner/pulls/1360
Reviewed-by: Zettat123 <[email protected]>
2026-10-29 16:46:53 +00:00
silverwind c6567296a0 fix: stop the docker proxy probe from leaking its container (#1357)
When the probe's deadline expired during `ContainerCreate`, dockerd still created the container but the probe never got its ID, leaving it in `Created` with its image's anonymous volumes. A slow daemon leaked one per job.

The probe now runs detached with its own timeout, so the create is never cancelled and the container is always removed. Callers still fall back after their own deadline, and the runner waits for pending probes before exiting.

Fixes: https://gitea.com/gitea/runner/issues/1353
Reviewed-on: https://gitea.com/gitea/runner/pulls/1357
Reviewed-by: Zettat123 <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-29 15:53:32 +00:00
Renovate Bot e3607b576e fix(deps): update module golang.org/x/net to v0.62.0 [security] (#1356)
This PR contains the following updates:

| Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| [golang.org/x/net](https://pkg.go.dev/golang.org/x/net) | [`v0.61.0` → `v0.62.0`](https://cs.opensource.google/go/x/net/+/refs/tags/v0.61.0...refs/tags/v0.62.0) | ![age](https://developer.mend.io/api/mc/badges/age/go/golang.org%2fx%2fnet/v0.62.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/golang.org%2fx%2fnet/v0.61.0/v0.62.0?slim=true) |

---

### Avoid quadratic complexity when coalescing text nodes in html in golang.org/x/net
[CVE-2026-78661](https://nvd.nist.gov/vuln/detail/CVE-2026-78661) / [GO-2026-6860](https://pkg.go.dev/vuln/GO-2026-6860)

<details>
<summary>More information</summary>

#### Details
When coalescing adjacent text nodes in addText and fosterParent, appending to an existing TextNode's Data via string concatenation (n.Data += text) reallocated and copied the accumulated string on every token. Inputs containing many short text runs separated by ignored end tags (such as "a</x>") caused O(n^2) allocations and byte copies.

Accumulate coalesced text in per-node byte slices during parsing and materialize Data strings once at the end of parse.

#### Severity
Unknown

#### References
- [https://go.dev/cl/849365](https://go.dev/cl/849365)
- [https://go.dev/issue/82224](https://go.dev/issue/82224)
- [https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs](https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6860) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Check all ancestor namespace transitions when rendering raw text in html in golang.org/x/net
[CVE-2026-97045](https://nvd.nist.gov/vuln/detail/CVE-2026-97045) / [GO-2026-6866](https://pkg.go.dev/vuln/GO-2026-6866)

<details>
<summary>More information</summary>

#### Details
In childTextNodesAreLiteral, walking up the tree previously stopped at the nearest foreign ancestor once an HTML integration point was found. However, that integration point may itself be a descendant of an element that was foster-parented into a foreign namespace (such as <mglyph> or <malignmark> fostered under <mtext>), causing a parser to resolve the subtree in a different foreign namespace upon re-parsing the rendered output.

Continue walking the full ancestor chain and verify that every transition from a non-empty namespace occurs at an HTML integration point before emitting literal text.

#### Severity
Unknown

#### References
- [https://go.dev/cl/855121](https://go.dev/cl/855121)
- [https://go.dev/issue/82622](https://go.dev/issue/82622)
- [https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs](https://groups.google.com/g/golang-announce/c/sqrs2vTx_Xs)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6866) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xODEuNyIsInVwZGF0ZWRJblZlciI6IjQ0LjE4MS43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://gitea.com/gitea/runner/pulls/1356
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-10-28 06:33:58 +00:00
Lunny XiaoandZettat123 ffd80101a3 enhance(kubernetes): gzip copies into job containers over exec (#1355)
The Kubernetes backend copies actions and other files into the job container with `tar` over `pods/exec`, so the data goes through the API server, which on managed clusters often sits in other zones than the nodes (each copied byte then crosses zones twice). Actions are megabytes of JavaScript bundles that gzip about fivefold:

| Action | tar | gzip |
|---|---|---|
| actions/setup-go@v5 | 8.0 MB | 1.5 MB |
| actions/cache@v4 | 11.6 MB | 2.0 MB |
| actions/setup-java@v4 | 15.3 MB | 2.5 MB |
| docker/build-push-action@v6 | 12.9 MB | 3.0 MB |

The startup probe now also reports whether the job container has `gzip`. If it does, `CopyDir` and copies of 64 KiB or more are gzipped at `BestSpeed` and extracted with `gzip -dc | tar -xf -`. Smaller copies (run scripts, file command files) stay uncompressed, where gzip would save next to nothing and cost a process in the pod. Images without `gzip` keep the plain tar stream, so the image requirements do not change.

This is independent of #1354 and gets most of its traffic saving with no new moving parts.

---
*Written by Codet (AI coding agent) on behalf of @lunny.*

---------

Co-authored-by: Zettat123 <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1355
Reviewed-by: Zettat123 <[email protected]>
2026-10-28 06:00:51 +00:00
b712c91c2d test(e2e): cover host mode, reusable workflows, scheduling, reconnect and CLI flags (#1347)
Extends the e2e suite with scenarios for host labels, local `workflow_call`, label routing, `runner.capacity`, a dropped and restored connection to Gitea, and the daemon's `--once`, `--labels` and `shutdown_timeout` behaviour.

Stacked on https://gitea.com/gitea/runner/pulls/1346, so its commit shows here until that merges. The new scenarios pass locally against gitea/gitea:latest and main-nightly.

Assisted-by: Codet:codet
Co-authored-by: Zettat123 <[email protected]>
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1347
Reviewed-by: Zettat123 <[email protected]>
2026-10-28 04:32:50 +00:00
d77e216674 fix(exec): serve the v4 artifact API so upload-artifact v4 and later work (#1341)
`exec` forwarded the toolkit's ArtifactService calls to an instance it never registered, and its random runtime token held no backend IDs, so `actions/upload-artifact` and `actions/download-artifact` v4 and later failed.

The local artifact server now serves the v4 API, including the single Put Blob uploads docker/build-push-action uses. `exec` gives each workflow its own unsigned JWT runtime token and artifact run, so artifact names conflict and carry over between runs as on GitHub. Artifacts default to a temporary directory instead of the workspace that jobs copy.

Co-authored-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1341
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-10-28 03:52:53 +00:00
Lunny Xiao a76ccd7020 test(e2e): cover the register/daemon CLI and failing jobs (#1346)
The e2e suite started the runner in-process and never exercised failing jobs. This adds a scenario that builds the binary and drives `register` then `daemon` against the Gitea fixture, and one for failed, skipped, continue-on-error and timed-out jobs.

Verified locally against gitea/gitea:main-nightly.

Assisted-by: Codet:codet
Reviewed-on: https://gitea.com/gitea/runner/pulls/1346
Reviewed-by: Zettat123 <[email protected]>
2026-10-27 21:23:59 +00:00
3dbed63f55 fix: close the action header group before the action runs (#1345)
`uses:` steps and job hooks kept their "Run …" group open until they finished. Gitea doesn't nest groups, so the action's own first `::group::` closed the header and everything before it stayed hidden there, e.g. all of goreleaser's output in https://gitea.com/gitea/cli/actions/runs/1121069/jobs/1519275. The header group now closes right after its details, as in actions/runner.

The action header also matches actions/runner's otherwise now: it is titled by the `uses:` reference, lists the resolved inputs including `action.yml` defaults without empty ones, is shown for pre stages too, and post stages start with `Post job cleanup.`.

---------

Co-authored-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1345
Reviewed-by: silverwind <[email protected]>
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-10-26 17:42:16 +00:00
silverwindandbircni dafe8d8d0c fix: keep dry runs off the docker daemon and speed up the tests (#1349)
Without a docker daemon six `act/runner` tests failed instead of passing. A dry run still created and removed job networks, the job teardown dropped the dry-run flag along with every other context value, and the container mocks asked the daemon for the runner arch.

This also speeds up the tests and CI without dropping assertions. Most of the time went to git and docker round trips, which are now pinned, batched or coalesced, and the dind and e2e steps cache or overlap more of their setup. On CI, `act/runner` went from 77s to 55s and `test-dind` from 108s to 55s. Locally, `act/runner` without docker went from 53s to 7s.

Split out of https://gitea.com/gitea/runner/pulls/1348, which adds Windows CI on top. The go cache changes moved to https://gitea.com/gitea/runner/pulls/1350.

Written by Claude Code.

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1349
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-26 17:25:17 +00:00
Zettat123 8ae48ae033 fix: keep the shared action cache as extracted, re-fetch damaged trees (#1352)
The toolkit patch rewrote bundles in place in the shared action cache. Since https://gitea.com/gitea/runner/pulls/1316 trees are never refreshed, so a bundle left truncated failed every later job using it.

Patched bundles now only overwrite the job container's copy, so the runner no longer writes to an extracted tree.

Extraction records each file's size beside the tree, and a tree that no longer matches is fetched again, offline mode included, as a missing one always was. This also catches damage from outside the runner, such as storage losing a write, for a stat per file, a fraction of the copy each stage already makes. Trees from older runners have no record and download once more, except in offline mode, which keeps using them as before.

Generated with Claude Code.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1352
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: bircni <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-10-26 17:20:53 +00:00
Renovate Bot 8dd3776014 chore(deps): update dependencies (#1351)
Co-authored-by: Renovate Bot <[email protected]>
2026-10-26 07:06:33 +00:00
silverwind c6ca020281 ci: implement gitea's cache-seeder (#1350)
Like Gitea's CI, only a `cache-seeder` workflow on main saves the go caches, all other jobs only restore them. Caches are scoped per repository, so until now any pull request, forks included, could write a cache that other runs restore. The seeder compiles the tests without running them, so no test result is ever cached.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1350
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-24 17:04:51 +00:00
silverwind fe2a66dc25 fix: align runner behaviours (#1338)
Contains a number of small bug fixes where jobs behaved differently from GitHub or failed silently.

1. Hooks fail on a refused `::set-env::` or `::add-path::` and no longer inherit the last step's identity
1. Docker build, exec and wait errors are returned instead of swallowed, token-only registry credentials work, logs of `--tty` services decode
1. Env files split lines like GitHub on each OS, so a carriage return is kept on Linux and CRLF ends a line on Windows, empty heredoc names or delimiters are rejected
1. Post steps run after their pre step even when main was skipped, composites see undeclared inputs, `runs.args` wins without `with.args` being parsed
1. Docker actions skip evaluating defaults of inputs given in `with` and read `entryPoint` and `args` in any case, a lowercase `dockerfile` is found, undeclared inputs get GitHub's warning
1. A user-set `CI` is kept, `GITHUB_REPOSITORY_ID` and the other missing `GITHUB_*` variables are exported
1. `timeout-minutes` can read the step's own `env`, a step-level `ACTIONS_ALLOW_UNSECURE_COMMANDS: false` beats a job-level `true` while the runner's own opt-in still wins, `pwsh` quotes the script path
1. A step `env:` given as one expression such as `${{ fromJSON(matrix.env) }}` is applied, `hashFiles` fails with GitHub's messages on script errors and after 120 seconds, a job image without node still gets an empty hash
1. A step whose command files cannot be reset fails instead of importing stale ones, a timeout between steps fails the job instead of reporting success
1. One named volume can be mounted at several paths, host-mode PTY steps keep their last output when they fail, host scratch directories are removed when job setup fails
1. Cancelling a step right after it starts kills its whole process tree
1. Cache prefix restores pick the newest entry created within the same second, also on S3, cache commits with a wrong size are rejected
1. Artifact chunks are written at their offset, download URLs are escaped, listing a run without artifacts no longer crashes
1. Poll backoff reaches `fetch_interval_max`, proxy passwords are masked in more forms, secrets are also masked in step names and in their quoted form in the runner log, a failed final report is logged
1. On Windows hosts a differently cased unsecure-commands opt-in or proxy variable counts, and an early cancel no longer leaks a job object handle
1. Interactive `register` prompts for a missing token and stops on SIGTERM, `exec --list --detect-event` no longer panics without events, `config set` works on `cache.s3` keys

Reviewed-on: https://gitea.com/gitea/runner/pulls/1338
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-23 09:18:24 +00:00
238 changed files with 9727 additions and 3122 deletions

No files matched your search

+5
View File
@@ -45,8 +45,13 @@ cpu.out
/bin /bin
/dist /dist
/.tmp
/.git/hooks
/.git/logs
/.git/rr-cache
/.env /.env
/.runner /.runner
/.runner.lock
/config.yaml /config.yaml
/Dockerfile /Dockerfile
.DS_Store .DS_Store
+29
View File
@@ -0,0 +1,29 @@
name: go-setup
description: Set up go and restore its caches, which only the cache-seeder workflow saves
runs:
using: composite
steps:
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
id: setup-go
with:
go-version-file: go.mod
check-latest: true
cache: false
- if: ${{ github.workflow == 'cache-seeder' }}
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
- if: ${{ github.workflow != 'cache-seeder' }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
restore-keys: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-
+26
View File
@@ -0,0 +1,26 @@
name: cache-seeder
on:
workflow_dispatch:
push:
branches:
- main
paths:
- go.mod
- go.sum
- .golangci.yml
- Makefile
- .gitea/actions/go-setup/action.yml
- .gitea/workflows/cache-seeder.yml
jobs:
seed:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.gitea/actions/go-setup
- run: make deps-tools lint build
- name: compile the test binaries without running them
run: |
make test GOTEST_FLAGS='-race -exec true'
go test -tags e2e -exec true ./e2e/...
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7 - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7
with: with:
node-version: 24 node-version: 26
- run: make lint-pr-title - run: make lint-pr-title
env: env:
PR_TITLE: ${{ github.event.pull_request.title }} PR_TITLE: ${{ github.event.pull_request.title }}
+10 -2
View File
@@ -34,8 +34,15 @@ jobs:
with: with:
go-version-file: "go.mod" go-version-file: "go.mod"
check-latest: true check-latest: true
- name: Import GPG key
id: import_gpg
uses: crazy-max/ghaction-import-gpg@2dc316deee8e90f13e1a351ab510b4d5bc0c82cd # v7
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
passphrase: ${{ secrets.PASSPHRASE }}
fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0
- name: goreleaser - name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7 uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with: with:
distribution: goreleaser-pro distribution: goreleaser-pro
args: release --nightly args: release --nightly
@@ -47,6 +54,7 @@ jobs:
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }} R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
GORELEASER_FORCE_TOKEN: "gitea" GORELEASER_FORCE_TOKEN: "gitea"
GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GPG_FINGERPRINT: ${{ steps.import_gpg.outputs.fingerprint }}
release-image: release-image:
runs-on: ubuntu-latest runs-on: ubuntu-latest
@@ -79,7 +87,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub - name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with: with:
username: ${{ secrets.DOCKER_USERNAME }} username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }} password: ${{ secrets.DOCKER_PASSWORD }}
+2 -2
View File
@@ -35,7 +35,7 @@ jobs:
passphrase: ${{ secrets.PASSPHRASE }} passphrase: ${{ secrets.PASSPHRASE }}
fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0 fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0
- name: goreleaser - name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7 uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with: with:
distribution: goreleaser-pro distribution: goreleaser-pro
args: release args: release
@@ -82,7 +82,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub - name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with: with:
username: ${{ secrets.DOCKER_USERNAME }} username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }} password: ${{ secrets.DOCKER_PASSWORD }}
+37 -15
View File
@@ -17,25 +17,24 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 - uses: ./.gitea/actions/go-setup
with:
go-version-file: 'go.mod'
check-latest: true
- name: prepare anonymous docker config - name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json" run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
# Pre-pull act/runner's two largest base images so a slow pull can't dominate `make test`; # Pre-pull act/runner's two largest base images so a slow pull can't dominate `make test`;
# the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host # the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host
# daemon retains them between runs, so this is usually a fast manifest re-check. # daemon retains them between runs, and their digest pins skip the registry when it has them.
- name: pre-pull test images - name: pre-pull test images
env: env:
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:d6aa754f16b3197301076f047b5def2f02ea1dbbc2ca920407d46d7ec7f87b20 # renovate: datasource=docker TEST_JOB_IMAGE: node:26-bookworm-slim@sha256:bc0a2afd3b74ce12222fc8bab99271e66257f021078f5f4a5d687a77d280c1d9 # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:df221db836e1754089190208cee7eeda94f233197056426eda74a43ab1abeac2 # renovate: datasource=docker TEST_SERVICE_IMAGE: nginx:alpine@sha256:a3554c0d0b48177b1413c26e01f695334428020d9df01243fbae3e11cfd17417 # renovate: datasource=docker
run: | run: |
for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do
for attempt in 1 2 3; do if [ "$(docker image inspect --format '{{.Os}}/{{.Architecture}}' "$image" 2>/dev/null)" != linux/amd64 ]; then
docker pull "$image" && break for attempt in 1 2 3; do
[ "$attempt" = 3 ] || sleep 5 docker pull "$image" && break
done [ "$attempt" = 3 ] || sleep 5
done
fi
docker tag "$image" "${image%@*}" docker tag "$image" "${image%@*}"
done done
- name: lint - name: lint
@@ -58,6 +57,32 @@ jobs:
make coverage-report make coverage-report
cat .tmp/coverage.md >> "$GITHUB_STEP_SUMMARY" cat .tmp/coverage.md >> "$GITHUB_STEP_SUMMARY"
windows:
name: test on windows
runs-on: ubuntu-latest-kvm
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.gitea/actions/go-setup
- name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
- name: windows vm cache key
id: windows-vm
run: echo "key=windows-vm-$(grep '^WINDOWS_TEST_IMAGE' Makefile | cat - scripts/test-windows.sh scripts/test-windows-setup.ps1 | sha256sum | cut -c1-16)" >> "$GITHUB_OUTPUT"
- uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/windows-vm
key: ${{ steps.windows-vm.outputs.key }}
- name: test on windows
id: test-windows
run: WINDOWS_TEST_CACHE="$HOME/.cache/windows-vm" make test-windows
env:
WINDOWS_TEST_TIMEOUT: 45
- if: always() && steps.test-windows.outputs.save-cache == 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/windows-vm
key: ${{ steps.windows-vm.outputs.key }}
e2e: e2e:
name: gitea compatibility (${{ matrix.gitea_image }}) name: gitea compatibility (${{ matrix.gitea_image }})
runs-on: ubuntu-latest runs-on: ubuntu-latest
@@ -68,10 +93,7 @@ jobs:
- gitea/gitea:main-nightly - gitea/gitea:main-nightly
steps: steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 - uses: ./.gitea/actions/go-setup
with:
go-version-file: 'go.mod'
check-latest: true
- name: prepare anonymous docker config - name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json" run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
- name: e2e compatibility - name: e2e compatibility
+1
View File
@@ -1,4 +1,5 @@
/gitea-runner /gitea-runner
/gitea-runner.exe
.env .env
!/act/runner/testdata/secrets/.env !/act/runner/testdata/secrets/.env
.runner .runner
+2
View File
@@ -39,6 +39,8 @@ linters:
deny: deny:
- pkg: golang.org/x/exp - pkg: golang.org/x/exp
desc: it's experimental and unreliable desc: it's experimental and unreliable
- pkg: unsafe
desc: use typed APIs instead
logging: logging:
files: files:
- $all - $all
+5 -2
View File
@@ -34,8 +34,6 @@ builds:
goarch: s390x goarch: s390x
- goos: windows - goos: windows
goarch: arm goarch: arm
- goos: windows
goarch: arm64
- goos: freebsd - goos: freebsd
goarch: s390x goarch: s390x
- goos: freebsd - goos: freebsd
@@ -82,6 +80,7 @@ builds:
publishers: publishers:
- name: cloudflare-r2 - name: cloudflare-r2
checksum: true checksum: true
signature: true
extra_files: extra_files:
- glob: ./**.xz - glob: ./**.xz
- glob: ./**.sha256 - glob: ./**.sha256
@@ -102,6 +101,10 @@ checksum:
extra_files: extra_files:
- glob: ./**.xz - glob: ./**.xz
signs:
- artifacts: checksum
args: ["--batch", "-u", "{{ .Env.GPG_FINGERPRINT }}", "--output", "${signature}", "--detach-sign", "${artifact}"]
snapshot: snapshot:
version_template: "{{ .Branch }}-devel" version_template: "{{ .Branch }}-devel"
+27 -6
View File
@@ -23,11 +23,14 @@ depending on the prefix:
covers both forms: it escapes `%` first, so a literal `%3B` becomes `%253B` that the extra covers both forms: it escapes `%` first, so a literal `%3B` becomes `%253B` that the extra
`##[…]` rules cannot match, and a raw `;` or `]` is never decoded. It is also what makes `##[…]` rules cannot match, and a raw `;` or `]` is never decoded. It is also what makes
multi-line work, `\n` becomes `%0A` and the UI turns it back into a line break. multi-line work, `\n` becomes `%0A` and the UI turns it back into a line break.
- **Forwarding a command from step output?** Leave the payload alone, it arrived escaped and is - **Forwarding a command from step output?** Preserve its escaping unless masking changes it.
decoded once. Decoding here double-decodes and destroys multi-line. Decoding for display happens in the UI.
- **No prefix?** Do not escape, and split multi-line values into one row each. - **No prefix?** Do not escape, and split multi-line values into one row each.
- **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the - **Interpolating a secret?** Masking runs after escaping. Recognised command payloads are
encoded forms too. normalised before masking to cover partially escaped values. Their original escaping is
preserved when no replacement occurs. `AppendSecretMasker` also registers encoded forms for
other output. Complete nonblank one-character values are masked, while one-byte fragments
of multiline values are skipped.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the command - Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the command
handler folds an annotation's location into its decoded message and escapes the result again. handler folds an annotation's location into its decoded message and escapes the result again.
@@ -57,7 +60,25 @@ Each lives in `internal/pkg/action/<name>` and is registered in `builtinActions`
`make test-e2e` runs the runner against `E2E_GITEA_IMAGE`. It defaults to the nightly image. `make test-e2e` runs the runner against `E2E_GITEA_IMAGE`. It defaults to the nightly image.
It requires Docker and is excluded from `make test`. CI runs stable and nightly variants in It requires Docker and is excluded from `make test`. CI runs stable and nightly variants in
parallel. parallel. Outside a container it runs `go test` in a Linux container on its own network, as CI does,
so both take the same paths. `E2E_ARGS` passes extra flags, e.g. `make test-e2e E2E_ARGS='-run TestCompatibility/dispatch'`.
The suite shares one Gitea and regular runner. Cache and ephemeral scenarios use isolated The `cli_*` scenarios build the binary and drive `register` and `daemon` as a user would, including `--once`,
`--labels` and shutdown with a running job. `failure_paths` covers failed, skipped and tolerated jobs, `job_timeout`
a job killed at the runner's deadline. `label_routing` and `capacity` cover scheduling, `reconnect` drops the
connection to Gitea through a proxy, and `host_mode` and `reusable_workflow` cover host labels and `workflow_call`.
`dispatch`, `failure_paths` and `reusable_workflow` run in every mode: through the daemon and `exec`, each on
Docker and Kubernetes.
The suite shares one Gitea, regular runner and k3s cluster. Cache and ephemeral scenarios use isolated
repository runners. A run path is `<workflow>@<ref>` and a log row is `<timestamp>Z <payload>`. repository runners. A run path is `<workflow>@<ref>` and a log row is `<timestamp>Z <payload>`.
## Windows tests
`make test-windows` cross-compiles the test binaries, installs Windows Server 2025 Core in a
[dockur/windows](https://github.com/dockur/windows) QEMU VM on the docker host and runs them there.
The host must be amd64 and allow privileged containers. `WINDOWS_TEST_TIMEOUT` sets the minutes
allowed for installing Windows and running the tests, 165 by default. The VM uses KVM when the host
has `/dev/kvm`, otherwise Windows is emulated and much slower. Docker-backed tests skip in the VM
and the race detector is off, it needs cgo. `WINDOWS_TEST_CACHE` names a directory to keep the
installed Windows in, so later runs skip the install. CI runs this on every pull request.
+10 -6
View File
@@ -12,19 +12,22 @@ ENV GOPROXY=${GOPROXY:-}
COPY . /opt/src/runner COPY . /opt/src/runner
WORKDIR /opt/src/runner WORKDIR /opt/src/runner
RUN make clean && make build RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
make clean && make build
### DIND VARIANT ### DIND VARIANT
# #
# #
FROM docker:29.9.1-dind AS dind FROM docker:29.9.3-dind AS dind
ARG VERSION=dev ARG VERSION=dev
LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner" LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}" LABEL org.opencontainers.image.version="${VERSION}"
RUN apk add --no-cache s6 bash git tzdata nftables # upgrade first: the docker base image can lag behind Alpine security fixes
RUN apk upgrade --no-cache && apk add --no-cache s6 bash git tzdata nftables
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh COPY scripts/run.sh /usr/local/bin/run.sh
@@ -39,7 +42,7 @@ ENTRYPOINT ["s6-svscan","/etc/s6"]
### DIND-ROOTLESS VARIANT ### DIND-ROOTLESS VARIANT
# #
# #
FROM docker:29.9.1-dind-rootless AS dind-rootless FROM docker:29.9.3-dind-rootless AS dind-rootless
ARG VERSION=dev ARG VERSION=dev
@@ -47,7 +50,8 @@ LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}" LABEL org.opencontainers.image.version="${VERSION}"
USER root USER root
RUN apk add --no-cache s6 bash git tzdata nftables # upgrade first: the docker base image can lag behind Alpine security fixes
RUN apk upgrade --no-cache && apk add --no-cache s6 bash git tzdata nftables
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh COPY scripts/run.sh /usr/local/bin/run.sh
@@ -73,7 +77,7 @@ ARG VERSION=dev
LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner" LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}" LABEL org.opencontainers.image.version="${VERSION}"
RUN apk add --no-cache tini bash git tzdata RUN apk upgrade --no-cache && apk add --no-cache tini bash git tzdata
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh COPY scripts/run.sh /usr/local/bin/run.sh
+10 -5
View File
@@ -1,13 +1,14 @@
DIST := dist DIST := dist
EXECUTABLE := gitea-runner
GO ?= go GO ?= go
EXECUTABLE := gitea-runner$(shell $(GO) env GOEXE)
HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" ) HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" )
PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.9-immutable@sha256:87f42f37fa85b421664c30b8f5f9cfc23920057f5e5fcadf0b6f68d9edc6d415 # renovate: datasource=docker PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.9-immutable@sha256:87f42f37fa85b421664c30b8f5f9cfc23920057f5e5fcadf0b6f68d9edc6d415 # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:0e62e56b382ebf485bff1e51a05cbe27c708545a2e91425467fd31eb3e249521 # renovate: datasource=docker E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:657378e5cfd898e36f820459c60ef8e69be8fcda9fd354e4381a8b654d091bbc # renovate: datasource=docker
SERVICE_IMAGE ?= nginx:1.31.7-alpine@sha256:892bb2152464f0c5d8972593a5f5afaf8054552580b699d95f8ff4af08c2d33d # renovate: datasource=docker SERVICE_IMAGE ?= nginx:1.31.7-alpine@sha256:892bb2152464f0c5d8972593a5f5afaf8054552580b699d95f8ff4af08c2d33d # renovate: datasource=docker
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
E2E_K3S_IMAGE ?= rancher/k3s:v1.37.1-k3s1@sha256:ca7f37d993d82ef0dcdcfecb2e0e2618ea541dbaffc620c8cedebe01a82acd0d # renovate: datasource=docker E2E_K3S_IMAGE ?= rancher/k3s:v1.37.1-k3s1@sha256:ca7f37d993d82ef0dcdcfecb2e0e2618ea541dbaffc620c8cedebe01a82acd0d # renovate: datasource=docker
WINDOWS_TEST_IMAGE ?= dockurr/windows:6.06@sha256:16fd7c39a3e48bb946d29c59dc6015942e5b900a1fa5d7e5552053bfc48ee2ea # renovate: datasource=docker
GOFILES := $(shell find . -type f -name "*.go" -o -name "go.mod" ! -name "generated.*") GOFILES := $(shell find . -type f -name "*.go" -o -name "go.mod" ! -name "generated.*")
@@ -15,8 +16,8 @@ DOCKER_IMAGE ?= gitea/runner
DOCKER_TAG ?= nightly DOCKER_TAG ?= nightly
DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG) DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]4.0 # renovate: datasource=go GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]5.0 # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.8.0 # renovate: datasource=go GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.9.0 # renovate: datasource=go
GOTEST_FLAGS ?= -race -timeout 20m -parallel 8 GOTEST_FLAGS ?= -race -timeout 20m -parallel 8
@@ -150,11 +151,15 @@ coverage-report: ## turn coverage.txt from `make test` into .tmp/coverage.md
test-dind: ## run the daemon-facing tests against the built dind image or podman (TARGET=dind|dind-rootless|podman) test-dind: ## run the daemon-facing tests against the built dind image or podman (TARGET=dind|dind-rootless|podman)
@PODMAN_TEST_IMAGE=$(PODMAN_TEST_IMAGE) ./scripts/test-dind.sh $(TARGET) @PODMAN_TEST_IMAGE=$(PODMAN_TEST_IMAGE) ./scripts/test-dind.sh $(TARGET)
.PHONY: test-windows
test-windows: ## run the tests on Windows Server Core in a QEMU VM on the docker host
@GO=$(GO) WINDOWS_TEST_IMAGE=$(WINDOWS_TEST_IMAGE) ./scripts/test-windows.sh
E2E_CONCURRENCY ?= 8 E2E_CONCURRENCY ?= 8
.PHONY: test-e2e .PHONY: test-e2e
test-e2e: ## run Gitea compatibility tests against E2E_GITEA_IMAGE test-e2e: ## run Gitea compatibility tests against E2E_GITEA_IMAGE
@E2E_CONCURRENCY=$(E2E_CONCURRENCY) E2E_GITEA_IMAGE=$(E2E_GITEA_IMAGE) E2E_K3S_IMAGE=$(E2E_K3S_IMAGE) E2E_JOB_IMAGE=$(E2E_JOB_IMAGE) GO=$(GO) SERVICE_IMAGE=$(SERVICE_IMAGE) ./tools/test-e2e.sh @E2E_CONCURRENCY=$(E2E_CONCURRENCY) E2E_GITEA_IMAGE=$(E2E_GITEA_IMAGE) E2E_K3S_IMAGE=$(E2E_K3S_IMAGE) E2E_JOB_IMAGE=$(E2E_JOB_IMAGE) GO=$(GO) SERVICE_IMAGE=$(SERVICE_IMAGE) ./tools/test-e2e.sh $(E2E_ARGS)
.PHONY: install .PHONY: install
install: $(GOFILES) ## install the runner binary via `go install` install: $(GOFILES) ## install the runner binary via `go install`
+145 -215
View File
@@ -1,26 +1,28 @@
# Gitea Runner # Gitea Runner
Gitea Runner runs the jobs of Gitea Actions workflows. Register it with a Gitea instance to run jobs in Docker containers, Kubernetes pods or directly on the host.
## Installation ## Installation
### Prerequisites ### Prerequisites
On the runner: - Linux, macOS or Windows, or FreeBSD.
- `git` 2.34.1 or newer.
- `docker`, or `podman` with `podman.socket` enabled, for `docker` labels and Docker actions. See Docker's [install instructions](https://docs.docker.com/engine/install/).
- `git` 2.34.1 or newer on `PATH`, to clone reusable workflows, and actions whose host serves no archive of them. In the job image, or on the host for host jobs:
- `docker` or `podman` with `podman.socket` enabled, for docker mode, Docker actions and `docker://` steps. To install Docker CE, follow the official [install instructions](https://docs.docker.com/engine/install/).
Where steps run, in the job image or on the runner in host mode: - `bash` or `sh`, or `pwsh` or `powershell` on Windows.
- `bash` or `sh` for `run` steps, `pwsh` or `powershell` on a Windows host.
- `node` for JavaScript actions and `hashFiles()`. - `node` for JavaScript actions and `hashFiles()`.
- `git` 2.34.1 or newer for [`builtin:checkout`](docs/builtin-actions.md). - `git` 2.34.1 or newer for [`builtin:checkout`](docs/builtin-actions.md).
- `git-lfs` and `ssh` for `builtin:checkout`, only when its `lfs` or `ssh-key` input is used. - `git-lfs` for `builtin:checkout` with `lfs: true`.
- `ssh` for `builtin:checkout` with `ssh-key`.
Kubernetes mode needs [a few more tools](docs/kubernetes.md) in job images. Kubernetes mode needs [a few more tools](docs/kubernetes.md) in job images.
### Download pre-built binary ### Download pre-built binary
Visit [here](https://dl.gitea.com/gitea-runner/) and download the right version for your platform. Download the binary for your platform from [dl.gitea.com](https://dl.gitea.com/gitea-runner/).
### Build from source ### Build from source
@@ -36,7 +38,7 @@ make docker
## Quickstart ## Quickstart
Actions are disabled by default, so you need to add the following to the configuration file of your Gitea instance to enable it: Enable Actions in the configuration file of your Gitea instance:
```ini ```ini
[actions] [actions]
@@ -49,70 +51,72 @@ ENABLED=true
./gitea-runner register ./gitea-runner register
``` ```
And you will be asked to input: It asks for:
1. Gitea instance URL, like `http://192.168.8.8:3000/`. You should use your gitea instance ROOT_URL as the instance argument 1. The Gitea instance URL. Use its `ROOT_URL`, not `localhost` or `127.0.0.1`.
and you should not use `localhost` or `127.0.0.1` as instance IP; 2. A runner token, from `/admin/actions/runners` on your instance.
2. Runner token, you can get it from `http://192.168.8.8:3000/admin/actions/runners`; 3. A runner name, the hostname when left blank.
3. Runner name, you can just leave it blank; 4. The runner [labels](#labels), the defaults when left blank.
4. Runner labels, you can just leave it blank.
The process looks like: To register without prompts:
```text
INFO Registering runner, arch=amd64, os=darwin, version=0.1.5.
WARN Runner in user-mode.
INFO Enter the Gitea instance URL (for example, https://gitea.com/):
http://192.168.8.8:3000/
INFO Enter the runner token:
fe884e8027dc292970d4e0303fe82b14xxxxxxxx
INFO Enter the runner name (if set empty, use hostname: Test.local):
INFO Enter the runner labels, leave blank to use the default labels (comma-separated, for example, ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest):
INFO Registering runner, name=Test.local, instance=http://192.168.8.8:3000/, labels=[ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04 ubuntu-20.04:docker://docker.gitea.com/runner-images:ubuntu-20.04].
DEBU Successfully pinged the Gitea instance server
INFO Runner registered successfully.
```
You can also register with command line arguments.
```bash ```bash
./gitea-runner register --instance http://192.168.8.8:3000 --token <my_runner_token> --no-interactive ./gitea-runner register --instance https://gitea.example.com --token <token> --no-interactive
``` ```
If the registry succeed, it will run immediately. Next time, you could run the runner directly.
### Run ### Run
```bash ```bash
./gitea-runner daemon ./gitea-runner daemon
``` ```
## Deployment
### Windows service
Register the runner first, keeping `gitea-runner.exe`, `config.yaml` and the `.runner` file in one directory. Then install and start the service from an elevated Command Prompt:
```bat
sc.exe create gitea-runner binPath= "\"C:\gitea-runner\gitea-runner.exe\" daemon --config \"C:\gitea-runner\config.yaml\"" start= auto
sc.exe start gitea-runner
```
The service logs to `gitea-runner.log` next to the executable. Its account needs write access to that directory and access to the tools jobs use.
`sc.exe stop gitea-runner` lets running jobs finish within `runner.shutdown_timeout`. To allow the same on shutdown, raise the service's preshutdown timeout above it, here to six minutes. This takes effect after the next reboot:
```bat
reg.exe add HKLM\SYSTEM\CurrentControlSet\Services\gitea-runner /v PreshutdownTimeout /t REG_DWORD /d 360000 /f
```
### Run with docker ### Run with docker
The recommended Docker-in-Docker image runs its own Docker daemon for jobs. Jobs never get the host's Docker socket, which is equivalent to root access on the host. The recommended Docker-in-Docker image runs its own Docker daemon for jobs. Jobs never get the host's Docker socket, which is equivalent to root access on the host.
```bash ```bash
docker run --privileged -e GITEA_INSTANCE_URL=https://your_gitea.com -e GITEA_RUNNER_REGISTRATION_TOKEN=<your_token> \ docker run --privileged -e GITEA_INSTANCE_URL=https://your_gitea.com -e GITEA_RUNNER_REGISTRATION_TOKEN=<your_token> \
-v "$PWD/data:/data" -v runner-docker:/var/lib/docker --name my_runner gitea/runner:nightly-dind -v "$PWD/data:/data" -v runner-docker:/var/lib/docker --name my_runner gitea/runner:latest-dind
``` ```
The volumes keep the runner's registration and the job image cache across container recreation. The volumes keep the registration and the image cache across container recreation. The image reads more variables, such as `GITEA_RUNNER_LABELS` and `CONFIG_FILE`, see [scripts/run.sh](scripts/run.sh) and the [Docker example](examples/docker/README.md).
### Image flavours #### Image flavours
| Tag | Docker daemon for jobs | | Tag | Docker daemon for jobs |
| --- | --- | | --- | --- |
| `latest-dind` (recommended) | Bundled, needs `--privileged`. | | `latest-dind` (recommended) | Bundled, needs `--privileged`. |
| `latest-dind-rootless` | Bundled, runs as a non-root user (UID 1000) for stronger isolation. Needs `--privileged` and [rootless Docker](https://docs.docker.com/engine/security/rootless/) support on the host. Keeps its image cache in `/home/rootless/.local/share/docker`. | | `latest-dind-rootless` | Bundled, runs as UID 1000 for stronger isolation. Needs `--privileged` and [rootless Docker](https://docs.docker.com/engine/security/rootless/) support on the host. Its image cache is in `/home/rootless/.local/share/docker`. |
| `latest` | None. Mount the host's socket with `-v /var/run/docker.sock:/var/run/docker.sock`, or set `DOCKER_HOST`. | | `latest` | None. Mount the host's `/var/run/docker.sock`, or set `DOCKER_HOST`. |
Versioned and `nightly` tags carry the same suffixes, for example `nightly-dind`. The images target Docker, not Podman. Versioned and `nightly` tags take the same suffixes, like `nightly-dind`. The images target Docker, not Podman.
### Configuration ### Example deployments
The runner reads a YAML file. Without one, every option keeps its default. See the [examples](examples) directory for more deployments, among them systemd, Kubernetes and VMs.
## Configuration
The runner reads an optional YAML file:
```bash ```bash
./gitea-runner config init # write config.yaml, with no option set ./gitea-runner config init # write config.yaml, with no option set
@@ -120,37 +124,68 @@ The runner reads a YAML file. Without one, every option keeps its default.
./gitea-runner -c config.yaml daemon # -c also works on register and cache-server ./gitea-runner -c config.yaml daemon # -c also works on register and cache-server
``` ```
`config generate` prints [config.example.yaml](internal/pkg/config/config.example.yaml). Every value in it is commented out, so copy the lines you want to change into your own file and uncomment them. `config generate` prints [config.example.yaml](internal/pkg/config/config.example.yaml), which documents every option. Copy the lines you need into your file and uncomment them.
#### Editing a config file ### Editing a config file
`config` edits a file in place, which is handy in provisioning scripts: `config` edits a file in place and keeps its comments, which suits provisioning scripts:
```bash ```bash
./gitea-runner config set runner.capacity 4 ./gitea-runner config set runner.capacity 4
./gitea-runner config set runner.timeout 90m # written as 1h30m0s
./gitea-runner config set runner.envs.MY_VAR value ./gitea-runner config set runner.envs.MY_VAR value
./gitea-runner config add runner.labels 'ubuntu:docker://node:22' ./gitea-runner config add runner.labels 'ubuntu:docker://node:22'
./gitea-runner config remove runner.labels 'ubuntu:docker://node:22' ./gitea-runner config remove runner.labels 'ubuntu:docker://node:22'
./gitea-runner config get runner.labels ./gitea-runner config get runner.labels
``` ```
A key is its dotted YAML path. An unknown key, a value of the wrong type, or `add`/`remove` on anything but a list is refused before the file is touched. `set` replaces a whole list when you give it several values. Keys are dotted YAML paths. Unknown keys and invalid values are refused without touching the file. Without `-c`, it uses `config.yaml` from the working directory, then from the binary's directory.
An edit keeps the comments and the key order of the file. Indentation becomes two spaces, and a blank line between two values is dropped. ### Environment variables
`config get`, `set`, `add` and `remove` use `config.yaml` (or `config.yml`) from the working directory, then from the directory of the binary, and print their choice to stderr. `config init` writes `config.yaml` in the working directory, and refuses to overwrite an existing config without `--force`. Pass `-c` for another path. Config values can read the runner's environment as `${NAME}`, for example `capacity: ${RUNNER_CAPACITY}`. An unset variable fails the load, and `$${NAME}` stays a literal `${NAME}`.
#### Tool cache The override variables of earlier releases, such as `GITEA_RUNNER_CAPACITY`, are gone. Set those options in the file instead.
Setup actions like `setup-go` install tools into `RUNNER_TOOL_CACHE`, which is `/opt/hostedtoolcache` inside a job. `runner.tool_cache_mode` selects what backs it: ### Labels
| Mode | Tool cache | Trade-off | Labels decide which jobs a runner accepts and how it runs them. A job's `runs-on` is matched against the label names, and the first match picks where it runs.
| --- | --- | --- |
| `none` (default) | Per job, provided by the job image | A version the image lacks is downloaded in every job |
| `shared` | One volume reused by every job | Two jobs writing the same tool version at once corrupt it, so use it only with `runner.capacity: 1` |
With `none`, tools must come from the job image. Install them into `/opt/hostedtoolcache/<tool>/<version>/<arch>`, with an empty `<arch>.complete` file next to the directory: A label is written as `<name>[:<schema>[:<args>]]`, with one of these schemas:
- `docker://<image>` runs the job in a container of `<image>`.
- `kubernetes://<image>` runs it as a Kubernetes pod, see [docs/kubernetes.md](docs/kubernetes.md).
- `host`, the default, runs its steps directly on the runner's machine.
With these labels, `runs-on: ubuntu-latest` runs in a container and `runs-on: macos` on the host:
```text
ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,macos:host
```
[gitea/runner-images](https://gitea.com/gitea/runner-images) has suitable images. Jobs matching no label run in `runner.default_image`, or on the host when docker is unavailable.
Host jobs still need a Docker daemon for Docker actions and `docker://` steps. Windows hosts do not support those, as on GitHub. A step's background processes end with the job unless it opts out, see [host mode](docs/host-mode.md).
#### Registration vs config labels
Labels are set at registration, with `--labels` or the prompt. To change them later, use one of these, the leftmost set one wins:
```
daemon --labels / GITEA_RUNNER_LABELS > runner.labels in the config file > labels in the .runner file
```
Changed labels are saved to the `.runner` file and sent to Gitea on startup. When `runner.labels` is set, `register` uses it and ignores `--labels`.
### Tool cache
Setup actions like `setup-go` install tools into `RUNNER_TOOL_CACHE`. `runner.tool_cache_mode` selects what backs it:
| Mode | Tool cache |
| --- | --- |
| `none` (default) | Provided by the job image. Versions it lacks are downloaded in every job. |
| `shared` | One volume for all jobs. Only safe with `runner.capacity: 1`. |
To ship tools in a job image, install them to `/opt/hostedtoolcache/<tool>/<version>/<arch>` with an empty `<arch>.complete` file next to it, as the `-full` [runner images](https://gitea.com/gitea/runner-images) do:
```dockerfile ```dockerfile
RUN GO=$(curl -fsSL 'https://go.dev/dl/?mode=json' | grep -oP '"version": "\Kgo1\.26\.[0-9]*' | head -1); \ RUN GO=$(curl -fsSL 'https://go.dev/dl/?mode=json' | grep -oP '"version": "\Kgo1\.26\.[0-9]*' | head -1); \
@@ -161,83 +196,29 @@ RUN GO=$(curl -fsSL 'https://go.dev/dl/?mode=json' | grep -oP '"version": "\Kgo1
touch "${DIR}.complete" touch "${DIR}.complete"
``` ```
A workflow requesting a minor version, `go-version: "1.26"`, resolves to the newest matching version in the cache, so a patch update in the image still hits it. ### Proxy
Of the [runner images](https://gitea.com/gitea/runner-images), the `-full` flavour is the one that ships tools in this layout. Set the proxy in the runner's environment, for example with systemd `Environment=` or `docker run -e`:
`gitea-runner exec` reads no config file and takes `--tool-cache-mode` instead, defaulting to `none`. ```sh
http_proxy=http://proxy.example:3128
#### Environment variables https_proxy=http://proxy.example:3128
no_proxy=gitea.internal,.example.local
Earlier releases let a few environment variables (`GITEA_DEBUG`, `GITEA_TRACE`, `GITEA_RUNNER_CAPACITY`, `GITEA_RUNNER_FILE`, `GITEA_RUNNER_ENVIRON`, `GITEA_RUNNER_ENV_FILE`) override parts of the config. They are gone, use the YAML file for all settings. Config values can read the runner's environment as `${NAME}`, for example `capacity: ${RUNNER_CAPACITY}`. An unset `NAME` fails the load, and `$${NAME}` is a literal `${NAME}`. The Docker images still read their own variables, such as `RUNNER_STATE_FILE`, see [scripts/run.sh](scripts/run.sh) and the container documentation below.
### Labels
Labels decide **which jobs a runner accepts** and **how it runs them**. A job's `runs-on` is matched against the runner's label names; the first match wins and selects the execution environment for that job.
A label is written as:
```text
<name>[:<schema>[:<args>]]
``` ```
| Part | Meaning | The runner uses it and passes it on to jobs. Jobs always reach the cache server, `localhost`, their service containers and a `tcp://` Docker daemon directly. Add Gitea to `no_proxy` if it should be reached directly too.
| --- | --- |
| `name` | The name a workflow refers to in `runs-on`, e.g. `ubuntu-latest`. |
| `schema` | `docker`, `kubernetes` or `host`. Defaults to `host` when omitted. |
| `args` | Only used by the `docker` and `kubernetes` schemas: the image to run the job in. |
Three schemas are supported: To change the proxy for jobs, use `runner.envs`, a step's `env:` or `$GITHUB_ENV`. Workflow and job level `env:` have no effect on it.
- **`docker://<image>`** — the job runs inside a container created from `<image>`: The Docker daemon pulls images with its own proxy settings. The `dind` images pass the variables above to it, for other daemons see [the Docker documentation](https://docs.docker.com/engine/daemon/proxy/).
```text A password in the proxy URL is hidden in logs, but every step can read it.
ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest
```
- **`kubernetes://<image>`** — the job runs as a Kubernetes pod created from `<image>`, see **[docs/kubernetes.md](docs/kubernetes.md)**: ## Jobs
```text ### Service containers
ubuntu-latest:kubernetes://docker.gitea.com/runner-images:ubuntu-latest
```
- **`host`** — the job's steps run directly on the machine the runner is on, using the tools installed there: When a service declares a healthcheck, in its image or `options`, the job waits for it to become healthy:
```text
macos:host
```
So with the labels
```text
ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,macos:host
```
a workflow with `runs-on: ubuntu-latest` is executed in the `runner-images:ubuntu-latest` container, and one with `runs-on: macos` is executed directly on the host.
Names may themselves contain a colon, for example `pool:e57e18d4-10d4-406f-93bf-60f127221bdd`. Only `host`, `docker` and `kubernetes` are treated as schemas.
Jobs run on `runner.default_image` when no runner label matches their `runs-on`: as a pod on runners with `kubernetes` labels, otherwise in a container, or on the host when docker is unavailable. Suitable images are listed at [gitea/runner-images](https://gitea.com/gitea/runner-images).
Labels are chosen at registration time (`--labels`, or the interactive prompt) and can be changed afterwards by editing `runner.labels` in the config file, or in the Gitea UI under the runner's settings.
#### Registration vs config labels
If `runner.labels` is set in the YAML file, those labels are used during `register` and the `--labels` CLI flag is ignored.
The `daemon` command also accepts `--labels` (which defaults to the `GITEA_RUNNER_LABELS` environment variable), so the labels of an already registered runner can be changed without deleting its registration file. The most explicit source wins:
```
--labels / GITEA_RUNNER_LABELS > runner.labels in the config file > labels in the .runner file
```
Whenever the resulting labels differ from the ones in the registration file, they are written back to it and re-declared to the Gitea instance on startup.
> **Note:** A runner that only exposes `host` labels still needs access to a Docker daemon (e.g. a mounted `/var/run/docker.sock`) whenever a job uses a `docker://` action or a service container. `host` labels only change where the job's own steps run; container-based steps and actions are still executed with Docker.
#### Service containers
A job's `services` are started before its steps run. When a service's image or its `options` declare a healthcheck, the runner waits for it to report healthy, so a workflow does not have to poll for its own services:
```yaml ```yaml
services: services:
@@ -249,63 +230,32 @@ services:
--health-retries 10 --health-retries 10
``` ```
A service that reports unhealthy fails the job right away, with its container log. One that never becomes healthy fails it after `container.service_ready_timeout` (default `5m`, negative disables the wait). A service that exits without declaring a healthcheck only gets its log and a warning. A service that turns unhealthy, or is not healthy within `container.service_ready_timeout`, fails the job. Steps reach a service by its id and port, like `psql -h postgres -p 5432`.
A job in a container reaches a service by its id on the job network, on the port the service listens on, for example `psql -h postgres -p 5432`. The started containers also fill the `job` context: `job.container.{id,network}` and `job.services.<id>.{id,network,ports}`, where `ports` maps a container port to the host port Docker published it on, for the services that publish one. Unlike GitHub, jobs running on the host start no services. Give them a `container:` to use services.
Unlike GitHub, a job whose steps run on the host (a `host` label without `container:`) starts no service containers, so `job.services` and `job.container` stay empty. Give such a job a `container:` when it needs services. ### Docker from a job (`GITEA_DOCKER_WORKSPACE`)
#### Docker from a job (`GITEA_DOCKER_WORKSPACE`) Containers a job starts can bind-mount the workspace by its path, like `docker run -v "$PWD:/src"`. Where the Docker daemon sees another path, for example a remote `DOCKER_HOST`, prefix the bind with `GITEA_DOCKER_WORKSPACE`:
Containers a job starts through the Docker socket can bind-mount the workspace by the job's own path, as on a host, for example `docker run -v "$PWD:/src"`. Without the Docker proxy, for example with a remote `DOCKER_HOST`, use `GITEA_DOCKER_WORKSPACE`, the path the daemon sees, as the prefix of workspace binds, with `.` as the fallback for local use:
```yaml ```yaml
volumes: volumes:
- ${GITEA_DOCKER_WORKSPACE:-.}/data:/app/data - ${GITEA_DOCKER_WORKSPACE:-.}/data:/app/data
``` ```
Linux container jobs use a Docker proxy when socket sharing and permissions allow it. Other setups use the daemon socket directly. Host jobs use their existing Docker access, where `.` works. The proxy removes the containers, networks and volumes a job creates, `docker compose` services included, after post steps and the completed hook. Set `container.sweep: false` when jobs deploy services to the runner's Docker daemon. Set `container.sweep: true` to remove the containers, networks and volumes most Linux container jobs create when they end, `docker compose` services included. Leave it off if jobs deploy services to the runner's Docker daemon.
#### Proxy ### Built-in actions (`uses: builtin:<name>`)
Set these variables in the runner's environment, with systemd `Environment=`, `docker run -e`, or Kubernetes `env:`: `uses: builtin:checkout` checks out a repository without downloading `actions/checkout` or needing Node in the job image, see [docs/builtin-actions.md](docs/builtin-actions.md).
```sh ### Caching (`actions/cache`)
http_proxy=http://proxy.example:3128
https_proxy=http://proxy.example:3128
no_proxy=gitea.internal,.example.local
```
The runner uses them for its own requests and gives them to every job, in lower and upper case. Each runner runs its own cache server. As on GitHub, a job restores caches of its own ref, its pull request's base branch and the default branch.
These hosts are added to `no_proxy` for jobs, so they are always reached directly: #### Sharing a cache between runners
- the built-in cache server, whose address is assigned at startup Run one `gitea-runner cache-server` and point every runner at it with the same secret, for example from `openssl rand -hex 32`:
- `localhost`, `127.0.0.1` and `::1`
- the job's service containers
- the Docker daemon, when it is reached over `tcp://`
Gitea is not added. Add it to `no_proxy` yourself if it should be reached directly.
For one step, set it in the step's `env:`, for the rest of a job, write it to `$GITHUB_ENV`. Setting it at workflow or job level or in `container.env` has no effect. To change it for the whole runner, set it in `runner.envs`. A `no_proxy` set there is added to the list above instead of replacing it.
Images are pulled by the Docker daemon, which needs its own proxy setting. In the `dind` images the daemon runs in the same container and reads the variables above. For any other daemon, see [the Docker documentation](https://docs.docker.com/engine/daemon/proxy/). The runner logs a warning at startup if it has a proxy and the daemon does not.
Dockerfile actions are built with these variables as build arguments, so their `RUN` steps can reach the network.
A password in a proxy URL is hidden in job logs. Any step can still read it, because the step is given the proxy URL in its environment.
#### Built-in actions (`uses: builtin:<name>`)
Use `uses: builtin:checkout` to check out a repository without downloading `actions/checkout` or needing Node in the job image, see **[docs/builtin-actions.md](docs/builtin-actions.md)**.
#### Caching (`actions/cache`)
Each runner starts its own cache server, so runners do not share cached entries. A runner in a container on the jobs' Docker daemon attaches itself to each job network, so jobs reach its cache without extra setup.
**Sharing a cache between runners**
Run one `gitea-runner cache-server` and point every runner at it, using the same secret everywhere, for example from `openssl rand -hex 32`:
```yaml ```yaml
# cache-server.yaml # cache-server.yaml
@@ -326,13 +276,11 @@ cache:
external_secret: "<secret>" external_secret: "<secret>"
``` ```
`--dir`, `--host` and `--port` override the matching `cache` keys. Eviction settings take effect on the cache server, not on the runners. Runners can also share one `cache.dir` on a file system with working file locks, at the cost of slower requests.
Runners can also share one `cache.dir` on a file system with working file locks, at the cost of slower cache requests. #### Storing the cache in an S3-compatible bucket
**Storing the cache in an S3-compatible bucket** Runners can share a cache in an S3-compatible bucket, such as AWS S3, MinIO or RustFS, without a cache server:
The cache can also be stored in an S3-compatible bucket such as AWS S3, MinIO or RustFS, with no cache server. Runners with the same settings use the same cache:
```yaml ```yaml
cache: cache:
@@ -346,66 +294,48 @@ cache:
secret_key_file: "/run/secrets/cache-s3-secret-key" secret_key_file: "/run/secrets/cache-s3-secret-key"
``` ```
A cache saved on one runner can be restored on the others within 30 seconds. The eviction settings below do not apply to the bucket, so add lifecycle rules to it that expire old caches and abort incomplete multipart uploads. Eviction does not apply to the bucket. Add lifecycle rules that expire old caches and abort incomplete multipart uploads.
**Eviction** #### Eviction
Entries not read or written within `retention` (default `168h`) are removed. A repository over `repo_size_limit` (default `10GB`) loses its least recently used entries, and `size_limit` (off by default) caps the whole cache the same way. Entries in use are never removed. The cache also keeps 1024 MiB free on its volume, or `health_check.min_free_disk_space_mb` when health checks are enabled. See [config.example.yaml](internal/pkg/config/config.example.yaml) for all options. Entries unused for `cache.retention` (default `168h`) are removed. A repository over `cache.repo_size_limit` (default `10GB`) loses its least recently used entries. See [config.example.yaml](internal/pkg/config/config.example.yaml) for more limits.
**Cache service v2** #### Cache service v2
`actions/cache` v3.4.0, v4.2.0 and later use the cache service v2 API, which the runner serves by default. These actions fall back to v1 on hosts they do not recognize as GitHub, so the runner removes that check from them while a job runs. This also lets the stock `actions/upload-artifact` v4.4.0 and `actions/download-artifact` v4.1.8 and later work without the `gitea-upload-artifact` fork. With `runner.patch_actions: false`, the cache stays on v1 and the artifact actions fail. The runner serves the cache service v2 API used by current `actions/cache` releases. It patches them to use it on Gitea, which also makes the stock `actions/upload-artifact` and `actions/download-artifact` v4 work. Artifact calls then go through the cache server, so jobs must be able to reach it.
With v2, artifact calls also go through the cache server, so jobs must be able to reach it. `cache.v2: false` sends them to Gitea directly, unless the Gitea URL has a path or `runner.insecure` is set with HTTPS. ### Job hooks (`runner.hooks.job_started`, `runner.hooks.job_completed`)
#### Official Docker image Hook scripts run where the job's steps run, before the first step and after the last. They match GitHub's `ACTIONS_RUNNER_HOOK_JOB_STARTED` and `ACTIONS_RUNNER_HOOK_JOB_COMPLETED`.
Besides `GITEA_INSTANCE_URL` and `GITEA_RUNNER_REGISTRATION_TOKEN`, the image entrypoint supports optional variables such as `CONFIG_FILE` (passed through as `-c`), `GITEA_RUNNER_LABELS`, `GITEA_RUNNER_EPHEMERAL`, `GITEA_RUNNER_ONCE`, `GITEA_RUNNER_NAME`, `GITEA_MAX_REG_ATTEMPTS`, `RUNNER_STATE_FILE`, and `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`. See [scripts/run.sh](scripts/run.sh) for exact behavior. Use them for per-job setup no workflow should carry, like registry logins or masking runner-wide secrets. They block the job, and a non-zero exit fails it. See [docs/job-hooks.md](docs/job-hooks.md).
For a fuller container-oriented walkthrough, see [examples/docker](examples/docker/README.md). ### Secret masking
While the runner is idle it cleans up after earlier jobs: Secrets and `::add-mask::` values are hidden in job logs, [local job logs](#local-job-logs-logjobdir), job summaries and container names. As on GitHub, a job output containing one is dropped with a warning.
- when `container.bind_workdir` is enabled, stale task workspace directories older than `runner.workdir_cleanup_age` are removed (default: `24h`; set `0` to disable)
- only purely numeric subdirectories under `container.workdir_parent` are treated as task workspaces and may be removed
- cleanup assumes `container.workdir_parent` is not shared across multiple runners
- on runners that use docker, per-job networks and volumes left behind by jobs the runner did not live to tear down are removed once unused and 10 minutes old, identified by the `com.gitea.runner.uuid` label carrying this runner's uuid
- on runners with `kubernetes` labels, job pods with that label older than `runner.workdir_cleanup_age` are removed the same way, together with their Secrets
- cleanup runs every `runner.idle_cleanup_interval` (default: `10m`; set `0` to disable), and setting either knob to `0` disables all of the above
#### Post-task script (`runner.post_task_script`) ## Operation
Optional host script that runs **after** each task's built-in cleanup (post-steps, container teardown, bind-workdir removal). Use it for extra machine housekeeping — Docker pruning, disk cleanup, and similar. ### Idle cleanup
**While the script runs, the runner stops task heartbeats and stays offline from Gitea's perspective until the script exits (or hits `runner.post_task_script_timeout`, default `5m`).** A script that blocks without exiting keeps the runner from taking new work for up to that timeout. Script output goes to the runner log, not the job log; a non-zero exit is warned but does not change the job result. While idle, the runner removes leftovers of earlier jobs every `runner.idle_cleanup_interval` (default `10m`):
On Windows, use `.exe`, `.bat`, or `.cmd` paths; **PowerShell (`.ps1`) is not supported yet** as the configured path — wrap commands in a `.cmd` file instead. - Workspaces older than `runner.workdir_cleanup_age` (default `24h`), when `container.bind_workdir` is on. This assumes no other runner shares `container.workdir_parent`.
- Unused Docker networks and volumes of jobs the runner did not tear down.
- Kubernetes job pods older than `runner.workdir_cleanup_age`, with their Secrets.
See **[docs/post-task-script.md](docs/post-task-script.md)** for lifecycle details, environment variables, timeout interaction, and platform notes. Set either option to `0` to turn cleanup off.
#### Job hooks (`runner.hooks.job_started`, `runner.hooks.job_completed`) ### Post-task script (`runner.post_task_script`)
Optional scripts that run **inside the job environment** (the job container, or the host in host mode), before the job's first step and after its last one. They are the equivalent of GitHub's `ACTIONS_RUNNER_HOOK_JOB_STARTED` / `ACTIONS_RUNNER_HOOK_JOB_COMPLETED`, which are read when the settings are unset. A host script that runs after each task's cleanup, for housekeeping like Docker pruning. The runner takes no new jobs until it exits or hits `runner.post_task_script_timeout` (default `5m`). Its output goes to the runner log.
Because they run where the steps run and see the job's environment, they are the place for per-job setup no workflow should have to carry: registry logins, mirror configuration, or masking runner-wide secrets with `::add-mask::`. Their output is part of the job log and is scanned for workflow commands, and they can export to the job through `$GITHUB_ENV` and `$GITHUB_PATH`. On Windows, use an `.exe`, `.bat` or `.cmd` file, PowerShell scripts are not supported. See [docs/post-task-script.md](docs/post-task-script.md).
Both hooks are synchronous and block the job while they run. Either one exiting non-zero fails the job, and there is no per-hook timeout. ### Local job logs (`log.job.dir`)
See **[docs/job-hooks.md](docs/job-hooks.md)** for the execution order, environment, and platform notes. Set `log.job.dir` to keep a copy of every task's log, masked as in Gitea, with the job's result on the last line. Copies are kept for `log.job.retention` (default `168h`), so keep it above `runner.timeout`. Only the runner's user can read them.
#### OpenTelemetry (`OTEL_*`) ### OpenTelemetry (`OTEL_*`)
Set `OTEL_EXPORTER_OTLP_ENDPOINT` in the runner's environment to send a trace of every job, and the runner's metrics, to an OpenTelemetry collector. See **[docs/telemetry.md](docs/telemetry.md)**. Set `OTEL_EXPORTER_OTLP_ENDPOINT` in the runner's environment to send job traces and runner metrics to an OpenTelemetry collector. See [docs/telemetry.md](docs/telemetry.md).
#### Local job logs (`log.job.dir`)
Set `log.job.dir` to a path and the runner writes a copy of every task's log there as `<start time>-task-<id>.log`: the rows exactly as Gitea received them, with the same secrets masked and the job's result on the last line. Off by default, and what Gitea shows does not change.
#### Secret masking
A job's secrets and its `::add-mask::` values are hidden from what the runner writes and uploads: the job log, the local copy above, job summaries, and the names of the containers it creates. A job output carrying one is skipped with a warning rather than sent masked, as GitHub does, so a downstream `needs.<job>.outputs.<name>` reading it is empty.
`log.job.retention` (default `168h`) is how long a log is kept, expired ones being deleted as new tasks start, and `log.job.max_size` (default `1GB`) caps one log. Keep `retention` above `runner.timeout` so a long job cannot outlive its own log, and prefer local disk, the file is written while the job runs. Only the runner's own user can read it.
### Example Deployments
Check out the [examples](examples) directory for sample deployment types.
-2
View File
@@ -6,7 +6,5 @@
// //
// Inspired by https://github.com/sp-ricard-valverde/github-act-cache-server // Inspired by https://github.com/sp-ricard-valverde/github-act-cache-server
// //
// TODO: Authorization
// TODO: Restrictions for accessing a cache, see https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache
// TODO: Force deleting cache entries, see https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#force-deleting-cache-entries // TODO: Force deleting cache entries, see https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#force-deleting-cache-entries
package artifactcache package artifactcache
+121 -74
View File
@@ -15,6 +15,7 @@ import (
"encoding/json/v2" "encoding/json/v2"
"errors" "errors"
"fmt" "fmt"
"io"
"net" "net"
"net/http" "net/http"
"net/http/httputil" "net/http/httputil"
@@ -58,6 +59,10 @@ type credKey struct{}
type JobCredential struct { type JobCredential struct {
Repo string `json:"repo"` Repo string `json:"repo"`
Ref string `json:"ref"`
BaseRef string `json:"base_ref"`
DefaultRef string `json:"default_ref"`
// Results is the instance whose artifact service this server forwards for the job, and // Results is the instance whose artifact service this server forwards for the job, and
// InsecureTLS how the runner reaches it; see results.go. // InsecureTLS how the runner reaches it; see results.go.
Results string `json:"-"` Results string `json:"-"`
@@ -67,6 +72,10 @@ type JobCredential struct {
PublicURL string `json:"public_url"` PublicURL string `json:"public_url"`
} }
func (cred JobCredential) owns(cache *Cache) bool {
return cache.Repo == cred.Repo && cache.Ref == cred.Ref
}
// credEntry holds a registered job's credential along with an active // credEntry holds a registered job's credential along with an active
// registration count. RegisterJob is reference-counted so that if two tasks // registration count. RegisterJob is reference-counted so that if two tasks
// briefly share an ACTIONS_RUNTIME_TOKEN — e.g. a runner that retries a task // briefly share an ACTIONS_RUNTIME_TOKEN — e.g. a runner that retries a task
@@ -95,7 +104,8 @@ type Handler struct {
stopSync context.CancelFunc stopSync context.CancelFunc
syncing sync.WaitGroup syncing sync.WaitGroup
outboundIP string outboundIP string
instanceTransport http.RoundTripper
// internalSecret guards /_internal/{register,revoke}. When set, a remote // internalSecret guards /_internal/{register,revoke}. When set, a remote
// runner can use these endpoints to pre-register per-job // runner can use these endpoints to pre-register per-job
@@ -115,7 +125,7 @@ type Handler struct {
policy Policy policy Policy
// freeDisk is a field so tests can drive evictForFreeSpace without a full volume. // freeDisk is a field so tests can drive makeRoom without a full volume.
freeDisk func(string) (uint64, error) freeDisk func(string) (uint64, error)
} }
@@ -134,19 +144,21 @@ type Options struct {
// in-process RegisterJob method directly. // in-process RegisterJob method directly.
InternalSecret string InternalSecret string
Policy Policy Policy Policy
Logger *logging.Logger Logger *logging.Logger
InstanceTransport http.RoundTripper // reaches the instance artifact requests are forwarded to, nil for http.DefaultTransport
} }
// StartHandler starts an embedded cache or forwards cache requests to Upstream. // StartHandler starts an embedded cache or forwards cache requests to Upstream.
func StartHandler(opts Options) (*Handler, error) { func StartHandler(opts Options) (*Handler, error) {
dir, logger := opts.Dir, opts.Logger dir, logger := opts.Dir, opts.Logger
h := &Handler{ h := &Handler{
creds: make(map[string]*credEntry), creds: make(map[string]*credEntry),
internalSecret: opts.InternalSecret, internalSecret: opts.InternalSecret,
policy: opts.Policy.withDefaults(), policy: opts.Policy.withDefaults(),
freeDisk: disk.FreeBytes, freeDisk: disk.FreeBytes,
dbLock: make(chan struct{}, 1), dbLock: make(chan struct{}, 1),
instanceTransport: opts.InstanceTransport,
} }
if logger == nil { if logger == nil {
@@ -652,7 +664,7 @@ func (h *Handler) find(w http.ResponseWriter, r *http.Request) {
} }
defer db.Close() defer db.Close()
cache, err := h.lookupCache(db, cred.Repo, keys, version) cache, err := h.lookupCache(db, cred, keys, version)
if err != nil { if err != nil {
h.responseJSON(w, r, 500, err) h.responseJSON(w, r, 500, err)
return return
@@ -668,26 +680,27 @@ func (h *Handler) find(w http.ResponseWriter, r *http.Request) {
}) })
} }
// lookupCache returns the entry to restore for these keys, or (nil, nil) when there is none: // lookupCache returns the entry to restore or (nil, nil), dropping matches whose blob was pruned.
// either nothing matched, or the match had lost its blob to a prune, in which case the dangling func (h *Handler) lookupCache(db *cacheDB, cred JobCredential, keys []string, version string) (*Cache, error) {
// entry is dropped on the way out. for {
func (h *Handler) lookupCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) { cache, err := findCache(db, cred, keys, version)
cache, err := findCache(db, repo, keys, version) if err != nil || cache == nil {
if err != nil || cache == nil { return nil, err
return nil, err }
ok, err := h.storage.Exist(cache.ID)
if err != nil {
return nil, err
}
if ok {
// Handing out a download URL counts as access, or eviction could drop the entry between
// this call and the GET that follows it.
h.touch(db, cache)
return cache, nil
}
if !h.deleteCache(db, cache) {
return nil, nil //nolint:nilnil // absence is not an error here
}
} }
ok, err := h.storage.Exist(cache.ID)
if err != nil {
return nil, err
}
if !ok {
h.deleteCache(db, cache)
return nil, nil //nolint:nilnil // absence is not an error here
}
// Handing out a download URL counts as access, or eviction could drop the entry between
// this call and the GET that follows it.
h.touch(db, cache)
return cache, nil
} }
// POST /_apis/artifactcache/caches // POST /_apis/artifactcache/caches
@@ -699,7 +712,7 @@ func (h *Handler) reserve(w http.ResponseWriter, r *http.Request) {
return return
} }
cache := &Cache{Repo: cred.Repo, Key: api.Key, Version: api.Version, Size: api.Size} cache := &Cache{Repo: cred.Repo, Ref: cred.Ref, Key: api.Key, Version: api.Version, Size: api.Size}
if cache.Size == 0 { if cache.Size == 0 {
cache.Size = -1 cache.Size = -1
} }
@@ -710,9 +723,10 @@ func (h *Handler) reserve(w http.ResponseWriter, r *http.Request) {
} }
defer db.Close() defer db.Close()
now := time.Now().Unix() now := time.Now()
cache.CreatedAt = now cache.CreatedAt = now.Unix()
cache.UsedAt = now cache.CreatedAtNanos = now.Nanosecond()
cache.UsedAt = now.Unix()
if err := h.insertCache(db, cache); err != nil { if err := h.insertCache(db, cache); err != nil {
h.responseJSON(w, r, 500, err) h.responseJSON(w, r, 500, err)
return return
@@ -747,7 +761,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
return return
} }
if cache.Repo != cred.Repo { if !cred.owns(cache) {
h.responseJSON(w, r, 403, fmt.Errorf("cache %d: forbidden", id)) h.responseJSON(w, r, 403, fmt.Errorf("cache %d: forbidden", id))
return return
} }
@@ -762,6 +776,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
h.responseJSON(w, r, 400, err) h.responseJSON(w, r, 400, err)
return return
} }
h.makeRoom(r.ContentLength)
if err := h.storage.Write(cache.ID, start, r.Body); err != nil { if err := h.storage.Write(cache.ID, start, r.Body); err != nil {
h.responseJSON(w, r, 500, err) h.responseJSON(w, r, 500, err)
return return
@@ -778,6 +793,17 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
h.responseJSON(w, r, 400, err) h.responseJSON(w, r, 400, err)
return return
} }
var request struct {
Size *int64 `json:"size"`
}
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
if err == nil && len(body) > 0 {
err = json.Unmarshal(body, &request)
}
if err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err)
return
}
db, err := h.openDB() db, err := h.openDB()
if err != nil { if err != nil {
@@ -795,7 +821,7 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
return return
} }
if cache.Repo != cred.Repo { if !cred.owns(cache) {
h.responseJSON(w, r, 403, fmt.Errorf("cache %d: forbidden", id)) h.responseJSON(w, r, 403, fmt.Errorf("cache %d: forbidden", id))
return return
} }
@@ -805,6 +831,14 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
return return
} }
if request.Size != nil {
if *request.Size < 0 || cache.Size >= 0 && cache.Size != *request.Size {
h.responseJSON(w, r, http.StatusBadRequest, errors.New("invalid cache size"))
return
}
cache.Size = *request.Size
}
db.Close() db.Close()
if err := h.commitCache(cache); err != nil { if err := h.commitCache(cache); err != nil {
@@ -845,11 +879,8 @@ func (h *Handler) commitCache(cache *Cache) error {
if h.s3 != nil { if h.s3 != nil {
return nil // lifecycle rules on the bucket bound its size return nil // lifecycle rules on the bucket bound its size
} }
// A commit is the only thing that grows the store, so the only thing that can push the
// volume under the floor.
h.evictRepo(db, cache.Repo) h.evictRepo(db, cache.Repo)
h.evictTotal(db) h.evictTotal(db)
h.evictForFreeSpace(db)
return nil return nil
} }
@@ -1050,25 +1081,35 @@ func (h *Handler) signedArtifactURL(cred JobCredential, cacheID uint64, exp time
return h.signedURL(cred, apiPath+"/artifacts", "", cacheID, exp) return h.signedURL(cred, apiPath+"/artifacts", "", cacheID, exp)
} }
// if not found, return (nil, nil) instead of an error. // findCache tries every key in one ref before the next ref, and returns (nil, nil) on a miss.
func findCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) { func findCache(db *cacheDB, cred JobCredential, keys []string, version string) (*Cache, error) {
for _, prefix := range keys { refs := []string{cred.Ref}
// if a key in the list matches exactly, don't return partial matches for _, ref := range []string{cred.BaseRef, cred.DefaultRef} {
exact, err := findExactCache(db, repo, prefix, version, true) if ref != "" && !slices.Contains(refs, ref) {
if err != nil { refs = append(refs, ref)
return nil, err
} }
if exact != nil { }
return exact, nil caches, err := db.find(func(cache *Cache) bool {
} return cache.Repo == cred.Repo && cache.Version == version && cache.Complete && slices.Contains(refs, cache.Ref) &&
caches, err := db.find(func(cache *Cache) bool { slices.ContainsFunc(keys, func(prefix string) bool { return strings.HasPrefix(cache.Key, prefix) })
return cache.Repo == repo && strings.HasPrefix(cache.Key, prefix) && cache.Version == version && cache.Complete })
}) if err != nil {
if err != nil { return nil, fmt.Errorf("find cache: %w", err)
return nil, fmt.Errorf("find cache: %w", err) }
} for _, ref := range refs {
if cache := latest(caches, func(cache *Cache) int64 { return cache.CreatedAt }); cache != nil { for _, prefix := range keys {
return cache, nil var exact, partial []*Cache
for _, cache := range caches {
if cache.Ref == ref && cache.Key == prefix {
exact = append(exact, cache)
} else if cache.Ref == ref && strings.HasPrefix(cache.Key, prefix) {
partial = append(partial, cache)
}
}
// if a key in the list matches exactly, don't return partial matches
if cache := cmp.Or(latest(exact, (*Cache).createdAtNano), latest(partial, (*Cache).createdAtNano)); cache != nil {
return cache, nil
}
} }
} }
return nil, nil //nolint:nilnil // pre-existing issue from nektos/act return nil, nil //nolint:nilnil // pre-existing issue from nektos/act
@@ -1078,7 +1119,9 @@ func latest(caches []*Cache, stamp func(*Cache) int64) *Cache {
if len(caches) == 0 { if len(caches) == 0 {
return nil return nil
} }
return slices.MaxFunc(caches, func(left, right *Cache) int { return cmp.Compare(stamp(left), stamp(right)) }) return slices.MaxFunc(caches, func(left, right *Cache) int {
return cmp.Or(cmp.Compare(stamp(left), stamp(right)), cmp.Compare(left.ID, right.ID))
})
} }
// findExactCache returns the entry for exactly this key and version, or (nil, nil) if there is // findExactCache returns the entry for exactly this key and version, or (nil, nil) if there is
@@ -1088,16 +1131,16 @@ func latest(caches []*Cache, stamp func(*Cache) int64) *Cache {
// A completed entry is the one to restore, sorted by when it was written. An incomplete one is a // A completed entry is the one to restore, sorted by when it was written. An incomplete one is a
// reservation being uploaded to, sorted by when it was last written to, because the upload route // reservation being uploaded to, sorted by when it was last written to, because the upload route
// touches UsedAt on every part. // touches UsedAt on every part.
func findExactCache(db *cacheDB, repo, key, version string, complete bool) (*Cache, error) { func findExactCache(db *cacheDB, cred JobCredential, key, version string, complete bool) (*Cache, error) {
caches, err := db.find(func(cache *Cache) bool { caches, err := db.find(func(cache *Cache) bool {
return cache.Key == key && cache.Repo == repo && cache.Version == version && cache.Complete == complete return cred.owns(cache) && cache.Key == key && cache.Version == version && cache.Complete == complete
}) })
if err != nil { if err != nil {
return nil, fmt.Errorf("find cache: %w", err) return nil, fmt.Errorf("find cache: %w", err)
} }
stamp := func(cache *Cache) int64 { return cache.UsedAt } stamp := func(cache *Cache) int64 { return cache.UsedAt }
if complete { if complete {
stamp = func(cache *Cache) int64 { return cache.CreatedAt } stamp = (*Cache).createdAtNano
} }
return latest(caches, stamp), nil return latest(caches, stamp), nil
} }
@@ -1131,9 +1174,6 @@ func (h *Handler) touchCache(id uint64, requireIncomplete bool) error {
defer db.Close() defer db.Close()
cache, err := db.get(id) cache, err := db.get(id)
if errors.Is(err, errCacheNotFound) {
return nil
}
if err != nil { if err != nil {
return err return err
} }
@@ -1218,23 +1258,29 @@ func (h *Handler) gcCache() {
h.evictExpired(db) h.evictExpired(db)
h.evictSuperseded(db) h.evictSuperseded(db)
h.evictOversized(db) h.evictOversized(db)
h.evictForFreeSpace(db) _ = db.Close()
h.makeRoom(0)
} }
// evictForFreeSpace bounds the volume itself, so it also covers bytes the cache never // makeRoom evicts the least recently used entries to keep need bytes above the floor, as GitHub saves new over old.
// accounted for. func (h *Handler) makeRoom(need int64) {
func (h *Handler) evictForFreeSpace(db *cacheDB) {
free, err := h.freeDisk(h.dir) free, err := h.freeDisk(h.dir)
if err != nil { if err != nil {
h.logger.Debugf("free disk check: %v", err) // unsupported platform, treat as unavailable rather than full h.logger.Debugf("free disk check: %v", err) // unsupported platform, treat as unavailable rather than full
return return
} }
if free >= uint64(h.policy.MinFreeDisk) { shortfall := h.policy.MinFreeDisk + max(need, 0) - int64(free)
if h.s3 != nil || shortfall <= 0 {
return return
} }
db, err := h.openDB()
if err != nil {
return
}
defer db.Close()
caches := h.completedByUse(db) caches := h.completedByUse(db)
total, shortfall := totalSize(caches), h.policy.MinFreeDisk-int64(free) total := totalSize(caches)
if total <= shortfall { if total <= shortfall {
// Say so, or shedding everything and still being short reads as the backstop working. // Say so, or shedding everything and still being short reads as the backstop working.
h.logger.Warnf("cache volume is %d MiB short of the free space floor with only %d MiB of cache on it; something else is filling it", shortfall/miB, total/miB) h.logger.Warnf("cache volume is %d MiB short of the free space floor with only %d MiB of cache on it; something else is filling it", shortfall/miB, total/miB)
@@ -1257,17 +1303,18 @@ func (h *Handler) evictExpired(db *cacheDB) {
h.sweep(db, func(cache *Cache) bool { return cache.UsedAt < expired }) h.sweep(db, func(cache *Cache) bool { return cache.UsedAt < expired })
} }
// evictSuperseded removes entries a newer one with the same key and version replaced. The // evictSuperseded removes entries a newer one with the same repo, ref, key and version replaced.
// grouping includes Repo so two repos sharing a (key, version) do not evict each other.
func (h *Handler) evictSuperseded(db *cacheDB) { func (h *Handler) evictSuperseded(db *cacheDB) {
type entry struct{ repo, key, version string } type entry struct{ repo, ref, key, version string }
groups := make(map[entry][]*Cache) groups := make(map[entry][]*Cache)
for _, cache := range h.caches(db, func(cache *Cache) bool { return cache.Complete }) { for _, cache := range h.caches(db, func(cache *Cache) bool { return cache.Complete }) {
group := entry{cache.Repo, cache.Key, cache.Version} group := entry{cache.Repo, cache.Ref, cache.Key, cache.Version}
groups[group] = append(groups[group], cache) groups[group] = append(groups[group], cache)
} }
for _, caches := range groups { for _, caches := range groups {
slices.SortFunc(caches, func(left, right *Cache) int { return cmp.Compare(left.CreatedAt, right.CreatedAt) }) slices.SortFunc(caches, func(left, right *Cache) int {
return cmp.Or(cmp.Compare(left.createdAtNano(), right.createdAtNano()), cmp.Compare(left.ID, right.ID))
})
for _, cache := range caches[:len(caches)-1] { for _, cache := range caches[:len(caches)-1] {
if inUse(cache) { if inUse(cache) {
continue continue
+162 -62
View File
@@ -15,8 +15,10 @@ import (
"net/http/httptest" "net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
"testing" "testing"
"testing/iotest"
"testing/synctest" "testing/synctest"
"time" "time"
@@ -47,10 +49,12 @@ func signArtifactURL(h *Handler, id int64) string {
} }
func TestHandler_ExternalURL(t *testing.T) { func TestHandler_ExternalURL(t *testing.T) {
t.Parallel()
assert.Equal(t, "http://[2001:db8::1]:8080", (&Handler{outboundIP: "2001:db8::1", port: 8080}).ExternalURL()) assert.Equal(t, "http://[2001:db8::1]:8080", (&Handler{outboundIP: "2001:db8::1", port: 8080}).ExternalURL())
} }
func TestHandler_Upstream(t *testing.T) { func TestHandler_Upstream(t *testing.T) {
t.Parallel()
var upstream *httptest.Server var upstream *httptest.Server
upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, http.MethodPut, r.Method) assert.Equal(t, http.MethodPut, r.Method)
@@ -100,6 +104,7 @@ func TestHandler_Upstream(t *testing.T) {
} }
func TestHandler(t *testing.T) { func TestHandler(t *testing.T) {
t.Parallel()
const version = "c19da02a2bd7e77277f1ac29ab45c09b7d46a4ee758284e26bb3045ad11d9d20" const version = "c19da02a2bd7e77277f1ac29ab45c09b7d46a4ee758284e26bb3045ad11d9d20"
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir}) handler, err := StartHandler(Options{Dir: dir})
@@ -302,6 +307,37 @@ func TestHandler(t *testing.T) {
} }
}) })
t.Run("chunked commit validates size omitted at reservation", func(t *testing.T) {
for _, size := range []int64{5, 10} {
t.Run(strconv.FormatInt(size, 10), func(t *testing.T) {
id := reserveCache(t, testClient, base, strings.ToLower(t.Name()), version, 0)
req, err := http.NewRequest(http.MethodPatch, fmt.Sprintf("%s/caches/%d", base, id), strings.NewReader("cache"))
require.NoError(t, err)
req.Header.Set("Content-Range", "bytes 0-4/*")
resp, err := testClient.Do(req)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
resp, err = testClient.Post(fmt.Sprintf("%s/caches/%d", base, id), "application/json", io.MultiReader(strings.NewReader(fmt.Sprintf(`{"size":%d}`, size))))
require.NoError(t, err)
resp.Body.Close()
if size == 5 {
require.Equal(t, http.StatusOK, resp.StatusCode)
} else {
require.Equal(t, http.StatusInternalServerError, resp.StatusCode)
}
db, err := handler.openDB()
require.NoError(t, err)
defer db.Close()
cache, err := db.get(id)
require.NoError(t, err)
require.Equal(t, size == 5, cache.Complete)
})
}
})
t.Run("get with bad id", func(t *testing.T) { t.Run("get with bad id", func(t *testing.T) {
resp, err := testClient.Get(base + "/artifacts/invalid_id") resp, err := testClient.Get(base + "/artifacts/invalid_id")
require.NoError(t, err) require.NoError(t, err)
@@ -438,6 +474,7 @@ func uploadCacheNormally(t *testing.T, base, key, version string, content []byte
} }
func TestHandler_gcCache(t *testing.T) { func TestHandler_gcCache(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir, Policy: Policy{Retention: testRetention}}) handler, err := StartHandler(Options{Dir: dir, Policy: Policy{Retention: testRetention}})
require.NoError(t, err) require.NoError(t, err)
@@ -540,6 +577,7 @@ func TestHandler_gcCache(t *testing.T) {
} }
func TestHandler_evictPolicy(t *testing.T) { func TestHandler_evictPolicy(t *testing.T) {
t.Parallel()
now := time.Now() now := time.Now()
stale := func(d time.Duration) int64 { return now.Add(-d).Unix() } stale := func(d time.Duration) int64 { return now.Add(-d).Unix() }
mib := func(n int64) int64 { return n * miB } mib := func(n int64) int64 { return n * miB }
@@ -636,7 +674,8 @@ func TestHandler_evictPolicy(t *testing.T) {
} }
} }
func TestHandler_evictForFreeSpace(t *testing.T) { func TestHandler_makeRoom(t *testing.T) {
t.Parallel()
free := func(n int64) func(string) (uint64, error) { free := func(n int64) func(string) (uint64, error) {
return func(string) (uint64, error) { return uint64(n), nil } return func(string) (uint64, error) { return uint64(n), nil }
} }
@@ -644,14 +683,16 @@ func TestHandler_evictForFreeSpace(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
freeDisk func(string) (uint64, error) freeDisk func(string) (uint64, error)
upload int64
kept []string kept []string
}{ }{
{"ample free space evicts nothing", free(defaultMinFreeDisk), []string{"oldest", "middle", "newest"}}, {"ample free space evicts nothing", free(defaultMinFreeDisk), 0, []string{"oldest", "middle", "newest"}},
{"a small shortfall sheds one entry", free(defaultMinFreeDisk - 4*miB), []string{"middle", "newest"}}, {"room for an upload sheds the oldest entry", free(defaultMinFreeDisk), 4 * miB, []string{"middle", "newest"}},
{"a shortfall the cache cannot cover sheds all of it", free(0), nil}, {"a shortfall the cache cannot cover sheds all of it", free(0), 0, nil},
{ {
"an unreadable volume is treated as unavailable, not as full", "an unreadable volume is treated as unavailable, not as full",
func(string) (uint64, error) { return 0, errors.New("unsupported") }, func(string) (uint64, error) { return 0, errors.New("unsupported") },
0,
[]string{"oldest", "middle", "newest"}, []string{"oldest", "middle", "newest"},
}, },
} { } {
@@ -664,11 +705,7 @@ func TestHandler_evictForFreeSpace(t *testing.T) {
} }
handler := newTestHandler(t, Policy{}, entries...) handler := newTestHandler(t, Policy{}, entries...)
handler.freeDisk = tc.freeDisk handler.freeDisk = tc.freeDisk
handler.makeRoom(tc.upload)
db, err := handler.openDB()
require.NoError(t, err)
handler.evictForFreeSpace(db)
require.NoError(t, db.Close())
assert.ElementsMatch(t, tc.kept, keptKeys(t, handler, entries)) assert.ElementsMatch(t, tc.kept, keptKeys(t, handler, entries))
}) })
@@ -676,6 +713,7 @@ func TestHandler_evictForFreeSpace(t *testing.T) {
} }
func TestHandler_SweepKeepsEntryWhenBlobSurvives(t *testing.T) { func TestHandler_SweepKeepsEntryWhenBlobSurvives(t *testing.T) {
t.Parallel()
cache := &Cache{Key: "stuck", Complete: true, UsedAt: time.Now().Add(-(testRetention + time.Hour)).Unix()} cache := &Cache{Key: "stuck", Complete: true, UsedAt: time.Now().Add(-(testRetention + time.Hour)).Unix()}
handler := newTestHandler(t, Policy{Retention: testRetention}, cache) handler := newTestHandler(t, Policy{Retention: testRetention}, cache)
@@ -690,6 +728,7 @@ func TestHandler_SweepKeepsEntryWhenBlobSurvives(t *testing.T) {
} }
func TestHandler_FindProtectsFromEviction(t *testing.T) { func TestHandler_FindProtectsFromEviction(t *testing.T) {
t.Parallel()
wanted := &Cache{Repo: testRepo, Key: "wanted", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-3 * time.Hour).Unix()} wanted := &Cache{Repo: testRepo, Key: "wanted", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-3 * time.Hour).Unix()}
other := &Cache{Repo: testRepo, Key: "other", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-2 * time.Hour).Unix()} other := &Cache{Repo: testRepo, Key: "other", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-2 * time.Hour).Unix()}
newest := &Cache{Repo: testRepo, Key: "newest", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-time.Hour).Unix()} newest := &Cache{Repo: testRepo, Key: "newest", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-time.Hour).Unix()}
@@ -709,7 +748,67 @@ func TestHandler_FindProtectsFromEviction(t *testing.T) {
assert.Equal(t, []string{wanted.Key}, keptKeys(t, handler, []*Cache{wanted, other}), "the entry just promised to a job survives, the next least recently used goes instead") assert.Equal(t, []string{wanted.Key}, keptKeys(t, handler, []*Cache{wanted, other}), "the entry just promised to a job survives, the next least recently used goes instead")
} }
func TestHandler_PrefixUsesNewestReservationWithinSecond(t *testing.T) {
t.Parallel()
created := time.Now().Add(-time.Hour).Unix()
older := &Cache{Repo: testRepo, Key: "deps-old", Version: "v", Complete: true, CreatedAt: created}
newer := &Cache{Repo: testRepo, Key: "deps-new", Version: "v", Complete: true, CreatedAt: created}
handler := newTestHandler(t, Policy{}, older, newer)
db, err := handler.openDB()
require.NoError(t, err)
defer db.Close()
cache, err := findCache(db, JobCredential{Repo: testRepo}, []string{"deps-"}, "v")
require.NoError(t, err)
require.Equal(t, newer, cache)
}
func TestHandler_RestoreRefs(t *testing.T) {
t.Parallel()
const main, release, pull = "refs/heads/main", "refs/heads/release", "refs/pull/1/head"
now := time.Now().Unix()
entry := func(ref, key string, age int64) *Cache {
return &Cache{Repo: testRepo, Ref: ref, Key: key, Version: "v", Complete: true, CreatedAt: now - 3600 + age, UsedAt: now}
}
exactOnMain := entry(main, "deps-exact", -1)
onRelease := entry(release, "deps-release", 0)
onMain := entry(main, "deps-main", 1)
onPull := entry(pull, "deps-pull", 2)
unscoped := entry("", "deps-unscoped", 3)
handler := newTestHandler(t, Policy{}, exactOnMain, onRelease, onMain, onPull, unscoped, entry("refs/pull/2/head", "deps-lost", 4))
for _, cache := range []*Cache{exactOnMain, onRelease, onMain, onPull, unscoped} {
writeBlob(t, handler, cache.ID)
}
db, err := handler.openDB()
require.NoError(t, err)
defer db.Close()
for _, tc := range []struct {
name string
cred JobCredential
keys []string
want *Cache
}{
{"the default branch never restores what a pull request wrote", JobCredential{Ref: main, DefaultRef: main}, []string{"deps-"}, onMain},
{"a pull request restores its own entry first", JobCredential{Ref: pull, BaseRef: release, DefaultRef: main}, []string{"deps-"}, onPull},
{"a pull request falls back past a lost blob to its base branch before the default branch", JobCredential{Ref: "refs/pull/2/head", BaseRef: release, DefaultRef: main}, []string{"deps-"}, onRelease},
{"a branch does not restore a sibling branch", JobCredential{Ref: "refs/heads/other", DefaultRef: main}, []string{"deps-"}, onMain},
{"a prefix in the own ref wins over an exact key in the default branch", JobCredential{Ref: release, DefaultRef: main}, []string{"deps-exact", "deps-"}, onRelease},
{"a job with a ref never restores entries saved without one", JobCredential{Ref: "refs/heads/other"}, []string{"deps-"}, nil},
{"a job without a ref only restores entries saved without one", JobCredential{}, []string{"deps-"}, unscoped},
} {
t.Run(tc.name, func(t *testing.T) {
tc.cred.Repo = testRepo
cache, err := handler.lookupCache(db, tc.cred, tc.keys, "v")
require.NoError(t, err)
assert.Equal(t, tc.want, cache)
})
}
}
func TestHandler_evictOnCommit(t *testing.T) { func TestHandler_evictOnCommit(t *testing.T) {
t.Parallel()
full := &Cache{Repo: testRepo, Key: "full", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-time.Hour).Unix()} full := &Cache{Repo: testRepo, Key: "full", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-time.Hour).Unix()}
handler := newTestHandler(t, Policy{RepoSizeLimit: 4 * miB}, full) handler := newTestHandler(t, Policy{RepoSizeLimit: 4 * miB}, full)
@@ -719,6 +818,7 @@ func TestHandler_evictOnCommit(t *testing.T) {
} }
func TestHandler_gcCacheInterval(t *testing.T) { func TestHandler_gcCacheInterval(t *testing.T) {
t.Parallel()
cache := &Cache{Key: "temp", UsedAt: time.Now().Add(-time.Hour).Unix()} cache := &Cache{Key: "temp", UsedAt: time.Now().Add(-time.Hour).Unix()}
handler := newTestHandler(t, Policy{SweepInterval: 30 * time.Minute}, cache) handler := newTestHandler(t, Policy{SweepInterval: 30 * time.Minute}, cache)
@@ -728,6 +828,7 @@ func TestHandler_gcCacheInterval(t *testing.T) {
} }
func TestHandler_touchCacheSkipsFreshCompletedEntries(t *testing.T) { func TestHandler_touchCacheSkipsFreshCompletedEntries(t *testing.T) {
t.Parallel()
fresh := &Cache{Repo: testRepo, Key: "fresh", Version: "v", Complete: true, UsedAt: time.Now().Unix()} fresh := &Cache{Repo: testRepo, Key: "fresh", Version: "v", Complete: true, UsedAt: time.Now().Unix()}
pending := &Cache{Repo: testRepo, Key: "pending", Version: "v", UsedAt: time.Now().Add(-time.Second).Unix()} pending := &Cache{Repo: testRepo, Key: "pending", Version: "v", UsedAt: time.Now().Add(-time.Second).Unix()}
handler := newTestHandler(t, Policy{}, fresh, pending) handler := newTestHandler(t, Policy{}, fresh, pending)
@@ -750,6 +851,9 @@ func TestHandler_touchCacheSkipsFreshCompletedEntries(t *testing.T) {
func newTestHandler(t *testing.T, policy Policy, entries ...*Cache) *Handler { func newTestHandler(t *testing.T, policy Policy, entries ...*Cache) *Handler {
t.Helper() t.Helper()
handler := startTestHandler(t, Options{Policy: policy}) handler := startTestHandler(t, Options{Policy: policy})
if len(entries) == 0 {
return handler
}
db, err := handler.openDB() db, err := handler.openDB()
require.NoError(t, err) require.NoError(t, err)
@@ -786,6 +890,13 @@ func keptKeys(t *testing.T, handler *Handler, entries []*Cache) []string {
return kept return kept
} }
func TestStorageDropsFailedWrites(t *testing.T) {
storage, err := NewStorage(t.TempDir())
require.NoError(t, err)
require.Error(t, storage.Write(1, 0, io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(io.ErrUnexpectedEOF))))
assert.NoFileExists(t, storage.tempName(1, 0))
}
func writeBlob(t *testing.T, handler *Handler, id uint64) { func writeBlob(t *testing.T, handler *Handler, id uint64) {
t.Helper() t.Helper()
require.NoError(t, handler.storage.Write(id, 0, strings.NewReader("a"))) require.NoError(t, handler.storage.Write(id, 0, strings.NewReader("a")))
@@ -794,6 +905,7 @@ func writeBlob(t *testing.T, handler *Handler, id uint64) {
} }
func TestHandler_RejectsBearer(t *testing.T) { func TestHandler_RejectsBearer(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir}) handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err) require.NoError(t, err)
@@ -830,6 +942,11 @@ func TestHandler_RejectsBearer(t *testing.T) {
} }
func TestHandler_UnregisterRevokes(t *testing.T) { func TestHandler_UnregisterRevokes(t *testing.T) {
t.Parallel()
handler, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache")})
require.NoError(t, err)
defer handler.Close()
for _, tc := range []struct { for _, tc := range []struct {
name string name string
token string token string
@@ -840,9 +957,6 @@ func TestHandler_UnregisterRevokes(t *testing.T) {
{"reference counted", "shared", "v", 2}, {"reference counted", "shared", "v", 2},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
handler, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache")})
require.NoError(t, err)
defer handler.Close()
revokers := make([]func(), tc.registrations) revokers := make([]func(), tc.registrations)
for i := range revokers { for i := range revokers {
revokers[i] = handler.RegisterJob(tc.token, JobCredential{Repo: testRepo}) revokers[i] = handler.RegisterJob(tc.token, JobCredential{Repo: testRepo})
@@ -871,12 +985,14 @@ func TestHandler_UnregisterRevokes(t *testing.T) {
} }
func TestHandler_CrossRepoIsolation(t *testing.T) { func TestHandler_CrossRepoIsolation(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir}) handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err) require.NoError(t, err)
defer handler.Close() defer handler.Close()
handler.RegisterJob("token-a", JobCredential{Repo: "owner/repoA"}) handler.RegisterJob("token-a", JobCredential{Repo: "owner/repoA"})
handler.RegisterJob("token-b", JobCredential{Repo: "owner/repoB"}) handler.RegisterJob("token-b", JobCredential{Repo: "owner/repoB"})
handler.RegisterJob("token-a-pull", JobCredential{Repo: "owner/repoA", Ref: "refs/pull/1/head"})
base := handler.ExternalURL() + apiPath base := handler.ExternalURL() + apiPath
key := "shared-key" key := "shared-key"
@@ -911,16 +1027,24 @@ func TestHandler_CrossRepoIsolation(t *testing.T) {
resp.Body.Close() resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode) assert.Equal(t, http.StatusOK, resp.StatusCode)
req, err = http.NewRequest(http.MethodPatch, fmt.Sprintf("%s/caches/%d", base, id), bytes.NewReader([]byte("poison"))) for _, client := range []*http.Client{clientB, {Transport: &bearerTransport{token: "token-a-pull"}}} {
require.NoError(t, err) req, err = http.NewRequest(http.MethodPatch, fmt.Sprintf("%s/caches/%d", base, id), bytes.NewReader([]byte("poison")))
req.Header.Set("Content-Range", "bytes 0-5/*") require.NoError(t, err)
resp, err = clientB.Do(req) req.Header.Set("Content-Range", "bytes 0-5/*")
require.NoError(t, err) resp, err = client.Do(req)
resp.Body.Close() require.NoError(t, err)
assert.Equal(t, http.StatusForbidden, resp.StatusCode) resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
resp, err = client.Post(fmt.Sprintf("%s/caches/%d", base, id), "", nil)
require.NoError(t, err)
resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
}
} }
func TestHandler_ArtifactSignature(t *testing.T) { func TestHandler_ArtifactSignature(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir}) handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err) require.NoError(t, err)
@@ -929,9 +1053,8 @@ func TestHandler_ArtifactSignature(t *testing.T) {
base := handler.ExternalURL() + apiPath base := handler.ExternalURL() + apiPath
good := signArtifactURL(handler, 1) good := signArtifactURL(handler, 1)
other, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache2")}) secret := bytes.Clone(handler.secret)
require.NoError(t, err) secret[0] ^= 1
defer other.Close()
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -940,7 +1063,7 @@ func TestHandler_ArtifactSignature(t *testing.T) {
{"missing signature", fmt.Sprintf("%s/artifacts/%d", base, 1)}, {"missing signature", fmt.Sprintf("%s/artifacts/%d", base, 1)},
{"tampered signature", good[:len(good)-4] + "dead"}, {"tampered signature", good[:len(good)-4] + "dead"},
{"expired signature", handler.signedArtifactURL(JobCredential{}, 1, time.Now().Add(-time.Second))}, {"expired signature", handler.signedArtifactURL(JobCredential{}, 1, time.Now().Add(-time.Second))},
{"signature from a different server", base + strings.SplitN(signArtifactURL(other, 1), apiPath, 2)[1]}, {"signature from a different server", base + strings.SplitN(signArtifactURL(&Handler{secret: secret}, 1), apiPath, 2)[1]},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
resp, err := testClient.Get(tc.url) resp, err := testClient.Get(tc.url)
@@ -952,6 +1075,7 @@ func TestHandler_ArtifactSignature(t *testing.T) {
} }
func TestHandler_SecretPersistsAcrossRestarts(t *testing.T) { func TestHandler_SecretPersistsAcrossRestarts(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
first, err := StartHandler(Options{Dir: dir, OutboundIP: "127.0.0.1"}) first, err := StartHandler(Options{Dir: dir, OutboundIP: "127.0.0.1"})
@@ -968,6 +1092,7 @@ func TestHandler_SecretPersistsAcrossRestarts(t *testing.T) {
} }
func TestHandler_SeesWritesOfOtherProcesses(t *testing.T) { func TestHandler_SeesWritesOfOtherProcesses(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
insert := func(by *Handler, cache *Cache) { insert := func(by *Handler, cache *Cache) {
db, err := by.openDB() db, err := by.openDB()
@@ -985,6 +1110,7 @@ func TestHandler_SeesWritesOfOtherProcesses(t *testing.T) {
} }
func TestHandler_ArtifactSignatureDownload(t *testing.T) { func TestHandler_ArtifactSignatureDownload(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir}) handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err) require.NoError(t, err)
@@ -1018,51 +1144,24 @@ func TestHandler_ArtifactSignatureDownload(t *testing.T) {
assert.Equal(t, content, body) assert.Equal(t, content, body)
} }
func TestHandler_GC_PerRepoDedup(t *testing.T) { func TestHandler_GC_DedupPerRepoAndRef(t *testing.T) {
dir := filepath.Join(t.TempDir(), "artifactcache") t.Parallel()
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
defer handler.Close()
handler.RegisterJob("tok-a", JobCredential{Repo: "owner/repoA"})
handler.RegisterJob("tok-b", JobCredential{Repo: "owner/repoB"})
key := "shared-dedup-key"
version := "c19da02a2bd7e77277f1ac29ab45c09b7d46a4ee758284e26bb3045ad11d9d20"
db, err := handler.openDB()
require.NoError(t, err)
now := time.Now().Unix() now := time.Now().Unix()
stale := time.Now().Add(-inUseGrace - time.Minute).Unix() stale := time.Now().Add(-inUseGrace - time.Minute).Unix()
a := &Cache{Repo: "owner/repoA", Key: key, Version: version, Complete: true, CreatedAt: stale, UsedAt: stale, Size: 1} entries := []*Cache{
b := &Cache{Repo: "owner/repoB", Key: key, Version: version, Complete: true, CreatedAt: now, UsedAt: now, Size: 1} {Repo: "owner/repoA", Key: "shared", Version: "v", Complete: true, CreatedAt: stale, UsedAt: stale, Size: 1},
require.NoError(t, handler.insertCache(db, a)) {Repo: "owner/repoB", Key: "shared", Version: "v", Complete: true, CreatedAt: now, UsedAt: now, Size: 1},
require.NoError(t, handler.insertCache(db, b)) {Repo: "owner/repoA", Ref: "refs/pull/1/head", Key: "shared", Version: "v", Complete: true, CreatedAt: now, UsedAt: now, Size: 1},
require.NoError(t, handler.storage.Write(a.ID, 0, strings.NewReader("a"))) }
_, err = handler.storage.Commit(a.ID, 1) handler := newTestHandler(t, Policy{}, entries...)
require.NoError(t, err)
require.NoError(t, handler.storage.Write(b.ID, 0, strings.NewReader("b")))
_, err = handler.storage.Commit(b.ID, 1)
require.NoError(t, err)
require.NoError(t, db.Close())
handler.gcAt = time.Time{} handler.gcAt = time.Time{}
handler.gcCache() handler.gcCache()
db, err = handler.openDB() assert.Len(t, keptKeys(t, handler, entries), len(entries))
require.NoError(t, err)
defer db.Close()
after, err := db.find(func(cache *Cache) bool { return cache.Key == key && cache.Version == version })
require.NoError(t, err)
repos := make(map[string]bool)
for _, c := range after {
repos[c.Repo] = true
}
assert.True(t, repos["owner/repoA"], "repoA's cache must survive dedup against repoB")
assert.True(t, repos["owner/repoB"], "repoB's cache must survive dedup against repoA")
} }
func TestHandler_InternalAPI_Disabled(t *testing.T) { func TestHandler_InternalAPI_Disabled(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir}) handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err) require.NoError(t, err)
@@ -1077,6 +1176,7 @@ func TestHandler_InternalAPI_Disabled(t *testing.T) {
} }
func TestHandler_InternalAPI_AuthAndUsage(t *testing.T) { func TestHandler_InternalAPI_AuthAndUsage(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache") dir := filepath.Join(t.TempDir(), "artifactcache")
const secret = "internal-secret" const secret = "internal-secret"
handler, err := StartHandler(Options{Dir: dir, InternalSecret: secret}) handler, err := StartHandler(Options{Dir: dir, InternalSecret: secret})
@@ -1122,10 +1222,10 @@ func TestHandler_InternalAPI_AuthAndUsage(t *testing.T) {
} }
assert.Equal(t, http.StatusUnauthorized, probe("via-internal-api")) assert.Equal(t, http.StatusUnauthorized, probe("via-internal-api"))
assert.Equal(t, http.StatusOK, post("/_internal/register", secret, `{"token":"via-internal-api","repo":"owner/repo","public_url":"http://runner.example","results":"https://gitea.example","insecure_tls":true}`)) assert.Equal(t, http.StatusOK, post("/_internal/register", secret, `{"token":"via-internal-api","repo":"owner/repo","ref":"refs/pull/1/head","base_ref":"refs/heads/release","default_ref":"refs/heads/main","public_url":"http://runner.example","results":"https://gitea.example","insecure_tls":true}`))
cred, ok := handler.lookupCredential("via-internal-api") cred, ok := handler.lookupCredential("via-internal-api")
require.True(t, ok) require.True(t, ok)
assert.Equal(t, JobCredential{Repo: testRepo, PublicURL: "http://runner.example"}, cred) assert.Equal(t, JobCredential{Repo: testRepo, Ref: "refs/pull/1/head", BaseRef: "refs/heads/release", DefaultRef: "refs/heads/main", PublicURL: "http://runner.example"}, cred)
assert.NotEqual(t, http.StatusUnauthorized, probe("via-internal-api")) assert.NotEqual(t, http.StatusUnauthorized, probe("via-internal-api"))
assert.Equal(t, http.StatusOK, post("/_internal/revoke", secret, `{"token":"via-internal-api"}`)) assert.Equal(t, http.StatusOK, post("/_internal/revoke", secret, `{"token":"via-internal-api"}`))
assert.Equal(t, http.StatusUnauthorized, probe("via-internal-api")) assert.Equal(t, http.StatusUnauthorized, probe("via-internal-api"))
+16 -13
View File
@@ -73,7 +73,7 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
// An exact (key, version) match means the entry is already cached; the client then skips // An exact (key, version) match means the entry is already cached; the client then skips
// the upload. A prefix match must not count here, or a shorter key would be reported as // the upload. A prefix match must not count here, or a shorter key would be reported as
// existing and silently never saved. // existing and silently never saved.
if existing, err := findExactCache(db, cred.Repo, req.Key, req.Version, true); err != nil { if existing, err := findExactCache(db, cred, req.Key, req.Version, true); err != nil {
h.twirpError(w, r, twirpInternal, err) h.twirpError(w, r, twirpInternal, err)
return return
} else if existing != nil { } else if existing != nil {
@@ -84,7 +84,7 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
// A second live reservation is finalized by whichever job calls last, against the other's upload. // A second live reservation is finalized by whichever job calls last, against the other's upload.
owner := hashedToken(bearerToken(r)) owner := hashedToken(bearerToken(r))
if pending, err := findExactCache(db, cred.Repo, req.Key, req.Version, false); err != nil { if pending, err := findExactCache(db, cred, req.Key, req.Version, false); err != nil {
h.twirpError(w, r, twirpInternal, err) h.twirpError(w, r, twirpInternal, err)
return return
} else if pending != nil && pending.UsedAt > time.Now().Add(-uploadStallTimeout).Unix() { } else if pending != nil && pending.UsedAt > time.Now().Add(-uploadStallTimeout).Unix() {
@@ -100,15 +100,17 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
return return
} }
now := time.Now().Unix() now := time.Now()
cache := &Cache{ cache := &Cache{
Repo: cred.Repo, Repo: cred.Repo,
Key: req.Key, Ref: cred.Ref,
Version: req.Version, Key: req.Key,
Owner: owner, Version: req.Version,
Size: -1, // the size is only known at finalize time Owner: owner,
CreatedAt: now, Size: -1, // the size is only known at finalize time
UsedAt: now, CreatedAt: now.Unix(),
CreatedAtNanos: now.Nanosecond(),
UsedAt: now.Unix(),
} }
if err := h.insertCache(db, cache); err != nil { if err := h.insertCache(db, cache); err != nil {
h.twirpError(w, r, twirpInternal, err) h.twirpError(w, r, twirpInternal, err)
@@ -136,13 +138,13 @@ func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Requ
} }
defer db.Close() defer db.Close()
cache, err := findExactCache(db, cred.Repo, req.Key, req.Version, false) cache, err := findExactCache(db, cred, req.Key, req.Version, false)
if err == nil && cache != nil && cache.Owner != hashedToken(bearerToken(r)) { if err == nil && cache != nil && cache.Owner != hashedToken(bearerToken(r)) {
cache = nil // not the reservation this job made, so not this job's to commit cache = nil // not the reservation this job made, so not this job's to commit
} }
if err == nil && cache == nil { if err == nil && cache == nil {
// A retry whose first response was lost finds the entry already committed. // A retry whose first response was lost finds the entry already committed.
cache, err = findExactCache(db, cred.Repo, req.Key, req.Version, true) cache, err = findExactCache(db, cred, req.Key, req.Version, true)
} }
if err != nil { if err != nil {
h.twirpError(w, r, twirpInternal, err) h.twirpError(w, r, twirpInternal, err)
@@ -185,7 +187,7 @@ func (h *Handler) v2GetCacheEntryDownloadURL(w http.ResponseWriter, r *http.Requ
} }
defer db.Close() defer db.Close()
cache, err := h.lookupCache(db, cred.Repo, req.keys(), req.Version) cache, err := h.lookupCache(db, cred, req.keys(), req.Version)
if err != nil { if err != nil {
h.twirpError(w, r, twirpInternal, err) h.twirpError(w, r, twirpInternal, err)
return return
@@ -216,6 +218,7 @@ func (h *Handler) v2UploadBlob(w http.ResponseWriter, r *http.Request) {
h.responseJSON(w, r, http.StatusBadRequest, err) h.responseJSON(w, r, http.StatusBadRequest, err)
return return
} }
h.makeRoom(r.ContentLength)
query := r.URL.Query() query := r.URL.Query()
switch strings.ToLower(query.Get("comp")) { switch strings.ToLower(query.Get("comp")) {
+26
View File
@@ -14,6 +14,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"testing" "testing"
"time"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -78,11 +79,14 @@ func saveV2(t *testing.T, handler *Handler, key, version string, content []byte)
} }
func TestCacheServiceV2RoundTrip(t *testing.T) { func TestCacheServiceV2RoundTrip(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
content := []byte("the cached archive") content := []byte("the cached archive")
unsigned := fmt.Sprintf("%s%s/1", handler.ExternalURL(), blobPath) unsigned := fmt.Sprintf("%s%s/1", handler.ExternalURL(), blobPath)
assert.Equal(t, http.StatusUnauthorized, putBlob(t, unsigned, content)) assert.Equal(t, http.StatusUnauthorized, putBlob(t, unsigned, content))
unreserved := handler.signedURL(JobCredential{Repo: testRepo}, blobPath, blobUploadPurpose, 1, time.Now().Add(time.Minute))
assert.Equal(t, http.StatusBadRequest, putBlob(t, unreserved, content))
finalized, uploadURL := saveV2(t, handler, "deps-v1", "abc123", content) finalized, uploadURL := saveV2(t, handler, "deps-v1", "abc123", content)
require.Equal(t, true, finalized["ok"]) require.Equal(t, true, finalized["ok"])
@@ -103,6 +107,7 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
} }
func TestCacheServiceV2BlockUpload(t *testing.T) { func TestCacheServiceV2BlockUpload(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "blocks", "version": "v1"}) created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "blocks", "version": "v1"})
@@ -150,6 +155,7 @@ func TestCacheServiceV2BlockUpload(t *testing.T) {
} }
func TestCacheServiceV2Lookups(t *testing.T) { func TestCacheServiceV2Lookups(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
saved, _ := saveV2(t, handler, "deps-abc", "v1", []byte("x")) saved, _ := saveV2(t, handler, "deps-abc", "v1", []byte("x"))
require.Equal(t, true, saved["ok"]) require.Equal(t, true, saved["ok"])
@@ -228,9 +234,28 @@ func TestCacheServiceV2Lookups(t *testing.T) {
got := v2Call(t, handler, otherClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-abc", "version": "v1"}) got := v2Call(t, handler, otherClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-abc", "version": "v1"})
assert.Equal(t, false, got["ok"]) assert.Equal(t, false, got["ok"])
}) })
t.Run("each ref saves and restores its own entry for the same key", func(t *testing.T) {
clients := map[string]*http.Client{"x": testClient}
for _, ref := range []string{"y", "z"} {
handler.RegisterJob(ref, JobCredential{Repo: testRepo, Ref: "refs/pull/" + ref + "/head"})
clients[ref] = &http.Client{Transport: &bearerTransport{token: ref}}
created := v2Call(t, handler, clients[ref], "CreateCacheEntry", map[string]any{"key": "deps-abc", "version": "v1"})
require.Equal(t, true, created["ok"])
require.Equal(t, http.StatusCreated, putBlob(t, created["signed_upload_url"].(string), []byte(ref)))
}
for _, ref := range []string{"y", "z"} {
require.Equal(t, true, v2Call(t, handler, clients[ref], "FinalizeCacheEntryUpload", map[string]any{"key": "deps-abc", "version": "v1", "size_bytes": 1})["ok"])
}
for content, client := range clients {
got := v2Call(t, handler, client, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-abc", "version": "v1"})
assert.Equal(t, content, string(getURL(t, got["signed_download_url"].(string))))
}
})
} }
func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) { func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
tests := []struct { tests := []struct {
@@ -267,6 +292,7 @@ func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
} }
func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) { func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{}) handler := newTestHandler(t, Policy{})
content := []byte("the cached archive") content := []byte("the cached archive")
+17 -9
View File
@@ -4,6 +4,8 @@
package artifactcache package artifactcache
import "time"
type Request struct { type Request struct {
Key string `json:"key" ` Key string `json:"key" `
Version string `json:"version"` Version string `json:"version"`
@@ -11,13 +13,19 @@ type Request struct {
} }
type Cache struct { type Cache struct {
ID uint64 `json:"id"` ID uint64 `json:"id"`
Repo string `json:"repo"` Repo string `json:"repo"`
Key string `json:"key"` Ref string `json:"ref,omitzero"`
Version string `json:"version"` Key string `json:"key"`
Size int64 `json:"cacheSize"` Version string `json:"version"`
Owner string `json:"owner"` // hashed bearer, so a job's own retry is not another job's reservation Size int64 `json:"cacheSize"`
Complete bool `json:"complete"` Owner string `json:"owner"` // hashed bearer, so a job's own retry is not another job's reservation
UsedAt int64 `json:"usedAt"` Complete bool `json:"complete"`
CreatedAt int64 `json:"createdAt"` UsedAt int64 `json:"usedAt"`
CreatedAt int64 `json:"createdAt"`
CreatedAtNanos int `json:"createdAtNanos,omitzero"`
}
func (cache *Cache) createdAtNano() int64 {
return cache.CreatedAt*int64(time.Second) + int64(cache.CreatedAtNanos)
} }
+1 -1
View File
@@ -67,7 +67,7 @@ func (h *Handler) forwardOrNotFound(w http.ResponseWriter, r *http.Request) {
h.twirpError(w, r, twirpInternal, fmt.Errorf("cache server cannot reach the artifact service at %s: %w", target, err)) h.twirpError(w, r, twirpInternal, fmt.Errorf("cache server cannot reach the artifact service at %s: %w", target, err))
}, },
} }
if cred.InsecureTLS { if proxy.Transport = h.instanceTransport; proxy.Transport == nil && cred.InsecureTLS { // the instance transport carries runner.insecure and the client cert
proxy.Transport = insecureTransport proxy.Transport = insecureTransport
} }
proxy.ServeHTTP(w, r) proxy.ServeHTTP(w, r)
+18
View File
@@ -17,6 +17,7 @@ import (
) )
func TestFrontResultsService(t *testing.T) { func TestFrontResultsService(t *testing.T) {
t.Parallel()
var gotHost, gotPath, gotProto string var gotHost, gotPath, gotProto string
gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotHost, gotPath, gotProto = r.Host, r.URL.Path, r.Header.Get("X-Forwarded-Proto") gotHost, gotPath, gotProto = r.Host, r.URL.Path, r.Header.Get("X-Forwarded-Proto")
@@ -132,3 +133,20 @@ func TestFrontResultsService(t *testing.T) {
}) })
} }
} }
func TestFrontResultsServiceForwardsThroughTheInstanceTransport(t *testing.T) {
t.Parallel()
gitea := httptest.NewTLSServer(http.HandlerFunc(func(writer http.ResponseWriter, _request *http.Request) { _, _ = io.WriteString(writer, "{}") }))
defer gitea.Close()
handler, err := StartHandler(Options{Dir: t.TempDir(), OutboundIP: "127.0.0.1", InstanceTransport: gitea.Client().Transport})
require.NoError(t, err)
defer handler.Close()
defer handler.RegisterJob("token", JobCredential{Results: gitea.URL})()
req, err := http.NewRequestWithContext(t.Context(), http.MethodPost, handler.ExternalURL()+artifactServicePath+"CreateArtifact", nil)
require.NoError(t, err)
resp, err := (&http.Client{Transport: &bearerTransport{token: "token"}}).Do(req)
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
}
+23
View File
@@ -8,6 +8,7 @@ import (
"io" "io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strconv"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -19,6 +20,7 @@ import (
) )
func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) { func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
t.Parallel()
var objects sync.Map var objects sync.Map
var uploads atomic.Int32 var uploads atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
@@ -51,14 +53,33 @@ func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
saver, restorer := startTestHandler(t, opts), startTestHandler(t, opts) saver, restorer := startTestHandler(t, opts), startTestHandler(t, opts)
entry := map[string]any{"key": "deps", "version": "v1"} entry := map[string]any{"key": "deps", "version": "v1"}
started := time.Now()
finalized, _ := saveV2(t, saver, "deps", "v1", []byte("archive")) finalized, _ := saveV2(t, saver, "deps", "v1", []byte("archive"))
require.Equal(t, true, finalized["ok"]) require.Equal(t, true, finalized["ok"])
assert.EqualValues(t, 1, uploads.Load()) assert.EqualValues(t, 1, uploads.Load())
id, err := strconv.ParseUint(finalized["entry_id"].(string), 10, 64)
require.NoError(t, err)
cache, err := saver.s3.fetch(t.Context(), id)
require.NoError(t, err)
assert.WithinRange(t, time.Unix(cache.CreatedAt, int64(cache.CreatedAtNanos)), started, time.Now())
require.NoError(t, restorer.importMetadata(t.Context())) require.NoError(t, restorer.importMetadata(t.Context()))
found := v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", entry) found := v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", entry)
require.Equal(t, true, found["ok"]) require.Equal(t, true, found["ok"])
assert.Equal(t, []byte("archive"), getURL(t, found["signed_download_url"].(string))) assert.Equal(t, []byte("archive"), getURL(t, found["signed_download_url"].(string)))
for _, cache := range []*Cache{
{ID: 2, Repo: testRepo, Key: "deps-old", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt},
{ID: 1, Repo: testRepo, Key: "deps-new", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt, CreatedAtNanos: 1},
} {
require.NoError(t, saver.s3.publish(cache))
objects.Store(saver.s3.blobKey(cache.ID), []byte(cache.Key))
}
require.NoError(t, restorer.importMetadata(t.Context()))
found = v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-", "version": "v1"})
require.Equal(t, true, found["ok"])
assert.Equal(t, "deps-new", found["matched_key"])
assert.Equal(t, []byte("deps-new"), getURL(t, found["signed_download_url"].(string)))
objects.Clear() objects.Clear()
require.NoError(t, restorer.importMetadata(t.Context())) require.NoError(t, restorer.importMetadata(t.Context()))
assert.Equal(t, true, v2Call(t, restorer, testClient, "CreateCacheEntry", entry)["ok"]) assert.Equal(t, true, v2Call(t, restorer, testClient, "CreateCacheEntry", entry)["ok"])
@@ -71,6 +92,7 @@ func (transport s3TestTransport) RoundTrip(req *http.Request) (*http.Response, e
} }
func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) { func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) {
t.Parallel()
for _, failingPart := range []string{"", "2"} { for _, failingPart := range []string{"", "2"} {
var parts, aborts atomic.Int32 var parts, aborts atomic.Int32
var completion struct { var completion struct {
@@ -110,6 +132,7 @@ func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) {
} }
func TestS3StorageSignsLikeAWS(t *testing.T) { func TestS3StorageSignsLikeAWS(t *testing.T) {
t.Parallel()
storage := &s3Storage{opts: S3Options{Region: "us-east-1", AccessKey: "AKIAIOSFODNN7EXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}} storage := &s3Storage{opts: S3Options{Region: "us-east-1", AccessKey: "AKIAIOSFODNN7EXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}}
req := httptest.NewRequest(http.MethodGet, "https://examplebucket.s3.amazonaws.com/?max-keys=2&prefix=J", nil) req := httptest.NewRequest(http.MethodGet, "https://examplebucket.s3.amazonaws.com/?max-keys=2&prefix=J", nil)
req.Header.Set("X-Amz-Content-Sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855") req.Header.Set("X-Amz-Content-Sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
+25 -27
View File
@@ -9,6 +9,7 @@ import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json/v2" "encoding/json/v2"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -51,9 +52,10 @@ func (s *Storage) writeFile(name string, reader io.Reader) error {
if err != nil { if err != nil {
return err return err
} }
defer file.Close()
_, err = io.Copy(file, reader) _, err = io.Copy(file, reader)
if err = errors.Join(err, file.Close()); err != nil {
_ = os.Remove(name)
}
return err return err
} }
@@ -102,50 +104,46 @@ func (s *Storage) Commit(id uint64, size int64) (int64, error) {
return 0, err return 0, err
} }
written, err := assemble(name, tempNames) written, err := assemble(name, tempNames)
if err != nil { if err == nil && size >= 0 && written != size { // old clients like actions/cache@v2 send no size
return 0, err err = fmt.Errorf("broken file: %v != %v", written, size)
} }
// If size is less than 0, it means the size is unknown. if err != nil {
// We can't check the size of the file, just skip the check.
// It happens when the request comes from old versions of actions, like `actions/cache@v2`.
if size >= 0 && written != size {
_ = os.Remove(name) _ = os.Remove(name)
return 0, fmt.Errorf("broken file: %v != %v", written, size) return 0, err
} }
return written, nil return written, nil
} }
// assemble concatenates the uploaded parts into name. A single part, which is what the v2 API // assemble appends the parts into the first, deleting each once copied, so it never needs twice the space.
// produces below the client's block threshold, is already the whole archive and is moved.
func assemble(name string, tempNames []string) (int64, error) { func assemble(name string, tempNames []string) (int64, error) {
if len(tempNames) == 1 { if len(tempNames) == 0 {
info, err := os.Stat(tempNames[0]) return 0, errors.New("nothing was uploaded")
if err != nil {
return 0, err
}
return info.Size(), os.Rename(tempNames[0], name)
} }
if err := os.Rename(tempNames[0], name); err != nil {
file, err := os.Create(name) return 0, err
}
file, err := os.OpenFile(name, os.O_WRONLY|os.O_APPEND, 0)
if err != nil { if err != nil {
return 0, err return 0, err
} }
defer file.Close() defer file.Close()
for _, v := range tempNames[1:] {
var written int64 part, err := os.Open(v)
for _, v := range tempNames {
f, err := os.Open(v)
if err != nil { if err != nil {
return 0, err return 0, err
} }
n, err := io.Copy(file, f) _, err = io.Copy(file, part)
_ = f.Close() _ = part.Close()
if err != nil { if err != nil {
return 0, err return 0, err
} }
written += n _ = os.Remove(v)
} }
return written, nil info, err := file.Stat()
if err != nil {
return 0, err
}
return info.Size(), file.Close()
} }
func (s *Storage) Serve(w http.ResponseWriter, r *http.Request, id uint64) { func (s *Storage) Serve(w http.ResponseWriter, r *http.Request, id uint64) {
+21 -7
View File
@@ -13,6 +13,7 @@ import (
"io/fs" "io/fs"
"net" "net"
"net/http" "net/http"
"net/url"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -84,16 +85,25 @@ func uploads(router *http.ServeMux, baseDir string) {
if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil { if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil {
panic(err) panic(err)
} }
flags := os.O_CREATE | os.O_WRONLY | os.O_TRUNC var start int64
appendUpload := req.Header.Get("Content-Range") if contentRange := req.Header.Get("Content-Range"); contentRange != "" {
if appendUpload != "" && !strings.HasPrefix(appendUpload, "bytes 0-") { if _, err := fmt.Sscanf(contentRange, "bytes %d-", &start); err != nil || start < 0 {
flags = os.O_CREATE | os.O_WRONLY | os.O_APPEND http.Error(w, "invalid content range", http.StatusBadRequest)
return
}
}
flags := os.O_CREATE | os.O_WRONLY
if start == 0 {
flags |= os.O_TRUNC
} }
file, err := os.OpenFile(safePath, flags, 0o644) file, err := os.OpenFile(safePath, flags, 0o644)
if err != nil { if err != nil {
panic(err) panic(err)
} }
defer file.Close() defer file.Close()
if _, err := file.Seek(start, io.SeekStart); err != nil {
panic(err)
}
if req.Body == nil { if req.Body == nil {
panic(errors.New("no body given")) panic(errors.New("no body given"))
} }
@@ -120,7 +130,7 @@ func downloads(router *http.ServeMux, baseDir string) {
runID := req.PathValue("runId") runID := req.PathValue("runId")
entries, err := os.ReadDir(safeResolve(baseDir, runID)) entries, err := os.ReadDir(safeResolve(baseDir, runID))
if err != nil { if err != nil && !os.IsNotExist(err) {
panic(err) panic(err)
} }
@@ -145,6 +155,9 @@ func downloads(router *http.ServeMux, baseDir string) {
var files []ContainerItem var files []ContainerItem
err := filepath.WalkDir(safePath, func(path string, entry fs.DirEntry, err error) error { err := filepath.WalkDir(safePath, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if !entry.IsDir() { if !entry.IsDir() {
rel, err := filepath.Rel(safePath, path) rel, err := filepath.Rel(safePath, path)
if err != nil { if err != nil {
@@ -157,12 +170,12 @@ func downloads(router *http.ServeMux, baseDir string) {
files = append(files, ContainerItem{ files = append(files, ContainerItem{
Path: filepath.ToSlash(filepath.Join(itemPath, rel)), Path: filepath.ToSlash(filepath.Join(itemPath, rel)),
ItemType: "file", ItemType: "file",
ContentLocation: fmt.Sprintf("http://%s/artifact/%s/%s/%s", req.Host, container, itemPath, filepath.ToSlash(rel)), ContentLocation: (&url.URL{Scheme: "http", Host: req.Host, Path: "/artifact/" + container + "/" + itemPath + "/" + filepath.ToSlash(rel)}).String(),
}) })
} }
return nil return nil
}) })
if err != nil { if err != nil && !os.IsNotExist(err) {
panic(err) panic(err)
} }
@@ -205,6 +218,7 @@ func Serve(ctx context.Context, artifactPath, addr, port string) context.CancelF
logger.Debugf("Artifacts base path '%s'", artifactPath) logger.Debugf("Artifacts base path '%s'", artifactPath)
uploads(router, artifactPath) uploads(router, artifactPath)
downloads(router, artifactPath) downloads(router, artifactPath)
registerV4(router, artifactPath)
server := &http.Server{ server := &http.Server{
Addr: net.JoinHostPort(addr, port), Addr: net.JoinHostPort(addr, port),
+113 -8
View File
@@ -28,6 +28,7 @@ func TestArtifactFlow(t *testing.T) {
router := http.NewServeMux() router := http.NewServeMux()
uploads(router, artifactPath) uploads(router, artifactPath)
downloads(router, artifactPath) downloads(router, artifactPath)
registerV4(router, artifactPath)
server := httptest.NewServer(router) server := httptest.NewServer(router)
defer server.Close() defer server.Close()
@@ -48,8 +49,17 @@ func TestArtifactFlow(t *testing.T) {
return resp.StatusCode, data return resp.StatusCode, data
} }
t.Run("no-artifacts", func(t *testing.T) {
status, data := request(t, http.MethodGet, baseURL+"/_apis/pipelines/workflows/0/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data))
var list NamedFileContainerResourceURLResponse
require.NoError(t, json.Unmarshal(data, &list))
require.Zero(t, list.Count)
require.Empty(t, list.Value)
})
t.Run("upload-and-download", func(t *testing.T) { t.Run("upload-and-download", func(t *testing.T) {
const runID, item, content = "1", "my-artifact/data.txt", "hello artifact\n" const runID, item, content = "1", "my-artifact/data#1%25.txt", "hello artifact\n"
status, data := request(t, http.MethodPost, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil) status, data := request(t, http.MethodPost, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data)) require.Equal(t, http.StatusOK, status, string(data))
@@ -81,13 +91,13 @@ func TestArtifactFlow(t *testing.T) {
require.NoError(t, json.Unmarshal(data, &items)) require.NoError(t, json.Unmarshal(data, &items))
require.Len(t, items.Value, 1) require.Len(t, items.Value, 1)
require.Equal(t, "file", items.Value[0].ItemType) require.Equal(t, "file", items.Value[0].ItemType)
require.Equal(t, "my-artifact/data.txt", items.Value[0].Path) require.Equal(t, item, items.Value[0].Path)
status, data = request(t, http.MethodGet, items.Value[0].ContentLocation, nil, nil) status, data = request(t, http.MethodGet, items.Value[0].ContentLocation, nil, nil)
require.Equal(t, http.StatusOK, status) require.Equal(t, http.StatusOK, status)
require.Equal(t, content, string(data)) require.Equal(t, content, string(data))
stored, err := os.ReadFile(filepath.Join(artifactPath, runID, "my-artifact", "data.txt")) stored, err := os.ReadFile(filepath.Join(artifactPath, runID, filepath.FromSlash(item)))
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, content, string(stored)) require.Equal(t, content, string(stored))
}) })
@@ -95,16 +105,25 @@ func TestArtifactFlow(t *testing.T) {
t.Run("content-range", func(t *testing.T) { t.Run("content-range", func(t *testing.T) {
const rawURL = "/upload/4?itemPath=chunks.txt" const rawURL = "/upload/4?itemPath=chunks.txt"
status, data := request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("first"), status, data := request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("first"),
http.Header{"Content-Range": []string{"bytes 0-4/11"}}) http.Header{"Content-Range": []string{"bytes 0-4/12"}})
require.Equal(t, http.StatusOK, status, string(data)) require.Equal(t, http.StatusOK, status, string(data))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"), for range 2 {
http.Header{"Content-Range": []string{"bytes 5-11/11"}}) status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"),
require.Equal(t, http.StatusOK, status, string(data)) http.Header{"Content-Range": []string{"bytes 5-11/12"}})
require.Equal(t, http.StatusOK, status, string(data))
}
stored, err := os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt")) stored, err := os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, "first-second", string(stored)) require.Equal(t, "first-second", string(stored))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader(""),
http.Header{"Content-Range": []string{"bytes 0--1/0"}})
require.Equal(t, http.StatusOK, status, string(data))
stored, err = os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err)
require.Empty(t, stored)
}) })
t.Run("gzip-roundtrip", func(t *testing.T) { t.Run("gzip-roundtrip", func(t *testing.T) {
@@ -156,6 +175,92 @@ func TestArtifactFlow(t *testing.T) {
require.Equal(t, http.StatusOK, status) require.Equal(t, http.StatusOK, status)
require.Equal(t, content, string(data)) require.Equal(t, content, string(data))
}) })
t.Run("v4", func(t *testing.T) {
const artifactRef = `"workflow_run_backend_id":"7","name":"50% art@v1"`
call := func(method, body string) (int, map[string]any) {
status, data := request(t, http.MethodPost, baseURL+artifactServicePath+"/"+method, strings.NewReader(body), nil)
var out map[string]any
require.NoError(t, json.Unmarshal(data, &out), string(data))
return status, out
}
put := func(rawURL, body string) int {
status, _ := request(t, http.MethodPut, rawURL, strings.NewReader(body), nil)
return status
}
download := func() (string, http.Header) {
status, signed := call("GetSignedArtifactURL", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, signed)
resp, err := client.Get(signed["signedUrl"].(string))
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
return string(body), resp.Header
}
status, created := call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, created)
upload := created["signedUploadUrl"].(string)
require.Equal(t, http.StatusCreated, put(upload+"&comp=block&blockid=Yg", "world"))
require.Equal(t, http.StatusCreated, put(upload+"&comp=block&blockid=YQ", "hello "))
for attempt := range 2 {
require.Equal(t, http.StatusCreated, put(upload+"&comp=blocklist", `<?xml version="1.0"?><BlockList><Latest>YQ</Latest><Latest>Yg</Latest></BlockList>`), "commit attempt %d", attempt)
}
digest := "sha256:b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"
status, _ = call("FinalizeArtifact", "{"+artifactRef+`,"size":"10"}`)
require.Equal(t, http.StatusBadRequest, status)
status, finalized := call("FinalizeArtifact", "{"+artifactRef+`,"size":"11","hash":"`+digest+`"}`)
require.Equal(t, http.StatusOK, status, finalized)
id := finalized["artifactId"].(string)
status, refinalized := call("FinalizeArtifact", "{"+artifactRef+`,"size":"10"}`)
require.Equal(t, http.StatusOK, status)
require.Equal(t, id, refinalized["artifactId"])
status, conflict := call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusConflict, status)
require.Equal(t, "already_exists", conflict["code"])
status, listed := call("ListArtifacts", `{"workflowRunBackendId":"7","idFilter":"`+id+`"}`)
require.Equal(t, http.StatusOK, status)
artifacts := listed["artifacts"].([]any)
require.Len(t, artifacts, 1)
artifact := artifacts[0].(map[string]any)
require.Regexp(t, `^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$`, artifact["createdAt"])
delete(artifact, "createdAt")
require.Equal(t, map[string]any{"workflowRunBackendId": "7", "name": "50% art@v1", "size": "11", "digest": digest, "databaseId": id}, artifact)
_, other := call("ListArtifacts", `{"workflow_run_backend_id":"8"}`)
require.Empty(t, other["artifacts"])
body, header := download()
require.Equal(t, "hello world", body)
require.Equal(t, "application/zip", header.Get("Content-Type"))
require.Equal(t, "attachment; filename*=UTF-8''50%25%20art%40v1.zip", header.Get("Content-Disposition"))
status, _ = call("DeleteArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status)
status, missing := call("GetSignedArtifactURL", "{"+artifactRef+"}")
require.Equal(t, http.StatusNotFound, status)
require.Equal(t, "not_found", missing["code"])
status, created = call("CreateArtifact", "{"+artifactRef+"}")
require.Equal(t, http.StatusOK, status, created)
status, _ = request(t, http.MethodPut, created["signedUploadUrl"].(string), strings.NewReader("gz"), http.Header{"X-Ms-Blob-Content-Type": {"application/gzip"}})
require.Equal(t, http.StatusCreated, status)
status, _ = call("FinalizeArtifact", "{"+artifactRef+`,"size":"2"}`)
require.Equal(t, http.StatusOK, status)
body, header = download()
require.Equal(t, "gz", body)
require.Equal(t, "application/gzip", header.Get("Content-Type"))
require.Equal(t, "attachment; filename*=UTF-8''50%25%20art%40v1", header.Get("Content-Disposition"))
rerun := http.NewServeMux()
registerV4(rerun, artifactPath)
recorder := httptest.NewRecorder()
rerun.ServeHTTP(recorder, httptest.NewRequest(http.MethodPost, artifactServicePath+"/CreateArtifact", strings.NewReader("{"+artifactRef+"}")))
require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String())
})
} }
func TestSafeResolve(t *testing.T) { func TestSafeResolve(t *testing.T) {
@@ -176,7 +281,7 @@ func TestSafeResolve(t *testing.T) {
for name, tc := range tests { for name, tc := range tests {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
require.Equal(t, tc.want, safeResolve(baseDir, tc.input)) require.Equal(t, filepath.FromSlash(tc.want), safeResolve(baseDir, tc.input))
}) })
} }
} }
+329
View File
@@ -0,0 +1,329 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import (
"cmp"
"crypto/sha256"
"encoding/hex"
"encoding/json/v2"
"encoding/xml"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"net/url"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"time"
)
type serverV4 struct { // the subset of Gitea's v4 artifact service that the toolkit uses, unauthenticated like the v3 routes beside it
dir string
mu sync.Mutex
lastID int64
created map[string]bool
}
const (
zipMimeType = "application/zip"
maxTwirpBytes = 1 << 20
)
func registerV4(router *http.ServeMux, baseDir string) {
server := &serverV4{dir: filepath.Join(baseDir, "v4"), lastID: time.Now().UnixMilli(), created: map[string]bool{}} // IDs stay unique across invocations sharing baseDir
router.HandleFunc("POST "+artifactServicePath+"/CreateArtifact", twirp(server.createArtifact))
router.HandleFunc("POST "+artifactServicePath+"/FinalizeArtifact", twirp(server.finalizeArtifact))
router.HandleFunc("POST "+artifactServicePath+"/ListArtifacts", twirp(server.listArtifacts))
router.HandleFunc("POST "+artifactServicePath+"/GetSignedArtifactURL", twirp(server.getSignedArtifactURL))
router.HandleFunc("POST "+artifactServicePath+"/DeleteArtifact", twirp(server.deleteArtifact))
router.HandleFunc("PUT "+artifactServicePath+"/UploadArtifact", server.uploadArtifact)
router.HandleFunc("GET "+artifactServicePath+"/DownloadArtifact", server.downloadArtifact)
}
type twirpError struct {
status int
code string
msg string
}
func (e *twirpError) Error() string { return e.msg }
func notFound(name string) error {
return &twirpError{http.StatusNotFound, "not_found", fmt.Sprintf("artifact %q not found", name)}
}
func badRequest(format string, args ...any) error {
return &twirpError{http.StatusBadRequest, "invalid_argument", fmt.Sprintf(format, args...)}
}
func twirp[Req, Resp any](handle func(*http.Request, Req) (Resp, error)) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var req Req
var resp Resp
err := json.UnmarshalRead(io.LimitReader(r.Body, maxTwirpBytes), &req, json.MatchCaseInsensitiveNames(true)) // also matches the toolkit's proto field names
if err != nil {
err = badRequest("malformed request: %v", err)
} else {
resp, err = handle(r, req)
}
w.Header().Set("Content-Type", "application/json")
if err != nil {
terr := &twirpError{http.StatusInternalServerError, "internal", err.Error()}
errors.As(err, &terr)
w.WriteHeader(terr.status)
_ = json.MarshalWrite(w, map[string]string{"code": terr.code, "msg": terr.msg})
return
}
_ = json.MarshalWrite(w, resp)
}
}
func fileName(name string) string { // safe on any filesystem for any client-chosen name, case-insensitive ones included
sum := sha256.Sum256([]byte(name))
return hex.EncodeToString(sum[:])
}
func (s *serverV4) runDir(runID string) string {
return filepath.Join(s.dir, fileName(runID))
}
func (s *serverV4) artifactDir(runID, name string) string {
return filepath.Join(s.runDir(runID), fileName(name))
}
func (s *serverV4) urlDir(r *http.Request) string {
return s.artifactDir(r.URL.Query().Get("run"), r.URL.Query().Get("name"))
}
func readMeta(dir string) (*monolithArtifact, error) {
data, err := os.ReadFile(filepath.Join(dir, "meta.json"))
if err != nil {
return nil, err
}
var meta monolithArtifact
return &meta, json.Unmarshal(data, &meta)
}
func writeMeta(dir string, meta *monolithArtifact) error {
data, err := json.Marshal(meta)
if err != nil {
return err
}
return os.WriteFile(filepath.Join(dir, "meta.json"), data, 0o644)
}
func blobURL(r *http.Request, endpoint, runID, name string) string {
return "http://" + r.Host + artifactServicePath + "/" + endpoint + "?" + url.Values{"run": {runID}, "name": {name}}.Encode()
}
func (s *serverV4) createArtifact(r *http.Request, req createArtifactRequest) (createArtifactResponse, error) { // an exec invocation re-runs its workflows, so it replaces earlier invocations' artifacts but conflicts with its own
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
s.mu.Lock()
defer s.mu.Unlock()
if s.created[dir] {
return createArtifactResponse{}, &twirpError{http.StatusConflict, "already_exists", "an artifact with this name already exists on the workflow run"}
}
if err := os.RemoveAll(dir); err != nil {
return createArtifactResponse{}, err
}
if err := os.MkdirAll(filepath.Join(dir, "blocks"), 0o755); err != nil {
return createArtifactResponse{}, err
}
s.lastID++
if err := writeMeta(dir, &monolithArtifact{
WorkflowRunBackendID: req.WorkflowRunBackendID,
DatabaseID: s.lastID,
Name: req.Name,
CreatedAt: time.Now().UTC().Truncate(time.Second),
}); err != nil {
return createArtifactResponse{}, err
}
s.created[dir] = true
return createArtifactResponse{Ok: true, SignedUploadURL: blobURL(r, "UploadArtifact", req.WorkflowRunBackendID, req.Name)}, nil
}
type blockList struct {
Blocks []string `xml:",any"` // Latest, Committed and Uncommitted all name a staged block
}
func (s *serverV4) uploadArtifact(w http.ResponseWriter, r *http.Request) {
dir := s.urlDir(r)
if meta, err := readMeta(dir); err != nil || meta.Digest != "" {
http.Error(w, "no artifact awaits an upload", http.StatusNotFound)
return
}
contentType := r.Header.Get("x-ms-blob-content-type")
var err error
switch comp := r.URL.Query().Get("comp"); comp {
case "block":
err = writeFile(filepath.Join(dir, "blocks", fileName(r.URL.Query().Get("blockid"))), r.Body)
case "blocklist":
err = commitBlob(dir, contentType, func(out io.Writer) error { return joinBlocks(out, dir, r.Body) })
case "": // Put Blob, which docker/build-push-action uses
err = commitBlob(dir, contentType, func(out io.Writer) error {
_, err := io.Copy(out, r.Body)
return err
})
default:
http.Error(w, "unsupported comp "+comp, http.StatusBadRequest)
return
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusCreated)
}
func writeFile(name string, body io.Reader) error {
file, err := os.Create(name)
if err != nil {
return err
}
_, err = io.Copy(file, body)
return errors.Join(err, file.Close())
}
func commitBlob(dir, contentType string, write func(io.Writer) error) error { // replaces the blob whole and keeps its content type, as Azure does
out, err := os.CreateTemp(dir, "blob")
if err != nil {
return err
}
if err = errors.Join(write(out), out.Close()); err == nil {
err = os.Rename(out.Name(), filepath.Join(dir, "blob"))
}
if err != nil {
return errors.Join(err, os.Remove(out.Name()))
}
return os.WriteFile(filepath.Join(dir, "content-type"), []byte(contentType), 0o644)
}
func joinBlocks(out io.Writer, dir string, body io.Reader) error { // keeps the blocks for finalizeArtifact, so that a retried commit can join them again
var list blockList
if err := xml.NewDecoder(io.LimitReader(body, maxTwirpBytes)).Decode(&list); err != nil {
return err
}
for _, id := range list.Blocks {
if err := appendBlock(out, filepath.Join(dir, "blocks", fileName(id))); err != nil {
return err
}
}
return nil
}
func appendBlock(out io.Writer, name string) error {
in, err := os.Open(name)
if err != nil {
return err
}
defer in.Close()
_, err = io.Copy(out, in)
return err
}
func (s *serverV4) finalizeArtifact(_ *http.Request, req finalizeArtifactRequest) (finalizeArtifactResponse, error) {
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
meta, err := readMeta(dir)
if err != nil {
return finalizeArtifactResponse{}, notFound(req.Name)
} else if meta.Digest != "" {
return finalizeArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, nil
}
blob, err := os.Open(filepath.Join(dir, "blob"))
if err != nil {
return finalizeArtifactResponse{}, badRequest("artifact %q has no upload", req.Name)
}
defer blob.Close()
hash := sha256.New()
size, err := io.Copy(hash, blob)
if err != nil {
return finalizeArtifactResponse{}, err
}
digest := "sha256:" + hex.EncodeToString(hash.Sum(nil))
if size != req.Size {
return finalizeArtifactResponse{}, badRequest("size %d does not match the %d bytes uploaded", req.Size, size)
}
if req.Hash != "" && req.Hash != digest {
return finalizeArtifactResponse{}, badRequest("hash %s does not match the uploaded %s", req.Hash, digest)
}
s.mu.Lock() // only after hashing, so finalizing one large artifact does not stall the others
defer s.mu.Unlock()
if current, err := readMeta(dir); err != nil || current.DatabaseID != meta.DatabaseID {
return finalizeArtifactResponse{}, notFound(req.Name)
}
meta.Size, meta.Digest = size, digest
if err := writeMeta(dir, meta); err != nil {
return finalizeArtifactResponse{}, err
}
return finalizeArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, os.RemoveAll(filepath.Join(dir, "blocks"))
}
func (s *serverV4) listArtifacts(_ *http.Request, req listArtifactsRequest) (listArtifactsResponse, error) {
run := s.runDir(req.WorkflowRunBackendID)
entries, err := os.ReadDir(run)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return listArtifactsResponse{}, err
}
var resp listArtifactsResponse
for _, entry := range entries {
meta, err := readMeta(filepath.Join(run, entry.Name()))
if err != nil || meta.Digest == "" ||
req.NameFilter != "" && req.NameFilter != meta.Name || req.IDFilter != 0 && req.IDFilter != meta.DatabaseID {
continue
}
resp.Artifacts = append(resp.Artifacts, *meta)
}
slices.SortFunc(resp.Artifacts, func(left, right monolithArtifact) int { return cmp.Compare(left.DatabaseID, right.DatabaseID) })
return resp, nil
}
func (s *serverV4) getSignedArtifactURL(r *http.Request, req getSignedArtifactURLRequest) (getSignedArtifactURLResponse, error) {
if meta, err := readMeta(s.artifactDir(req.WorkflowRunBackendID, req.Name)); err != nil || meta.Digest == "" {
return getSignedArtifactURLResponse{}, notFound(req.Name)
}
return getSignedArtifactURLResponse{SignedURL: blobURL(r, "DownloadArtifact", req.WorkflowRunBackendID, req.Name)}, nil
}
func (s *serverV4) downloadArtifact(w http.ResponseWriter, r *http.Request) {
dir := s.urlDir(r)
meta, err := readMeta(dir)
if err != nil || meta.Digest == "" {
http.Error(w, "artifact not found", http.StatusNotFound)
return
}
blob, err := os.Open(filepath.Join(dir, "blob"))
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
defer blob.Close()
data, _ := os.ReadFile(filepath.Join(dir, "content-type"))
contentType, filename := string(data), meta.Name
if !strings.Contains(contentType, "/") {
contentType = zipMimeType
}
if contentType == zipMimeType && !strings.HasSuffix(filename, ".zip") {
filename += ".zip"
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Disposition", "attachment; filename*=UTF-8''"+strings.ReplaceAll(url.QueryEscape(filename), "+", "%20")) // RFC 5987 attr-chars only, which the toolkit and mime.ParseMediaType both accept
http.ServeContent(w, r, "", meta.CreatedAt, blob)
}
func (s *serverV4) deleteArtifact(_ *http.Request, req deleteArtifactRequest) (deleteArtifactResponse, error) {
dir := s.artifactDir(req.WorkflowRunBackendID, req.Name)
s.mu.Lock()
defer s.mu.Unlock()
meta, err := readMeta(dir)
if err != nil {
return deleteArtifactResponse{}, notFound(req.Name)
}
delete(s.created, dir)
return deleteArtifactResponse{Ok: true, ArtifactID: meta.DatabaseID}, os.RemoveAll(dir)
}
+23
View File
@@ -0,0 +1,23 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import (
"crypto/rand"
"encoding/base64"
"encoding/json/v2"
"time"
)
func LocalRuntimeToken(run string, lifetime time.Duration) string { // an unsecured JWT with the claims the toolkit and buildx read, its run hashed for scp and its jti an unguessable bearer
now := time.Now()
claims, _ := json.Marshal(map[string]any{
"scp": "Actions.Results:" + fileName(run) + ":1",
"ac": `[{"Scope":"","Permission":3}]`,
"nbf": now.Unix(),
"exp": now.Add(lifetime).Unix(),
"jti": rand.Text(),
})
return base64.RawURLEncoding.EncodeToString([]byte(`{"alg":"none","typ":"JWT"}`)) + "." + base64.RawURLEncoding.EncodeToString(claims) + "."
}
+55
View File
@@ -0,0 +1,55 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifacts
import "time"
const artifactServicePath = "/twirp/github.actions.results.api.v1.ArtifactService"
type ( // the ArtifactService messages in the protobuf JSON mapping, which writes int64 as strings
createArtifactResponse struct {
Ok bool `json:"ok"`
SignedUploadURL string `json:"signedUploadUrl"`
}
finalizeArtifactRequest struct {
WorkflowRunBackendID string
Name string
Size int64 `json:",string"`
Hash string
}
finalizeArtifactResponse struct {
Ok bool `json:"ok"`
ArtifactID int64 `json:"artifactId,string"`
}
listArtifactsRequest struct {
WorkflowRunBackendID string
NameFilter string
IDFilter int64 `json:",string"`
}
listArtifactsResponse struct {
Artifacts []monolithArtifact `json:"artifacts"`
}
monolithArtifact struct {
WorkflowRunBackendID string `json:"workflowRunBackendId"` // the toolkit sends it back to GetSignedArtifactURL
DatabaseID int64 `json:"databaseId,string"`
Name string `json:"name"`
Size int64 `json:"size,string"`
CreatedAt time.Time `json:"createdAt"` // whole seconds, the toolkit rejects the trimmed fractions json writes
Digest string `json:"digest"` // empty until the upload is finalized
}
getSignedArtifactURLRequest struct {
WorkflowRunBackendID string
Name string
}
getSignedArtifactURLResponse struct {
SignedURL string `json:"signedUrl"`
}
createArtifactRequest = getSignedArtifactURLRequest
deleteArtifactRequest = getSignedArtifactURLRequest
deleteArtifactResponse = finalizeArtifactResponse
)
+28 -1
View File
@@ -7,7 +7,9 @@ package common
import ( import (
"context" "context"
"fmt" "fmt"
"math/rand/v2"
"runtime/debug" "runtime/debug"
"time"
"gitea.com/gitea/runner/act/common/logging" "gitea.com/gitea/runner/act/common/logging"
) )
@@ -158,13 +160,38 @@ func (e Executor) IfBool(conditional bool) Executor {
}) })
} }
// RetryBackoff is the wait between Retry's attempts, random between 1 and 10 seconds like GitHub's runner.
var RetryBackoff = func() time.Duration { return time.Second + rand.N(9*time.Second) }
// Retry runs the executor up to attempts times until it succeeds.
func (e Executor) Retry(attempts int) Executor {
return func(ctx context.Context) error {
for attempt := 1; ; attempt++ {
err := e(ctx)
if err == nil || attempt == attempts || ctx.Err() != nil {
return err
}
backoff := RetryBackoff()
Logger(ctx).Warnf("%v, retrying in %s", err, backoff.Round(time.Second))
select {
case <-ctx.Done():
return err
case <-time.After(backoff):
}
}
}
}
// Finally adds an executor to run after other executor // Finally adds an executor to run after other executor
func (e Executor) Finally(finally Executor) Executor { func (e Executor) Finally(finally Executor) Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
err := e(ctx) err := e(ctx)
err2 := finally(ctx) err2 := finally(ctx)
if err2 != nil { if err2 != nil {
return fmt.Errorf("error occurred running finally: %v (original error: %v)", err2, err) if err != nil {
return fmt.Errorf("error occurred running finally: %w (original error: %w)", err2, err)
}
return fmt.Errorf("error occurred running finally: %w (original error: %v)", err2, err)
} }
return err return err
} }
+35 -4
View File
@@ -7,9 +7,11 @@ package common
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"sync" "sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -133,10 +135,39 @@ func TestExecutorConditionalsAndFinally(t *testing.T) {
assert.Equal(t, []string{"if-true", "if-not", "if-bool", "main", "finally"}, calls) assert.Equal(t, []string{"if-true", "if-not", "if-bool", "main", "finally"}, calls)
} }
func TestExecutorRetryStopsOnSuccessOrAfterAllAttempts(t *testing.T) {
backoff := RetryBackoff
RetryBackoff = func() time.Duration { return 0 }
t.Cleanup(func() { RetryBackoff = backoff })
transient := errors.New("transient")
for _, tc := range []struct {
failures, calls int
err error
}{
{failures: 2, calls: 3},
{failures: 5, calls: 3, err: transient},
} {
var calls int
err := Executor(func(context.Context) error {
calls++
if calls <= tc.failures {
return transient
}
return nil
}).Retry(3)(t.Context())
require.Equal(t, tc.err, err)
require.Equal(t, tc.calls, calls)
}
}
func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) { func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
mainErr := errors.New("main failed")
finalErr := errors.New("cleanup failed") finalErr := errors.New("cleanup failed")
for _, mainErr := range []error{context.Canceled, nil} {
err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context()) err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context())
require.EqualError(t, err, "error occurred running finally: cleanup failed (original error: main failed)") require.EqualError(t, err, fmt.Sprintf("error occurred running finally: cleanup failed (original error: %v)", mainErr))
require.ErrorIs(t, err, finalErr)
if mainErr != nil {
require.ErrorIs(t, err, mainErr)
}
}
} }
+246 -16
View File
@@ -8,7 +8,9 @@ import (
"bytes" "bytes"
"compress/gzip" "compress/gzip"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt"
"io" "io"
"math/rand/v2" "math/rand/v2"
"net" "net"
@@ -28,6 +30,17 @@ import (
var actionFetches sync.Map var actionFetches sync.Map
var symlink = (*os.Root).Symlink
var symlinkFallback = runtime.GOOS == "windows"
const treeSizes = ".sizes" // beside the tree rather than in it, which is copied into job containers as it is
type treeFile struct {
Name []byte // not a string, which JSON would turn into UTF-8, as git allows any bytes but NUL
Size int64
}
type actionFetch struct { type actionFetch struct {
done chan struct{} done chan struct{}
dir, sha string dir, sha string
@@ -86,9 +99,14 @@ func fetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, s
record := filepath.Join(input.Dir, "ref") record := filepath.Join(input.Dir, "ref")
data, _ := os.ReadFile(record) data, _ := os.ReadFile(record)
sha, etag, _ := strings.Cut(string(data), "\n") sha, etag, _ := strings.Cut(string(data), "\n")
if !isFullObjectID(sha) || !exists(tree(sha)) { switch {
case !isFullObjectID(sha):
sha, etag = "", "" sha, etag = "", ""
} else if input.OfflineMode || isFullObjectID(input.Ref) { case input.OfflineMode && exists(tree(sha)) && !exists(tree(sha)+treeSizes):
return tree(sha), sha, nil // from an older runner, with no sizes to check, so offline mode uses it as before
case !treeIntact(input.Dir, sha):
sha, etag = "", ""
case input.OfflineMode || isFullObjectID(input.Ref):
return tree(sha), sha, nil return tree(sha), sha, nil
} }
@@ -173,7 +191,7 @@ func downloadArchiveOnce(ctx context.Context, client *http.Client, archive, toke
newEtag := response.Header.Get("ETag") newEtag := response.Header.Get("ETag")
gz, err := gzip.NewReader(response.Body) gz, err := gzip.NewReader(response.Body)
if err == nil { if err == nil {
sha, err = extractTree(gz, dir, !strings.HasSuffix(newEtag, `-false"`)) // Gitea's mark of an unprefixed archive sha, err = extractTree(ctx, gz, dir, !strings.HasSuffix(newEtag, `-false"`)) // Gitea's mark of an unprefixed archive
} }
var netErr net.Error var netErr net.Error
if errors.Is(err, io.ErrUnexpectedEOF) || errors.As(err, &netErr) { if errors.Is(err, io.ErrUnexpectedEOF) || errors.As(err, &netErr) {
@@ -191,18 +209,18 @@ func exportClone(ctx context.Context, input NewGitCloneExecutorInput) (string, e
clone := input clone := input
clone.Dir = filepath.Join(input.Dir, "git") clone.Dir = filepath.Join(input.Dir, "git")
sha, _, err := resolveClone(ctx, clone) sha, _, err := resolveClone(ctx, clone)
if err != nil || exists(filepath.Join(input.Dir, sha)) { if err != nil || treeIntact(input.Dir, sha) {
return sha, err return sha, err
} }
archive, err := runGit(ctx, clone.Dir, nil, "archive", "--format=tar", sha) archive, err := runGit(ctx, clone.Dir, nil, "archive", "--format=tar", sha)
if err != nil { if err != nil {
return "", err return "", err
} }
_, err = extractTree(bytes.NewReader(archive), input.Dir, false) _, err = extractTree(ctx, bytes.NewReader(archive), input.Dir, false)
return sha, err return sha, err
} }
func extractTree(archive io.Reader, dir string, strip bool) (string, error) { // into dir/<commit of the pax header>, strip dropping the directory wrapping the files func extractTree(ctx context.Context, archive io.Reader, dir string, strip bool) (string, error) { // into dir/<commit of the pax header>, strip dropping the directory wrapping the files
temp, err := os.MkdirTemp(dir, "tmp-") temp, err := os.MkdirTemp(dir, "tmp-")
if err != nil { if err != nil {
return "", err return "", err
@@ -215,6 +233,9 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
defer root.Close() defer root.Close()
sha := "" sha := ""
sizes := map[string]int64{}
links := map[string]string{}
var deferred []string
reader := tar.NewReader(archive) reader := tar.NewReader(archive)
for { for {
header, err := reader.Next() header, err := reader.Next()
@@ -225,7 +246,7 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
return "", err return "", err
} }
if header.Typeflag == tar.TypeXGlobalHeader { if header.Typeflag == tar.TypeXGlobalHeader {
if sha = header.PAXRecords["comment"]; isFullObjectID(sha) && exists(filepath.Join(dir, sha)) { if sha = header.PAXRecords["comment"]; isFullObjectID(sha) && treeIntact(dir, sha) {
return sha, nil return sha, nil
} }
continue continue
@@ -241,11 +262,10 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
case tar.TypeDir: case tar.TypeDir:
err = root.MkdirAll(name, 0o755) err = root.MkdirAll(name, 0o755)
case tar.TypeReg: case tar.TypeReg:
err = writeTreeFile(root, name, header.Mode, reader) sizes[name], err = writeTreeFile(root, name, header.Mode, reader)
case tar.TypeSymlink: case tar.TypeSymlink:
if err = root.Symlink(header.Linkname, name); err != nil && runtime.GOOS == "windows" { links[filepath.Clean(name)] = header.Linkname
err = writeTreeFile(root, name, 0, strings.NewReader(header.Linkname)) deferred = append(deferred, filepath.Clean(name))
}
} }
if err != nil { if err != nil {
return "", err return "", err
@@ -254,23 +274,233 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
if !isFullObjectID(sha) { if !isFullObjectID(sha) {
return "", errors.New("the archive names no commit") return "", errors.New("the archive names no commit")
} }
linksByParent := map[string][]string{}
if symlinkFallback {
for name := range links {
linksByParent[filepath.Dir(name)] = append(linksByParent[filepath.Dir(name)], filepath.Base(name))
}
}
for requireTarget := symlinkFallback; len(deferred) > 0; {
unresolved := deferred[:0]
for _, name := range deferred {
if err := ctx.Err(); err != nil {
return "", err
}
if requireTarget {
if _, err := root.Stat(filepath.Dir(name) + string(filepath.Separator) + links[name]); err != nil {
unresolved = append(unresolved, name)
continue
}
}
if err := symlink(root, links[name], name); err == nil || !symlinkFallback {
if err != nil {
return "", err
}
continue
}
if err := copyTreeLink(ctx, root, name, links, linksByParent, sizes); err != nil {
return "", fmt.Errorf("materialize symlink %q: %w", filepath.ToSlash(name), err)
}
}
if len(unresolved) == len(deferred) {
requireTarget = false
}
deferred = unresolved
}
if err := root.Close(); err != nil { if err := root.Close(); err != nil {
return "", err return "", err
} }
return sha, os.Rename(temp, filepath.Join(dir, sha)) return sha, installTree(ctx, dir, sha, temp, sizes)
} }
func writeTreeFile(root *os.Root, name string, mode int64, content io.Reader) error { func resolveTreeLink(name string, links map[string]string) (string, bool) {
if !filepath.IsLocal(name) {
return "", false
}
parts := strings.Split(name, string(filepath.Separator))
var resolved []string
for count := 0; len(parts) > 0; {
part := parts[0]
parts = parts[1:]
switch part {
case "", ".":
continue
case "..":
if len(resolved) == 0 {
return "", false
}
resolved = resolved[:len(resolved)-1]
continue
}
target, ok := links[filepath.Join(append(resolved, part)...)]
if !ok {
resolved = append(resolved, part)
continue
}
count++
if target == "" || os.IsPathSeparator(target[0]) || filepath.VolumeName(target) != "" || count > 255 {
return "", false
}
parts = append(strings.Split(filepath.FromSlash(target), string(filepath.Separator)), parts...)
}
return filepath.Join(append([]string{"."}, resolved...)...), true
}
var (
errTreeLinkTarget = errors.New("unmaterializable symlink target")
errTreeLinkCycle = fmt.Errorf("cyclic %w", errTreeLinkTarget)
)
func copyTreeLink(ctx context.Context, root *os.Root, name string, links map[string]string, linksByParent map[string][]string, sizes map[string]int64) error {
writeLinkText := func(destination, target string) error { // as git writes links it cannot create
if err := root.RemoveAll(destination); err != nil {
return err
}
for written := range sizes {
if written == destination || strings.HasPrefix(written, destination+string(filepath.Separator)) {
delete(sizes, written)
}
}
size, err := writeTreeFile(root, destination, 0o644, strings.NewReader(target))
sizes[destination] = size
return err
}
active := map[string]bool{}
var copyTarget func(string, string) error
copyTarget = func(source, destination string) error {
if err := ctx.Err(); err != nil {
return err
}
source, ok := resolveTreeLink(source, links)
if !ok {
return errTreeLinkTarget
}
file, err := root.Open(source)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return errTreeLinkTarget
}
return err
}
defer file.Close()
info, err := file.Stat()
if err != nil {
return err
}
if info.Mode().IsRegular() {
sizes[destination], err = writeTreeFile(root, destination, int64(info.Mode().Perm()), file)
return err
}
if !info.IsDir() {
return errTreeLinkTarget
}
if active[source] {
return errTreeLinkCycle
}
active[source] = true
defer delete(active, source)
entries, err := file.ReadDir(-1)
if err != nil {
return err
}
children := map[string]bool{}
for _, entry := range entries {
children[entry.Name()] = true
}
for _, child := range linksByParent[source] {
children[child] = true
}
if err := root.MkdirAll(destination, info.Mode().Perm()); err != nil {
return err
}
for child := range children {
childSource, childDestination := filepath.Join(source, child), filepath.Join(destination, child)
err := copyTarget(childSource, childDestination)
if target, isLink := links[childSource]; isLink && errors.Is(err, errTreeLinkTarget) && !errors.Is(err, errTreeLinkCycle) {
err = writeLinkText(childDestination, target)
}
if err != nil {
return err
}
}
return nil
}
if err := copyTarget(name, name); !errors.Is(err, errTreeLinkTarget) {
return err
}
if err := ctx.Err(); err != nil {
return err
}
return writeLinkText(name, links[name])
}
var lockTree = AcquireCloneLock // replaced in tests to see installTree wait for a job holding the tree
// installTree moves temp into place as dir/sha, replacing a tree that failed treeIntact.
func installTree(ctx context.Context, dir, sha, temp string, sizes map[string]int64) error {
files := make([]treeFile, 0, len(sizes))
for name, size := range sizes {
files = append(files, treeFile{[]byte(name), size})
}
data, err := json.Marshal(files)
if err != nil {
return err
}
// Written before the swap: it describes the commit, not a copy of it, so it holds for whichever tree is left.
if err := os.WriteFile(filepath.Join(dir, sha+treeSizes), data, 0o600); err != nil { // a torn record fails treeIntact, so it needs no rename
return err
}
tree := filepath.Join(dir, sha)
if !exists(tree) {
return os.Rename(temp, tree)
}
stale, err := os.MkdirTemp(dir, "tmp-")
if err != nil {
return err
}
defer os.RemoveAll(stale) // after the unlock, as no job reads the old tree once it is moved
unlock, err := lockTree(ctx, tree) // jobs copying the old tree hold it
if err != nil {
return err
}
defer unlock()
if err := os.Rename(tree, filepath.Join(stale, sha)); err != nil {
return err
}
return os.Rename(temp, tree)
}
// treeIntact reports whether dir/sha still holds each file extracted into it at the size it was written.
// Only sizes are compared, not content: a stat per file is cheap enough for every job, hashing is not.
func treeIntact(dir, sha string) bool {
data, err := os.ReadFile(filepath.Join(dir, sha+treeSizes))
if err != nil || !exists(filepath.Join(dir, sha)) {
return false
}
var files []treeFile
if json.Unmarshal(data, &files) != nil {
return false
}
for _, file := range files {
info, err := os.Lstat(filepath.Join(dir, sha, filepath.FromSlash(string(file.Name))))
if err != nil || !info.Mode().IsRegular() || info.Size() != file.Size {
return false
}
}
return true
}
func writeTreeFile(root *os.Root, name string, mode int64, content io.Reader) (int64, error) {
perm := os.FileMode(0o644) perm := os.FileMode(0o644)
if mode&0o111 != 0 { if mode&0o111 != 0 {
perm = 0o755 perm = 0o755
} }
file, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm) file, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm)
if err != nil { if err != nil {
return err return 0, err
} }
_, err = io.Copy(file, content) n, err := io.Copy(file, content)
return errors.Join(err, file.Close()) return n, errors.Join(err, file.Close())
} }
func exists(name string) bool { func exists(name string) bool {
+272 -14
View File
@@ -8,6 +8,7 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/base64" "encoding/base64"
"errors"
"fmt" "fmt"
"net/http" "net/http"
"net/http/cgi" "net/http/cgi"
@@ -15,6 +16,7 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"runtime"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -34,13 +36,15 @@ func TestFetchAction(t *testing.T) {
root := t.TempDir() root := t.TempDir()
remote := filepath.Join(root, "org", "action") remote := filepath.Join(root, "org", "action")
require.NoError(t, gitCmd("init", "--initial-branch=main", remote)) require.NoError(t, os.CopyFS(remote, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, os.WriteFile(filepath.Join(remote, "action.yml"), []byte("name: action\n"), 0o644)) require.NoError(t, os.WriteFile(filepath.Join(remote, "action.yml"), []byte("name: action\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(remote, "run.sh"), []byte("#!/bin/sh\n"), 0o755)) require.NoError(t, os.WriteFile(filepath.Join(remote, "run.sh"), []byte("#!/bin/sh\n"), 0o755))
require.NoError(t, os.Symlink("action.yml", filepath.Join(remote, "link"))) require.NoError(t, os.Symlink("action.yml", filepath.Join(remote, "link")))
require.NoError(t, gitCmd("-C", remote, "add", ".")) require.NoError(t, gitCmd("-C", remote, "add", "."))
require.NoError(t, gitCmd("-C", remote, "update-index", "--chmod=+x", "run.sh"))
require.NoError(t, gitCmd("-C", remote, "commit", "-m", "initial")) require.NoError(t, gitCmd("-C", remote, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", remote, "tag", "v1")) require.NoError(t, gitCmd("-C", remote, "tag", "v1"))
expectedSha := gitRevParse(t, remote, "main")
gitPath, err := exec.LookPath("git") gitPath, err := exec.LookPath("git")
require.NoError(t, err) require.NoError(t, err)
@@ -48,6 +52,8 @@ func TestFetchAction(t *testing.T) {
var mu sync.Mutex var mu sync.Mutex
var archiveRequests []http.Header var archiveRequests []http.Header
var mirror *httptest.Server var mirror *httptest.Server
archiveCache := map[string][]byte{}
revisions := map[string]string{}
requests, archives, failures := 0, "", 0 requests, archives, failures := 0, "", 0
handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
mu.Lock() mu.Lock()
@@ -74,7 +80,14 @@ func TestFetchAction(t *testing.T) {
http.Redirect(writer, request, mirror.URL+request.URL.Path, http.StatusFound) http.Redirect(writer, request, mirror.URL+request.URL.Path, http.StatusFound)
return return
} }
sha, _ := gitOutput(request.Context(), remote, "rev-parse", strings.TrimSuffix(ref, ".tar.gz")) sha, ok := revisions[ref]
if !ok {
var err error
sha, err = gitOutput(request.Context(), remote, "rev-parse", strings.TrimSuffix(ref, ".tar.gz"))
if err == nil {
revisions[ref] = sha
}
}
etag := fmt.Sprintf(`W/"%s-tar.gz-%t"`, sha, archives != "unprefixed") etag := fmt.Sprintf(`W/"%s-tar.gz-%t"`, sha, archives != "unprefixed")
if request.Header.Get("If-None-Match") == etag { if request.Header.Get("If-None-Match") == etag {
writer.WriteHeader(http.StatusNotModified) writer.WriteHeader(http.StatusNotModified)
@@ -84,7 +97,12 @@ func TestFetchAction(t *testing.T) {
if archives != "unprefixed" { if archives != "unprefixed" {
args = append(args, "--prefix=action/") args = append(args, "--prefix=action/")
} }
archive, _ := runGit(request.Context(), remote, nil, args...) archiveKey := strings.Join(args, "\x00")
archive, ok := archiveCache[archiveKey]
if !ok {
archive, _ = runGit(request.Context(), remote, nil, args...)
archiveCache[archiveKey] = archive
}
writer.Header().Set("ETag", etag) writer.Header().Set("ETag", etag)
_, _ = writer.Write(archive) _, _ = writer.Write(archive)
}) })
@@ -107,7 +125,6 @@ func TestFetchAction(t *testing.T) {
} }
t.Run("shares identical fetches and isolates cancellation, panics and differing inputs", func(t *testing.T) { t.Run("shares identical fetches and isolates cancellation, panics and differing inputs", func(t *testing.T) {
expectedSha := gitRevParse(t, remote, "main")
for _, scenario := range []string{"shared", "leader cancelled", "leader panics", "different token"} { for _, scenario := range []string{"shared", "leader cancelled", "leader panics", "different token"} {
t.Run(scenario, func(t *testing.T) { t.Run(scenario, func(t *testing.T) {
serve("prefixed", 0) serve("prefixed", 0)
@@ -180,12 +197,14 @@ func TestFetchAction(t *testing.T) {
input := newInput("v1") input := newInput("v1")
tree, sha, err := FetchAction(t.Context(), input) tree, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "v1"), sha) assert.Equal(t, expectedSha, sha)
assert.Equal(t, filepath.Join(input.Dir, sha), tree) assert.Equal(t, filepath.Join(input.Dir, sha), tree)
assert.NoDirExists(t, filepath.Join(tree, ".git")) assert.NoDirExists(t, filepath.Join(tree, ".git"))
info, err := os.Stat(filepath.Join(tree, "run.sh")) info, err := os.Stat(filepath.Join(tree, "run.sh"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, os.FileMode(0o755), info.Mode().Perm()) if runtime.GOOS != "windows" {
assert.EqualValues(t, 0o755, info.Mode().Perm())
}
link, err := os.Readlink(filepath.Join(tree, "link")) link, err := os.Readlink(filepath.Join(tree, "link"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "action.yml", link) assert.Equal(t, "action.yml", link)
@@ -196,9 +215,13 @@ func TestFetchAction(t *testing.T) {
require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved")) require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved"))
require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1")) require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1"))
expectedSha = gitRevParse(t, remote, "v1")
mu.Lock()
clear(revisions)
mu.Unlock()
moved, movedSha, err := FetchAction(t.Context(), input) moved, movedSha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "v1"), movedSha) assert.Equal(t, expectedSha, movedSha)
assert.NotEqual(t, tree, moved) assert.NotEqual(t, tree, moved)
assert.FileExists(t, filepath.Join(tree, "action.yml")) assert.FileExists(t, filepath.Join(tree, "action.yml"))
@@ -206,10 +229,17 @@ func TestFetchAction(t *testing.T) {
_, offlineSha, err := FetchAction(t.Context(), input) _, offlineSha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, movedSha, offlineSha) assert.Equal(t, movedSha, offlineSha)
require.NoError(t, os.Truncate(filepath.Join(moved, "action.yml"), 4)) // fetched again, offline or not
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err)
require.NoError(t, os.Remove(moved+treeSizes)) // from an older runner, used as it is offline
reused, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, moved, reused)
requests, headers := served() requests, headers := served()
assert.Equal(t, 3, requests) assert.Equal(t, 4, requests)
require.Len(t, headers, 3) require.Len(t, headers, 4)
assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), headers[0].Get("Authorization")) assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), headers[0].Get("Authorization"))
assert.Empty(t, headers[0].Get("If-None-Match")) assert.Empty(t, headers[0].Get("If-None-Match"))
assert.Equal(t, fmt.Sprintf(`W/"%s-tar.gz-true"`, sha), headers[1].Get("If-None-Match")) assert.Equal(t, fmt.Sprintf(`W/"%s-tar.gz-true"`, sha), headers[1].Get("If-None-Match"))
@@ -218,7 +248,7 @@ func TestFetchAction(t *testing.T) {
t.Run("downloads a pinned commit once and rejects a short one", func(t *testing.T) { t.Run("downloads a pinned commit once and rejects a short one", func(t *testing.T) {
serve("prefixed", 0) serve("prefixed", 0)
sha := gitRevParse(t, remote, "main") sha := expectedSha
input := newInput(sha) input := newInput(sha)
for range 2 { for range 2 {
tree, got, err := FetchAction(t.Context(), input) tree, got, err := FetchAction(t.Context(), input)
@@ -233,12 +263,74 @@ func TestFetchAction(t *testing.T) {
assert.NoDirExists(t, filepath.Join(input.Dir, "git")) assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
}) })
t.Run("downloads again over a tree whose files changed size or that has no record, not under a job copying it", func(t *testing.T) {
sha := expectedSha
for _, mode := range []string{"prefixed", "none"} {
serve(mode, 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.WriteFile(filepath.Join(tree, "trampoline.js"), nil, 0o644)) // the runner's own, no damage
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err, mode)
for _, damage := range []func() error{
func() error { return os.Truncate(yml, 4) },
func() error { return os.Remove(tree + treeSizes) },
} {
require.NoError(t, damage(), mode)
_, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
body, err := os.ReadFile(yml)
require.NoError(t, err, mode)
assert.Equal(t, "name: action\n", string(body), mode)
}
_, headers := served()
assert.Len(t, headers, 3, mode)
}
serve("prefixed", 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.Truncate(yml, 4))
unlock, err := AcquireCloneLock(t.Context(), tree)
require.NoError(t, err)
release := sync.OnceFunc(unlock)
defer release()
waiting := make(chan struct{})
lockTree = func(ctx context.Context, dir string) (func(), error) {
close(waiting)
return AcquireCloneLock(ctx, dir)
}
t.Cleanup(func() { lockTree = AcquireCloneLock })
done := make(chan error, 1)
go func() {
_, _, err := FetchAction(t.Context(), input)
done <- err
}()
select {
case err := <-done:
t.Fatalf("FetchAction replaced the tree without waiting for the job holding it: %v", err)
case <-waiting:
}
info, err := os.Stat(yml)
require.NoError(t, err)
assert.Equal(t, int64(4), info.Size(), "the tree a job holds is left alone")
release()
require.NoError(t, <-done)
body, err := os.ReadFile(yml)
require.NoError(t, err)
assert.Equal(t, "name: action\n", string(body))
})
t.Run("retries a failing download and drops the token on a redirect to another origin", func(t *testing.T) { t.Run("retries a failing download and drops the token on a redirect to another origin", func(t *testing.T) {
serve("redirect", 2) serve("redirect", 2)
input := newInput("main") input := newInput("main")
_, sha, err := FetchAction(t.Context(), input) _, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "main"), sha) assert.Equal(t, expectedSha, sha)
assert.NoDirExists(t, filepath.Join(input.Dir, "git")) assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
_, headers := served() _, headers := served()
require.Len(t, headers, 4) require.Len(t, headers, 4)
@@ -251,7 +343,7 @@ func TestFetchAction(t *testing.T) {
serve(mode, 0) serve(mode, 0)
tree, sha, err := FetchAction(t.Context(), newInput("main")) tree, sha, err := FetchAction(t.Context(), newInput("main"))
require.NoError(t, err, mode) require.NoError(t, err, mode)
assert.Equal(t, gitRevParse(t, remote, "main"), sha, mode) assert.Equal(t, expectedSha, sha, mode)
assert.FileExists(t, filepath.Join(tree, "action.yml"), mode) assert.FileExists(t, filepath.Join(tree, "action.yml"), mode)
assert.NoDirExists(t, filepath.Join(tree, ".git"), mode) assert.NoDirExists(t, filepath.Join(tree, ".git"), mode)
_, headers := served() _, headers := served()
@@ -266,7 +358,7 @@ func TestFetchAction(t *testing.T) {
input.URL = "https://example.invalid/org/action" input.URL = "https://example.invalid/org/action"
_, sha, err := FetchAction(t.Context(), input) _, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "main"), sha) assert.Equal(t, expectedSha, sha)
_, headers := served() _, headers := served()
assert.Len(t, headers, 1) assert.Len(t, headers, 1)
assert.NoDirExists(t, filepath.Join(input.Dir, "git")) assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
@@ -306,10 +398,176 @@ func TestExtractTreeConfinesEntries(t *testing.T) {
require.NoError(t, writer.Close()) require.NoError(t, writer.Close())
dir := t.TempDir() dir := t.TempDir()
_, err := extractTree(&archive, dir, false) _, err := extractTree(t.Context(), &archive, dir, false)
require.Error(t, err) require.Error(t, err)
assert.NoFileExists(t, filepath.Join(dir, "escape")) assert.NoFileExists(t, filepath.Join(dir, "escape"))
assert.NoFileExists(t, filepath.Join(outside, "escape")) assert.NoFileExists(t, filepath.Join(outside, "escape"))
}) })
} }
} }
func TestExtractTreeCopiesDeferredLinks(t *testing.T) {
originalSymlink, originalFallback := symlink, symlinkFallback
symlink = func(_root *os.Root, _target, _name string) error { return errors.New("symlink privilege unavailable") }
symlinkFallback = true
t.Cleanup(func() { symlink, symlinkFallback = originalSymlink, originalFallback })
for _, test := range []struct {
name string
links [][2]string
files []string
text bool
}{
{name: "file chain", links: [][2]string{{"index.js", "entry.js"}, {"entry.js", "dist/index.js"}}, files: []string{"index.js", "entry.js"}},
{name: "directory chain", links: [][2]string{{"lib", "bundle"}, {"bundle", "dist"}, {"dist/entry.js", "index.js"}}, files: []string{"lib/index.js", "lib/entry.js", "lib/nested/index.js", "bundle/index.js", "bundle/entry.js", "bundle/nested/index.js"}},
{name: "directory in target path", links: [][2]string{{"index.js", "bundle/nested/index.js"}, {"bundle", "dist"}}, files: []string{"index.js", "bundle/nested/index.js"}},
{name: "parent after directory link", links: [][2]string{{"index.js", "bundle/../index.js"}, {"bundle", "dist/nested"}}, files: []string{"index.js", "bundle/index.js"}},
{name: "outside", links: [][2]string{{"index.js", "../outside.js"}}, text: true},
{name: "absolute", links: [][2]string{{"index.js", t.TempDir()}}, text: true},
{name: "dangling", links: [][2]string{{"index.js", "missing.js"}}, text: true},
{name: "dangling child of directory link", links: [][2]string{{"lib", "dist"}, {"dist/optional", "missing.js"}}, files: []string{"lib/index.js", "lib/nested/index.js"}},
{name: "cycle", links: [][2]string{{"index.js", "entry.js"}, {"entry.js", "index.js"}}, text: true},
{name: "directory cycle", links: [][2]string{{"dist/loop", "."}}, text: true},
} {
t.Run(test.name, func(t *testing.T) {
sha := strings.Repeat("a", 40)
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": sha}}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "dist", Typeflag: tar.TypeDir, Mode: 0o755}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "dist/nested", Typeflag: tar.TypeDir, Mode: 0o755}))
for _, link := range test.links {
require.NoError(t, writer.WriteHeader(&tar.Header{Name: link[0], Typeflag: tar.TypeSymlink, Linkname: link[1]}))
}
content := "console.log('action');\n"
for _, name := range []string{"dist/index.js", "dist/nested/index.js"} {
require.NoError(t, writer.WriteHeader(&tar.Header{Name: name, Typeflag: tar.TypeReg, Mode: 0o755, Size: int64(len(content))}))
_, err := writer.Write([]byte(content))
require.NoError(t, err)
}
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err := extractTree(t.Context(), &archive, dir, false)
require.NoError(t, err)
assert.True(t, treeIntact(dir, sha))
if test.text {
for _, link := range test.links {
data, err := os.ReadFile(filepath.Join(dir, sha, link[0]))
require.NoError(t, err)
assert.Equal(t, link[1], string(data))
info, err := os.Lstat(filepath.Join(dir, sha, link[0]))
require.NoError(t, err)
assert.True(t, info.Mode().IsRegular())
}
}
for _, name := range test.files {
data, err := os.ReadFile(filepath.Join(dir, sha, name))
require.NoError(t, err)
assert.Equal(t, content, string(data))
info, err := os.Lstat(filepath.Join(dir, sha, name))
require.NoError(t, err)
assert.True(t, info.Mode().IsRegular())
if runtime.GOOS != "windows" {
assert.Equal(t, os.FileMode(0o755), info.Mode().Perm())
}
require.NoError(t, os.WriteFile(filepath.Join(dir, sha, name), nil, 0o644))
assert.False(t, treeIntact(dir, sha), name)
require.NoError(t, os.WriteFile(filepath.Join(dir, sha, name), []byte(content), 0o755))
}
})
}
}
func TestTreeIntactFindsNamesThatAreNotUTF8(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("macOS and Windows store no name that is not UTF-8")
}
sha := strings.Repeat("a", 40)
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": sha}}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "caf\xe9.js", Typeflag: tar.TypeReg, Mode: 0o644, Size: 1, Format: tar.FormatGNU}))
_, err := writer.Write([]byte("x"))
require.NoError(t, err)
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err = extractTree(t.Context(), &archive, dir, false)
require.NoError(t, err)
require.FileExists(t, filepath.Join(dir, sha, "caf\xe9.js"))
assert.True(t, treeIntact(dir, sha))
}
func TestExtractTreeCreatesLinksInPlatformOrder(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("requires symlink privileges")
}
original, originalFallback := symlink, symlinkFallback
var created []string
symlink = func(root *os.Root, target, name string) error {
created = append(created, name)
return root.Symlink(target, name)
}
t.Cleanup(func() { symlink, symlinkFallback = original, originalFallback })
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": strings.Repeat("a", 40)}}))
for _, entry := range []*tar.Header{
{Name: "a", Typeflag: tar.TypeSymlink, Linkname: "b"},
{Name: "b", Typeflag: tar.TypeSymlink, Linkname: "dist"},
{Name: "dangling", Typeflag: tar.TypeSymlink, Linkname: "missing"},
{Name: "dist", Typeflag: tar.TypeDir, Mode: 0o755},
} {
require.NoError(t, writer.WriteHeader(entry))
}
require.NoError(t, writer.Close())
for _, test := range []struct {
name string
fallback bool
want []string
}{
{name: "native", want: []string{"a", "b", "dangling"}},
{name: "fallback", fallback: true, want: []string{"b", "a", "dangling"}},
} {
t.Run(test.name, func(t *testing.T) {
symlinkFallback = test.fallback
created = nil
_, err := extractTree(t.Context(), bytes.NewReader(archive.Bytes()), t.TempDir(), false)
require.NoError(t, err)
assert.Equal(t, test.want, created)
})
}
}
func TestCopyTreeLinkCancellationBetweenEntries(t *testing.T) {
root, err := os.OpenRoot(t.TempDir())
require.NoError(t, err)
defer root.Close()
require.NoError(t, root.Mkdir("source", 0o755))
for _, name := range []string{"first", "second"} {
_, err := writeTreeFile(root, filepath.Join("source", name), 0o644, strings.NewReader("content"))
require.NoError(t, err)
}
sizes := map[string]int64{}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
observed := &treeCopyContext{Context: ctx, check: func() {
if len(sizes) == 1 {
cancel()
}
}}
require.ErrorIs(t, copyTreeLink(observed, root, "copy", map[string]string{"copy": "source"}, nil, sizes), context.Canceled)
assert.Len(t, sizes, 1)
}
type treeCopyContext struct {
context.Context
check func()
}
func (ctx *treeCopyContext) Err() error {
ctx.check()
return ctx.Context.Err()
}
+55 -11
View File
@@ -5,11 +5,13 @@
package git package git
import ( import (
"bytes"
"context" "context"
"crypto/sha256" "crypto/sha256"
"encoding/base64" "encoding/base64"
"errors" "errors"
"fmt" "fmt"
"io"
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
@@ -26,6 +28,7 @@ import (
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging" "gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/internal/pkg/lock" "gitea.com/gitea/runner/internal/pkg/lock"
"gitea.com/gitea/runner/internal/pkg/process"
) )
var ( var (
@@ -182,6 +185,7 @@ type NewGitCloneExecutorInput struct {
// For Gitea // For Gitea
InsecureSkipTLS bool InsecureSkipTLS bool
ClientCertFile, ClientKeyFile string // PEM, absolute as git runs in other directories ClientCertFile, ClientKeyFile string // PEM, absolute as git runs in other directories
CAFile string // PEM, absolute, replaces git's CA bundle
Transport http.RoundTripper // for FetchAction's archive downloads, nil for http.DefaultTransport Transport http.RoundTripper // for FetchAction's archive downloads, nil for http.DefaultTransport
} }
@@ -320,9 +324,18 @@ func cachedRef(ref string) string {
func resolveRef(ctx context.Context, dir, ref string) (string, error) { func resolveRef(ctx context.Context, dir, ref string) (string, error) {
if !isFullObjectID(ref) { if !isFullObjectID(ref) {
for _, name := range []string{cachedRef(ref), "refs/tags/" + ref, "refs/remotes/origin/" + ref} { revisions := []string{cachedRef(ref) + "^{commit}", "refs/tags/" + ref + "^{commit}", "refs/remotes/origin/" + ref + "^{commit}"}
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", name+"^{commit}"); err == nil { if strings.ContainsAny(ref, "\x00\n") {
return sha, nil for _, revision := range revisions {
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", revision); err == nil {
return sha, nil
}
}
} else if output, err := runGitWithStdin(ctx, dir, nil, strings.NewReader(strings.Join(revisions, "\n")+"\n"), "cat-file", "--batch-check=%(objectname)"); err == nil {
for sha := range strings.SplitSeq(string(output), "\n") {
if isFullObjectID(sha) {
return sha, nil
}
} }
} }
} }
@@ -377,6 +390,21 @@ func RemoteConfig(token string, insecureSkipTLS bool) []string {
return config return config
} }
// SSLConfig uses OpenSSL on Windows, as Schannel takes no PEM key and ignores sslCAInfo.
func SSLConfig(prefix, certFile, keyFile, caFile, executionOS string) []string {
var config []string
if certFile != "" {
config = append(config, prefix+"sslCert="+certFile, prefix+"sslKey="+keyFile)
}
if caFile != "" {
config = append(config, prefix+"sslCAInfo="+caFile)
}
if config != nil && strings.EqualFold(executionOS, "windows") {
config = append(config, prefix+"sslBackend=openssl")
}
return config
}
// ConfigEnv applies `-c` style entries through the environment, keeping secrets out of argv. // ConfigEnv applies `-c` style entries through the environment, keeping secrets out of argv.
func ConfigEnv(config ...string) map[string]string { func ConfigEnv(config ...string) map[string]string {
env := map[string]string{"GIT_CONFIG_COUNT": strconv.Itoa(len(config))} env := map[string]string{"GIT_CONFIG_COUNT": strconv.Itoa(len(config))}
@@ -427,17 +455,20 @@ func ListFiles(ctx context.Context, dir string) ([]string, error) {
} }
func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, args ...string) ([]byte, error) { func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, args ...string) ([]byte, error) {
return runGitWithStdin(ctx, dir, input, nil, args...)
}
func runGitWithStdin(ctx context.Context, dir string, input *NewGitCloneExecutorInput, stdin io.Reader, args ...string) ([]byte, error) {
config := []string{"core.autocrlf=false", "core.eol=lf"} // keep action files LF config := []string{"core.autocrlf=false", "core.eol=lf"} // keep action files LF
if input != nil { if input != nil {
config = append(config, RemoteConfig(input.Token, input.InsecureSkipTLS)...) config = append(config, RemoteConfig(input.Token, input.InsecureSkipTLS)...)
if input.ClientCertFile != "" { config = append(config, SSLConfig("http.", input.ClientCertFile, input.ClientKeyFile, input.CAFile, runtime.GOOS)...)
config = append(config, "http.sslCert="+input.ClientCertFile, "http.sslKey="+input.ClientKeyFile)
}
} }
if dir != "" { if dir != "" {
args = append([]string{"-C", dir}, args...) args = append([]string{"-C", dir}, args...)
} }
cmd := exec.CommandContext(ctx, "git", args...) cmd := exec.CommandContext(ctx, "git", args...)
cmd.Stdin = stdin
cmd.Env = slices.DeleteFunc(os.Environ(), func(entry string) bool { cmd.Env = slices.DeleteFunc(os.Environ(), func(entry string) bool {
name, _, _ := strings.Cut(entry, "=") name, _, _ := strings.Cut(entry, "=")
return slices.Contains(LocalEnvVars, name) return slices.Contains(LocalEnvVars, name)
@@ -446,15 +477,28 @@ func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, ar
cmd.Env = append(cmd.Env, key+"="+value) cmd.Env = append(cmd.Env, key+"="+value)
} }
cmd.Env = append(cmd.Env, "GIT_TERMINAL_PROMPT=0") cmd.Env = append(cmd.Env, "GIT_TERMINAL_PROMPT=0")
if runtime.GOOS != "windows" { var treeKill *process.TreeKill
if runtime.GOOS == "windows" {
treeKill = process.NewTreeKill(cmd) // a cancelled git leaves its remote helper holding the repository otherwise
} else {
cmd.Cancel = func() error { return cmd.Process.Signal(os.Interrupt) } // lets git remove its lock files cmd.Cancel = func() error { return cmd.Process.Signal(os.Interrupt) } // lets git remove its lock files
} }
cmd.WaitDelay = time.Second cmd.WaitDelay = time.Second
var stdout bytes.Buffer
var stderr strings.Builder var stderr strings.Builder
cmd.Stdout = &stdout
cmd.Stderr = &stderr cmd.Stderr = &stderr
output, err := cmd.Output() err := cmd.Start()
if err != nil { if err == nil {
return output, fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String())) if treeKill != nil {
if killer, captureErr := treeKill.Capture(cmd.Process); captureErr == nil {
defer killer.Close()
}
}
err = cmd.Wait()
} }
return output, nil if err != nil {
return stdout.Bytes(), fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String()))
}
return stdout.Bytes(), nil
} }
+105 -63
View File
@@ -29,6 +29,47 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
var gitFixtureDir string
func TestMain(m *testing.M) {
var err error
gitFixtureDir, err = os.MkdirTemp("", "runner-git-fixture-")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
code := 1
if err = gitCmd("init", "--quiet", "--template=", "--bare", "--initial-branch=main", filepath.Join(gitFixtureDir, "remote")); err == nil {
err = gitCmd("init", "--quiet", "--template=", "--initial-branch=main", filepath.Join(gitFixtureDir, "work"))
}
if err == nil {
code = m.Run()
} else {
fmt.Fprintln(os.Stderr, err)
}
if err := os.RemoveAll(gitFixtureDir); err != nil {
fmt.Fprintln(os.Stderr, err)
code = 1
}
os.Exit(code)
}
func TestSSLConfig(t *testing.T) {
for _, testcase := range []struct {
prefix, certFile, caFile, executionOS string
expected []string
}{
{"http.https://example.com/.", "cert.pem", "ca.pem", "Windows", []string{"http.https://example.com/.sslCert=cert.pem", "http.https://example.com/.sslKey=key.pem", "http.https://example.com/.sslCAInfo=ca.pem", "http.https://example.com/.sslBackend=openssl"}},
{"http.", "", "ca.pem", "windows", []string{"http.sslCAInfo=ca.pem", "http.sslBackend=openssl"}},
{"http.", "cert.pem", "ca.pem", "Linux", []string{"http.sslCert=cert.pem", "http.sslKey=key.pem", "http.sslCAInfo=ca.pem"}},
{"http.", "", "", "Windows", nil},
} {
t.Run(testcase.prefix+testcase.certFile+testcase.caFile+testcase.executionOS, func(t *testing.T) {
assert.Equal(t, testcase.expected, SSLConfig(testcase.prefix, testcase.certFile, "key.pem", testcase.caFile, testcase.executionOS))
})
}
}
func TestFindGitSlug(t *testing.T) { func TestFindGitSlug(t *testing.T) {
for _, testcase := range []struct { for _, testcase := range []struct {
remoteURL string remoteURL string
@@ -71,8 +112,7 @@ func TestFindGitMetadataOfSHA256RepositoryUsesOrigin(t *testing.T) {
t.Parallel() t.Parallel()
basedir := t.TempDir() basedir := t.TempDir()
const remoteURL = "https://github.com/owner/repo.git" const remoteURL = "https://github.com/owner/repo.git"
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", basedir)) require.NoError(t, gitCmd("init", "--template=", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, os.RemoveAll(filepath.Join(basedir, ".git", "hooks")))
require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init")) require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init"))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL)) require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL))
require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git")) require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git"))
@@ -141,9 +181,8 @@ func TestGitFindRef(t *testing.T) {
} { } {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
dir := filepath.Join(basedir, name) dir := filepath.Join(basedir, name)
require.NoError(t, os.MkdirAll(dir, 0o755)) require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "init", "--initial-branch=master")) require.NoError(t, os.WriteFile(filepath.Join(dir, ".git", "HEAD"), []byte("ref: refs/heads/master\n"), 0o644))
require.NoError(t, os.RemoveAll(filepath.Join(dir, ".git", "hooks")))
for _, command := range tt.commands { for _, command := range tt.commands {
require.NoError(t, gitCmd(append([]string{"-C", dir}, command...)...)) require.NoError(t, gitCmd(append([]string{"-C", dir}, command...)...))
} }
@@ -161,25 +200,24 @@ func TestGitFindRef(t *testing.T) {
func TestGitCloneExecutor(t *testing.T) { func TestGitCloneExecutor(t *testing.T) {
remoteDir, workDir := newGitRemote(t) remoteDir, workDir := newGitRemote(t)
require.NoError(t, os.WriteFile(filepath.Join(workDir, "action.sh"), []byte("#!/bin/sh\necho hi\n"), 0o755)) require.NoError(t, os.WriteFile(filepath.Join(workDir, "action.sh"), []byte("#!/bin/sh\necho hi\n"), 0o755))
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh")) require.NoError(t, os.WriteFile(filepath.Join(workDir, ".gitattributes"), []byte("*.sh text\n"), 0o644))
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh", ".gitattributes"))
require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial")) require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "tag", "v2")) require.NoError(t, gitCmd("-C", workDir, "tag", "v2"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v2"))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails")) require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit")) require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "act-fails"))
fullSha := gitRevParse(t, workDir, "main") fullSha := gitRevParse(t, workDir, "main")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:refs/heads/"+fullSha[:4])) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", "v2", "act-fails", "main:refs/heads/"+fullSha[:4]))
hostConfig := filepath.Join(t.TempDir(), "gitconfig") hostConfig := filepath.Join(t.TempDir(), "gitconfig")
require.NoError(t, os.WriteFile(hostConfig, []byte("[core]\n\tautocrlf = true\n[url \""+remoteDir+"\"]\n\tinsteadOf = https://example.invalid/action\n"), 0o644)) require.NoError(t, os.WriteFile(hostConfig, []byte("[core]\n\tautocrlf = true\n\teol = crlf\n[url "+strconv.Quote(remoteDir)+"]\n\tinsteadOf = https://example.invalid/action\n"), 0o644))
t.Setenv("GIT_CONFIG_GLOBAL", hostConfig) t.Setenv("GIT_CONFIG_GLOBAL", hostConfig)
t.Setenv("GIT_DIR", t.TempDir()) t.Setenv("GIT_DIR", t.TempDir())
var clones sync.WaitGroup
for name, tt := range map[string]struct { for name, tt := range map[string]struct {
Err error Err error
Ref string Ref string
@@ -207,40 +245,43 @@ func TestGitCloneExecutor(t *testing.T) {
Ref: fullSha[:4], Ref: fullSha[:4],
}, },
} { } {
t.Run(name, func(t *testing.T) { clones.Go(func() {
dir := t.TempDir() t.Run(name, func(t *testing.T) {
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{ dir := t.TempDir()
URL: "https://example.invalid/action", clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
Ref: tt.Ref, URL: "https://example.invalid/action",
Dir: dir, Ref: tt.Ref,
}) Dir: dir,
})
err := clone(t.Context()) err := clone(t.Context())
if tt.Err != nil { if tt.Err != nil {
require.Error(t, err) require.Error(t, err)
assert.Equal(t, tt.Err, err) assert.Equal(t, tt.Err, err)
assert.Equal(t, tt.Err, clone(t.Context()), "a retry on the cache keeps the error") assert.Equal(t, tt.Err, clone(t.Context()), "a retry on the cache keeps the error")
return return
} }
require.NoError(t, err) require.NoError(t, err)
marker := filepath.Join(dir, "marker") marker := filepath.Join(dir, "marker")
require.NoError(t, os.WriteFile(marker, nil, 0o644)) require.NoError(t, os.WriteFile(marker, nil, 0o644))
require.NoError(t, clone(t.Context())) require.NoError(t, clone(t.Context()))
assert.FileExists(t, marker, "an insteadOf rewrite must not evict the cache") assert.FileExists(t, marker, "an insteadOf rewrite must not evict the cache")
script, err := os.ReadFile(filepath.Join(dir, "action.sh")) script, err := os.ReadFile(filepath.Join(dir, "action.sh"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "#!/bin/sh\necho hi\n", string(script), "host core.autocrlf must not convert action files") assert.Equal(t, "#!/bin/sh\necho hi\n", string(script), "host core.autocrlf must not convert action files")
})
}) })
} }
clones.Wait()
} }
func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) { func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
t.Parallel() t.Parallel()
createRemote := func(message string) string { createRemote := func(message string) string {
remoteDir, workDir := newGitRemote(t, message) dir := t.TempDir()
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", message))
return remoteDir return dir
} }
oldRemoteDir := createRemote("old-action") oldRemoteDir := createRemote("old-action")
@@ -360,31 +401,30 @@ func TestGitCloneExecutorShallow(t *testing.T) {
remoteDir, workDir := newGitRemote(t, "c1", "c2", "c3") remoteDir, workDir := newGitRemote(t, "c1", "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "tag", "v1")) require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1") unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", "v1"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v1"))
shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") } shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") }
for name, ref := range map[string]string{"branch is cloned shallowly": "main", "tag is cloned shallowly": "v1"} { for name, ref := range map[string]string{"branch is cloned shallowly": "main", "tag is cloned shallowly": "v1"} {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir() dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{ require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: ref, Dir: dir, Depth: 1, URL: remoteDir, Ref: ref, Dir: dir, Depth: 1,
})(t.Context())) })(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow") assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present") assert.Equal(t, []string{"c3"}, gitSubjects(t, dir), "only the tip commit should be present")
assert.Equal(t, "c3", gitHeadSubject(t, dir))
}) })
} }
unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string { unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string {
remote := filepath.Join(t.TempDir(), "remote.git") remote := filepath.Join(t.TempDir(), "remote.git")
require.NoError(t, gitCmd("clone", "--bare", remoteDir, remote)) require.NoError(t, gitCmd("clone", "--bare", "--template=", "--config", "uploadpack.allowAnySHA1InWant="+strconv.FormatBool(allowUnadvertised), remoteDir, remote))
require.NoError(t, gitCmd("-C", remote, "config", "uploadpack.allowAnySHA1InWant", strconv.FormatBool(allowUnadvertised)))
return remote return remote
} }
t.Run("commit hash resolves when the remote accepts a direct shallow fetch", func(t *testing.T) { t.Run("commit hash resolves when the remote accepts a direct shallow fetch", func(t *testing.T) {
t.Parallel()
dir := t.TempDir() dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{ require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: unadvertisedRemote(t, false), Ref: unadvertisedSHA, Dir: dir, Depth: 1, URL: unadvertisedRemote(t, false), Ref: unadvertisedSHA, Dir: dir, Depth: 1,
@@ -393,6 +433,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
}) })
t.Run("commit hash is fetched shallowly, moved to another hash, then reused without reaching the remote", func(t *testing.T) { t.Run("commit hash is fetched shallowly, moved to another hash, then reused without reaching the remote", func(t *testing.T) {
t.Parallel()
remote := unadvertisedRemote(t, true) remote := unadvertisedRemote(t, true)
dir := t.TempDir() dir := t.TempDir()
cloneAt := func(ref string) error { cloneAt := func(ref string) error {
@@ -403,7 +444,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
require.NoError(t, cloneAt(unadvertisedSHA)) require.NoError(t, cloneAt(unadvertisedSHA))
assert.FileExists(t, shallowMarker(dir)) assert.FileExists(t, shallowMarker(dir))
assert.Equal(t, 1, gitRevCount(t, dir)) assert.Len(t, gitSubjects(t, dir), 1)
assert.Equal(t, unadvertisedSHA, gitRevParse(t, dir, "HEAD")) assert.Equal(t, unadvertisedSHA, gitRevParse(t, dir, "HEAD"))
olderSHA := gitRevParse(t, workDir, "HEAD~2") olderSHA := gitRevParse(t, workDir, "HEAD~2")
@@ -416,6 +457,8 @@ func TestGitCloneExecutorShallow(t *testing.T) {
}) })
t.Run("moving hexadecimal-named branch updates while staying shallow", func(t *testing.T) { t.Run("moving hexadecimal-named branch updates while staying shallow", func(t *testing.T) {
t.Parallel()
remoteDir, workDir := newGitRemote(t, "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef")) require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef"))
dir := t.TempDir() dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{ clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
@@ -428,19 +471,17 @@ func TestGitCloneExecutorShallow(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef")) require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef"))
require.NoError(t, clone(t.Context())) require.NoError(t, clone(t.Context()))
assert.Equal(t, "c4", gitHeadSubject(t, dir), "reused shallow clone should update to the new tip") assert.Equal(t, []string{"c4"}, gitSubjects(t, dir), "reused shallow clone should update to the new tip")
assert.FileExists(t, shallowMarker(dir), "repo should remain shallow after update") assert.FileExists(t, shallowMarker(dir), "repo should remain shallow after update")
assert.Equal(t, 1, gitRevCount(t, dir))
}) })
t.Run("sha256 full hash is cloned shallowly, reused without the remote, and its 40-character prefix is short", func(t *testing.T) { t.Run("sha256 full hash is cloned shallowly, reused without the remote, and its 40-character prefix is short", func(t *testing.T) {
t.Parallel()
remote := filepath.Join(t.TempDir(), "remote.git") remote := filepath.Join(t.TempDir(), "remote.git")
require.NoError(t, gitCmd("init", "--bare", "--object-format=sha256", "--initial-branch=main", remote))
work := t.TempDir() work := t.TempDir()
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", work)) require.NoError(t, gitCmd("init", "--template=", "--object-format=sha256", "--initial-branch=main", work))
require.NoError(t, gitCmd("-C", work, "remote", "add", "origin", remote))
require.NoError(t, gitCmd("-C", work, "commit", "--allow-empty", "-m", "pinned")) require.NoError(t, gitCmd("-C", work, "commit", "--allow-empty", "-m", "pinned"))
require.NoError(t, gitCmd("-C", work, "push", "origin", "main")) require.NoError(t, gitCmd("clone", "--bare", "--template=", work, remote))
sha := gitRevParse(t, work, "HEAD") sha := gitRevParse(t, work, "HEAD")
dir := t.TempDir() dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{URL: remote, Ref: sha, Dir: dir, Depth: 1}) clone := NewGitCloneExecutor(NewGitCloneExecutorInput{URL: remote, Ref: sha, Dir: dir, Depth: 1})
@@ -459,11 +500,11 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
pinned := gitRevParse(t, workDir, "HEAD") pinned := gitRevParse(t, workDir, "HEAD")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy")) require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy"))
require.NoError(t, gitCmd("-C", workDir, "tag", pinned)) require.NoError(t, gitCmd("-C", workDir, "tag", pinned))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", pinned))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", pinned))
for name, depth := range map[string]int{"shallow": 1, "full clone": 0} { for name, depth := range map[string]int{"shallow": 1, "full clone": 0} {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir() dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{ require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: pinned, Dir: dir, Depth: depth, URL: remoteDir, Ref: pinned, Dir: dir, Depth: depth,
@@ -476,9 +517,12 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
func newGitRemote(t *testing.T, messages ...string) (string, string) { func newGitRemote(t *testing.T, messages ...string) (string, string) {
t.Helper() t.Helper()
remoteDir, workDir := t.TempDir(), t.TempDir() remoteDir, workDir := t.TempDir(), t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir)) require.NoError(t, os.CopyFS(remoteDir, os.DirFS(filepath.Join(gitFixtureDir, "remote"))))
require.NoError(t, gitCmd("clone", remoteDir, workDir)) require.NoError(t, os.CopyFS(workDir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main")) config, err := os.ReadFile(filepath.Join(workDir, ".git", "config"))
require.NoError(t, err)
require.NoError(t, os.WriteFile(filepath.Join(workDir, ".git", "config"),
fmt.Appendf(config, "\n[remote \"origin\"]\n\turl = %s\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n", strconv.Quote(remoteDir)), 0o644))
for _, message := range messages { for _, message := range messages {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message)) require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
} }
@@ -492,13 +536,11 @@ func gitRevParse(t *testing.T, dir, rev string) string {
return strings.TrimSpace(string(out)) return strings.TrimSpace(string(out))
} }
func gitRevCount(t *testing.T, dir string) int { func gitSubjects(t *testing.T, dir string) []string {
t.Helper() t.Helper()
out, err := exec.Command("git", "-C", dir, "rev-list", "--count", "HEAD").Output() out, err := exec.Command("git", "-C", dir, "log", "--format=%s").Output()
require.NoError(t, err) require.NoError(t, err)
n, err := strconv.Atoi(strings.TrimSpace(string(out))) return strings.Split(strings.TrimSpace(string(out)), "\n")
require.NoError(t, err)
return n
} }
func gitHeadSubject(t *testing.T, dir string) string { func gitHeadSubject(t *testing.T, dir string) string {
@@ -632,7 +674,7 @@ func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
}) })
dir := filepath.Join(t.TempDir(), "cached-action") dir := filepath.Join(t.TempDir(), "cached-action")
require.NoError(t, gitCmd("init", dir)) require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "remote", "add", "origin", server.URL)) require.NoError(t, gitCmd("-C", dir, "remote", "add", "origin", server.URL))
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
@@ -645,8 +687,8 @@ func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
select { select {
case <-reached: case <-reached:
case <-time.After(10 * time.Second): case err := <-done:
t.Fatal("the executor never reached the remote") t.Fatalf("the executor finished before reaching the remote: %v", err)
} }
assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), authorization) assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), authorization)
cancel() cancel()
+20
View File
@@ -0,0 +1,20 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package common
import "strings"
// LookupFold finds name in values, ignoring case where the environment does, as GitHub's inputs always do.
func LookupFold[V any](values map[string]V, name string, caseInsensitive bool) (V, bool) {
if value, ok := values[name]; ok || !caseInsensitive {
return value, ok
}
for key, value := range values {
if strings.EqualFold(key, name) {
return value, true
}
}
var zero V
return zero, false
}
+3
View File
@@ -10,6 +10,7 @@ import (
"fmt" "fmt"
"io" "io"
"path" "path"
"runtime"
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -20,6 +21,8 @@ import (
"github.com/moby/moby/api/types/container" "github.com/moby/moby/api/types/container"
) )
var hostOS = runtime.GOOS // replaced in tests
// ExitCodeError reports a non-zero process exit code from a container command. // ExitCodeError reports a non-zero process exit code from a container command.
type ExitCodeError int type ExitCodeError int
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2021 The nektos/act Authors. All rights reserved. // Copyright 2021 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !windows && !(WITHOUT_DOCKER || !(linux || darwin || netbsd)) //go:build !windows && !(WITHOUT_DOCKER || !(linux || darwin || netbsd || freebsd))
package container package container
+12 -6
View File
@@ -2,7 +2,7 @@
// Copyright 2020 The nektos/act Authors. All rights reserved. // Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -48,6 +48,7 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
options := client.ImageBuildOptions{ options := client.ImageBuildOptions{
Tags: []string{input.ImageTag}, Tags: []string{input.ImageTag},
Labels: map[string]string{"com.gitea.runner.image": input.ImageTag}, // own image ID per tag, so removing one never deletes another job's identical build
Remove: true, Remove: true,
AuthConfigs: LoadDockerAuthConfigs(ctx), AuthConfigs: LoadDockerAuthConfigs(ctx),
Dockerfile: input.Dockerfile, Dockerfile: input.Dockerfile,
@@ -74,8 +75,10 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Creating image from context dir '%s' with tag '%s' and platform '%s'", input.ContextDir, input.ImageTag, input.Platform) logger.Debugf("Creating image from context dir '%s' with tag '%s' and platform '%s'", input.ContextDir, input.ImageTag, input.Platform)
resp, err := cli.ImageBuild(ctx, buildContext, options) resp, err := cli.ImageBuild(ctx, buildContext, options)
if err != nil {
return logDockerResponse(logger, resp.Body, err != nil) return err
}
return logDockerResponse(logger, resp.Body, false)
} }
} }
@@ -123,6 +126,10 @@ func createBuildContext(ctx context.Context, contextDir, relDockerfile string) (
} }
func writeBuildContext(ctx context.Context, w io.Writer, contextDir string, includes []string, pm *patternmatcher.PatternMatcher) error { func writeBuildContext(ctx context.Context, w io.Writer, contextDir string, includes []string, pm *patternmatcher.PatternMatcher) error {
contextDir, err := filepath.EvalSymlinks(contextDir)
if err != nil {
return err
}
tw := tar.NewWriter(w) tw := tar.NewWriter(w)
seen := map[string]bool{} seen := map[string]bool{}
hardlinks := map[uint64]string{} hardlinks := map[uint64]string{}
@@ -130,8 +137,7 @@ func writeBuildContext(ctx context.Context, w io.Writer, contextDir string, incl
for _, include := range includes { for _, include := range includes {
include = filepath.FromSlash(include) include = filepath.FromSlash(include)
var parents []parentMatch var parents []parentMatch
root := strings.TrimSuffix(contextDir, string(filepath.Separator)) + string(filepath.Separator) + include // unlike filepath.Join, follows a symlinked context dir as docker build does err := filepath.WalkDir(filepath.Join(contextDir, include), func(file string, entry fs.DirEntry, err error) error {
err := filepath.WalkDir(root, func(file string, entry fs.DirEntry, err error) error {
if err != nil { if err != nil {
return leaveOut(ctx, file, err) return leaveOut(ctx, file, err)
} }
@@ -208,7 +214,7 @@ func addBuildContextEntry(ctx context.Context, tw *tar.Writer, file, rel string,
return leaveOut(ctx, file, err) return leaveOut(ctx, file, err)
} }
} }
header, err := tar.FileInfoHeader(noNameLookups{info}, link) header, err := tar.FileInfoHeader(noNameLookups{info}, filepath.ToSlash(link))
if err != nil { if err != nil {
return err return err
} }
+28 -3
View File
@@ -1,34 +1,59 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
import ( import (
"archive/tar" "archive/tar"
"io" "io"
"net/http"
"net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
func TestDockerBuildLabelsImageWithItsTagAndReportsDaemonError(t *testing.T) {
var labels string
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("API-Version", "1.47")
if strings.HasSuffix(request.URL.Path, "/build") {
labels = request.URL.Query().Get("labels")
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(writer, `{"message":"build rejected"}`)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
t.Setenv("DOCKER_CONFIG", t.TempDir())
t.Setenv("DOCKER_AUTH_CONFIG", "")
require.ErrorContains(t, NewDockerBuildExecutor(NewDockerBuildExecutorInput{
ImageTag: "action:test", Dockerfile: "Dockerfile", BuildContext: strings.NewReader(""),
})(t.Context()), "build rejected")
require.JSONEq(t, `{"com.gitea.runner.image":"action:test"}`, labels)
}
func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile(t *testing.T) { func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
require.NoError(t, os.Mkdir(filepath.Join(dir, "build"), 0o755))
for name, content := range map[string]string{ for name, content := range map[string]string{
".dockerignore": "*.log\nbuild\n!build/keep.txt\nDockerfile", "Dockerfile": "FROM scratch", "a.txt": "a", ".dockerignore": "*.log\nbuild\n!build/keep.txt\nDockerfile", "Dockerfile": "FROM scratch", "a.txt": "a",
"a.log": "log", "build/drop.txt": "drop", "build/keep.txt": "keep", "a.log": "log", "build/drop.txt": "drop", "build/keep.txt": "keep",
} { } {
require.NoError(t, os.MkdirAll(filepath.Join(dir, filepath.Dir(name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644)) require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
} }
require.NoError(t, os.Mkdir(filepath.Join(dir, "empty"), 0o755)) require.NoError(t, os.Mkdir(filepath.Join(dir, "empty"), 0o755))
contexts := []string{dir} contexts := []string{dir}
if link := filepath.Join(t.TempDir(), "link"); os.Symlink(dir, link) == nil { if link := filepath.Join(t.TempDir(), "link"); os.Symlink(dir, link) == nil {
contexts = append(contexts, link+string(filepath.Separator)) contexts = append(contexts, link, link+string(filepath.Separator))
} }
for _, contextDir := range contexts { for _, contextDir := range contexts {
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd) //go:build !WITHOUT_DOCKER && (linux || darwin || netbsd || freebsd)
package container package container
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd) //go:build !WITHOUT_DOCKER && (linux || darwin || netbsd || freebsd)
package container package container
+9 -5
View File
@@ -2,7 +2,7 @@
// Copyright 2022 The nektos/act Authors. All rights reserved. // Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
// This file is based on https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts.go with: // This file is based on https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts.go with:
// * appended with license information // * appended with license information
@@ -351,7 +351,7 @@ type containerConfig struct {
// parse parses the args for the specified command and generates a Config, // parse parses the args for the specified command and generates a Config,
// a HostConfig and returns them with the specified command. // a HostConfig and returns them with the specified command.
// If the specified args are not valid, it will return an error. // If the specified args are not valid, it will return an error.
func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*containerConfig, error) { //nolint:gocyclo,unparam // verbatim copy from docker/cli func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*containerConfig, error) { //nolint:gocyclo // verbatim copy from docker/cli
var ( var (
attachStdin = copts.attach.Get("stdin") attachStdin = copts.attach.Get("stdin")
attachStdout = copts.attach.Get("stdout") attachStdout = copts.attach.Get("stdout")
@@ -724,6 +724,10 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
}, nil }, nil
} }
func isUserDefinedNetwork(mode container.NetworkMode) bool { // Docker's helper uses the host OS instead of the daemon's.
return !mode.IsDefault() && mode != network.NetworkBridge && mode != network.NetworkHost && !mode.IsNone() && !mode.IsContainer()
}
// parseNetworkOpts converts --network advanced options to endpoint-specs, and combines // parseNetworkOpts converts --network advanced options to endpoint-specs, and combines
// them with the old --network-alias and --links. If returns an error if conflicting options // them with the old --network-alias and --links. If returns an error if conflicting options
// are found. // are found.
@@ -752,7 +756,7 @@ func parseNetworkOpts(copts *containerOptions) (map[string]*network.EndpointSett
} }
for i, n := range copts.netMode.Value() { for i, n := range copts.netMode.Value() {
if container.NetworkMode(n.Target).IsUserDefined() { if isUserDefinedNetwork(container.NetworkMode(n.Target)) {
hasUserDefined = true hasUserDefined = true
} else { } else {
hasNonUserDefined = true hasNonUserDefined = true
@@ -814,7 +818,7 @@ func applyContainerOptions(n *opts.NetworkAttachmentOpts, copts *containerOption
} }
// For a user-defined network, "--link" is an endpoint option, it creates an alias. But, // For a user-defined network, "--link" is an endpoint option, it creates an alias. But,
// for the default bridge it defines a legacy-link. // for the default bridge it defines a legacy-link.
if container.NetworkMode(n.Target).IsUserDefined() && copts.links.Len() > 0 { if isUserDefinedNetwork(container.NetworkMode(n.Target)) && copts.links.Len() > 0 {
n.Links = slices.Clone(copts.links.GetSlice()) n.Links = slices.Clone(copts.links.GetSlice())
} }
if copts.ipv4Address != nil { if copts.ipv4Address != nil {
@@ -840,7 +844,7 @@ func parseNetworkAttachmentOpt(ep opts.NetworkAttachmentOpts) (*network.Endpoint
if strings.TrimSpace(ep.Target) == "" { if strings.TrimSpace(ep.Target) == "" {
return nil, errors.New("no name set for network") return nil, errors.New("no name set for network")
} }
if !container.NetworkMode(ep.Target).IsUserDefined() { if !isUserDefinedNetwork(container.NetworkMode(ep.Target)) {
if len(ep.Aliases) > 0 { if len(ep.Aliases) > 0 {
return nil, errors.New("network-scoped aliases are only supported for user-defined networks") return nil, errors.New("network-scoped aliases are only supported for user-defined networks")
} }
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2020 The nektos/act Authors. All rights reserved. // Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
+25 -23
View File
@@ -5,14 +5,17 @@
package container package container
import ( import (
"bytes"
"context"
"fmt" "fmt"
"os"
"os/exec"
"strings" "strings"
"testing" "testing"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging" "gitea.com/gitea/runner/act/common/logging"
mobyclient "github.com/moby/moby/client"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -21,48 +24,47 @@ func init() {
logging.SetLevel(logging.LevelDebug) logging.SetLevel(logging.LevelDebug)
} }
// buildScratchImage builds a tiny empty image for the given platform locally (FROM scratch, no func loadScratchImage(t *testing.T, platform string) string {
// network or emulation since there is nothing to run) and returns its tag, removing it after
// the test.
func buildScratchImage(t *testing.T, platform string) string {
t.Helper() t.Helper()
tag := fmt.Sprintf("act-test-exists-%s:latest", strings.TrimPrefix(platform, "linux/")) tag := fmt.Sprintf("act-test-exists-%s-%d:latest", strings.TrimPrefix(platform, "linux/"), time.Now().UnixNano())
cmd := exec.Command("docker", "build", "--platform", platform, "-t", tag, "-") var archive bytes.Buffer
cmd.Stdin = strings.NewReader("FROM scratch\nLABEL act-test=1\n") require.NoError(t, writeFilesTar(t.Context(), &archive, 0, 0,
// Force BuildKit: it records the requested architecture in the image config for a &FileEntry{Name: "config.json", Mode: 0o600, Body: fmt.Sprintf(`{"architecture":%q,"os":"linux","rootfs":{"type":"layers","diff_ids":[]}}`, strings.TrimPrefix(platform, "linux/"))},
// FROM-scratch build, whereas the classic builder ignores --platform and tags it with the &FileEntry{Name: "manifest.json", Mode: 0o600, Body: fmt.Sprintf(`[{"Config":"config.json","RepoTags":[%q],"Layers":[]}]`, tag)},
// host arch, which would break the per-platform existence assertions below. ))
cmd.Env = append(os.Environ(), "DOCKER_BUILDKIT=1") cli, err := GetDockerClient(t.Context())
out, err := cmd.CombinedOutput() require.NoError(t, err)
require.NoError(t, err, string(out)) t.Cleanup(func() { _ = cli.Close() })
t.Cleanup(func() { _ = exec.Command("docker", "rmi", "-f", tag).Run() }) response, err := cli.ImageLoad(t.Context(), &archive)
require.NoError(t, err)
defer response.Close()
require.NoError(t, logDockerResponse(common.Logger(t.Context()), response, false))
t.Cleanup(func() {
_, _ = cli.ImageRemove(context.WithoutCancel(t.Context()), tag, mobyclient.ImageRemoveOptions{Force: true})
})
return tag return tag
} }
func TestImageExistsLocally(t *testing.T) { func TestImageExistsLocally(t *testing.T) {
t.Parallel()
requireDocker(t) requireDocker(t)
ctx := t.Context() ctx := t.Context()
// a non-existent image is reported absent
missing, err := ImageExistsLocally(ctx, "library/alpine:this-random-tag-will-never-exist", "linux/amd64") missing, err := ImageExistsLocally(ctx, "library/alpine:this-random-tag-will-never-exist", "linux/amd64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, missing) assert.False(t, missing)
// Build tiny images for two architectures locally so per-platform existence can be checked amd64Ref := loadScratchImage(t, "linux/amd64")
// offline (formerly pulled node:24-bookworm-slim for amd64 and arm64 over the network). arm64Ref := loadScratchImage(t, "linux/arm64")
amd64Ref := buildScratchImage(t, "linux/amd64")
arm64Ref := buildScratchImage(t, "linux/arm64")
amd64Exists, err := ImageExistsLocally(ctx, amd64Ref, "linux/amd64") amd64Exists, err := ImageExistsLocally(ctx, amd64Ref, "linux/amd64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, amd64Exists) assert.True(t, amd64Exists)
// a non-host architecture image is detected for its own architecture
arm64Exists, err := ImageExistsLocally(ctx, arm64Ref, "linux/arm64") arm64Exists, err := ImageExistsLocally(ctx, arm64Ref, "linux/arm64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, arm64Exists) assert.True(t, arm64Exists)
// a present image is reported absent for a different platform
wrongPlatform, err := ImageExistsLocally(ctx, amd64Ref, "linux/arm64") wrongPlatform, err := ImageExistsLocally(ctx, amd64Ref, "linux/arm64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, wrongPlatform) assert.False(t, wrongPlatform)
+3 -2
View File
@@ -2,7 +2,7 @@
// Copyright 2020 The nektos/act Authors. All rights reserved. // Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -33,6 +33,7 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
defer dockerResponse.Close() defer dockerResponse.Close()
scanner := bufio.NewScanner(dockerResponse) scanner := bufio.NewScanner(dockerResponse)
scanner.Buffer(nil, 64*1024*1024) // a build echoes long instructions in one record
for scanner.Scan() { for scanner.Scan() {
line := scanner.Bytes() line := scanner.Bytes()
var msg dockerMessage var msg dockerMessage
@@ -65,7 +66,7 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
} }
} }
return nil return scanner.Err()
} }
func writeLog(logger *logging.Logger, isError bool, format string, args ...any) { func writeLog(logger *logging.Logger, isError bool, format string, args ...any) {
+9 -9
View File
@@ -2,7 +2,7 @@
// Copyright 2023 The nektos/act Authors. All rights reserved. // Copyright 2023 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -85,14 +85,14 @@ networks:
} }
func NewDockerNetworkCreateExecutor(name string, opts NewDockerNetworkCreateExecutorInput) common.Executor { func NewDockerNetworkCreateExecutor(name string, opts NewDockerNetworkCreateExecutorInput) common.Executor {
return func(ctx context.Context) error { return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx) cli, err := GetDockerClient(ctx)
if err != nil { if err != nil {
return err return err
} }
defer cli.Close() defer cli.Close()
return createDockerNetwork(ctx, cli, name, opts) return createDockerNetwork(ctx, cli, name, opts)
} }).IfNot(common.Dryrun)
} }
func createDockerNetwork(ctx context.Context, cli client.APIClient, name string, opts NewDockerNetworkCreateExecutorInput) error { func createDockerNetwork(ctx context.Context, cli client.APIClient, name string, opts NewDockerNetworkCreateExecutorInput) error {
@@ -157,7 +157,7 @@ func isAddressPoolExhausted(err error) bool {
} }
func NewDockerNetworkConnectExecutor(networkName, container string) common.Executor { func NewDockerNetworkConnectExecutor(networkName, container string) common.Executor {
return func(ctx context.Context) error { return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx) cli, err := GetDockerClient(ctx)
if err != nil { if err != nil {
return err return err
@@ -167,7 +167,7 @@ func NewDockerNetworkConnectExecutor(networkName, container string) common.Execu
common.Logger(ctx).Warnf("cannot connect %s to network %s, so jobs on it cannot reach the cache: %v", container, networkName, err) common.Logger(ctx).Warnf("cannot connect %s to network %s, so jobs on it cannot reach the cache: %v", container, networkName, err)
} }
return nil return nil
} }).IfNot(common.Dryrun)
} }
func connectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error { func connectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error {
@@ -182,14 +182,14 @@ func connectNetwork(ctx context.Context, cli client.APIClient, networkName, cont
} }
func NewDockerNetworkDisconnectExecutor(networkName, container string) common.Executor { func NewDockerNetworkDisconnectExecutor(networkName, container string) common.Executor {
return func(ctx context.Context) error { return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx) cli, err := GetDockerClient(ctx)
if err != nil { if err != nil {
return err return err
} }
defer cli.Close() defer cli.Close()
return disconnectNetwork(ctx, cli, networkName, container) return disconnectNetwork(ctx, cli, networkName, container)
} }).IfNot(common.Dryrun)
} }
func disconnectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error { func disconnectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error {
@@ -201,14 +201,14 @@ func disconnectNetwork(ctx context.Context, cli client.APIClient, networkName, c
} }
func NewDockerNetworkRemoveExecutor(name string) common.Executor { func NewDockerNetworkRemoveExecutor(name string) common.Executor {
return func(ctx context.Context) error { return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx) cli, err := GetDockerClient(ctx)
if err != nil { if err != nil {
return err return err
} }
defer cli.Close() defer cli.Close()
return removeNetwork(ctx, cli, name) return removeNetwork(ctx, cli, name)
} }).IfNot(common.Dryrun)
} }
func removeNetwork(ctx context.Context, cli client.APIClient, name string) error { func removeNetwork(ctx context.Context, cli client.APIClient, name string) error {
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2025 The nektos/act Authors. All rights reserved. // Copyright 2025 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
+105 -30
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -40,7 +40,9 @@ const (
jobLabel = "com.gitea.runner.job." jobLabel = "com.gitea.runner.job."
maxCreateBody = 8 << 20 maxCreateBody = 8 << 20
dockerProxyProbeTimeout = 5 * time.Second dockerProxyProbeTimeout = 5 * time.Second
dockerProxyProbeWorkerTimeout = time.Minute
dockerProxyProbeTarget = "/gitea-runner-probe"
) )
var ( var (
@@ -48,20 +50,27 @@ var (
rawStreamPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers/[^/]+/attach|exec/[^/]+/start)$`) rawStreamPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers/[^/]+/attach|exec/[^/]+/start)$`)
) )
var dockerProxyDirectories sync.Map var dockerProxyProbes sync.Map
type dockerProxyDirectory struct { type dockerProxyDirectory struct {
dir, daemonDir string dir, daemonDir string
socket os.FileInfo socket os.FileInfo
} }
type dockerProxyProbe struct {
directory dockerProxyDirectory
done chan struct{}
seen bool
err error
}
func NewDockerProxy(ctx context.Context, job string) *DockerProxy { func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if host := os.Getenv("DOCKER_HOST"); runtime.GOOS != "linux" || host != "" && !strings.HasPrefix(host, "unix://") { if host := os.Getenv("DOCKER_HOST"); runtime.GOOS != "linux" || host != "" && !strings.HasPrefix(host, "unix://") {
return nil return nil
} }
probeCtx, cancel := context.WithTimeout(ctx, dockerProxyProbeTimeout) probeCtx, cancel := context.WithTimeout(ctx, dockerProxyProbeTimeout)
defer cancel() defer cancel()
cli, err := GetDockerClient(probeCtx) cli, err := client.New(client.FromEnv) // negotiates on first request, so a cached probe costs no round trip
if err != nil { if err != nil {
return nil return nil
} }
@@ -76,42 +85,72 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
} }
workdir, _ := os.Getwd() workdir, _ := os.Getwd()
key := [3]string{cli.DaemonHost(), workdir, os.TempDir()} key := [3]string{cli.DaemonHost(), workdir, os.TempDir()}
cached, _ := dockerProxyDirectories.Load(key) probe, err := probeDockerProxyDirectory(probeCtx, cli, key, socket)
directory, seen := cached.(dockerProxyDirectory)
seen = seen && os.SameFile(directory.socket, socket) // daemon replacement invalidates cached paths
dir, daemonDir := directory.dir, directory.daemonDir
if !seen {
dir, daemonDir = runnerContainerWorkdir(probeCtx, cli)
seen = daemonDir != ""
if !seen {
if dir, err = filepath.Abs(os.TempDir()); err == nil {
daemonDir = dir
seen, err = daemonSeesDir(probeCtx, cli, dir, daemonDir)
}
}
}
if err != nil { if err != nil {
common.Logger(ctx).Infof("docker proxy probe failed, jobs get the daemon socket directly: %v", err) common.Logger(ctx).Infof("docker proxy probe failed, jobs get the daemon socket directly: %v", err)
return nil return nil
} }
if !seen { if !probe.seen {
common.Logger(ctx).Infof("the docker daemon cannot reach the runner's temporary or working directory, jobs get the daemon socket directly") common.Logger(ctx).Infof("the docker daemon cannot reach the runner's temporary or working directory, jobs get the daemon socket directly")
return nil return nil
} }
if ctx.Err() != nil { if ctx.Err() != nil {
return nil return nil
} }
proxy, err := StartDockerProxy(daemonSocket, dir, job) proxy, err := StartDockerProxy(daemonSocket, probe.directory.dir, job)
if err != nil { if err != nil {
dockerProxyDirectories.Delete(key) dockerProxyProbes.CompareAndDelete(key, probe)
common.Logger(ctx).Warnf("docker proxy not started, the job gets the daemon socket directly: %v", err) common.Logger(ctx).Warnf("docker proxy not started, the job gets the daemon socket directly: %v", err)
return nil return nil
} }
dockerProxyDirectories.Store(key, dockerProxyDirectory{dir, daemonDir, socket}) proxy.Socket = probe.directory.daemonDir + strings.TrimPrefix(proxy.Socket, probe.directory.dir)
proxy.Socket = daemonDir + strings.TrimPrefix(proxy.Socket, dir)
return proxy return proxy
} }
func probeDockerProxyDirectory(ctx context.Context, cli client.APIClient, key [3]string, socket os.FileInfo) (*dockerProxyProbe, error) {
for {
probe := &dockerProxyProbe{directory: dockerProxyDirectory{socket: socket}, done: make(chan struct{})}
actual, loaded := dockerProxyProbes.LoadOrStore(key, probe)
if loaded {
probe, _ = actual.(*dockerProxyProbe)
if !os.SameFile(probe.directory.socket, socket) { // daemon replacement invalidates cached paths
dockerProxyProbes.CompareAndDelete(key, probe)
continue
}
} else {
go func() { // outlives callers' deadlines so a slow daemon still gets its result cached
defer cli.Close()
ctx, cancel := context.WithTimeout(context.Background(), dockerProxyProbeWorkerTimeout)
defer cancel()
probe.directory.dir, probe.directory.daemonDir = runnerContainerWorkdir(ctx, cli)
probe.seen = probe.directory.daemonDir != ""
if !probe.seen {
if probe.directory.dir, probe.err = filepath.Abs(key[2]); probe.err == nil {
probe.directory.daemonDir = probe.directory.dir
probe.seen, probe.err = daemonSeesDir(ctx, cli, probe.directory.dir, probe.directory.daemonDir)
}
}
if probe.seen {
dockerProxyProbes.LoadOrStore(key, probe)
} else {
dockerProxyProbes.CompareAndDelete(key, probe)
}
close(probe.done)
removeStaleProbeContainers(cli)
}()
}
select {
case <-ctx.Done():
return &dockerProxyProbe{}, ctx.Err()
case <-probe.done:
}
if ctx.Err() == nil && errors.Is(probe.err, context.DeadlineExceeded) {
continue
}
return probe, probe.err
}
}
// runnerContainerWorkdir looks the runner's container up by hostname to find the daemon's path to its working directory. // runnerContainerWorkdir looks the runner's container up by hostname to find the daemon's path to its working directory.
func runnerContainerWorkdir(ctx context.Context, cli client.APIClient) (workdir, daemonDir string) { func runnerContainerWorkdir(ctx context.Context, cli client.APIClient) (workdir, daemonDir string) {
workdir, err := os.Getwd() workdir, err := os.Getwd()
@@ -162,19 +201,25 @@ func daemonSeesDir(ctx context.Context, cli client.APIClient, dir, daemonDir str
if len(images.Items) == 0 { if len(images.Items) == 0 {
return false, errors.New("no image available for the docker proxy probe") return false, errors.New("no image available for the docker proxy probe")
} }
created, err := cli.ContainerCreate(ctx, client.ContainerCreateOptions{ var created client.ContainerCreateResult
Config: &container.Config{Image: images.Items[0].ID, Cmd: []string{"true"}}, for _, image := range images.Items {
HostConfig: &container.HostConfig{Mounts: []mount.Mount{ created, err = cli.ContainerCreate(ctx, client.ContainerCreateOptions{
{Type: mount.TypeBind, Source: daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}, // not the marker itself, podman creates a missing bind source where docker rejects it Config: &container.Config{Image: image.ID, Cmd: []string{"true"}},
}}, HostConfig: &container.HostConfig{Mounts: []mount.Mount{
}) {Type: mount.TypeBind, Source: daemonDir, Target: dockerProxyProbeTarget, ReadOnly: true}, // not the marker itself, podman creates a missing bind source where docker rejects it
}},
})
if !cerrdefs.IsNotFound(err) {
break
}
}
if cerrdefs.IsInvalidArgument(err) { if cerrdefs.IsInvalidArgument(err) {
return false, nil return false, nil
} }
if err != nil { if err != nil {
return false, err return false, err
} }
seen, err := containerSeesMarker(ctx, cli, dir, created.ID, "/gitea-runner-probe") seen, err := containerSeesMarker(ctx, cli, dir, created.ID, dockerProxyProbeTarget)
cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), dockerProxyProbeTimeout) cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), dockerProxyProbeTimeout)
defer cancel() defer cancel()
if _, removeErr := cli.ContainerRemove(cleanupCtx, created.ID, client.ContainerRemoveOptions{Force: true, RemoveVolumes: true}); removeErr != nil { if _, removeErr := cli.ContainerRemove(cleanupCtx, created.ID, client.ContainerRemoveOptions{Force: true, RemoveVolumes: true}); removeErr != nil {
@@ -183,6 +228,36 @@ func daemonSeesDir(ctx context.Context, cli client.APIClient, dir, daemonDir str
return seen, err return seen, err
} }
// removeStaleProbeContainers removes probe containers left by abandoned creates once any probe in flight when listed has ended.
func removeStaleProbeContainers(cli client.APIClient) {
ctx, cancel := context.WithTimeout(context.Background(), dockerProxyProbeTimeout)
defer cancel()
listed, err := cli.ContainerList(ctx, client.ContainerListOptions{
All: true,
Filters: make(client.Filters).Add("status", string(container.StateCreated)).Add("volume", dockerProxyProbeTarget),
})
if err != nil {
common.Logger(ctx).Warnf("listing stale docker proxy probe containers failed: %v", err)
return
}
listed.Items = slices.DeleteFunc(listed.Items, func(stale container.Summary) bool {
return stale.State != container.StateCreated || !slices.ContainsFunc(stale.Mounts, func(mountPoint container.MountPoint) bool {
return mountPoint.Type == mount.TypeBind && !mountPoint.RW && mountPoint.Destination == dockerProxyProbeTarget
})
})
if len(listed.Items) == 0 {
return
}
time.Sleep(dockerProxyProbeWorkerTimeout + dockerProxyProbeTimeout)
for _, stale := range listed.Items {
ctx, cancel := context.WithTimeout(context.Background(), dockerProxyProbeTimeout)
if err := (&containerReference{cli: cli, id: stale.ID}).remove()(ctx); err != nil {
common.Logger(ctx).Warnf("removing stale docker proxy probe container failed: %v", err)
}
cancel()
}
}
// StartDockerProxy serves a job's docker socket in dir, labelling what the job creates through it. // StartDockerProxy serves a job's docker socket in dir, labelling what the job creates through it.
func StartDockerProxy(daemonSocket, dir, job string) (*DockerProxy, error) { func StartDockerProxy(daemonSocket, dir, job string) (*DockerProxy, error) {
info, err := os.Stat(daemonSocket) info, err := os.Stat(daemonSocket)
+232 -16
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -22,6 +22,7 @@ import (
"strings" "strings"
"sync/atomic" "sync/atomic"
"testing" "testing"
"testing/synctest"
"time" "time"
cerrdefs "github.com/containerd/errdefs" cerrdefs "github.com/containerd/errdefs"
@@ -93,6 +94,8 @@ func TestDockerProxy(t *testing.T) {
return return
} }
_, _ = w.Write([]byte(`[{"Id":"probe-image"}]`)) _, _ = w.Write([]byte(`[{"Id":"probe-image"}]`))
case strings.HasSuffix(r.URL.Path, "/containers/json"):
_, _ = w.Write([]byte(`[]`))
case strings.HasSuffix(r.URL.Path, "/json"): case strings.HasSuffix(r.URL.Path, "/json"):
http.NotFound(w, r) http.NotFound(w, r)
case strings.HasSuffix(r.URL.Path, "/archive"): case strings.HasSuffix(r.URL.Path, "/archive"):
@@ -234,7 +237,8 @@ func TestDockerProxy(t *testing.T) {
t.Skip("automatic proxy discovery requires Linux") t.Skip("automatic proxy discovery requires Linux")
} }
t.Setenv("DOCKER_HOST", "unix://"+daemonSocket) t.Setenv("DOCKER_HOST", "unix://"+daemonSocket)
t.Setenv("TMPDIR", shortTempDir(t)) tempDir := shortTempDir(t)
t.Setenv("TMPDIR", tempDir)
require.Nil(t, NewDockerProxy(t.Context(), "failed")) require.Nil(t, NewDockerProxy(t.Context(), "failed"))
for _, job := range []string{"first", "second"} { for _, job := range []string{"first", "second"} {
proxy := NewDockerProxy(t.Context(), job) proxy := NewDockerProxy(t.Context(), job)
@@ -242,12 +246,10 @@ func TestDockerProxy(t *testing.T) {
require.NoError(t, proxy.Close(context.Background())) require.NoError(t, proxy.Close(context.Background()))
} }
assert.EqualValues(t, 2, probes.Load()) assert.EqualValues(t, 2, probes.Load())
dockerProxyDirectories.Range(func(key, value any) bool { workdir, err := os.Getwd()
directory, _ := value.(dockerProxyDirectory) require.NoError(t, err)
directory.socket, _ = os.Stat(t.TempDir()) cached, _ := dockerProxyProbes.Load([3]string{"unix://" + daemonSocket, workdir, tempDir})
dockerProxyDirectories.Store(key, directory) cached.(*dockerProxyProbe).directory.socket, _ = os.Stat(t.TempDir())
return true
})
proxy := NewDockerProxy(t.Context(), "replaced daemon") proxy := NewDockerProxy(t.Context(), "replaced daemon")
require.NotNil(t, proxy) require.NotNil(t, proxy)
require.NoError(t, proxy.Close(context.Background())) require.NoError(t, proxy.Close(context.Background()))
@@ -371,6 +373,9 @@ func TestDockerProxyWithDaemon(t *testing.T) {
job := "proxy-test-" + strconv.FormatInt(time.Now().UnixNano(), 36) job := "proxy-test-" + strconv.FormatInt(time.Now().UnixNano(), 36)
proxy, err := StartDockerProxy(daemonSocketPath(t, direct), dir, job) proxy, err := StartDockerProxy(daemonSocketPath(t, direct), dir, job)
if os.Geteuid() != 0 && errors.Is(err, os.ErrPermission) {
t.Skipf("non-root runners that cannot take the daemon socket's ownership get it directly: %v", err)
}
require.NoError(t, err) require.NoError(t, err)
t.Cleanup(func() { _ = proxy.Close(context.Background()) }) t.Cleanup(func() { _ = proxy.Close(context.Background()) })
viaProxy, err := mobyclient.New(mobyclient.WithHost("unix://" + proxy.Socket)) viaProxy, err := mobyclient.New(mobyclient.WithHost("unix://" + proxy.Socket))
@@ -472,10 +477,27 @@ func BenchmarkDockerProxy(b *testing.B) {
type probeClient struct { type probeClient struct {
mobyclient.APIClient mobyclient.APIClient
create func(mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) create func(context.Context, mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error)
remove func(context.Context, string, mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) remove func(context.Context, string, mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error)
stat func(string) error stat func(string) error
mounts []container.MountPoint mounts []container.MountPoint
containers []container.Summary
}
func (c *probeClient) Info(_ctx context.Context, _opts mobyclient.InfoOptions) (mobyclient.SystemInfoResult, error) {
return mobyclient.SystemInfoResult{}, nil
}
func (c *probeClient) Close() error {
return nil
}
func (c *probeClient) ContainerList(context.Context, mobyclient.ContainerListOptions) (mobyclient.ContainerListResult, error) {
return mobyclient.ContainerListResult{Items: c.containers}, nil
}
func (c *probeClient) ContainerKill(context.Context, string, mobyclient.ContainerKillOptions) (mobyclient.ContainerKillResult, error) {
return mobyclient.ContainerKillResult{}, cerrdefs.ErrConflict
} }
func (c *probeClient) ContainerInspect(context.Context, string, mobyclient.ContainerInspectOptions) (mobyclient.ContainerInspectResult, error) { func (c *probeClient) ContainerInspect(context.Context, string, mobyclient.ContainerInspectOptions) (mobyclient.ContainerInspectResult, error) {
@@ -483,11 +505,11 @@ func (c *probeClient) ContainerInspect(context.Context, string, mobyclient.Conta
} }
func (c *probeClient) ImageList(context.Context, mobyclient.ImageListOptions) (mobyclient.ImageListResult, error) { func (c *probeClient) ImageList(context.Context, mobyclient.ImageListOptions) (mobyclient.ImageListResult, error) {
return mobyclient.ImageListResult{Items: []image.Summary{{ID: "probe-image"}}}, nil return mobyclient.ImageListResult{Items: []image.Summary{{ID: "removed-image"}, {ID: "probe-image"}}}, nil
} }
func (c *probeClient) ContainerCreate(_ context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) { func (c *probeClient) ContainerCreate(ctx context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
return c.create(opts) return c.create(ctx, opts)
} }
func (c *probeClient) ContainerRemove(ctx context.Context, id string, opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) { func (c *probeClient) ContainerRemove(ctx context.Context, id string, opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
@@ -498,9 +520,193 @@ func (c *probeClient) ContainerStatPath(_ context.Context, _ string, opts mobycl
return mobyclient.ContainerStatPathResult{}, c.stat(opts.Path) return mobyclient.ContainerStatPathResult{}, c.stat(opts.Path)
} }
func TestDockerProxyDirectoryProbes(t *testing.T) {
type result struct {
seen bool
err error
}
for name, createErr := range map[string]error{"unavailable directory": nil, "daemon error": errors.New("probe failed")} {
t.Run(name, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
key := [3]string{"daemon", dir, dir}
release := make(chan struct{})
var calls, removals atomic.Int32
visible := false
cli := &probeClient{
create: func(_ctx context.Context, _opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls.Add(1)
<-release
return mobyclient.ContainerCreateResult{ID: "probe"}, createErr
},
stat: func(_path string) error {
if visible {
return nil
} else {
return cerrdefs.ErrNotFound
}
},
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removals.Add(1)
return mobyclient.ContainerRemoveResult{}, nil
},
}
results := make(chan result, 8)
for range cap(results) {
go func() {
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
results <- result{probe.seen, err}
}()
}
synctest.Wait()
assert.EqualValues(t, 1, calls.Load())
replacement, err := os.Stat(t.TempDir())
require.NoError(t, err)
replaced := make(chan result, 1)
go func() {
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, replacement)
replaced <- result{probe.seen, err}
}()
synctest.Wait()
assert.EqualValues(t, 2, calls.Load())
close(release)
for range cap(results) {
result := <-results
require.ErrorIs(t, result.err, createErr)
assert.False(t, result.seen)
}
replacementResult := <-replaced
require.ErrorIs(t, replacementResult.err, createErr)
assert.False(t, replacementResult.seen)
assert.EqualValues(t, 2, calls.Load())
if createErr == nil {
assert.EqualValues(t, 2, removals.Load())
} else {
assert.Zero(t, removals.Load())
}
createErr, visible = nil, true
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
require.NoError(t, err)
assert.True(t, probe.seen)
assert.Equal(t, dir, probe.directory.dir)
assert.Equal(t, dir, probe.directory.daemonDir)
assert.EqualValues(t, 3, calls.Load())
entries, err := os.ReadDir(dir)
require.NoError(t, err)
assert.Empty(t, entries)
})
})
}
t.Run("cancelled caller leaves the probe to finish for the others", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
key := [3]string{"daemon", dir, dir}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
release := make(chan struct{})
var calls, removals atomic.Int32
cli := &probeClient{
create: func(ctx context.Context, _opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls.Add(1)
<-release
return mobyclient.ContainerCreateResult{ID: "probe"}, ctx.Err()
},
stat: func(_path string) error { return nil },
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removals.Add(1)
return mobyclient.ContainerRemoveResult{}, nil
},
}
leader, follower, late := make(chan result, 1), make(chan result, 1), make(chan result, 1)
go func() {
probe, err := probeDockerProxyDirectory(ctx, cli, key, socket)
leader <- result{probe.seen, err}
}()
synctest.Wait()
go func() {
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
follower <- result{probe.seen, err}
}()
synctest.Wait()
cancel()
leaderResult := <-leader
require.ErrorIs(t, leaderResult.err, context.Canceled)
assert.False(t, leaderResult.seen)
go func() {
probe, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
late <- result{probe.seen, err}
}()
synctest.Wait()
close(release)
for _, results := range []chan result{follower, late} {
result := <-results
require.NoError(t, result.err)
assert.True(t, result.seen)
}
assert.EqualValues(t, 1, calls.Load())
assert.EqualValues(t, 1, removals.Load())
_, cached := dockerProxyProbes.Load(key)
assert.True(t, cached)
})
})
t.Run("caller joining a probe that ran out of time retries", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
var calls atomic.Int32
cli := &probeClient{
create: func(ctx context.Context, _opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
if calls.Add(1) == 1 {
<-ctx.Done()
return mobyclient.ContainerCreateResult{}, ctx.Err()
}
return mobyclient.ContainerCreateResult{ID: "probe"}, nil
},
stat: func(_path string) error { return nil },
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
return mobyclient.ContainerRemoveResult{}, nil
},
}
probe, err := probeDockerProxyDirectory(t.Context(), cli, [3]string{"daemon", dir, dir}, socket)
require.NoError(t, err)
assert.True(t, probe.seen)
assert.EqualValues(t, 2, calls.Load())
})
})
}
func TestRemoveStaleProbeContainers(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
probeMount := container.MountPoint{Type: mount.TypeBind, Destination: "/gitea-runner-probe"}
writableMount := container.MountPoint{Type: mount.TypeBind, Destination: "/gitea-runner-probe", RW: true}
var removed []string
cli := &probeClient{
containers: []container.Summary{
{ID: "abandoned", State: container.StateCreated, Mounts: []container.MountPoint{probeMount}},
{ID: "foreign", State: container.StateCreated, Mounts: []container.MountPoint{writableMount}},
{ID: "started", State: container.StateExited, Mounts: []container.MountPoint{probeMount}},
},
remove: func(_ctx context.Context, id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removed = append(removed, id)
return mobyclient.ContainerRemoveResult{}, nil
},
}
start := time.Now()
removeStaleProbeContainers(cli)
assert.Equal(t, []string{"abandoned"}, removed)
assert.GreaterOrEqual(t, time.Since(start), dockerProxyProbeWorkerTimeout+dockerProxyProbeTimeout)
})
}
func TestDaemonSeesDir(t *testing.T) { func TestDaemonSeesDir(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
markers := make(map[string]bool) markers := make(map[string]bool)
for _, testCase := range []struct { for _, testCase := range []struct {
name string name string
daemonDir string daemonDir string
@@ -517,9 +723,14 @@ func TestDaemonSeesDir(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
defer cancel() defer cancel()
removed := false removed := false
var attemptedImages []string
cli := &probeClient{ cli := &probeClient{
create: func(opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) { create: func(_ctx context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
assert.Equal(t, []mount.Mount{{Type: mount.TypeBind, Source: testCase.daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}}, opts.HostConfig.Mounts) assert.Equal(t, []mount.Mount{{Type: mount.TypeBind, Source: testCase.daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}}, opts.HostConfig.Mounts)
attemptedImages = append(attemptedImages, opts.Config.Image)
if opts.Config.Image == "removed-image" {
return mobyclient.ContainerCreateResult{}, cerrdefs.ErrNotFound
}
cancel() cancel()
if _, err := os.Stat(testCase.daemonDir); err != nil { if _, err := os.Stat(testCase.daemonDir); err != nil {
return mobyclient.ContainerCreateResult{}, cerrdefs.ErrInvalidArgument return mobyclient.ContainerCreateResult{}, cerrdefs.ErrInvalidArgument
@@ -547,8 +758,10 @@ func TestDaemonSeesDir(t *testing.T) {
} }
seen, err := daemonSeesDir(ctx, cli, dir, testCase.daemonDir) seen, err := daemonSeesDir(ctx, cli, dir, testCase.daemonDir)
require.ErrorIs(t, err, testCase.removeErr) require.ErrorIs(t, err, testCase.removeErr)
assert.Equal(t, []string{"removed-image", "probe-image"}, attemptedImages)
assert.Equal(t, testCase.seen, seen) assert.Equal(t, testCase.seen, seen)
assert.Equal(t, testCase.removed, removed) assert.Equal(t, testCase.removed, removed)
entries, err := os.ReadDir(dir) entries, err := os.ReadDir(dir)
require.NoError(t, err) require.NoError(t, err)
assert.Empty(t, entries) assert.Empty(t, entries)
@@ -557,6 +770,9 @@ func TestDaemonSeesDir(t *testing.T) {
} }
func TestRunnerContainerWorkdir(t *testing.T) { func TestRunnerContainerWorkdir(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("a runner container is Linux")
}
workdir := t.TempDir() workdir := t.TempDir()
t.Chdir(workdir) t.Chdir(workdir)
volume := container.MountPoint{Type: mount.TypeVolume, Source: "/daemon/_data", Destination: filepath.Dir(workdir)} volume := container.MountPoint{Type: mount.TypeVolume, Source: "/daemon/_data", Destination: filepath.Dir(workdir)}
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd) //go:build !WITHOUT_DOCKER && (linux || darwin || netbsd || freebsd)
package container package container
+4 -4
View File
@@ -2,7 +2,7 @@
// Copyright 2020 The nektos/act Authors. All rights reserved. // Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -22,7 +22,7 @@ import (
// NewDockerPullExecutor function to create a run executor for the container // NewDockerPullExecutor function to create a run executor for the container
func NewDockerPullExecutor(input NewDockerPullExecutorInput) common.Executor { func NewDockerPullExecutor(input NewDockerPullExecutorInput) common.Executor {
return func(ctx context.Context) error { return common.Executor(func(ctx context.Context) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
logger.Debugf("docker pull %v", input.Image) logger.Debugf("docker pull %v", input.Image)
@@ -83,7 +83,7 @@ func NewDockerPullExecutor(input NewDockerPullExecutorInput) common.Executor {
return nil return nil
} }
return fmt.Errorf("failed to pull image '%s' (%s): %w", imageRef, input.Platform, err) return fmt.Errorf("failed to pull image '%s' (%s): %w", imageRef, input.Platform, err)
} }).Retry(3)
} }
func getImagePullOptions(ctx context.Context, input NewDockerPullExecutorInput) (client.ImagePullOptions, error) { func getImagePullOptions(ctx context.Context, input NewDockerPullExecutorInput) (client.ImagePullOptions, error) {
@@ -114,7 +114,7 @@ func getImagePullOptions(ctx context.Context, input NewDockerPullExecutorInput)
if err != nil { if err != nil {
return imagePullOptions, err return imagePullOptions, err
} }
if authConfig.Username == "" && authConfig.Password == "" { if authConfig.Username == "" && authConfig.Password == "" && authConfig.IdentityToken == "" && authConfig.RegistryToken == "" {
return imagePullOptions, nil return imagePullOptions, nil
} }
logger.Info("using DockerAuthConfig authentication for docker pull") logger.Info("using DockerAuthConfig authentication for docker pull")
+24
View File
@@ -6,12 +6,17 @@ package container
import ( import (
"io" "io"
"os"
"path/filepath"
"strings" "strings"
"testing" "testing"
"testing/iotest"
"gitea.com/gitea/runner/act/common/logging" "gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest" "gitea.com/gitea/runner/act/common/logging/logtest"
"github.com/moby/moby/api/pkg/authconfig"
"github.com/moby/moby/api/types/registry"
assert "github.com/stretchr/testify/assert" assert "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -64,6 +69,21 @@ func TestGetImagePullOptions(t *testing.T) {
}) })
assert.NoError(t, err, "Failed to create ImagePullOptions") //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err, "Failed to create ImagePullOptions") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, "eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwicGFzc3dvcmQiOiJwYXNzd29yZFxuIiwic2VydmVyYWRkcmVzcyI6Imh0dHBzOi8vaW5kZXguZG9ja2VyLmlvL3YxLyJ9", options.RegistryAuth, "RegistryAuth should be taken from local docker config") assert.Equal(t, "eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwicGFzc3dvcmQiOiJwYXNzd29yZFxuIiwic2VydmVyYWRkcmVzcyI6Imh0dHBzOi8vaW5kZXguZG9ja2VyLmlvL3YxLyJ9", options.RegistryAuth, "RegistryAuth should be taken from local docker config")
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"auths":{"identity.example.com":{"identitytoken":"identity-token"},"registry.example.com":{"registrytoken":"registry-token"}}}`), 0o600))
t.Setenv("DOCKER_CONFIG", dir)
for host, want := range map[string]registry.AuthConfig{
"identity.example.com": {IdentityToken: "identity-token", ServerAddress: "identity.example.com"},
"registry.example.com": {RegistryToken: "registry-token", ServerAddress: "registry.example.com"},
} {
options, err := getImagePullOptions(ctx, NewDockerPullExecutorInput{Image: host + "/app"})
require.NoError(t, err)
require.NotEmpty(t, options.RegistryAuth, host)
got, err := authconfig.Decode(options.RegistryAuth)
require.NoError(t, err)
assert.Equal(t, want, *got)
}
} }
// A digest-pinned image is immutable, so its local copy is always current. // A digest-pinned image is immutable, so its local copy is always current.
@@ -82,4 +102,8 @@ func TestLogDockerResponseError(t *testing.T) {
require.EqualError(t, streamErr(`{"error":"toomanyrequests: rate limit exceeded"}`), "toomanyrequests: rate limit exceeded") require.EqualError(t, streamErr(`{"error":"toomanyrequests: rate limit exceeded"}`), "toomanyrequests: rate limit exceeded")
require.EqualError(t, streamErr(`{"errorDetail":{"message":"unexpected EOF"}}`), "unexpected EOF") require.EqualError(t, streamErr(`{"errorDetail":{"message":"unexpected EOF"}}`), "unexpected EOF")
require.NoError(t, streamErr(`{"status":"Downloading"}`)) require.NoError(t, streamErr(`{"status":"Downloading"}`))
require.NoError(t, streamErr(`{"stream":"`+strings.Repeat("x", 70000)+`"}`))
require.ErrorIs(t, logDockerResponse(logger, io.NopCloser(io.MultiReader(
strings.NewReader(`{"status":"Downloading"}`+"\n"), iotest.ErrReader(io.ErrUnexpectedEOF),
)), false), io.ErrUnexpectedEOF)
} }
+74 -30
View File
@@ -2,7 +2,7 @@
// Copyright 2020 The nektos/act Authors. All rights reserved. // Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -205,18 +205,30 @@ func (cr *containerReference) DumpLogs(ctx context.Context) error {
if cr.id == "" { if cr.id == "" {
return cr.missingContainerError("read its logs") return cr.missingContainerError("read its logs")
} }
tty, err := cr.containerTTY(ctx)
if err != nil {
return err
}
logs, err := cr.cli.ContainerLogs(ctx, cr.id, client.ContainerLogsOptions{ShowStdout: true, ShowStderr: true}) logs, err := cr.cli.ContainerLogs(ctx, cr.id, client.ContainerLogsOptions{ShowStdout: true, ShowStderr: true})
if err != nil { if err != nil {
return err return err
} }
defer logs.Close() defer logs.Close()
return cr.copyOutput(logs) return cr.copyOutput(logs, tty)
}
func (cr *containerReference) containerTTY(ctx context.Context) (bool, error) {
inspect, err := cr.cli.ContainerInspect(ctx, cr.id, client.ContainerInspectOptions{})
if err != nil {
return false, fmt.Errorf("failed to inspect container: %w", err)
}
return inspect.Container.Config != nil && inspect.Container.Config.Tty, nil
} }
// copyOutput writes a container stream to the writers the container was created with, // copyOutput writes a container stream to the writers the container was created with,
// demultiplexing it unless the container has a TTY, which sends a single raw stream. // demultiplexing it unless the container has a TTY, which sends a single raw stream.
func (cr *containerReference) copyOutput(stream io.Reader) error { func (cr *containerReference) copyOutput(stream io.Reader, tty bool) error {
outWriter := cr.input.Stdout outWriter := cr.input.Stdout
if outWriter == nil { if outWriter == nil {
outWriter = os.Stdout outWriter = os.Stdout
@@ -227,7 +239,7 @@ func (cr *containerReference) copyOutput(stream io.Reader) error {
} }
var err error var err error
if !cr.input.AllocatePTY || os.Getenv("NORAW") != "" { if !tty || os.Getenv("NORAW") != "" {
_, err = stdcopy.StdCopy(outWriter, errWriter, stream) _, err = stdcopy.StdCopy(outWriter, errWriter, stream)
} else { } else {
_, err = io.Copy(outWriter, stream) _, err = io.Copy(outWriter, stream)
@@ -296,7 +308,7 @@ func containerInfoFromInspect(inspect container.InspectResponse) *Info {
} }
func (cr *containerReference) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor { func (cr *containerReference) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(cr, srcPath, env).IfNot(common.Dryrun) return parseEnvFile(cr, srcPath, env, false).IfNot(common.Dryrun)
} }
func (cr *containerReference) UpdateFromImageEnv(env *map[string]string) common.Executor { func (cr *containerReference) UpdateFromImageEnv(env *map[string]string) common.Executor {
@@ -354,6 +366,7 @@ type containerReference struct {
id string id string
input *NewContainerInput input *NewContainerInput
pullPolicy string pullPolicy string
daemonOS string
UID int UID int
GID int GID int
// attachDone is closed by the attach() streaming goroutine once it has // attachDone is closed by the attach() streaming goroutine once it has
@@ -363,13 +376,18 @@ type containerReference struct {
LinuxContainerEnvironmentExtensions LinuxContainerEnvironmentExtensions
} }
func GetDockerClient(ctx context.Context) (cli client.APIClient, err error) { func GetDockerClient(ctx context.Context) (client.APIClient, error) {
cli, _, err := newDockerClient(ctx)
return cli, err
}
func newDockerClient(ctx context.Context) (cli client.APIClient, daemonOS string, err error) {
dockerHost := os.Getenv("DOCKER_HOST") dockerHost := os.Getenv("DOCKER_HOST")
if strings.HasPrefix(dockerHost, "ssh://") { if strings.HasPrefix(dockerHost, "ssh://") {
var args []string var args []string
if args, err = sshArgs(dockerHost); err != nil { if args, err = sshArgs(dockerHost); err != nil {
return nil, err return nil, "", err
} }
cli, err = client.New( cli, err = client.New(
client.WithHost("http://docker.example.com"), // the dialer below ignores the address client.WithHost("http://docker.example.com"), // the dialer below ignores the address
@@ -379,16 +397,17 @@ func GetDockerClient(ctx context.Context) (cli client.APIClient, err error) {
cli, err = client.New(client.FromEnv) cli, err = client.New(client.FromEnv)
} }
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to connect to docker daemon: %w", err) return nil, "", fmt.Errorf("failed to connect to docker daemon: %w", err)
} }
// Best-effort API version negotiation, matching the old client.NegotiateAPIVersion // Best-effort API version negotiation, matching the old client.NegotiateAPIVersion
// behaviour. Ping failures here are non-fatal: the connection is exercised again on // behaviour. Ping failures here are non-fatal: the connection is exercised again on
// the first real call, and the client falls back to the daemon's default API version. // the first real call, and the client falls back to the daemon's default API version.
if _, err := cli.Ping(ctx, client.PingOptions{NegotiateAPIVersion: true}); err != nil { ping, err := cli.Ping(ctx, client.PingOptions{NegotiateAPIVersion: true})
if err != nil {
common.Logger(ctx).Warnf("docker daemon ping during version negotiation failed, continuing: %v", err) common.Logger(ctx).Warnf("docker daemon ping during version negotiation failed, continuing: %v", err)
} }
return cli, nil return cli, ping.OSType, nil
} }
func GetHostInfo(ctx context.Context) (system.Info, error) { func GetHostInfo(ctx context.Context) (system.Info, error) {
@@ -428,16 +447,32 @@ func RunnerArch(ctx context.Context) string {
return info.Architecture return info.Architecture
} }
func (cr *containerReference) targetOS(ctx context.Context) string {
if cr.daemonOS == "" {
cr.daemonOS = "linux"
if cr.cli != nil {
if result, err := cr.cli.Info(ctx, client.InfoOptions{}); err == nil && result.Info.OSType != "" {
cr.daemonOS = result.Info.OSType
}
}
}
return cr.daemonOS
}
func (cr *containerReference) markExecutable(ctx context.Context) bool {
return hostOS == "windows" && cr.targetOS(ctx) == "linux"
}
func (cr *containerReference) connect() common.Executor { func (cr *containerReference) connect() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
if cr.cli != nil { if cr.cli != nil {
return nil return nil
} }
cli, err := GetDockerClient(ctx) cli, daemonOS, err := newDockerClient(ctx)
if err != nil { if err != nil {
return err return err
} }
cr.cli = cli cr.cli, cr.daemonOS = cli, daemonOS
return nil return nil
} }
} }
@@ -447,6 +482,7 @@ func (cr *containerReference) Close() common.Executor {
if cr.cli != nil { if cr.cli != nil {
err := cr.cli.Close() err := cr.cli.Close()
cr.cli = nil cr.cli = nil
cr.daemonOS = ""
if err != nil { if err != nil {
return fmt.Errorf("failed to close client: %w", err) return fmt.Errorf("failed to close client: %w", err)
} }
@@ -499,7 +535,7 @@ func (cr *containerReference) remove() common.Executor {
if idOrName == "" && cr.input != nil { if idOrName == "" && cr.input != nil {
idOrName = cr.input.Name idOrName = cr.input.Name
} }
if idOrName == "" { if idOrName == "" || cr.cli == nil {
return nil return nil
} }
@@ -591,7 +627,8 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
copts.privileged = false copts.privileged = false
} }
containerConfig, err := parse(flags, copts, runtime.GOOS) serverOS := cr.targetOS(ctx)
containerConfig, err := parse(flags, copts, serverOS)
if err != nil { if err != nil {
return fmt.Errorf("cannot process container options: '%s': '%w'", options, err) return fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
} }
@@ -606,7 +643,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
// For Gitea, forcing --privileged off is not enough, other options reach the host too // For Gitea, forcing --privileged off is not enough, other options reach the host too
if !hostConfig.Privileged { if !hostConfig.Privileged {
trusted, err := parseOptionsHostConfig(cr.input.RunnerOptions) trusted, err := parseOptionsHostConfig(cr.input.RunnerOptions, serverOS)
if err != nil { if err != nil {
return err return err
} }
@@ -719,9 +756,7 @@ func (cr *containerReference) create(capAdd, capDrop []string) common.Executor {
cr.sanitizeConfig(ctx, hostConfig) cr.sanitizeConfig(ctx, hostConfig)
var networkingConfig *network.NetworkingConfig var networkingConfig *network.NetworkingConfig
n := hostConfig.NetworkMode if isUserDefinedNetwork(hostConfig.NetworkMode) && len(input.NetworkAliases) > 0 {
// IsUserDefined and IsHost are broken on windows
if n.IsUserDefined() && n != "host" && len(input.NetworkAliases) > 0 {
networkingConfig = &network.NetworkingConfig{ networkingConfig = &network.NetworkingConfig{
EndpointsConfig: map[string]*network.EndpointSettings{ EndpointsConfig: map[string]*network.EndpointSettings{
input.NetworkMode: {Aliases: input.NetworkAliases}, input.NetworkMode: {Aliases: input.NetworkAliases},
@@ -851,7 +886,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
cmdResponse := make(chan error, 1) cmdResponse := make(chan error, 1)
go func() { go func() {
cmdResponse <- cr.copyOutput(resp.Reader) cmdResponse <- cr.copyOutput(resp.Reader, cr.input.AllocatePTY)
}() }()
select { select {
@@ -867,7 +902,10 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
// truncate the tail of the log. The goroutine exits once the hijacked // truncate the tail of the log. The goroutine exits once the hijacked
// stream is closed by resp.Close() in the caller's defer. // stream is closed by resp.Close() in the caller's defer.
select { select {
case <-cmdResponse: case err := <-cmdResponse:
if err != nil {
logger.Debugf("Failed to drain command output after cancellation: %v", err)
}
case <-time.After(drainGracePeriod): case <-time.After(drainGracePeriod):
logger.Warn("Timed out draining command output after cancellation") logger.Warn("Timed out draining command output after cancellation")
} }
@@ -876,11 +914,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
// from executing // from executing
return ctx.Err() return ctx.Err()
case err := <-cmdResponse: case err := <-cmdResponse:
if err != nil { return err
logger.Error(err)
}
return nil
} }
} }
@@ -897,7 +931,7 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
} }
logger.Debugf("Writing tarball %s from %s", tarFile.Name(), srcPath) logger.Debugf("Writing tarball %s from %s", tarFile.Name(), srcPath)
defer func() { defer func() {
if err := tarFile.Close(); !errors.Is(err, os.ErrClosed) { if err := tarFile.Close(); err != nil && !errors.Is(err, os.ErrClosed) {
logger.Error(err) logger.Error(err)
} }
if err := os.Remove(tarFile.Name()); err != nil { if err := os.Remove(tarFile.Name()); err != nil {
@@ -908,7 +942,7 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
if actPath := cr.GetActPath(); strings.HasPrefix(dstPath+"/", actPath+"/") { if actPath := cr.GetActPath(); strings.HasPrefix(dstPath+"/", actPath+"/") {
extractionRoot = actPath // keeps tar entries from crossing the image's /var/run symlink extractionRoot = actPath // keeps tar entries from crossing the image's /var/run symlink
} }
if err := writeDirTar(ctx, tarFile, path.Join("/", strings.TrimPrefix(dstPath, extractionRoot)), srcPath, useGitIgnore, skipGitDir, cr.UID, cr.GID); err != nil { if err := writeDirTar(ctx, tarFile, path.Join("/", strings.TrimPrefix(dstPath, extractionRoot)), srcPath, useGitIgnore, skipGitDir, cr.UID, cr.GID, cr.markExecutable(ctx)); err != nil {
return err return err
} }
@@ -944,6 +978,10 @@ func (cr *containerReference) upload(ctx context.Context, dstPath string, conten
func (cr *containerReference) attach() common.Executor { func (cr *containerReference) attach() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
tty, err := cr.containerTTY(ctx)
if err != nil {
return err
}
out, err := cr.cli.ContainerAttach(ctx, cr.id, client.ContainerAttachOptions{ out, err := cr.cli.ContainerAttach(ctx, cr.id, client.ContainerAttachOptions{
Stream: true, Stream: true,
Stdout: true, Stdout: true,
@@ -954,9 +992,12 @@ func (cr *containerReference) attach() common.Executor {
} }
done := make(chan struct{}) done := make(chan struct{})
cr.attachDone = done cr.attachDone = done
stopClose := context.AfterFunc(ctx, out.Close)
go func() { go func() {
defer close(done) defer close(done)
if copyErr := cr.copyOutput(out.Reader); copyErr != nil { defer out.Close()
defer stopClose()
if copyErr := cr.copyOutput(out.Reader, tty); copyErr != nil && ctx.Err() == nil {
common.Logger(ctx).Error(copyErr) common.Logger(ctx).Error(copyErr)
} }
}() }()
@@ -991,6 +1032,9 @@ func (cr *containerReference) wait() common.Executor {
return fmt.Errorf("failed to wait for container: %w", err) return fmt.Errorf("failed to wait for container: %w", err)
} }
case status := <-waitResult.Result: case status := <-waitResult.Result:
if status.Error != nil {
return fmt.Errorf("failed to wait for container: %s", status.Error.Message)
}
statusCode = status.StatusCode statusCode = status.StatusCode
} }
@@ -1069,12 +1113,12 @@ func resetOption[T any](logger *logging.Logger, option string, field *T, trusted
// parseOptionsHostConfig parses one options string on its own, to see what it alone asks for. // parseOptionsHostConfig parses one options string on its own, to see what it alone asks for.
// Even "" goes through the parser, or its empty slices and maps would differ from a real parse. // Even "" goes through the parser, or its empty slices and maps would differ from a real parse.
func parseOptionsHostConfig(options string) (*container.HostConfig, error) { func parseOptionsHostConfig(options, serverOS string) (*container.HostConfig, error) {
flags, copts, _, err := parseContainerOptions(options) flags, copts, _, err := parseContainerOptions(options)
if err != nil { if err != nil {
return nil, err return nil, err
} }
containerConfig, err := parse(flags, copts, runtime.GOOS) containerConfig, err := parse(flags, copts, serverOS)
if err != nil { if err != nil {
return nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err) return nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
} }
+240 -41
View File
@@ -27,10 +27,12 @@ import (
"gitea.com/gitea/runner/act/common/logging/logtest" "gitea.com/gitea/runner/act/common/logging/logtest"
cerrdefs "github.com/containerd/errdefs" cerrdefs "github.com/containerd/errdefs"
"github.com/kballard/go-shellquote"
"github.com/moby/moby/api/pkg/stdcopy" "github.com/moby/moby/api/pkg/stdcopy"
"github.com/moby/moby/api/types/container" "github.com/moby/moby/api/types/container"
"github.com/moby/moby/api/types/mount" "github.com/moby/moby/api/types/mount"
"github.com/moby/moby/api/types/network" "github.com/moby/moby/api/types/network"
"github.com/moby/moby/api/types/system"
mobyclient "github.com/moby/moby/client" mobyclient "github.com/moby/moby/client"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
@@ -65,6 +67,14 @@ func TestDockerEnvironmentUsesContainerPath(t *testing.T) {
type mockDockerClient struct { type mockDockerClient struct {
mobyclient.APIClient mobyclient.APIClient
mock.Mock mock.Mock
info system.Info
infoErr error
infoCalls int
}
func (m *mockDockerClient) Info(_ctx context.Context, _opts mobyclient.InfoOptions) (mobyclient.SystemInfoResult, error) {
m.infoCalls++
return mobyclient.SystemInfoResult{Info: m.info}, m.infoErr
} }
func (m *mockDockerClient) ServerVersion(ctx context.Context, opts mobyclient.ServerVersionOptions) (mobyclient.ServerVersionResult, error) { func (m *mockDockerClient) ServerVersion(ctx context.Context, opts mobyclient.ServerVersionOptions) (mobyclient.ServerVersionResult, error) {
@@ -92,6 +102,16 @@ func (m *mockDockerClient) ContainerAttach(ctx context.Context, containerID stri
return args.Get(0).(mobyclient.ContainerAttachResult), args.Error(1) return args.Get(0).(mobyclient.ContainerAttachResult), args.Error(1)
} }
func (m *mockDockerClient) ContainerLogs(ctx context.Context, containerID string, opts mobyclient.ContainerLogsOptions) (mobyclient.ContainerLogsResult, error) {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerLogsResult), args.Error(1)
}
func (m *mockDockerClient) ContainerStart(ctx context.Context, containerID string, opts mobyclient.ContainerStartOptions) (mobyclient.ContainerStartResult, error) {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerStartResult), args.Error(1)
}
func (m *mockDockerClient) ContainerWait(ctx context.Context, containerID string, opts mobyclient.ContainerWaitOptions) mobyclient.ContainerWaitResult { func (m *mockDockerClient) ContainerWait(ctx context.Context, containerID string, opts mobyclient.ContainerWaitOptions) mobyclient.ContainerWaitResult {
args := m.Called(ctx, containerID, opts) args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerWaitResult) return args.Get(0).(mobyclient.ContainerWaitResult)
@@ -158,6 +178,7 @@ func (r *interruptReader) Read(_ []byte) (int, error) {
type mockConn struct { type mockConn struct {
net.Conn net.Conn
mock.Mock mock.Mock
closed atomic.Bool
} }
func (m *mockConn) Write(b []byte) (n int, err error) { func (m *mockConn) Write(b []byte) (n int, err error) {
@@ -166,6 +187,7 @@ func (m *mockConn) Write(b []byte) (n int, err error) {
} }
func (m *mockConn) Close() (err error) { func (m *mockConn) Close() (err error) {
m.closed.Store(true)
return nil return nil
} }
@@ -236,6 +258,21 @@ func TestDockerExecFailure(t *testing.T) {
client.AssertExpectations(t) client.AssertExpectations(t)
} }
func TestDockerExecStreamFailure(t *testing.T) {
ctx := t.Context()
client := &mockDockerClient{}
client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).
Return(mobyclient.ExecCreateResult{ID: "id"}, nil)
client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).
Return(mobyclient.ExecAttachResult{
Conn: &mockConn{},
Reader: bufio.NewReader(bytes.NewReader(stdcopyFrame(stdcopy.Systemerr, "unable to start exec"))),
}, nil)
require.ErrorContains(t, (&containerReference{id: "123", cli: client, input: &NewContainerInput{}}).
exec([]string{"sh"}, nil, "", "")(ctx), "unable to start exec")
client.AssertExpectations(t)
}
// stdcopyFrame wraps payload in a single Docker multiplexed-stream frame, the // stdcopyFrame wraps payload in a single Docker multiplexed-stream frame, the
// format StdCopy expects: an 8-byte header (stream type + 4-byte big-endian // format StdCopy expects: an 8-byte header (stream type + 4-byte big-endian
// length) followed by the payload. // length) followed by the payload.
@@ -248,60 +285,88 @@ func stdcopyFrame(stream stdcopy.StdType, payload string) []byte {
} }
func TestDockerAttachFlushesTrailingLine(t *testing.T) { func TestDockerAttachFlushesTrailingLine(t *testing.T) {
ctx := t.Context() for _, testcase := range []struct {
name string
tty bool
}{
{name: "multiplexed"},
{name: "tty from container options", tty: true},
} {
t.Run(testcase.name, func(t *testing.T) {
ctx := t.Context()
output := []byte("line one\nlast line without newline")
if !testcase.tty {
output = stdcopyFrame(stdcopy.Stdout, string(output))
}
var lines []string
logWriter := common.NewLineWriter(func(line string) bool {
lines = append(lines, line)
return true
})
client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "123", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{Config: &container.Config{Tty: testcase.tty}}}, nil)
conn := &mockConn{}
client.On("ContainerAttach", ctx, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{
Conn: conn,
Reader: bufio.NewReader(bytes.NewReader(output)),
}, nil)
client.On("ContainerLogs", ctx, "123", mobyclient.ContainerLogsOptions{ShowStdout: true, ShowStderr: true}).
Return(io.NopCloser(bytes.NewReader(output)), nil)
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 0}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{id: "123", cli: client, input: &NewContainerInput{Stdout: logWriter, Stderr: logWriter}}
framed := bytes.NewBuffer(stdcopyFrame(stdcopy.Stdout, "line one\nlast line without newline")) require.NoError(t, cr.attach()(ctx))
require.NoError(t, cr.wait()(ctx))
var lines []string assert.Equal(t, []string{"line one\n", "last line without newline"}, lines)
logWriter := common.NewLineWriter(func(s string) bool { assert.True(t, conn.closed.Load())
lines = append(lines, s) require.NoError(t, cr.DumpLogs(ctx))
return true assert.Equal(t, []string{"line one\n", "last line without newline", "line one\n", "last line without newline"}, lines)
}) client.AssertExpectations(t)
})
client := &mockDockerClient{}
client.On("ContainerAttach", ctx, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{
Conn: &mockConn{},
Reader: bufio.NewReader(framed),
}, nil)
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 0}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Stdout: logWriter,
Stderr: logWriter,
},
} }
}
require.NoError(t, cr.attach()(ctx)) func TestDockerAttachStreamClosesWithItsContext(t *testing.T) {
require.NoError(t, cr.wait()(ctx)) conn, peer := net.Pipe()
t.Cleanup(func() { _ = conn.Close() })
assert.Equal(t, []string{"line one\n", "last line without newline"}, lines) t.Cleanup(func() { _ = peer.Close() })
client := &mockDockerClient{}
client.On("ContainerInspect", mock.Anything, "123", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{Config: &container.Config{}}}, nil)
client.On("ContainerAttach", mock.Anything, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{HijackedResponse: mobyclient.NewHijackedResponse(conn, "")}, nil)
client.On("ContainerStart", mock.Anything, "123", mobyclient.ContainerStartOptions{}).
Return(mobyclient.ContainerStartResult{}, errors.New("container start failed"))
cr := &containerReference{id: "123", cli: client, input: &NewContainerInput{}}
ctx, cancel := context.WithCancel(t.Context())
require.ErrorContains(t, cr.Start(true)(ctx), "container start failed")
cancel()
<-cr.attachDone
client.AssertExpectations(t) client.AssertExpectations(t)
} }
func TestDockerWaitFailure(t *testing.T) { func TestDockerWaitFailure(t *testing.T) {
ctx := t.Context() ctx := t.Context()
statusCh := make(chan container.WaitResponse, 1) statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 2} statusCh <- container.WaitResponse{StatusCode: 2}
client := &mockDockerClient{} client := &mockDockerClient{}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}). client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{Result: statusCh}) Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{id: "123", cli: client}
err := (&containerReference{id: "123", cli: client}).wait()(ctx) err := cr.wait()(ctx)
var exitErr ExitCodeError var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr) require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(2), exitErr) assert.Equal(t, ExitCodeError(2), exitErr)
assert.Equal(t, "Process completed with exit code 2.", err.Error()) assert.Equal(t, "Process completed with exit code 2.", err.Error())
statusCh <- container.WaitResponse{Error: &container.WaitExitError{Message: "container wait failed"}}
require.EqualError(t, cr.wait()(ctx), "failed to wait for container: container wait failed")
client.AssertExpectations(t) client.AssertExpectations(t)
} }
@@ -370,6 +435,13 @@ func TestRemoveWithoutIDUsesName(t *testing.T) {
client.AssertExpectations(t) client.AssertExpectations(t)
} }
func TestRemovePreservesClientCreationError(t *testing.T) {
t.Setenv("DOCKER_HOST", "://")
require.ErrorContains(t, (&containerReference{input: &NewContainerInput{Name: "job-1"}}).
Remove()(t.Context()), "failed to connect to docker daemon")
}
func TestFindRevalidatesStaleID(t *testing.T) { func TestFindRevalidatesStaleID(t *testing.T) {
ctx := t.Context() ctx := t.Context()
inspectOpts := mobyclient.ContainerInspectOptions{} inspectOpts := mobyclient.ContainerInspectOptions{}
@@ -468,6 +540,9 @@ func TestRejectsMissingContainer(t *testing.T) {
} }
func TestPublicCopyPipelineHandlesStaleID(t *testing.T) { func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
source := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(source, "x"), nil, 0o644))
for _, testcase := range []struct { for _, testcase := range []struct {
name, destination, extractionRoot, entry string name, destination, extractionRoot, entry string
directory bool directory bool
@@ -477,7 +552,8 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
{name: "sibling prefix", directory: true, destination: "/var/run/actor/action", extractionRoot: "/", entry: "var/run/actor/action/x"}, {name: "sibling prefix", directory: true, destination: "/var/run/actor/action", extractionRoot: "/", entry: "var/run/actor/action/x"},
} { } {
t.Run(testcase.name, func(t *testing.T) { t.Run(testcase.name, func(t *testing.T) {
ctx := t.Context() hook := logtest.NewWithLevel(logging.LevelError)
ctx := common.WithLogger(t.Context(), logging.New(hook))
client := &mockDockerClient{} client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "stale", mobyclient.ContainerInspectOptions{}). client.On("ContainerInspect", ctx, "stale", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound.WithMessage("gone")) Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound.WithMessage("gone"))
@@ -494,13 +570,12 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
cr := &containerReference{id: "stale", cli: client, input: &NewContainerInput{Name: "job-1"}} cr := &containerReference{id: "stale", cli: client, input: &NewContainerInput{Name: "job-1"}}
if testcase.directory { if testcase.directory {
source := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(source, "x"), nil, 0o644))
require.NoError(t, cr.CopyDir(testcase.destination, source+string(filepath.Separator), false, true)(ctx)) require.NoError(t, cr.CopyDir(testcase.destination, source+string(filepath.Separator), false, true)(ctx))
} else { } else {
require.NoError(t, cr.Copy(testcase.destination, &FileEntry{Name: "x", Mode: 0o644})(ctx)) require.NoError(t, cr.Copy(testcase.destination, &FileEntry{Name: "x", Mode: 0o644})(ctx))
} }
assert.Equal(t, "fresh", cr.id) assert.Equal(t, "fresh", cr.id)
assert.Empty(t, hook.AllRecords())
client.AssertExpectations(t) client.AssertExpectations(t)
}) })
} }
@@ -625,11 +700,11 @@ func TestMergeContainerConfigsDropsVolumeDriversFromWorkflows(t *testing.T) {
} }
func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) { func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
hostFile := filepath.Join(t.TempDir(), "host.env") hostFile := filepath.Join(t.TempDir(), "host file.env")
require.NoError(t, os.WriteFile(hostFile, []byte("STOLEN=from-the-host\n"), 0o600)) require.NoError(t, os.WriteFile(hostFile, []byte("STOLEN=from-the-host\n"), 0o600))
t.Setenv("RUNNER_SECRET", "s3cr3t") t.Setenv("RUNNER_SECRET", "s3cr3t")
for _, option := range []string{"--env-file " + hostFile, "--label-file " + hostFile} { for _, option := range []string{shellquote.Join("--env-file", hostFile), shellquote.Join("--label-file", hostFile)} {
logger := logging.New(logtest.New()) logger := logging.New(logtest.New())
cr := &containerReference{input: &NewContainerInput{NetworkMode: "bridge", WorkflowOptions: option}} cr := &containerReference{input: &NewContainerInput{NetworkMode: "bridge", WorkflowOptions: option}}
@@ -869,6 +944,7 @@ func TestContainerInfoFromInspect(t *testing.T) {
"/etc/hosts": "", "/etc/hosts": "",
"/etc/hostname": "", "/etc/hostname": "",
}, info.Mounts) }, info.Mounts)
assert.Equal(t, "/var/lib/docker/volumes/job/_data/subdir", info.DaemonPath("/workspace/owner/repo/subdir"))
}) })
} }
@@ -916,7 +992,7 @@ func TestDockerCreateKeepsInputEntrypoint(t *testing.T) {
RunnerOptions: "--entrypoint=runner-entrypoint", RunnerOptions: "--entrypoint=runner-entrypoint",
WorkflowOptions: "--entrypoint=" + optionEntrypoint + " --init", WorkflowOptions: "--entrypoint=" + optionEntrypoint + " --init",
}, },
cli: &probeClient{create: func(opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) { cli: &probeClient{create: func(_ctx context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
if len(entrypoint) > 0 { if len(entrypoint) > 0 {
assert.Equal(t, entrypoint, opts.Config.Entrypoint) assert.Equal(t, entrypoint, opts.Config.Entrypoint)
} else { } else {
@@ -934,6 +1010,39 @@ func TestDockerCreateKeepsInputEntrypoint(t *testing.T) {
} }
} }
func TestDockerCreateNetworkAliases(t *testing.T) {
for _, test := range []struct {
networkMode string
wantAliases bool
}{
{"job-network", true},
{"host", false},
{"bridge", false},
{"default", false},
{"none", false},
{"container:x", false},
} {
t.Run(test.networkMode, func(t *testing.T) {
cr := &containerReference{
input: &NewContainerInput{NetworkMode: test.networkMode, NetworkAliases: []string{"service"}},
cli: &probeClient{create: func(_ctx context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
assert.Equal(t, container.NetworkMode(test.networkMode), opts.HostConfig.NetworkMode)
if test.wantAliases {
assert.Equal(t, &network.NetworkingConfig{EndpointsConfig: map[string]*network.EndpointSettings{
"job-network": {Aliases: []string{"service"}},
}}, opts.NetworkingConfig)
} else {
assert.Nil(t, opts.NetworkingConfig)
}
return mobyclient.ContainerCreateResult{ID: "created"}, nil
}},
}
require.NoError(t, cr.create(nil, nil)(t.Context()))
assert.Equal(t, "created", cr.id)
})
}
}
func TestMergeContainerConfigsKeepsNetworkAliasesFromOptions(t *testing.T) { func TestMergeContainerConfigsKeepsNetworkAliasesFromOptions(t *testing.T) {
logger := logging.New(logtest.New()) logger := logging.New(logtest.New())
cr := &containerReference{input: &NewContainerInput{ cr := &containerReference{input: &NewContainerInput{
@@ -955,3 +1064,93 @@ func TestSupportsContainerImagePlatformDaemonError(t *testing.T) {
require.ErrorContains(t, err, "cannot connect to the Docker daemon") require.ErrorContains(t, err, "cannot connect to the Docker daemon")
assert.False(t, supported) assert.False(t, supported)
} }
func TestContainerOptionsUseDaemonOS(t *testing.T) {
for _, testcase := range []struct {
name string
serverOS string
infoErr error
device string
expected container.DeviceMapping
}{
{name: "Linux daemon", serverOS: "linux", device: "/dev/kvm:/dev/kvm:rwm", expected: container.DeviceMapping{PathOnHost: "/dev/kvm", PathInContainer: "/dev/kvm", CgroupPermissions: "rwm"}},
{name: "unknown daemon", device: "/dev/kvm:/dev/kvm:rwm", expected: container.DeviceMapping{PathOnHost: "/dev/kvm", PathInContainer: "/dev/kvm", CgroupPermissions: "rwm"}},
{name: "unavailable daemon info", infoErr: errors.New("unavailable"), device: "/dev/kvm:/dev/kvm:rwm", expected: container.DeviceMapping{PathOnHost: "/dev/kvm", PathInContainer: "/dev/kvm", CgroupPermissions: "rwm"}},
{name: "Windows daemon", serverOS: "windows", device: "class/01234567-89ab-cdef-0123-456789abcdef", expected: container.DeviceMapping{PathOnHost: "class/01234567-89ab-cdef-0123-456789abcdef"}},
} {
t.Run(testcase.name, func(t *testing.T) {
cli := &mockDockerClient{info: system.Info{OSType: testcase.serverOS}, infoErr: testcase.infoErr}
cr := &containerReference{cli: cli, input: &NewContainerInput{RunnerOptions: "--device=" + testcase.device, WorkflowOptions: "--device=" + testcase.device, NetworkMode: "bridge"}}
for range 2 {
hostConfig := &container.HostConfig{}
require.NoError(t, cr.mergeContainerConfigs(t.Context(), &container.Config{}, hostConfig))
assert.Equal(t, []container.DeviceMapping{testcase.expected}, hostConfig.Devices)
}
assert.Equal(t, 1, cli.infoCalls)
})
}
}
func TestConnectTakesDaemonOSFromPing(t *testing.T) {
var infos atomic.Int32
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("API-Version", "1.47")
writer.Header().Set("OSType", "windows")
if strings.HasSuffix(request.URL.Path, "/info") {
infos.Add(1)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
cr := &containerReference{input: &NewContainerInput{}}
require.NoError(t, cr.connect()(t.Context()))
assert.Equal(t, "windows", cr.targetOS(t.Context()))
assert.Zero(t, infos.Load())
}
func TestContainerCopyModes(t *testing.T) {
source := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(source, "run.sh"), []byte("#!/bin/sh\n"), 0o666))
require.NoError(t, os.Chmod(filepath.Join(source, "run.sh"), 0o666))
for _, testcase := range []struct {
hostOS, serverOS string
expected int64
}{
{"windows", "linux", 0o755},
{"windows", "windows", 0o666},
{"linux", "linux", 0o666},
} {
t.Run(testcase.hostOS+"-"+testcase.serverOS, func(t *testing.T) {
file := &FileEntry{Name: "run.sh", Mode: 0o600, Body: "#!/bin/sh\n"}
original := hostOS
hostOS = testcase.hostOS
t.Cleanup(func() { hostOS = original })
cli := &mockDockerClient{info: system.Info{OSType: testcase.serverOS}}
var modes []int64
cli.On("CopyToContainer", mock.Anything, "job", mock.Anything).
Run(func(args mock.Arguments) {
reader := tar.NewReader(args.Get(2).(mobyclient.CopyToContainerOptions).Content)
header, err := reader.Next()
require.NoError(t, err)
modes = append(modes, header.Mode)
assert.Equal(t, "run.sh", filepath.Base(header.Name))
content, err := io.ReadAll(reader)
require.NoError(t, err)
assert.Equal(t, file.Body, string(content))
_, err = reader.Next()
require.ErrorIs(t, err, io.EOF)
}).Return(mobyclient.CopyToContainerResult{}, nil).Twice()
cr := &containerReference{cli: cli, id: "job", input: &NewContainerInput{}}
require.NoError(t, cr.copyContent("/workspace", file)(t.Context()))
require.NoError(t, cr.copyDir("/workspace", source+string(filepath.Separator), false, true)(t.Context()))
assert.Equal(t, []int64{0o600, testcase.expected}, modes)
if testcase.hostOS == "windows" {
assert.Equal(t, 1, cli.infoCalls)
} else {
assert.Zero(t, cli.infoCalls)
}
cli.AssertExpectations(t)
})
}
}
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
+2 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
@@ -30,6 +30,7 @@ func TestSSHArgs(t *testing.T) {
} }
func TestDialCommandStreamsStdioAndReportsExit(t *testing.T) { func TestDialCommandStreamsStdioAndReportsExit(t *testing.T) {
t.Parallel()
if _, err := exec.LookPath("sh"); err != nil { if _, err := exec.LookPath("sh"); err != nil {
t.Skip("needs a POSIX shell") t.Skip("needs a POSIX shell")
} }
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (darwin || netbsd) //go:build !WITHOUT_DOCKER && (darwin || netbsd || freebsd)
package container package container
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2023 The nektos/act Authors. All rights reserved. // Copyright 2023 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build WITHOUT_DOCKER || !(linux || darwin || windows || netbsd) //go:build WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd)
package container package container
+1 -1
View File
@@ -2,7 +2,7 @@
// Copyright 2020 The nektos/act Authors. All rights reserved. // Copyright 2020 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package container package container
+12 -1
View File
@@ -4,7 +4,18 @@
package container package container
import "context" import (
"context"
"path"
"strings"
)
func JoinPath(elem ...string) string { // path.Join would collapse the leading // of a UNC act path on a Windows host
if len(elem) > 0 && strings.HasPrefix(elem[0], "//") {
return "/" + path.Join(elem...)
}
return path.Join(elem...)
}
type ExecutionsEnvironment interface { type ExecutionsEnvironment interface {
Container Container
+43 -139
View File
@@ -40,7 +40,8 @@ type HostEnvironment struct {
// when Workdir points at a caller-owned checkout (e.g. `act` local mode). // when Workdir points at a caller-owned checkout (e.g. `act` local mode).
CleanWorkdir bool CleanWorkdir bool
ActPath string ActPath string
CleanUp func() Root string
TrackingID string // RUNNER_TRACKING_ID of the job, a step with another value keeps its leftovers
StdOut io.Writer StdOut io.Writer
AllocatePTY bool // allocate a pseudo-TTY for each step's process AllocatePTY bool // allocate a pseudo-TTY for each step's process
@@ -56,7 +57,7 @@ func (e *HostEnvironment) processGroup(ctx context.Context) *process.Group {
e.procGroupOnce.Do(func() { e.procGroupOnce.Do(func() {
group, err := process.NewGroup() group, err := process.NewGroup()
if err != nil { if err != nil {
common.Logger(ctx).Warnf("could not create the job's process group; processes a step leaves behind can only be reclaimed by the workspace scan: %v", err) common.Logger(ctx).Warnf("could not create the job's process group, processes a step leaves behind may outlive the job: %v", err)
return return
} }
e.procGroup.Store(group) e.procGroup.Store(group)
@@ -225,12 +226,16 @@ func setupPty(cmd *exec.Cmd, cmdline string) (*os.File, *os.File, error) {
return ppty, tty, nil return ppty, tty, nil
} }
func writeKeepAlive(ppty io.Writer) { func writeKeepAlive(ctx context.Context, ppty io.Writer) {
c := 1 for ctx.Err() == nil {
var err error if count, err := ppty.Write([]byte{4}); count != 1 || err != nil {
for c == 1 && err == nil { return
c, err = ppty.Write([]byte{4}) }
<-time.After(time.Second) select {
case <-ctx.Done():
return
case <-time.After(time.Second):
}
} }
} }
@@ -257,6 +262,7 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
// Flush any buffered, not-yet-newline-terminated trailing line, as the docker backend // Flush any buffered, not-yet-newline-terminated trailing line, as the docker backend
// does in waitForCommand, so the final line of a command's output is not lost. // does in waitForCommand, so the final line of a command's output is not lost.
defer common.FlushWriter(e.StdOut) defer common.FlushWriter(e.StdOut)
process.UTF8Console()
f, err := lookupPathHost(command[0], env, e.StdOut) f, err := lookupPathHost(command[0], env, e.StdOut)
if err != nil { if err != nil {
@@ -266,7 +272,12 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
cmd.Path = f cmd.Path = f
cmd.Args = command cmd.Args = command
cmd.Stdout = e.StdOut cmd.Stdout = e.StdOut
cmd.Env = getEnvListFromMap(env) trackingID := e.TrackingID
if value, ok := common.LookupFold(env, "RUNNER_TRACKING_ID", e.IsEnvironmentCaseInsensitive()); ok {
trackingID = value
}
tracked := trackingID == e.TrackingID
cmd.Env = append(getEnvListFromMap(env), "RUNNER_TRACKING_ID="+trackingID)
cmd.Stderr = e.StdOut cmd.Stderr = e.StdOut
cmd.Dir = wd cmd.Dir = wd
cmd.SysProcAttr = process.SysProcAttr(cmdline, false) cmd.SysProcAttr = process.SysProcAttr(cmdline, false)
@@ -274,6 +285,7 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
// Kills the step's whole tree on cancellation and bounds the post-exit I/O // Kills the step's whole tree on cancellation and bounds the post-exit I/O
// wait, so an orphan holding cmd's stdout pipe cannot hang cmd.Wait(). // wait, so an orphan holding cmd's stdout pipe cannot hang cmd.Wait().
treeKill := process.NewTreeKill(cmd) treeKill := process.NewTreeKill(cmd)
treeKill.KeepLeftovers = !tracked
var ppty *os.File var ppty *os.File
var tty *os.File var tty *os.File
@@ -302,15 +314,15 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
defer finishLog() defer finishLog()
_, _ = io.Copy(writer, ppty) _, _ = io.Copy(writer, ppty)
}() }()
go writeKeepAlive(ppty) go writeKeepAlive(logctx, ppty)
} }
if err := cmd.Start(); err != nil { if err := cmd.Start(); err != nil {
return err return err
} }
// Assign before the step's Killer so the step's job nests inside the group's; if tracked {
// cancellation still scopes to this step's tree. if err := e.processGroup(ctx).Assign(cmd.Process); err != nil {
if err := e.processGroup(ctx).Assign(cmd.Process); err != nil { common.Logger(ctx).Warnf("could not assign the step's process to the job's process group, a process it leaves behind may outlive the job: %v", err)
common.Logger(ctx).Warnf("could not assign the step's process to the job's process group; a process it leaves behind may outlive the job: %v", err) }
} }
if k, kerr := treeKill.Capture(cmd.Process); kerr != nil { if k, kerr := treeKill.Capture(cmd.Process); kerr != nil {
common.Logger(ctx).Warnf("process tree kill setup failed, falling back to single-process kill: %v", kerr) common.Logger(ctx).Warnf("process tree kill setup failed, falling back to single-process kill: %v", kerr)
@@ -318,21 +330,21 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
defer k.Close() defer k.Close()
} }
err = treeKill.Wait(ctx, cmd) err = treeKill.Wait(ctx, cmd)
if err != nil {
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
return ExitCodeError(exitErr.ExitCode())
}
return err
}
if tty != nil { if tty != nil {
writer.AutoStop.Store(true) writer.AutoStop.Store(true)
if _, err := tty.WriteString("\x04"); err != nil { if _, err := tty.WriteString("\x04"); err != nil {
common.Logger(ctx).Debug("Failed to write EOT") common.Logger(ctx).Debug("Failed to write EOT")
} }
<-logctx.Done() select {
case <-logctx.Done():
case <-ctx.Done(): // a cancelled step does not wait on output nobody consumes
}
ppty.Close() ppty.Close()
ppty = nil ppty = nil
} }
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
return ExitCodeError(exitErr.ExitCode())
}
return err return err
} }
@@ -355,15 +367,13 @@ func (e *HostEnvironment) ExecWithCmdLine(command []string, cmdline string, env
} }
func (e *HostEnvironment) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor { func (e *HostEnvironment) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(e, srcPath, env) return parseEnvFile(e, srcPath, env, runtime.GOOS == "windows")
} }
// removeAll is a var so tests can substitute a blocking stub. // removeAll is a var so tests can substitute a blocking stub.
var removeAll = os.RemoveAll var removeAll = os.RemoveAll
// removeAllWithContext returns once the delete finishes or ctx is cancelled. On // Returns on cancel while an uninterruptible delete keeps running, freeing the runner's capacity slot.
// cancellation the goroutine leaks: a delete inside a syscall cannot be
// interrupted (see runWithTimeout).
func removeAllWithContext(ctx context.Context, path string) error { func removeAllWithContext(ctx context.Context, path string) error {
done := make(chan error, 1) done := make(chan error, 1)
go func() { done <- removeAll(path) }() go func() { done <- removeAll(path) }()
@@ -405,129 +415,22 @@ func removePathWithRetry(ctx context.Context, path string) error {
return lastErr return lastErr
} }
// buildWindowsWorkspaceKillScript builds a PowerShell command that taskkills
// every process tree whose ExecutablePath or CommandLine references one of the
// given workspace dirs, releasing file handles for cleanup. Win32_Process
// exposes both fields (Get-Process doesn't, wmic is deprecated); matching is on
// the dir+separator prefix via ordinal String methods, so a name-prefix sibling
// (job1 vs job10) is spared and path metacharacters stay literal.
func buildWindowsWorkspaceKillScript(dirs []string) string {
quoted := make([]string, len(dirs))
for i, d := range dirs {
// Single-quoted PowerShell literal; escape ' by doubling it.
quoted[i] = "'" + strings.ReplaceAll(d, "'", "''") + "'"
}
return `$paths = @(` + strings.Join(quoted, ",") + `)
$selfPid = $PID
Get-CimInstance Win32_Process -ErrorAction SilentlyContinue | Where-Object {
if ($_.ProcessId -eq $selfPid) { return $false }
foreach ($p in $paths) {
$prefix = $p + '\'
if ($_.ExecutablePath -and $_.ExecutablePath.StartsWith($prefix, [System.StringComparison]::OrdinalIgnoreCase)) { return $true }
if ($_.CommandLine -and $_.CommandLine.IndexOf($prefix, [System.StringComparison]::OrdinalIgnoreCase) -ge 0) { return $true }
}
return $false
} | ForEach-Object {
& taskkill.exe /PID $_.ProcessId /T /F 2>$null | Out-Null
}
`
}
func (e *HostEnvironment) terminateRunningProcesses(ctx context.Context) {
if runtime.GOOS != "windows" {
return
}
// Detached: exec.CommandContext won't start on a cancelled ctx, and a
// server cancel has already cancelled the parent ctx.
killCtx, killCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer killCancel()
logger := common.Logger(ctx)
// Dirs we own; a process referencing one is a leftover. ToolCache is shared
// across jobs, and Workdir may be a caller-owned checkout.
owned := []string{e.Path, e.TmpDir}
if e.CleanWorkdir {
owned = append(owned, e.Workdir)
}
dirs := make([]string, 0, len(owned))
for _, d := range owned {
if d == "" {
continue
}
abs, err := filepath.Abs(d)
if err != nil {
continue
}
dirs = append(dirs, abs)
}
if len(dirs) == 0 {
return
}
script := buildWindowsWorkspaceKillScript(dirs)
cmd := exec.CommandContext(killCtx, "powershell.exe", "-NoProfile", "-NonInteractive", "-Command", script)
out, err := cmd.CombinedOutput()
if err != nil {
logger.Debugf("workspace process-tree kill via PowerShell failed: %v output=%s", err, strings.TrimSpace(string(out)))
}
// Win32_Process exposes no working directory, so the scan above misses a
// process that merely runs in a workspace dir while pinning a handle on it.
if killed, err := process.KillProcessesWithCWDUnder(killCtx, dirs); err != nil {
logger.Debugf("workspace process kill by working directory reported errors: %v", err)
} else if killed > 0 {
logger.Debugf("terminated %d leftover process(es) by workspace working directory", killed)
}
}
// hostCleanupTimeout bounds each teardown phase so one stalled delete cannot // hostCleanupTimeout bounds each teardown phase so one stalled delete cannot
// wedge the runner slot. A var so tests can shrink it. // wedge the runner slot. A var so tests can shrink it.
var hostCleanupTimeout = 30 * time.Second var hostCleanupTimeout = 30 * time.Second
// runWithTimeout returns context.DeadlineExceeded once timeout elapses, leaking
// the goroutine: a delete blocked in a syscall (AV filter driver, dead network
// mount) cannot be interrupted, and leaking scratch state beats losing the
// runner's capacity slot forever. The idle stale-dir sweep reclaims it later.
func runWithTimeout(fn func(), timeout time.Duration) error {
done := make(chan struct{})
go func() {
defer close(done)
fn()
}()
timer := time.NewTimer(timeout)
defer timer.Stop()
select {
case <-done:
return nil
case <-timer.C:
return context.DeadlineExceeded
}
}
func (e *HostEnvironment) Remove() common.Executor { func (e *HostEnvironment) Remove() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
// End lingering processes before removing the workspace; on Windows their // End lingering processes before removing the workspace, on Windows their file locks block cleanup.
// file locks block cleanup. Closing the group is deterministic, the scan a net.
if err := e.procGroup.Load().Close(); err != nil { if err := e.procGroup.Load().Close(); err != nil {
logger.Debugf("closing the job's process group failed: %v", err) logger.Debugf("closing the job's process group failed: %v", err)
} }
e.terminateRunningProcesses(ctx) if killed, err := process.KillTrackedProcesses(e.TrackingID); err != nil {
logger.Debugf("killing the job's leftover processes reported errors: %v", err)
// Removes per-job misc state only, never the toolcache root. Bounded because } else if killed > 0 {
// CleanUp is a caller-supplied, typically unbounded os.RemoveAll. logger.Debugf("killed %d leftover process(es) of the job", killed)
if e.CleanUp != nil {
logger.Debugf("running host environment cleanup callback")
if err := runWithTimeout(e.CleanUp, hostCleanupTimeout); err != nil {
logger.Warnf("host environment cleanup did not finish within %s; continuing job completion, scratch state may be leaked and is reclaimed by the idle stale-dir sweep", hostCleanupTimeout)
} else {
logger.Debugf("host environment cleanup callback finished")
}
} }
// Detach: a cancelled ctx would skip removePathWithRetry's retries, // Detach: a cancelled ctx would skip removePathWithRetry's retries,
@@ -536,8 +439,8 @@ func (e *HostEnvironment) Remove() common.Executor {
defer rmCancel() defer rmCancel()
var errs []error var errs []error
if err := removePathWithRetry(rmCtx, e.Path); err != nil { if err := removePathWithRetry(rmCtx, cmp.Or(e.Root, e.Path)); err != nil {
logger.Warnf("failed to remove host misc state %s: %v", e.Path, err) logger.Warnf("failed to remove host misc state %s: %v", cmp.Or(e.Root, e.Path), err)
errs = append(errs, err) errs = append(errs, err)
} }
if e.CleanWorkdir { if e.CleanWorkdir {
@@ -609,6 +512,7 @@ func goOsToActionOs(os string) string {
"linux": "Linux", "linux": "Linux",
"darwin": "macOS", "darwin": "macOS",
"windows": "Windows", "windows": "Windows",
"freebsd": "FreeBSD",
}[os], os) }[os], os)
} }
+98 -111
View File
@@ -14,6 +14,7 @@ import (
"runtime" "runtime"
"strings" "strings"
"testing" "testing"
"testing/synctest"
"time" "time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -26,11 +27,16 @@ import (
var _ ExecutionsEnvironment = &HostEnvironment{} var _ ExecutionsEnvironment = &HostEnvironment{}
func TestActionPlatformNames(t *testing.T) { func TestActionPlatformNames(t *testing.T) {
assert.Equal(t, []string{"Linux", "macOS", "Windows"}, []string{goOsToActionOs("linux"), goOsToActionOs("darwin"), goOsToActionOs("windows")}) assert.Equal(t, []string{"Linux", "macOS", "Windows", "FreeBSD"}, []string{goOsToActionOs("linux"), goOsToActionOs("darwin"), goOsToActionOs("windows"), goOsToActionOs("freebsd")})
assert.Equal(t, []string{"X86", "X64", "ARM", "ARM64"}, []string{goArchToActionArch("386"), goArchToActionArch("amd64"), goArchToActionArch("arm"), goArchToActionArch("arm64")}) assert.Equal(t, []string{"X86", "X64", "ARM", "ARM64"}, []string{goArchToActionArch("386"), goArchToActionArch("amd64"), goArchToActionArch("arm"), goArchToActionArch("arm64")})
} }
func TestCopyDir(t *testing.T) { func TestCopyDir(t *testing.T) {
src := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.MkdirAll(filepath.Join(src, ".git", "objects"), 0o700))
require.NoError(t, os.WriteFile(filepath.Join(src, ".git", "objects", "pack"), []byte("x"), 0o600))
require.NoError(t, os.WriteFile(filepath.Join(src, "action.yml"), []byte("name: x\n"), 0o600))
for _, test := range []struct { for _, test := range []struct {
name string name string
useGitIgnore, skipGitDir bool useGitIgnore, skipGitDir bool
@@ -40,11 +46,6 @@ func TestCopyDir(t *testing.T) {
{name: "unfiltered outside a git repository", useGitIgnore: true}, {name: "unfiltered outside a git repository", useGitIgnore: true},
} { } {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
src := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.MkdirAll(filepath.Join(src, ".git", "objects"), 0o700))
require.NoError(t, os.WriteFile(filepath.Join(src, ".git", "objects", "pack"), []byte("x"), 0o600))
require.NoError(t, os.WriteFile(filepath.Join(src, "action.yml"), []byte("name: x\n"), 0o600))
dest := filepath.Join(t.TempDir(), "dest") dest := filepath.Join(t.TempDir(), "dest")
require.NoError(t, (&HostEnvironment{}).CopyDir(dest, src+string(filepath.Separator), test.useGitIgnore, test.skipGitDir)(t.Context())) require.NoError(t, (&HostEnvironment{}).CopyDir(dest, src+string(filepath.Separator), test.useGitIgnore, test.skipGitDir)(t.Context()))
@@ -79,20 +80,48 @@ func TestGetContainerArchive(t *testing.T) {
} }
func TestHostEnvironmentExecExitCode(t *testing.T) { func TestHostEnvironmentExecExitCode(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" { if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell") t.Skip("uses POSIX shell")
} }
ctx := t.Context() for name, allocatePTY := range map[string]bool{"without PTY": false, "with PTY": true} {
e := &HostEnvironment{Path: t.TempDir(), StdOut: io.Discard} t.Run(name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
reader, writer := io.Pipe()
defer reader.Close()
e := &HostEnvironment{Path: t.TempDir(), StdOut: writer, AllocatePTY: allocatePTY}
t.Cleanup(func() { assert.NoError(t, e.Remove()(t.Context())) })
err := e.Exec([]string{"sh", "-c", "exit 3"}, map[string]string{"PATH": os.Getenv("PATH")}, "", "")(ctx) result := make(chan error, 1)
var exitErr ExitCodeError go func() {
require.ErrorAs(t, err, &exitErr) defer writer.Close()
assert.Equal(t, ExitCodeError(3), exitErr) result <- e.Exec([]string{"sh", "-c", "printf 'failed\\n'; exit 3"}, map[string]string{"PATH": os.Getenv("PATH")}, "", "")(t.Context())
assert.Equal(t, "Process completed with exit code 3.", err.Error()) }()
first := make([]byte, 1)
_, err := io.ReadFull(reader, first)
require.NoError(t, err)
synctest.Wait()
select {
case err := <-result:
t.Fatalf("Exec returned before its output was consumed: %v", err)
default:
}
remaining, err := io.ReadAll(reader)
require.NoError(t, err)
err = <-result
var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(3), exitErr)
assert.Equal(t, "Process completed with exit code 3.", err.Error())
assert.Equal(t, "failed\n", string(first)+string(remaining))
})
})
}
} }
func TestHostEnvironmentAllocatePTY(t *testing.T) { func TestHostEnvironmentAllocatePTY(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" { if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell") t.Skip("uses POSIX shell")
} }
@@ -105,6 +134,7 @@ func TestHostEnvironmentAllocatePTY(t *testing.T) {
{name: "on", allocPTY: true, expect: "TTY"}, {name: "on", allocPTY: true, expect: "TTY"},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel()
buf := &bytes.Buffer{} buf := &bytes.Buffer{}
e := &HostEnvironment{ e := &HostEnvironment{
Path: t.TempDir(), Path: t.TempDir(),
@@ -131,10 +161,12 @@ func TestHostEnvironmentRemoveWorkdir(t *testing.T) {
Path: filepath.Join(miscRoot, "hostexecutor"), Path: filepath.Join(miscRoot, "hostexecutor"),
Workdir: workdir, Workdir: workdir,
CleanWorkdir: owned, CleanWorkdir: owned,
CleanUp: func() { _ = os.RemoveAll(miscRoot) }, Root: miscRoot,
} }
require.NoError(t, os.MkdirAll(e.Path, 0o700)) require.NoError(t, os.MkdirAll(e.Path, 0o700))
require.NoError(t, os.MkdirAll(filepath.Join(miscRoot, "tmp"), 0o700))
require.NoError(t, e.Remove()(t.Context())) require.NoError(t, e.Remove()(t.Context()))
assert.NoDirExists(t, miscRoot)
_, err := os.Stat(workdir) _, err := os.Stat(workdir)
if owned { if owned {
assert.ErrorIs(t, err, os.ErrNotExist) assert.ErrorIs(t, err, os.ErrNotExist)
@@ -146,108 +178,63 @@ func TestHostEnvironmentRemoveWorkdir(t *testing.T) {
} }
func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) { func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) {
release := make(chan struct{}) synctest.Test(t, func(t *testing.T) {
stubDone := make(chan struct{}) release := make(chan struct{})
stubDone := make(chan struct{})
orig := removeAll orig := removeAll
removeAll = func(string) error { removeAll = func(string) error {
defer close(stubDone) defer close(stubDone)
<-release <-release
return nil return nil
} }
t.Cleanup(func() { t.Cleanup(func() {
close(release) close(release)
<-stubDone <-stubDone
removeAll = orig removeAll = orig
})
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
err := removeAllWithContext(ctx, t.TempDir())
require.ErrorIs(t, err, context.DeadlineExceeded)
}) })
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
err := removeAllWithContext(ctx, t.TempDir())
require.ErrorIs(t, err, context.DeadlineExceeded)
} }
func TestHostEnvironmentRemoveDoesNotHang(t *testing.T) { func TestHostEnvironmentRemoveDoesNotHang(t *testing.T) {
for _, phase := range []string{"CleanUp callback", "path removal"} { synctest.Test(t, func(t *testing.T) {
t.Run(phase, func(t *testing.T) { origTimeout, origRemoveAll := hostCleanupTimeout, removeAll
origTimeout := hostCleanupTimeout hostCleanupTimeout = 100 * time.Millisecond
hostCleanupTimeout = 100 * time.Millisecond release, stubDone := make(chan struct{}), make(chan struct{})
t.Cleanup(func() { hostCleanupTimeout = origTimeout }) removeAll = func(string) error {
release, stubDone := make(chan struct{}), make(chan struct{}) defer close(stubDone)
block := func() { <-release
defer close(stubDone) return nil
<-release }
} t.Cleanup(func() {
e := &HostEnvironment{Path: filepath.Join(t.TempDir(), "misc", "hostexecutor"), StdOut: os.Stdout} close(release)
if phase == "CleanUp callback" { <-stubDone
e.CleanUp = block hostCleanupTimeout, removeAll = origTimeout, origRemoveAll
} else {
origRemoveAll := removeAll
removeAll = func(string) error {
block()
return nil
}
t.Cleanup(func() { removeAll = origRemoveAll })
}
t.Cleanup(func() {
close(release)
<-stubDone
})
require.NoError(t, os.MkdirAll(e.Path, 0o700))
done := make(chan error, 1)
go func() { done <- e.Remove()(common.WithLogger(t.Context(), logging.Default())) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatalf("Remove() hung on a stuck %s", phase)
}
}) })
e := &HostEnvironment{Path: filepath.Join(t.TempDir(), "misc", "hostexecutor"), StdOut: os.Stdout}
require.NoError(t, os.MkdirAll(e.Path, 0o700))
done := make(chan error, 1)
go func() { done <- e.Remove()(common.WithLogger(t.Context(), logging.Default())) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatal("Remove() hung on a stuck path removal")
}
})
}
func TestJoinPath(t *testing.T) {
assert.Equal(t, "//server/share/act/workflow/envs.txt", JoinPath("//server/share/act", "workflow/envs.txt"))
assert.Equal(t, "/var/run/act/workflow/envs.txt", JoinPath("/var/run/act/", "workflow", "envs.txt"))
for _, root := range []string{"//server/share/act", "/var/run/act", "C:/act"} {
assert.Equal(t, root+"/actions/key/main.go.out", JoinPath(root+"/actions/key", "main.go.out"))
assert.Equal(t, root+"/workflow", JoinPath(root+"/workflow/outputcmd.txt", ".."))
} }
} }
func TestBuildWindowsWorkspaceKillScript(t *testing.T) {
t.Run("single dir", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript([]string{`C:\workspace\job1`})
assert.Contains(t, s, `$paths = @('C:\workspace\job1')`)
assert.Contains(t, s, "$selfPid = $PID")
assert.Contains(t, s, "$_.ProcessId -eq $selfPid")
assert.Contains(t, s, `$prefix = $p + '\'`)
assert.Contains(t, s, "$_.ExecutablePath.StartsWith($prefix")
assert.Contains(t, s, "$_.CommandLine.IndexOf($prefix")
assert.Contains(t, s, "taskkill.exe /PID $_.ProcessId /T /F")
})
t.Run("multiple dirs comma-separated", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript([]string{
`C:\work\path`,
`C:\work\workdir`,
`C:\Users\runner\AppData\Local\Temp\job-42`,
})
assert.Contains(t, s, `'C:\work\path'`)
assert.Contains(t, s, `'C:\work\workdir'`)
assert.Contains(t, s, `'C:\Users\runner\AppData\Local\Temp\job-42'`)
assert.Contains(t, s, `'C:\work\path','C:\work\workdir',`)
})
t.Run("path with single quote is escaped", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript([]string{`C:\work\it's\path`})
assert.Contains(t, s, `'C:\work\it''s\path'`)
assert.NotContains(t, s, `'C:\work\it's\path'`)
})
t.Run("path with wildcard metacharacters is matched literally", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript([]string{`C:\work\[job]?1`})
assert.Contains(t, s, `'C:\work\[job]?1'`)
assert.NotContains(t, s, "-like")
assert.Contains(t, s, "StartsWith")
assert.Contains(t, s, "IndexOf")
})
t.Run("empty dir list still produces a valid script", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript(nil)
assert.Contains(t, s, "$paths = @()")
assert.Contains(t, s, "Get-CimInstance Win32_Process")
})
}
@@ -0,0 +1,112 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"context"
"fmt"
"io"
"net"
"os"
"os/exec"
"strings"
"testing"
"time"
"github.com/stretchr/testify/require"
"golang.org/x/sys/windows"
)
func TestHostEnvironmentTrackingCleanup(t *testing.T) {
for _, tracking := range []string{"default", "", "shared-helper"} {
t.Run("tracking="+tracking, func(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
defer cancel()
listener, err := net.Listen("tcp4", "127.0.0.1:0")
require.NoError(t, err)
defer listener.Close()
stop := context.AfterFunc(ctx, func() { _ = listener.Close() })
defer stop()
workspace := t.TempDir()
environment := &HostEnvironment{Path: t.TempDir(), TmpDir: workspace, Workdir: workspace, TrackingID: "gitea_test", StdOut: io.Discard}
env := map[string]string{}
for _, entry := range os.Environ() {
if name, value, ok := strings.Cut(entry, "="); ok && !strings.EqualFold(name, "RUNNER_TRACKING_ID") {
env[name] = value
}
}
env["GITEA_TRACKING_HELPER"] = "step"
env["GITEA_TRACKING_ADDRESS"] = listener.Addr().String()
if tracking == "default" {
env["RUNNER_TRACKING_ID"] = environment.TrackingID
env["runner_tracking_id"] = "shared-helper"
} else {
env["runner_tracking_id"] = tracking
}
result := make(chan error, 1)
go func() {
result <- environment.Exec([]string{os.Args[0], "-test.run=^TestHostTrackingHelper$"}, env, "", workspace)(ctx)
}()
var helpers []windows.Handle
for range 2 {
connection, err := listener.Accept()
require.NoError(t, err)
t.Cleanup(func() { _ = connection.Close() })
require.NoError(t, connection.SetReadDeadline(time.Now().Add(20*time.Second)))
var pid int
_, err = fmt.Fscanln(connection, &pid)
require.NoError(t, err)
handle, err := windows.OpenProcess(windows.SYNCHRONIZE|windows.PROCESS_TERMINATE, false, uint32(pid))
require.NoError(t, err)
t.Cleanup(func() {
_ = windows.TerminateProcess(handle, 1)
_, _ = windows.WaitForSingleObject(handle, 5000)
_ = windows.CloseHandle(handle)
})
helpers = append(helpers, handle)
}
select {
case err := <-result:
require.NoError(t, err)
case <-ctx.Done():
t.Fatal(ctx.Err())
}
require.NoError(t, environment.Remove()(ctx))
for _, handle := range helpers {
if tracking == "default" {
result, err := windows.WaitForSingleObject(handle, 5000)
require.NoError(t, err)
require.EqualValues(t, windows.WAIT_OBJECT_0, result)
} else {
result, err := windows.WaitForSingleObject(handle, 0)
require.NoError(t, err)
require.EqualValues(t, windows.WAIT_TIMEOUT, result)
}
}
})
}
}
func TestHostTrackingHelper(t *testing.T) {
mode := os.Getenv("GITEA_TRACKING_HELPER")
if mode == "" {
return
}
if mode != "descendant" {
child := exec.Command(os.Args[0], "-test.run=^TestHostTrackingHelper$")
child.Env = append(os.Environ(), "GITEA_TRACKING_HELPER="+map[string]string{"step": "helper", "helper": "descendant"}[mode])
require.NoError(t, child.Start())
if mode == "step" {
os.Exit(0)
}
}
connection, err := net.Dial("tcp4", os.Getenv("GITEA_TRACKING_ADDRESS"))
require.NoError(t, err)
_, err = fmt.Fprintln(connection, os.Getpid())
require.NoError(t, err)
_, err = io.Copy(io.Discard, connection)
require.NoError(t, err)
require.NoError(t, connection.Close())
os.Exit(0)
}
+97 -12
View File
@@ -7,12 +7,15 @@ import (
"bufio" "bufio"
"bytes" "bytes"
"cmp" "cmp"
"compress/gzip"
"context" "context"
"crypto/rand" "crypto/rand"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"maps" "maps"
"net"
"net/http" "net/http"
"net/url" "net/url"
"os" "os"
@@ -23,6 +26,7 @@ import (
"time" "time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"github.com/kballard/go-shellquote" "github.com/kballard/go-shellquote"
"github.com/spf13/pflag" "github.com/spf13/pflag"
@@ -36,6 +40,7 @@ type KubernetesOptions struct {
MaxLifetime time.Duration MaxLifetime time.Duration
ForcePull bool ForcePull bool
RunnerUUID string RunnerUUID string
Transfer *KubernetesTransfer
} }
// KubernetesPod runs a job container and its service containers as one pod. // KubernetesPod runs a job container and its service containers as one pod.
@@ -46,6 +51,10 @@ type KubernetesPod struct {
job *kubernetesContainer job *kubernetesContainer
services []*kubernetesContainer services []*kubernetesContainer
arch, path string arch, path string
ips []net.IP
transfer *transferServer
transferImage string
transferHelper string // empty when copies go through the API server
} }
type kubernetesContainer struct { type kubernetesContainer struct {
@@ -55,6 +64,7 @@ type kubernetesContainer struct {
name string name string
input *NewContainerInput input *NewContainerInput
probe map[string]any probe map[string]any
gzip bool // the container can gunzip, so copies into it go compressed
} }
type containerStatus struct { type containerStatus struct {
@@ -74,6 +84,7 @@ type podStatus struct {
Spec struct{ Containers []struct{} } Spec struct{ Containers []struct{} }
Status struct { Status struct {
Phase, Reason, Message string Phase, Reason, Message string
PodIPs []struct{ IP string }
Conditions []struct{ Type, Status, Message string } Conditions []struct{ Type, Status, Message string }
InitContainerStatuses, ContainerStatuses []containerStatus InitContainerStatuses, ContainerStatuses []containerStatus
} }
@@ -129,14 +140,24 @@ func (p *KubernetesPod) create(ctx context.Context) error {
labels[runnerUUIDLabel] = p.options.RunnerUUID labels[runnerUUIDLabel] = p.options.RunnerUUID
} }
metadata := map[string]any{"name": name, "labels": labels} metadata := map[string]any{"name": name, "labels": labels}
var runnerImage string
if hostname, err := os.Hostname(); p.client.inCluster && err == nil { if hostname, err := os.Hostname(); p.client.inCluster && err == nil {
var runner struct{ Metadata struct{ Name, UID string } } var runner struct {
Metadata struct{ Name, UID string }
Spec struct{ Containers []struct{ Image string } }
}
if err := p.client.do(ctx, http.MethodGet, p.client.path("pods/"+hostname), nil, &runner); err == nil { if err := p.client.do(ctx, http.MethodGet, p.client.path("pods/"+hostname), nil, &runner); err == nil {
metadata["ownerReferences"] = []any{map[string]any{"apiVersion": "v1", "kind": "Pod", "name": runner.Metadata.Name, "uid": runner.Metadata.UID}} metadata["ownerReferences"] = []any{map[string]any{"apiVersion": "v1", "kind": "Pod", "name": runner.Metadata.Name, "uid": runner.Metadata.UID}}
if len(runner.Spec.Containers) == 1 {
runnerImage = runner.Spec.Containers[0].Image
}
} else { } else {
common.Logger(ctx).Debugf("The runner is not pod %s, job pods get no owner: %v", hostname, err) common.Logger(ctx).Debugf("The runner is not pod %s, job pods get no owner: %v", hostname, err)
} }
} }
if err := p.setupTransfer(runnerImage); err != nil {
return err
}
secretData := map[string]string{} secretData := map[string]string{}
manifest, err := p.manifest(name, secretData, metadata) manifest, err := p.manifest(name, secretData, metadata)
if err != nil { if err != nil {
@@ -205,6 +226,18 @@ func (p *KubernetesPod) manifest(secret string, secretData map[string]string, me
} }
} }
spec := map[string]any{"restartPolicy": "Never", "containers": containers, "volumes": volumes} spec := map[string]any{"restartPolicy": "Never", "containers": containers, "volumes": volumes}
if p.transferHelper != "" {
transfer := map[string]any{
"name": "gitea-runner-transfer", "image": p.transferImage, "terminationMessagePolicy": "FallbackToLogsOnError",
"command": []string{"gitea-runner", KubernetesTransferCommand, "install", p.transferHelper},
"volumeMounts": []any{map[string]any{"name": "act", "mountPath": p.job.GetActPath()}},
"securityContext": map[string]any{"allowPrivilegeEscalation": false, "capabilities": map[string]any{"drop": []string{"ALL"}}},
}
if p.options.ForcePull {
transfer["imagePullPolicy"] = "Always"
}
spec["initContainers"] = []any{transfer}
}
if len(hostnames) > 0 { if len(hostnames) > 0 {
spec["hostAliases"] = []any{map[string]any{"ip": "127.0.0.1", "hostnames": hostnames}} spec["hostAliases"] = []any{map[string]any{"ip": "127.0.0.1", "hostnames": hostnames}}
} }
@@ -341,7 +374,13 @@ func (p *KubernetesPod) waitStarted(ctx context.Context) error {
return err return err
} }
if pod.Status.Phase == "Failed" { if pod.Status.Phase == "Failed" {
return fmt.Errorf("pod %s failed: %s", p.name, cmp.Or(pod.Status.Message, pod.Status.Reason)) reason := cmp.Or(pod.Status.Message, pod.Status.Reason)
for _, status := range append(pod.Status.InitContainerStatuses, pod.Status.ContainerStatuses...) {
if terminated := status.State.Terminated; reason == "" && terminated != nil && terminated.ExitCode != 0 {
reason = fmt.Sprintf("container %s exited with %d: %s", status.Name, terminated.ExitCode, strings.TrimSpace(terminated.Message))
}
}
return fmt.Errorf("pod %s failed: %s", p.name, reason)
} }
started := len(pod.Status.ContainerStatuses) == len(pod.Spec.Containers) started := len(pod.Status.ContainerStatuses) == len(pod.Spec.Containers)
for _, status := range append(pod.Status.InitContainerStatuses, pod.Status.ContainerStatuses...) { for _, status := range append(pod.Status.InitContainerStatuses, pod.Status.ContainerStatuses...) {
@@ -354,6 +393,9 @@ func (p *KubernetesPod) waitStarted(ctx context.Context) error {
} }
} }
if started { if started {
for _, ip := range pod.Status.PodIPs {
p.ips = append(p.ips, net.ParseIP(ip.IP))
}
return nil return nil
} }
for _, condition := range pod.Status.Conditions { for _, condition := range pod.Status.Conditions {
@@ -391,8 +433,11 @@ func (c *kubernetesContainer) Start(_ bool) common.Executor {
if err != nil { if err != nil {
return err return err
} }
machine, path, _ := strings.Cut(output, "\n") probed := strings.SplitN(output, "\n", 3)
c.pod.arch, c.pod.path = goArchToActionArch(machine), path for len(probed) < 3 {
probed = append(probed, "")
}
c.pod.arch, c.gzip, c.pod.path = goArchToActionArch(probed[0]), probed[1] == "gzip", probed[2]
return nil return nil
} }
} }
@@ -403,7 +448,7 @@ func (p *KubernetesPod) probe(ctx context.Context, container string) (string, er
deadline := time.Now().Add(time.Minute) deadline := time.Now().Add(time.Minute)
for delay := 100 * time.Millisecond; ; delay = min(2*delay, 2*time.Second) { for delay := 100 * time.Millisecond; ; delay = min(2*delay, 2*time.Second) {
var stdout bytes.Buffer var stdout bytes.Buffer
err := p.client.exec(ctx, p.name, container, []string{"sh", "-c", `printf '%s\n%s' "$(uname -m)" "$PATH"`}, nil, &stdout, io.Discard) err := p.client.exec(ctx, p.name, container, []string{"sh", "-c", `printf '%s\n%s\n%s' "$(uname -m)" "$(command -v gzip >/dev/null && echo gzip)" "$PATH"`}, nil, &stdout, io.Discard)
var exitErr ExitCodeError var exitErr ExitCodeError
var apiErr *apiError var apiErr *apiError
if err == nil || ctx.Err() != nil || errors.As(err, &exitErr) || time.Now().After(deadline) || if err == nil || ctx.Err() != nil || errors.As(err, &exitErr) || time.Now().After(deadline) ||
@@ -452,27 +497,67 @@ func (c *kubernetesContainer) Exec(command []string, env map[string]string, _, w
} }
} }
func (c *kubernetesContainer) extract(ctx context.Context, destPath string, write func(io.Writer) error) error { // compressCopyMin is the size from which Copy compresses and uses the transfer helper: below it that saves next to nothing on the wire and costs a process in the pod.
const compressCopyMin = 64 << 10
// extract streams the tar write produces into destPath, gzipped with compress, as copied actions shrink about fivefold and API server traffic often crosses zones.
func (c *kubernetesContainer) extract(ctx context.Context, destPath string, compress bool, write func(io.Writer) error) error {
if compress && c.pod.transferHelper != "" {
request, done := c.pod.transfer.offer(c.pod.ips, write)
stdin, _ := json.Marshal(request)
var stderr bytes.Buffer
err := c.pod.client.exec(ctx, c.pod.name, c.name, []string{c.pod.transferHelper, KubernetesTransferCommand, "fetch", destPath}, bytes.NewReader(stdin), io.Discard, &stderr)
taken, writeErr := done()
if err == nil || ctx.Err() != nil {
return err
}
if taken || !c.pod.options.Transfer.Fallback {
return fmt.Errorf("copy to %s over the pod network: %w: %s", destPath, errors.Join(writeErr, err), strings.TrimSpace(stderr.String()))
}
c.pod.transferHelper = "" // a pod that cannot reach the runner once will not later
logging.Warnf("Copying into pod %s over the pod network failed, copying through the API server instead: %v: %s", c.pod.name, err, strings.TrimSpace(stderr.String()))
}
compress = compress && c.gzip
reader, writer := io.Pipe() reader, writer := io.Pipe()
defer reader.Close() defer reader.Close()
go func() { writer.CloseWithError(write(writer)) }() go func() {
if compress {
writer.CloseWithError(writeGzip(writer, write))
} else {
writer.CloseWithError(write(writer))
}
}()
script := `mkdir -p "$1" && tar -xf - -C "$1"`
if compress {
script = `mkdir -p "$1" && gzip -dc | tar -xf - -C "$1"` // gzip -dc rather than tar -z, which GNU tar runs gzip for anyway
}
var stderr bytes.Buffer var stderr bytes.Buffer
if err := c.pod.client.exec(ctx, c.pod.name, c.name, []string{"sh", "-c", `mkdir -p "$1" && tar -xf - -C "$1"`, "sh", destPath}, reader, io.Discard, &stderr); err != nil { if err := c.pod.client.exec(ctx, c.pod.name, c.name, []string{"sh", "-c", script, "sh", destPath}, reader, io.Discard, &stderr); err != nil {
return fmt.Errorf("extract to %s: %w: %s", destPath, err, strings.TrimSpace(stderr.String())) return fmt.Errorf("extract to %s: %w: %s", destPath, err, strings.TrimSpace(stderr.String()))
} }
return nil return nil
} }
// writeGzip gzips what write produces into w, at a level fast enough to not slow copies down and nearly as small as the default.
func writeGzip(w io.Writer, write func(io.Writer) error) error {
compressed, _ := gzip.NewWriterLevel(w, gzip.BestSpeed)
return errors.Join(write(compressed), compressed.Close())
}
func (c *kubernetesContainer) Copy(destPath string, files ...*FileEntry) common.Executor { func (c *kubernetesContainer) Copy(destPath string, files ...*FileEntry) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
return c.extract(ctx, destPath, func(writer io.Writer) error { return writeFilesTar(ctx, writer, 0, 0, files...) }) size := 0
for _, file := range files {
size += len(file.Body)
}
return c.extract(ctx, destPath, size >= compressCopyMin, func(writer io.Writer) error { return writeFilesTar(ctx, writer, 0, 0, files...) })
} }
} }
func (c *kubernetesContainer) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor { func (c *kubernetesContainer) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
return c.extract(ctx, destPath, func(writer io.Writer) error { return c.extract(ctx, destPath, true, func(writer io.Writer) error { // directories are actions and workspaces, megabytes of compressible sources
return writeDirTar(ctx, writer, "/", srcPath, useGitIgnore, skipGitDir, 0, 0) return writeDirTar(ctx, writer, "/", srcPath, useGitIgnore, skipGitDir, 0, 0, hostOS == "windows")
}) })
} }
} }
@@ -500,7 +585,7 @@ func (c *kubernetesContainer) GetContainerArchive(ctx context.Context, srcPath s
} }
func (c *kubernetesContainer) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor { func (c *kubernetesContainer) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(c, srcPath, env) return parseEnvFile(c, srcPath, env, false)
} }
func (c *kubernetesContainer) UpdateFromImageEnv(env *map[string]string) common.Executor { func (c *kubernetesContainer) UpdateFromImageEnv(env *map[string]string) common.Executor {
+60 -10
View File
@@ -89,7 +89,7 @@ func execFailure(t *testing.T, code int, status string) http.HandlerFunc {
} }
} }
func fakeKubernetes(t *testing.T, podStatus string) (string, *fakeCluster) { func fakeKubernetes(t *testing.T, podStatus string, env ...string) (string, *fakeCluster) {
cluster := &fakeCluster{requests: map[string][]string{}} cluster := &fakeCluster{requests: map[string][]string{}}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer test-token" { if r.Header.Get("Authorization") != "Bearer test-token" {
@@ -109,7 +109,7 @@ func fakeKubernetes(t *testing.T, podStatus string) (string, *fakeCluster) {
if failure := cluster.nextExecFailure(); failure != nil { if failure := cluster.nextExecFailure(); failure != nil {
failure(w, r) failure(w, r)
} else { } else {
serveLocalExec(t, w, r) serveLocalExec(t, w, r, env)
} }
case strings.HasSuffix(r.URL.Path, "/log"): case strings.HasSuffix(r.URL.Path, "/log"):
_, _ = io.WriteString(w, "log of "+r.URL.Query().Get("container")) _, _ = io.WriteString(w, "log of "+r.URL.Query().Get("container"))
@@ -132,7 +132,7 @@ users: [{name: test, user: {tokenFile: token}}]
return kubeconfig, cluster return kubeconfig, cluster
} }
func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) { func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request, env []string) {
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{Subprotocols: []string{execProtocol}}) conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{Subprotocols: []string{execProtocol}})
if err != nil { if err != nil {
t.Error(err) t.Error(err)
@@ -154,7 +154,7 @@ func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) {
stdin.Write(frame[1:]) stdin.Write(frame[1:])
} }
cmd := exec.Command(command[0], command[1:]...) cmd := exec.Command(command[0], command[1:]...)
cmd.Env = append(os.Environ(), "COPYFILE_DISABLE=1") cmd.Env = append(append(os.Environ(), "COPYFILE_DISABLE=1"), env...)
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
cmd.Stdin, cmd.Stdout, cmd.Stderr = &stdin, &stdout, &stderr cmd.Stdin, cmd.Stdout, cmd.Stderr = &stdin, &stdout, &stderr
status := `{"status":"Success"}` status := `{"status":"Success"}`
@@ -183,15 +183,17 @@ func TestKubernetesServiceHostnames(t *testing.T) {
} }
func TestKubernetesPodRunsJobThroughAPI(t *testing.T) { func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" { if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell") t.Skip("uses POSIX shell")
} }
ctx := t.Context() ctx := t.Context()
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{},{}]},"status":{"phase":"Running","containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}},
{"name":"my-db","ready":true,"state":{"running":{}}}]}}`)
workdir, err := filepath.EvalSymlinks(t.TempDir()) workdir, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err) require.NoError(t, err)
path := workdir + ":" + os.Getenv("PATH")
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{},{}]},"status":{"phase":"Running","containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}},
{"name":"my-db","ready":true,"state":{"running":{}}}]}}`, "PATH="+path)
options := KubernetesOptions{Kubeconfig: kubeconfig, RunnerUUID: "runner-uuid", PodTemplates: []map[string]any{ options := KubernetesOptions{Kubeconfig: kubeconfig, RunnerUUID: "runner-uuid", PodTemplates: []map[string]any{
{"spec": map[string]any{ {"spec": map[string]any{
"automountServiceAccountToken": true, "automountServiceAccountToken": true,
@@ -252,13 +254,13 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/pods"][0], `"args":["postgres","-D","$$(PGDATA)"]`) assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/pods"][0], `"args":["postgres","-D","$$(PGDATA)"]`)
require.NoError(t, os.WriteFile(filepath.Join(workdir, "env"), []byte("#!/bin/sh\necho \"$@\" > \"$0-args\"\nexec /usr/bin/env \"$@\"\n"), 0o755)) require.NoError(t, os.WriteFile(filepath.Join(workdir, "env"), []byte("#!/bin/sh\necho \"$@\" > \"$0-args\"\nexec /usr/bin/env \"$@\"\n"), 0o755))
t.Setenv("PATH", workdir+":"+os.Getenv("PATH"))
cluster.failExec( cluster.failExec(
execFailure(t, http.StatusInternalServerError, `{"message":"error dialing backend: remote error: tls: internal error"}`), execFailure(t, http.StatusInternalServerError, `{"message":"error dialing backend: remote error: tls: internal error"}`),
execFailure(t, 0, `{"status":"Failure","reason":"InternalError","message":"Internal error occurred: error sending request: remote error: tls: internal error"}`), execFailure(t, 0, `{"status":"Failure","reason":"InternalError","message":"Internal error occurred: error sending request: remote error: tls: internal error"}`),
) )
require.NoError(t, job.Start(false)(ctx)) require.NoError(t, job.Start(false)(ctx))
assert.Len(t, cluster.requests["exec"], 3) assert.Len(t, cluster.requests["exec"], 3)
assert.True(t, pod.job.gzip)
stepEnv := map[string]string{"INPUT_WHO-TO-GREET": "it's\nme", "SECRET": "hunter2", "UID": "5"} stepEnv := map[string]string{"INPUT_WHO-TO-GREET": "it's\nme", "SECRET": "hunter2", "UID": "5"}
require.NoError(t, job.Exec([]string{"printenv", "SECRET"}, stepEnv, "", "")(ctx)) require.NoError(t, job.Exec([]string{"printenv", "SECRET"}, stepEnv, "", "")(ctx))
@@ -275,7 +277,9 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
assert.NotContains(t, query, "hunter2") assert.NotContains(t, query, "hunter2")
} }
lastExec := func() string { return cluster.requests["exec"][len(cluster.requests["exec"])-1] }
require.NoError(t, job.Copy(workdir+"/act/", &FileEntry{Name: "workflow/event.json", Mode: 0o644, Body: `{"ref":"main"}`})(ctx)) require.NoError(t, job.Copy(workdir+"/act/", &FileEntry{Name: "workflow/event.json", Mode: 0o644, Body: `{"ref":"main"}`})(ctx))
assert.NotContains(t, lastExec(), "gzip")
archive, err := job.GetContainerArchive(ctx, workdir+"/act/workflow/event.json") archive, err := job.GetContainerArchive(ctx, workdir+"/act/workflow/event.json")
require.NoError(t, err) require.NoError(t, err)
reader := tar.NewReader(archive) reader := tar.NewReader(archive)
@@ -288,6 +292,12 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
require.NoError(t, archive.Close()) require.NoError(t, archive.Close())
_, err = job.GetContainerArchive(ctx, workdir+"/missing") _, err = job.GetContainerArchive(ctx, workdir+"/missing")
require.Error(t, err) require.Error(t, err)
large := strings.Repeat("a", compressCopyMin)
require.NoError(t, job.Copy(workdir+"/act/", &FileEntry{Name: "workflow/large", Mode: 0o644, Body: large})(ctx))
assert.Contains(t, lastExec(), url.QueryEscape("gzip -dc |"))
content, err = os.ReadFile(workdir + "/act/workflow/large")
require.NoError(t, err)
assert.Equal(t, large, string(content))
actionDir := filepath.Join(t.TempDir(), "action") actionDir := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.MkdirAll(actionDir, 0o755)) require.NoError(t, os.MkdirAll(actionDir, 0o755))
@@ -296,11 +306,12 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
script, err := os.ReadFile(workdir + "/act/actions/sha/index.js") script, err := os.ReadFile(workdir + "/act/actions/sha/index.js")
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "main()", string(script)) assert.Equal(t, "main()", string(script))
assert.Contains(t, cluster.requests["exec"][len(cluster.requests["exec"])-1], "command="+url.QueryEscape(workdir+"/act/actions/sha/")) assert.Contains(t, lastExec(), "command="+url.QueryEscape(workdir+"/act/actions/sha/"))
assert.Contains(t, lastExec(), url.QueryEscape("gzip -dc |"))
pathEnv := map[string]string{} pathEnv := map[string]string{}
require.NoError(t, job.UpdateFromImageEnv(&pathEnv)(ctx)) require.NoError(t, job.UpdateFromImageEnv(&pathEnv)(ctx))
assert.Equal(t, os.Getenv("PATH"), pathEnv["PATH"]) assert.Equal(t, path, pathEnv["PATH"])
info, err := service.Inspect(ctx) info, err := service.Inspect(ctx)
require.NoError(t, err) require.NoError(t, err)
@@ -320,6 +331,38 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
assert.Empty(t, cluster.requests["DELETE /api/v1/namespaces/ci/secrets/new"]) assert.Empty(t, cluster.requests["DELETE /api/v1/namespaces/ci/secrets/new"])
} }
func TestKubernetesCopiesUncompressedWithoutGzip(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
ctx := t.Context()
workdir, bin := t.TempDir(), t.TempDir()
for _, tool := range []string{"uname", "tar", "mkdir"} { // all the job container needs but gzip, as on a minimal image
found, err := exec.LookPath(tool)
require.NoError(t, err)
require.NoError(t, os.Symlink(found, filepath.Join(bin, tool)))
}
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{}]},"status":{"phase":"Running","containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}}]}}`, "PATH="+bin)
pod, err := NewKubernetesPod(KubernetesOptions{Kubeconfig: kubeconfig})
require.NoError(t, err)
job := pod.JobContainer(&NewContainerInput{Image: "node", WorkingDir: workdir})
require.NoError(t, job.Create(nil, nil)(ctx))
require.NoError(t, job.Start(false)(ctx))
assert.False(t, pod.job.gzip)
assert.Equal(t, bin, pod.path)
actionDir := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.MkdirAll(actionDir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(actionDir, "index.js"), []byte("main()"), 0o644))
require.NoError(t, job.CopyDir(workdir+"/act/actions/sha/", actionDir+"/", false, false)(ctx))
script, err := os.ReadFile(workdir + "/act/actions/sha/index.js")
require.NoError(t, err)
assert.Equal(t, "main()", string(script))
assert.NotContains(t, cluster.requests["exec"][len(cluster.requests["exec"])-1], "gzip")
}
func TestKubernetesPodFailsBeforeSteps(t *testing.T) { func TestKubernetesPodFailsBeforeSteps(t *testing.T) {
kubeconfig, _ := fakeKubernetes(t, `{"spec":{"containers":[{}]},"status":{"containerStatuses":[ kubeconfig, _ := fakeKubernetes(t, `{"spec":{"containers":[{}]},"status":{"containerStatuses":[
{"name":"job","state":{"waiting":{"reason":"ErrImagePull","message":"no such image"}}}]}}`) {"name":"job","state":{"waiting":{"reason":"ErrImagePull","message":"no such image"}}}]}}`)
@@ -360,6 +403,13 @@ func TestKubernetesPodFailsBeforeSteps(t *testing.T) {
job := pod.JobContainer(&NewContainerInput{Image: "missing"}) job := pod.JobContainer(&NewContainerInput{Image: "missing"})
require.NoError(t, job.Create(nil, nil)(t.Context())) require.NoError(t, job.Create(nil, nil)(t.Context()))
require.ErrorContains(t, job.Start(false)(t.Context()), "ErrImagePull: no such image") require.ErrorContains(t, job.Start(false)(t.Context()), "ErrImagePull: no such image")
failed, _ := fakeKubernetes(t, `{"spec":{"containers":[{}]},"status":{"phase":"Failed","initContainerStatuses":[
{"name":"gitea-runner-transfer","state":{"terminated":{"exitCode":1,"message":"exec format error\n"}}}]}}`)
pod, err = NewKubernetesPod(KubernetesOptions{Kubeconfig: failed})
require.NoError(t, err)
job = pod.JobContainer(&NewContainerInput{Image: "node"})
require.NoError(t, job.Create(nil, nil)(t.Context()))
require.ErrorContains(t, job.Start(false)(t.Context()), "container gitea-runner-transfer exited with 1: exec format error")
plugin := filepath.Join(t.TempDir(), "kubeconfig") plugin := filepath.Join(t.TempDir(), "kubeconfig")
require.NoError(t, os.WriteFile(plugin, []byte(` require.NoError(t, os.WriteFile(plugin, []byte(`
+248
View File
@@ -0,0 +1,248 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"cmp"
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"math/big"
"net"
"net/http"
"os"
"os/exec"
"path"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
)
// KubernetesTransfer has job pods fetch large copies from the runner over the pod network instead of through the API server.
type KubernetesTransfer struct {
Host string // empty uses the outbound IP
Port uint16 // 0 picks a free port
Image string // has gitea-runner on its PATH, for an init container to copy into each job pod
Fallback bool // copy through the API server when a job pod cannot reach the runner, else fail
}
// KubernetesTransferCommand is the hidden runner subcommand job pods run the transfer helper with.
const KubernetesTransferCommand = "kubernetes-transfer"
// transferRequest goes to the helper on the exec stream's stdin, keeping the token out of the command line and the audit log.
type transferRequest struct {
URL string `json:"url"`
Pin string `json:"pin"` // hex SHA-256 of the runner's certificate
}
type transferServer struct {
base string
pin string
mu sync.Mutex
pending map[string]transferOffer
}
type transferOffer struct {
clients []net.IP
write func(io.Writer) error
result chan error
}
var (
transferServersMu sync.Mutex
transferServers = map[string]*transferServer{}
)
// startTransferServer returns the runner's server for these settings, started on first use.
func startTransferServer(host string, port uint16) (*transferServer, error) {
transferServersMu.Lock()
defer transferServersMu.Unlock()
configured := net.JoinHostPort(host, strconv.Itoa(int(port)))
if server, ok := transferServers[configured]; ok {
return server, nil
}
if host == "" {
ip := common.GetOutboundIP()
if ip == nil {
return nil, errors.New("no outbound IP for job pods to reach the runner at, set kubernetes.transfer.host")
}
host = ip.String()
}
privateKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
return nil, err
}
template := &x509.Certificate{SerialNumber: big.NewInt(1)} // trusted by its pin alone, so no CA, names or validity
certificate, err := x509.CreateCertificate(rand.Reader, template, template, &privateKey.PublicKey, privateKey)
if err != nil {
return nil, err
}
listener, err := net.Listen("tcp", net.JoinHostPort(host, strconv.Itoa(int(port))))
if err != nil {
return nil, fmt.Errorf("listen for job pod transfers: %w", err)
}
pin := sha256.Sum256(certificate)
server := &transferServer{base: "https://" + listener.Addr().String() + "/", pin: hex.EncodeToString(pin[:]), pending: map[string]transferOffer{}}
httpServer := &http.Server{
Handler: server,
TLSConfig: &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{certificate}, PrivateKey: privateKey}}, MinVersion: tls.VersionTLS13},
ReadHeaderTimeout: 10 * time.Second,
IdleTimeout: 30 * time.Second,
}
go func() { _ = httpServer.ServeTLS(listener, "", "") }()
transferServers[configured] = server
return server, nil
}
// offer serves write once to one of clients until done, which reports whether a client took it and what write failed with.
func (s *transferServer) offer(clients []net.IP, write func(io.Writer) error) (request transferRequest, done func() (taken bool, err error)) {
token := rand.Text()
result := make(chan error, 1)
s.mu.Lock()
s.pending[token] = transferOffer{clients: clients, write: write, result: result}
s.mu.Unlock()
return transferRequest{URL: s.base + token, Pin: s.pin}, func() (bool, error) {
s.mu.Lock()
_, pending := s.pending[token]
delete(s.pending, token)
s.mu.Unlock()
select {
case err := <-result:
return true, err
default:
return !pending, nil
}
}
}
func (s *transferServer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
host, _, _ := net.SplitHostPort(r.RemoteAddr)
token := strings.TrimPrefix(r.URL.Path, "/")
s.mu.Lock()
offer, ok := s.pending[token]
ok = ok && r.Method == http.MethodGet && slices.ContainsFunc(offer.clients, net.ParseIP(host).Equal)
if ok {
delete(s.pending, token)
}
s.mu.Unlock()
if !ok {
http.NotFound(w, r)
return
}
w.Header().Set("Content-Encoding", "gzip")
err := writeGzip(w, offer.write)
offer.result <- err
if err != nil {
panic(http.ErrAbortHandler) // a cut-off body fails the fetch, a short tar might not
}
}
// FetchKubernetesTransfer reads a transferRequest from request and extracts the tar it points to into dest.
func FetchKubernetesTransfer(ctx context.Context, request io.Reader, dest string) error {
var transfer transferRequest
if err := json.NewDecoder(request).Decode(&transfer); err != nil {
return fmt.Errorf("read transfer request: %w", err)
}
client := &http.Client{Transport: &http.Transport{
DialContext: (&net.Dialer{Timeout: 10 * time.Second}).DialContext,
TLSHandshakeTimeout: 10 * time.Second,
ResponseHeaderTimeout: time.Minute,
TLSClientConfig: &tls.Config{
MinVersion: tls.VersionTLS13,
InsecureSkipVerify: true, // VerifyConnection checks the pin instead
VerifyConnection: func(state tls.ConnectionState) error {
if sum := sha256.Sum256(state.PeerCertificates[0].Raw); hex.EncodeToString(sum[:]) != transfer.Pin {
return errors.New("runner certificate does not match the pin")
}
return nil
},
},
}}
httpRequest, err := http.NewRequestWithContext(ctx, http.MethodGet, transfer.URL, nil)
if err != nil {
return err
}
response, err := client.Do(httpRequest)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return fmt.Errorf("fetch: %s", response.Status)
}
if err := os.MkdirAll(dest, 0o777); err != nil {
return err
}
cmd := exec.CommandContext(ctx, "tar", "-xf", "-", "-C", dest)
cmd.Stdin, cmd.Stdout, cmd.Stderr = response.Body, os.Stderr, os.Stderr
if err := cmd.Run(); err != nil {
return err
}
_, err = io.Copy(io.Discard, response.Body) // tar stops at the first zero blocks, a cut-off body after them still fails
return err
}
// InstallKubernetesTransferHelper copies the running binary to target.
func InstallKubernetesTransferHelper(target string) error {
executable, err := os.Executable()
if err != nil {
return err
}
source, err := os.Open(executable)
if err != nil {
return err
}
defer source.Close()
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
return err
}
_ = os.Remove(target) // a file left in a reused act volume would keep its mode
file, err := os.OpenFile(target, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o755)
if err != nil {
return err
}
_, err = io.Copy(file, source)
return errors.Join(err, file.Close())
}
// setupTransfer turns on transfers for the pod, with the runner pod's image unless one is set.
func (p *KubernetesPod) setupTransfer(runnerImage string) error {
settings := p.options.Transfer
if settings == nil {
return nil
}
image := cmp.Or(settings.Image, runnerImage)
var err error
var server *transferServer
if image == "" {
err = errors.New("set kubernetes.transfer.image, the runner is no single-container pod in kubernetes.namespace to take it from")
} else {
server, err = startTransferServer(settings.Host, settings.Port)
}
if err != nil {
if !settings.Fallback {
return fmt.Errorf("kubernetes.transfer: %w", err)
}
logging.Warnf("Job pod copies go through the API server: %v", err)
return nil
}
p.transfer, p.transferImage = server, image
p.transferHelper = path.Join(p.job.GetActPath(), "bin", "gitea-runner")
return nil
}
+185
View File
@@ -0,0 +1,185 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"bytes"
"context"
"crypto/rand"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const transferHelperEnv = "GITEA_RUNNER_TEST_TRANSFER_HELPER"
func TestMain(m *testing.M) {
if os.Getenv(transferHelperEnv) == "1" && len(os.Args) == 4 && os.Args[1] == KubernetesTransferCommand && os.Args[2] == "fetch" {
if err := FetchKubernetesTransfer(context.Background(), os.Stdin, os.Args[3]); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
os.Exit(0)
}
os.Exit(m.Run())
}
func fetch(t *testing.T, request transferRequest) error {
t.Helper()
body, err := json.Marshal(request)
require.NoError(t, err)
return FetchKubernetesTransfer(t.Context(), bytes.NewReader(body), t.TempDir())
}
func TestTransferServer(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("extracts with tar")
}
server, err := startTransferServer("127.0.0.1", 0)
require.NoError(t, err)
again, err := startTransferServer("127.0.0.1", 0)
require.NoError(t, err)
assert.Same(t, server, again)
loopback := []net.IP{net.ParseIP("127.0.0.1")}
nothing := func(io.Writer) error { return nil }
t.Run("serves an offer gzipped once to an IP of its job pod", func(t *testing.T) {
request, done := server.offer([]net.IP{net.ParseIP("fd00::1"), loopback[0]}, nothing)
client := &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, DisableCompression: true}}
response, err := client.Get(request.URL)
require.NoError(t, err)
require.NoError(t, response.Body.Close())
assert.Equal(t, "gzip", response.Header.Get("Content-Encoding"))
require.ErrorContains(t, fetch(t, request), "404")
taken, _ := done()
assert.True(t, taken)
})
t.Run("another pod can neither take nor block an offer", func(t *testing.T) {
request, done := server.offer([]net.IP{net.ParseIP("10.1.2.3")}, nothing)
require.ErrorContains(t, fetch(t, request), "404")
taken, _ := done()
assert.False(t, taken)
})
t.Run("a failed write fails the fetch even past the end blocks tar stops at, and done reports it", func(t *testing.T) {
request, done := server.offer(loopback, func(w io.Writer) error {
_, _ = w.Write(make([]byte, 1024))
_, _ = io.CopyN(w, rand.Reader, 1<<20)
return errors.New("source gone")
})
require.Error(t, fetch(t, request))
taken, err := done()
assert.True(t, taken)
require.EqualError(t, err, "source gone")
})
t.Run("the helper refuses a runner whose certificate does not match its pin", func(t *testing.T) {
request, done := server.offer(loopback, nothing)
request.Pin = strings.Repeat("0", 64)
require.ErrorContains(t, fetch(t, request), "does not match the pin")
taken, _ := done()
assert.False(t, taken)
})
}
func TestKubernetesPodTransferImage(t *testing.T) {
newPod := func(image string, fallback bool) *KubernetesPod {
pod := &KubernetesPod{options: KubernetesOptions{Transfer: &KubernetesTransfer{Host: "127.0.0.1", Image: image, Fallback: fallback}}}
pod.JobContainer(&NewContainerInput{})
return pod
}
pod := newPod("", false)
require.NoError(t, pod.setupTransfer("gitea/runner:latest"))
assert.Equal(t, "gitea/runner:latest", pod.transferImage)
pod = newPod("registry.example/runner:1", false)
require.NoError(t, pod.setupTransfer("gitea/runner:latest"))
assert.Equal(t, "registry.example/runner:1", pod.transferImage)
pod = newPod("", true)
require.NoError(t, pod.setupTransfer(""))
assert.Empty(t, pod.transferHelper)
pod = newPod("", false)
require.ErrorContains(t, pod.setupTransfer(""), "kubernetes.transfer.image")
}
func TestKubernetesPodCopiesLargeFilesOverPodNetwork(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
ctx := t.Context()
workdir, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err)
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{}]},"status":{"phase":"Running","podIPs":[{"ip":"127.0.0.1"}],"containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}}]}}`, transferHelperEnv+"=1")
pod, err := NewKubernetesPod(KubernetesOptions{Kubeconfig: kubeconfig, ForcePull: true, Transfer: &KubernetesTransfer{Host: "127.0.0.1", Image: "gitea/runner:latest", Fallback: true}})
require.NoError(t, err)
job := pod.JobContainer(&NewContainerInput{Name: "job", Image: "node", WorkingDir: workdir, Stdout: io.Discard, Stderr: io.Discard})
require.NoError(t, job.Create(nil, nil)(ctx))
var manifest struct {
Spec struct {
InitContainers []struct {
Image, ImagePullPolicy string
Command []string
VolumeMounts []struct{ Name, MountPath string }
}
}
}
require.NoError(t, json.Unmarshal([]byte(cluster.requests["POST /api/v1/namespaces/ci/pods"][0]), &manifest))
require.Len(t, manifest.Spec.InitContainers, 1)
initContainer := manifest.Spec.InitContainers[0]
assert.Equal(t, "gitea/runner:latest", initContainer.Image)
assert.Equal(t, "Always", initContainer.ImagePullPolicy)
assert.Equal(t, []string{"gitea-runner", KubernetesTransferCommand, "install", "/var/run/act/bin/gitea-runner"}, initContainer.Command)
assert.Equal(t, []struct{ Name, MountPath string }{{"act", "/var/run/act"}}, initContainer.VolumeMounts)
require.NoError(t, job.Start(false)(ctx))
helper := filepath.Join(t.TempDir(), "bin", "gitea-runner")
require.NoError(t, InstallKubernetesTransferHelper(helper))
pod.transferHelper = helper
lastExec := func() string { return cluster.requests["exec"][len(cluster.requests["exec"])-1] }
large := &FileEntry{Name: "large", Mode: 0o644, Body: strings.Repeat("a", compressCopyMin)}
copyLarge := func(dir string) error { return job.Copy(filepath.Join(workdir, dir)+"/", large)(ctx) }
require.NoError(t, copyLarge("transferred"))
content, err := os.ReadFile(filepath.Join(workdir, "transferred", "large"))
require.NoError(t, err)
assert.Equal(t, large.Body, string(content))
assert.Contains(t, lastExec(), "command="+KubernetesTransferCommand)
assert.NotContains(t, lastExec(), "127.0.0.1")
require.NoError(t, job.Copy(workdir+"/act/", &FileEntry{Name: "event.json", Mode: 0o644, Body: `{}`})(ctx))
assert.NotContains(t, lastExec(), "command="+KubernetesTransferCommand)
require.NoError(t, os.WriteFile(filepath.Join(workdir, "file"), nil, 0o644))
require.ErrorContains(t, copyLarge("file"), "over the pod network")
assert.Equal(t, helper, pod.transferHelper)
unreachable := filepath.Join(workdir, "unreachable")
require.NoError(t, os.WriteFile(unreachable, []byte("#!/bin/sh\necho 'connection refused' >&2\nexit 1\n"), 0o755))
pod.transferHelper = unreachable
pod.options.Transfer.Fallback = false
require.ErrorContains(t, copyLarge("direct"), "connection refused")
assert.NoFileExists(t, filepath.Join(workdir, "direct", "large"))
pod.options.Transfer.Fallback = true
require.NoError(t, copyLarge("auto"))
assert.FileExists(t, filepath.Join(workdir, "auto", "large"))
assert.Empty(t, pod.transferHelper)
}
+37 -6
View File
@@ -7,6 +7,7 @@ package container
import ( import (
"archive/tar" "archive/tar"
"bufio" "bufio"
"bytes"
"context" "context"
"fmt" "fmt"
"io" "io"
@@ -18,8 +19,14 @@ import (
"golang.org/x/text/transform" "golang.org/x/text/transform"
) )
func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Executor { // parseEnvFile reads a GITHUB_ENV style file, splitting lines like GitHub's runner on Windows or elsewhere.
localEnv := *env func parseEnvFile(e Container, srcPath string, env *map[string]string, windows bool) common.Executor {
return ParseEnvFile(e, srcPath, func(name, value string) {
(*env)[name] = value
}, windows)
}
func ParseEnvFile(e Container, srcPath string, setEnv func(name, value string), windows bool) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
envTar, err := e.GetContainerArchive(ctx, srcPath) envTar, err := e.GetContainerArchive(ctx, srcPath)
if err != nil { if err != nil {
@@ -38,8 +45,22 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
s := bufio.NewScanner(decoded) s := bufio.NewScanner(decoded)
// Default 64 KiB max token size is too small for realistic env-file lines; allow up to 16 MiB. // Default 64 KiB max token size is too small for realistic env-file lines; allow up to 16 MiB.
s.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) s.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
scanned := 0
s.Split(func(data []byte, atEOF bool) (int, []byte, error) {
if index := bytes.IndexByte(data[scanned:], '\n'); index >= 0 {
advance := scanned + index + 1
scanned = 0
return advance, data[:advance], nil
}
scanned = len(data)
if atEOF && len(data) > 0 {
scanned = 0
return len(data), data, nil
}
return 0, nil, nil
})
for s.Scan() { for s.Scan() {
line := s.Text() line := trimEnvLineEnding(s.Text(), windows)
// GitHub's runner ignores blank lines // GitHub's runner ignores blank lines
if strings.TrimSpace(line) == "" { if strings.TrimSpace(line) == "" {
continue continue
@@ -48,14 +69,17 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
multiLineEnv := strings.Index(line, "<<") multiLineEnv := strings.Index(line, "<<")
switch { switch {
case singleLineEnv != -1 && (multiLineEnv == -1 || singleLineEnv < multiLineEnv): case singleLineEnv != -1 && (multiLineEnv == -1 || singleLineEnv < multiLineEnv):
localEnv[line[:singleLineEnv]] = line[singleLineEnv+1:] setEnv(line[:singleLineEnv], line[singleLineEnv+1:])
case multiLineEnv != -1: case multiLineEnv != -1:
var multiLineEnvContent []string var multiLineEnvContent []string
multiLineEnvDelimiter := line[multiLineEnv+2:] multiLineEnvDelimiter := line[multiLineEnv+2:]
if multiLineEnv == 0 || multiLineEnvDelimiter == "" {
return fmt.Errorf("invalid format '%v', name and delimiter must not be empty", line)
}
delimiterFound := false delimiterFound := false
for s.Scan() { for s.Scan() {
content := s.Text() content := s.Text()
if content == multiLineEnvDelimiter { if trimEnvLineEnding(content, windows) == multiLineEnvDelimiter {
delimiterFound = true delimiterFound = true
break break
} }
@@ -67,7 +91,7 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
if !delimiterFound { if !delimiterFound {
return fmt.Errorf("invalid format delimiter '%v' not found before end of file", multiLineEnvDelimiter) return fmt.Errorf("invalid format delimiter '%v' not found before end of file", multiLineEnvDelimiter)
} }
localEnv[line[:multiLineEnv]] = strings.Join(multiLineEnvContent, "\n") setEnv(line[:multiLineEnv], trimEnvLineEnding(strings.Join(multiLineEnvContent, ""), windows))
default: default:
return fmt.Errorf("invalid format '%v', expected a line with '=' or '<<'", line) return fmt.Errorf("invalid format '%v', expected a line with '=' or '<<'", line)
} }
@@ -78,3 +102,10 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
return nil return nil
} }
} }
func trimEnvLineEnding(line string, windows bool) string {
if windows && strings.HasSuffix(line, "\r\n") {
return line[:len(line)-2]
}
return strings.TrimSuffix(line, "\n")
}
+59 -20
View File
@@ -5,6 +5,7 @@ package container
import ( import (
"bufio" "bufio"
"bytes"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -15,13 +16,9 @@ import (
"golang.org/x/text/encoding/unicode" "golang.org/x/text/encoding/unicode"
) )
func newTestHostEnv(t *testing.T) (*HostEnvironment, string) {
t.Helper()
e := &HostEnvironment{Path: t.TempDir()}
return e, filepath.Join(e.Path, "envfile")
}
func TestParseEnvFile(t *testing.T) { func TestParseEnvFile(t *testing.T) {
t.Parallel()
hostEnv := &HostEnvironment{Path: t.TempDir()}
largeValue := strings.Repeat("x", 2*1024*1024) largeValue := strings.Repeat("x", 2*1024*1024)
encodeUTF16 := func(endianness unicode.Endianness) string { encodeUTF16 := func(endianness unicode.Endianness) string {
content, err := unicode.UTF16(endianness, unicode.UseBOM).NewEncoder().Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n")) content, err := unicode.UTF16(endianness, unicode.UseBOM).NewEncoder().Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n"))
@@ -33,41 +30,83 @@ func TestParseEnvFile(t *testing.T) {
want map[string]string want map[string]string
}{ }{
{"single line", "FOO=bar\nBAZ=qux\n", map[string]string{"FOO": "bar", "BAZ": "qux"}}, {"single line", "FOO=bar\nBAZ=qux\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"repeated names", "FOO=first\nfoo=middle\nFOO=last\n", map[string]string{"FOO": "last", "foo": "middle"}},
{"single line CRLF and final bare CR", "FOO=bar\r\nBAZ=qux\r", map[string]string{"FOO": "bar\r", "BAZ": "qux\r"}},
{"single line without final newline", "FOO=bar", map[string]string{"FOO": "bar"}},
{"blank lines", "\nFOO=bar\n\n \nBAZ=qux\n\n", map[string]string{"FOO": "bar", "BAZ": "qux"}}, {"blank lines", "\nFOO=bar\n\n \nBAZ=qux\n\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"multiline", "FOO<<EOF\nline1\nline2\nEOF\n", map[string]string{"FOO": "line1\nline2"}}, {"multiline", "FOO<<EOF\nline1\nline2\nEOF\n", map[string]string{"FOO": "line1\nline2"}},
{"multiline without final newline", "FOO<<EOF\nline1\r\nline2\r\nEOF", map[string]string{"FOO": "line1\r\nline2\r"}},
{"empty multiline without final newline", "FOO<<EOF\nEOF", map[string]string{"FOO": ""}},
{"multiline blank lines", "FOO<<EOF\nline1\n\nline2\nEOF\n", map[string]string{"FOO": "line1\n\nline2"}}, {"multiline blank lines", "FOO<<EOF\nline1\n\nline2\nEOF\n", map[string]string{"FOO": "line1\n\nline2"}},
{"multiline leading blank line", "FOO<<EOF\n\nline2\nEOF\n", map[string]string{"FOO": "\nline2"}}, {"multiline leading blank line", "FOO<<EOF\n\nline2\nEOF\n", map[string]string{"FOO": "\nline2"}},
{"multiline CRLF", "FOO<<EOF\r\nline1\r\nline2\r\nEOF\r\n", map[string]string{"FOO": "line1\r\nline2\r"}},
{"multiline mixed line endings", "FOO<<EOF\nline1\r\nline2\n\r\nEOF\n", map[string]string{"FOO": "line1\r\nline2\n\r"}},
{"large value within limit", "FOO<<EOF\n" + largeValue + "\nEOF\n", map[string]string{"FOO": largeValue}}, {"large value within limit", "FOO<<EOF\n" + largeValue + "\nEOF\n", map[string]string{"FOO": largeValue}},
{"UTF8 BOM", "\xef\xbb\xbfFOO=bar\n", map[string]string{"FOO": "bar"}}, {"UTF8 BOM", "\xef\xbb\xbfFOO=bar\n", map[string]string{"FOO": "bar"}},
{"UTF16 little endian", encodeUTF16(unicode.LittleEndian), map[string]string{"FOO": "bar", "MULTI": "line1"}}, {"UTF16 little endian", encodeUTF16(unicode.LittleEndian), map[string]string{"FOO": "bar\r", "MULTI": "line1\r"}},
{"UTF16 big endian", encodeUTF16(unicode.BigEndian), map[string]string{"FOO": "bar", "MULTI": "line1"}}, {"UTF16 big endian", encodeUTF16(unicode.BigEndian), map[string]string{"FOO": "bar\r", "MULTI": "line1\r"}},
} { } {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
e, envPath := newTestHostEnv(t) envPath := filepath.Join(hostEnv.Path, test.name)
require.NoError(t, os.WriteFile(envPath, []byte(test.content), 0o600)) require.NoError(t, os.WriteFile(envPath, []byte(test.content), 0o600))
env := map[string]string{} env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context())) require.NoError(t, parseEnvFile(hostEnv, envPath, &env, false)(t.Context()))
assert.Equal(t, test.want, env) assert.Equal(t, test.want, env)
}) })
} }
t.Run("ordered entries preserve repeated names", func(t *testing.T) {
envPath := filepath.Join(hostEnv.Path, "ordered")
require.NoError(t, os.WriteFile(envPath, []byte("FOO=first\r\nfoo<<EOF\r\nmiddle\r\nEOF\r\nFOO=last\r\n"), 0o600))
var entries [][2]string
require.NoError(t, ParseEnvFile(hostEnv, envPath, func(name, value string) {
entries = append(entries, [2]string{name, value})
}, true)(t.Context()))
assert.Equal(t, [][2]string{
{"FOO", "first"}, {"foo", "middle"}, {"FOO", "last"},
}, entries)
})
t.Run("Windows line endings", func(t *testing.T) {
envPath := filepath.Join(hostEnv.Path, "windows")
require.NoError(t, os.WriteFile(envPath, []byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nline2\nEOF\r\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(hostEnv, envPath, &env, true)(t.Context()))
assert.Equal(t, map[string]string{"FOO": "bar", "MULTI": "line1\r\nline2"}, env)
})
} }
func TestParseEnvFileLineExceedsBufferReportsScannerError(t *testing.T) { func TestParseEnvFileLineExceedsBufferReportsScannerError(t *testing.T) {
e, envPath := newTestHostEnv(t) t.Parallel()
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n"+strings.Repeat("x", 17*1024*1024)+"\nEOF\n"), 0o600)) hostEnv := &HostEnvironment{Path: t.TempDir()}
envPath := filepath.Join(hostEnv.Path, "envfile")
const header, footer = "FOO<<EOF\n", "\nEOF\n"
content := bytes.Repeat([]byte("x"), len(header)+17*1024*1024+len(footer))
copy(content, header)
copy(content[len(content)-len(footer):], footer)
require.NoError(t, os.WriteFile(envPath, content, 0o600))
env := map[string]string{} env := map[string]string{}
err := parseEnvFile(e, envPath, &env)(t.Context()) err := parseEnvFile(hostEnv, envPath, &env, false)(t.Context())
require.ErrorIs(t, err, bufio.ErrTooLong) require.ErrorIs(t, err, bufio.ErrTooLong)
assert.Contains(t, err.Error(), "reading env file") assert.Contains(t, err.Error(), "reading env file")
} }
func TestParseEnvFileMissingDelimiter(t *testing.T) { func TestParseEnvFileInvalidHeredoc(t *testing.T) {
e, envPath := newTestHostEnv(t) hostEnv := &HostEnvironment{Path: t.TempDir()}
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\nline1\nline2\n"), 0o600)) for name, content := range map[string]string{
"missing delimiter": "FOO<<EOF\nline1\nline2\n",
"empty name": "<<EOF\nline1\nEOF\n",
"empty delimiter": "FOO<<\nline1\n\n",
"delimiter with carriage return": "FOO<<EOF\nline1\nEOF\r\n",
} {
t.Run(name, func(t *testing.T) {
envPath := filepath.Join(hostEnv.Path, name)
require.NoError(t, os.WriteFile(envPath, []byte("APPLIED=before-error\n"+content), 0o600))
env := map[string]string{} env := map[string]string{}
err := parseEnvFile(e, envPath, &env)(t.Context()) err := parseEnvFile(hostEnv, envPath, &env, false)(t.Context())
require.Error(t, err) require.Error(t, err)
assert.Contains(t, err.Error(), "delimiter") assert.Contains(t, err.Error(), "delimiter")
assert.Equal(t, map[string]string{"APPLIED": "before-error"}, env)
})
}
} }
+2 -2
View File
@@ -36,7 +36,7 @@ func writeFilesTar(ctx context.Context, w io.Writer, uid, gid int, files ...*Fil
} }
// writeDirTar prefixes entries with dstPath for extraction at root. // writeDirTar prefixes entries with dstPath for extraction at root.
func writeDirTar(ctx context.Context, w io.Writer, dstPath, srcPath string, useGitIgnore, skipGitDir bool, uid, gid int) error { func writeDirTar(ctx context.Context, w io.Writer, dstPath, srcPath string, useGitIgnore, skipGitDir bool, uid, gid int, executable bool) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
tw := tar.NewWriter(w) tw := tar.NewWriter(w)
srcPrefix := strings.TrimSuffix(filepath.Dir(srcPath), string(filepath.Separator)) + string(filepath.Separator) srcPrefix := strings.TrimSuffix(filepath.Dir(srcPath), string(filepath.Separator)) + string(filepath.Separator)
@@ -47,7 +47,7 @@ func writeDirTar(ctx context.Context, w io.Writer, dstPath, srcPath string, useG
SrcPrefix: srcPrefix, SrcPrefix: srcPrefix,
SkipGitDir: skipGitDir, SkipGitDir: skipGitDir,
UseGitIgnore: useGitIgnore, UseGitIgnore: useGitIgnore,
Handler: &filecollector.TarCollector{TarWriter: tw, UID: uid, GID: gid, DstDir: dstPath[1:]}, Handler: &filecollector.TarCollector{TarWriter: tw, UID: uid, GID: gid, DstDir: dstPath[1:], Executable: executable},
}).CollectFiles(ctx)); err != nil { }).CollectFiles(ctx)); err != nil {
return err return err
} }
+9 -6
View File
@@ -27,22 +27,25 @@ type Handler interface {
} }
type TarCollector struct { type TarCollector struct {
TarWriter *tar.Writer TarWriter *tar.Writer
UID int UID int
GID int GID int
DstDir string DstDir string
Executable bool // Windows has no exec bit, so mark everything executable like docker build
} }
func (tc TarCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string, f io.Reader) error { func (tc TarCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string, f io.Reader) error {
// create a new dir/file header // create a new dir/file header
header, err := tar.FileInfoHeader(fi, linkName) header, err := tar.FileInfoHeader(fi, filepath.ToSlash(linkName))
if err != nil { if err != nil {
return err return err
} }
// update the name to correctly reflect the desired destination when untaring // update the name to correctly reflect the desired destination when untaring
header.Name = path.Join(tc.DstDir, fpath) header.Name = path.Join(tc.DstDir, fpath)
header.Mode = int64(fi.Mode()) if tc.Executable {
header.Mode = header.Mode&0o755 | 0o111
}
header.Uid = tc.UID header.Uid = tc.UID
header.Gid = tc.GID header.Gid = tc.GID
+13 -7
View File
@@ -28,10 +28,12 @@ func gitCmd(t *testing.T, args ...string) {
} }
func TestCollectFilesKeepsWhatGitTracksOrDoesNotIgnore(t *testing.T) { func TestCollectFilesKeepsWhatGitTracksOrDoesNotIgnore(t *testing.T) {
repoDir := filepath.Join(t.TempDir(), "mygitrepo") repoDir := t.TempDir()
for _, dir := range []string{repoDir, filepath.Join(repoDir, "sub"), filepath.Join(repoDir, "nested")} { gitCmd(t, "init", "-q", "--template=", repoDir)
gitCmd(t, "init", "-q", dir) for _, dir := range []string{"sub", "nested"} {
require.NoError(t, os.CopyFS(filepath.Join(repoDir, dir, ".git"), os.DirFS(filepath.Join(repoDir, ".git"))))
} }
require.NoError(t, os.Mkdir(filepath.Join(repoDir, "ign"), 0o755))
for name, body := range map[string]string{ for name, body := range map[string]string{
".gitignore": ".*\nign/\n", ".gitignore": ".*\nign/\n",
".env": "", ".env": "",
@@ -43,17 +45,18 @@ func TestCollectFilesKeepsWhatGitTracksOrDoesNotIgnore(t *testing.T) {
"sub/kept": "", "sub/kept": "",
"nested/kept": "", "nested/kept": "",
} { } {
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(repoDir, name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(repoDir, name), []byte(body), 0o644)) require.NoError(t, os.WriteFile(filepath.Join(repoDir, name), []byte(body), 0o644))
} }
require.NoError(t, os.Mkdir(filepath.Join(repoDir, "uninitialized"), 0o755)) require.NoError(t, os.Mkdir(filepath.Join(repoDir, "uninitialized"), 0o755))
gitCmd(t, "-C", repoDir, "add", "-f", ".gitignore", "Renamed", "ign/tracked") gitCmd(t, "-C", repoDir, "add", "-f", ".gitignore", "Renamed", "ign/tracked")
require.NoError(t, os.Rename(filepath.Join(repoDir, "Renamed"), filepath.Join(repoDir, "renamed"))) require.NoError(t, os.Rename(filepath.Join(repoDir, "Renamed"), filepath.Join(repoDir, "renamed")))
for _, gitlink := range []string{"sub", "uninitialized"} { gitCmd(t, "-C", repoDir, "update-index", "--add",
gitCmd(t, "-C", repoDir, "update-index", "--add", "--cacheinfo", "160000,4b825dc642cb6eb9a060e54bf8d69288fbee4904,"+gitlink) "--cacheinfo", "160000,4b825dc642cb6eb9a060e54bf8d69288fbee4904,sub",
} "--cacheinfo", "160000,4b825dc642cb6eb9a060e54bf8d69288fbee4904,uninitialized",
)
expected := map[string]string{".gitignore": "", "ign/tracked": "", "nested/kept": "", "renamed": "", "sub/.gitignore": "", "sub/kept": ""} expected := map[string]string{".gitignore": "", "ign/tracked": "", "nested/kept": "", "renamed": "", "sub/.gitignore": "", "sub/kept": ""}
if runtime.GOOS != "windows" { if runtime.GOOS != "windows" {
require.NoError(t, os.Chmod(filepath.Join(repoDir, "renamed"), 0o777|os.ModeSticky))
require.NoError(t, os.Symlink(".env", filepath.Join(repoDir, "test.env"))) require.NoError(t, os.Symlink(".env", filepath.Join(repoDir, "test.env")))
expected["test.env"] = ".env" expected["test.env"] = ".env"
} }
@@ -69,6 +72,9 @@ func TestCollectFilesKeepsWhatGitTracksOrDoesNotIgnore(t *testing.T) {
for header, err := tr.Next(); err != io.EOF; header, err = tr.Next() { for header, err := tr.Next(); err != io.EOF; header, err = tr.Next() {
require.NoError(t, err) require.NoError(t, err)
files[header.Name] = header.Linkname files[header.Name] = header.Linkname
if header.Name == "renamed" && runtime.GOOS != "windows" {
assert.Equal(t, int64(0o1777), header.Mode)
}
} }
assert.Contains(t, files, ".git/HEAD") assert.Contains(t, files, ".git/HEAD")
maps.DeleteFunc(files, func(name, _ string) bool { return strings.Contains("/"+name, "/.git/") }) maps.DeleteFunc(files, func(name, _ string) bool { return strings.Contains("/"+name, "/.git/") })
+23 -6
View File
@@ -28,12 +28,12 @@ var (
type pinnedKey struct{} type pinnedKey struct{}
// WithPinnedCheckout keeps a run's sha and ref fixed when HEAD moves. // WithPinnedCheckout keeps a run's sha, ref and repository fixed when HEAD moves, in pins unless ctx has some.
func WithPinnedCheckout(ctx context.Context) context.Context { func WithPinnedCheckout(ctx context.Context, pins *sync.Map) context.Context {
if ctx.Value(pinnedKey{}) != nil { if ctx.Value(pinnedKey{}) != nil {
return ctx return ctx
} }
return context.WithValue(ctx, pinnedKey{}, &sync.Map{}) return context.WithValue(ctx, pinnedKey{}, pins)
} }
func pinned(ctx context.Context, key string, lookup func() (string, error)) (string, error) { func pinned(ctx context.Context, key string, lookup func() (string, error)) (string, error) {
@@ -57,12 +57,29 @@ func SetRef(ctx context.Context, ghc *model.GithubContext, repoPath string) {
case "pull_request_target": case "pull_request_target":
ghc.Ref = "refs/heads/" + ghc.BaseRef ghc.Ref = "refs/heads/" + ghc.BaseRef
case "pull_request", "pull_request_review", "pull_request_review_comment": case "pull_request", "pull_request_review", "pull_request_review_comment":
ghc.Ref = fmt.Sprintf("refs/pull/%.0f/merge", ghc.Event["number"]) number := ghc.Event["number"]
if number == nil {
number = model.NestedMapLookup(ghc.Event, "pull_request", "number")
}
ghc.Ref = fmt.Sprintf("refs/pull/%.0f/merge", number)
if ghc.EventName == "pull_request" && ghc.Event["action"] == "closed" && model.NestedMapLookup(ghc.Event, "pull_request", "merged") == true {
ghc.Ref = "refs/heads/" + ghc.BaseRef
}
case "deployment", "deployment_status": case "deployment", "deployment_status":
ghc.Ref = model.AsString(model.NestedMapLookup(ghc.Event, "deployment", "ref")) ghc.Ref = model.AsString(model.NestedMapLookup(ghc.Event, "deployment", "ref"))
case "release": case "release":
ghc.Ref = "refs/tags/" + model.AsString(model.NestedMapLookup(ghc.Event, "release", "tag_name")) ghc.Ref = "refs/tags/" + model.AsString(model.NestedMapLookup(ghc.Event, "release", "tag_name"))
case "push", "create", "workflow_dispatch": case "create":
ghc.Ref = model.AsString(ghc.Event["ref"])
if ghc.Ref != "" && !strings.HasPrefix(ghc.Ref, "refs/") {
switch model.AsString(ghc.Event["ref_type"]) {
case "branch":
ghc.Ref = "refs/heads/" + ghc.Ref
case "tag":
ghc.Ref = "refs/tags/" + ghc.Ref
}
}
case "push", "workflow_dispatch":
ghc.Ref = model.AsString(ghc.Event["ref"]) ghc.Ref = model.AsString(ghc.Event["ref"])
default: default:
defaultBranch := model.AsString(model.NestedMapLookup(ghc.Event, "repository", "default_branch")) defaultBranch := model.AsString(model.NestedMapLookup(ghc.Event, "repository", "default_branch"))
@@ -131,7 +148,7 @@ func SetSha(ctx context.Context, ghc *model.GithubContext, repoPath string) {
// remote in repoPath when it is not set yet, and derives its owner. // remote in repoPath when it is not set yet, and derives its owner.
func SetRepositoryAndOwner(ctx context.Context, ghc *model.GithubContext, repoPath string) { func SetRepositoryAndOwner(ctx context.Context, ghc *model.GithubContext, repoPath string) {
if ghc.Repository == "" { if ghc.Repository == "" {
repo, err := findGithubRepo(ctx, repoPath) repo, err := pinned(ctx, "repo:"+repoPath, func() (string, error) { return findGithubRepo(ctx, repoPath) })
if err != nil { if err != nil {
common.Logger(ctx).Warnf("unable to get git repo (repoPath: %v): %v", repoPath, err) common.Logger(ctx).Warnf("unable to get git repo (repoPath: %v): %v", repoPath, err)
return return
+50 -2
View File
@@ -7,6 +7,7 @@ package ghcontext
import ( import (
"context" "context"
"errors" "errors"
"sync"
"testing" "testing"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
@@ -41,6 +42,30 @@ func TestSetRef(t *testing.T) {
ref: "refs/pull/1234/merge", ref: "refs/pull/1234/merge",
refName: "1234/merge", refName: "1234/merge",
}, },
{
eventName: "pull_request_review",
event: map[string]any{"pull_request": map[string]any{"number": 1234.}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request",
event: map[string]any{"number": 1234., "action": "closed", "pull_request": map[string]any{"merged": true}},
ref: "refs/heads/master",
refName: "master",
},
{
eventName: "pull_request",
event: map[string]any{"number": 1234., "action": "closed", "pull_request": map[string]any{"merged": false}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request_review_comment",
event: map[string]any{"pull_request": map[string]any{"number": 1234.}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{ {
eventName: "deployment", eventName: "deployment",
event: map[string]any{ event: map[string]any{
@@ -69,6 +94,24 @@ func TestSetRef(t *testing.T) {
ref: "refs/heads/somebranch", ref: "refs/heads/somebranch",
refName: "somebranch", refName: "somebranch",
}, },
{
eventName: "create",
event: map[string]any{"ref": "feature/new", "ref_type": "branch"},
ref: "refs/heads/feature/new",
refName: "feature/new",
},
{
eventName: "create",
event: map[string]any{"ref": "v1", "ref_type": "tag"},
ref: "refs/tags/v1",
refName: "v1",
},
{
eventName: "create",
event: map[string]any{"ref": "refs/tags/v1", "ref_type": "tag"},
ref: "refs/tags/v1",
refName: "v1",
},
{ {
eventName: "unknown", eventName: "unknown",
event: map[string]any{ event: map[string]any{
@@ -122,8 +165,10 @@ func TestSetRef(t *testing.T) {
func TestSetSha(t *testing.T) { func TestSetSha(t *testing.T) {
oldFindGitRef := findGitRef oldFindGitRef := findGitRef
oldFindGitRevision := findGitRevision oldFindGitRevision := findGitRevision
oldFindGithubRepo := findGithubRepo
defer func() { findGitRef = oldFindGitRef }() defer func() { findGitRef = oldFindGitRef }()
defer func() { findGitRevision = oldFindGitRevision }() defer func() { findGitRevision = oldFindGitRevision }()
defer func() { findGithubRepo = oldFindGithubRepo }()
findGitRevision = func(_ctx context.Context, _file string) (string, error) { findGitRevision = func(_ctx context.Context, _file string) (string, error) {
return "1234fakesha", nil return "1234fakesha", nil
@@ -189,16 +234,19 @@ func TestSetSha(t *testing.T) {
}) })
} }
t.Run("pinned checkout keeps the first sha and ref", func(t *testing.T) { t.Run("pinned checkout keeps the first sha, ref and repository", func(t *testing.T) {
ctx := WithPinnedCheckout(t.Context()) ctx := WithPinnedCheckout(t.Context(), &sync.Map{})
for _, head := range []string{"first", "moved"} { for _, head := range []string{"first", "moved"} {
findGitRevision = func(context.Context, string) (string, error) { return head, nil } findGitRevision = func(context.Context, string) (string, error) { return head, nil }
findGitRef = func(context.Context, string) (string, error) { return "refs/heads/" + head, nil } findGitRef = func(context.Context, string) (string, error) { return "refs/heads/" + head, nil }
findGithubRepo = func(context.Context, string) (string, error) { return "owner/" + head, nil }
ghc := &model.GithubContext{Event: map[string]any{}} ghc := &model.GithubContext{Event: map[string]any{}}
SetSha(ctx, ghc, "/some/dir") SetSha(ctx, ghc, "/some/dir")
SetRef(ctx, ghc, "/some/dir") SetRef(ctx, ghc, "/some/dir")
SetRepositoryAndOwner(ctx, ghc, "/some/dir")
assert.Equal(t, "first", ghc.Sha) assert.Equal(t, "first", ghc.Sha)
assert.Equal(t, "refs/heads/first", ghc.Ref) assert.Equal(t, "refs/heads/first", ghc.Ref)
assert.Equal(t, "owner/first", ghc.Repository)
} }
}) })
} }
+25
View File
@@ -0,0 +1,25 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package lookpath
import (
"path/filepath"
"strings"
)
func lookPathWindows(file string, env map[string]string, lookup func(string, map[string]string) (string, error)) (string, error) {
found, err := lookup(file, env)
isBash := strings.EqualFold(file, "bash") || strings.EqualFold(file, "bash.exe")
if !isBash || err == nil && !strings.EqualFold(filepath.Base(filepath.Dir(found)), "System32") { // System32's bash is the WSL launcher, Git's bash replaces it
return found, err
}
if git, gitErr := lookup("git.exe", env); gitErr == nil {
for _, relative := range []string{"../bin/bash.exe", "../usr/bin/bash.exe", "bash.exe"} {
if bash, bashErr := lookup(filepath.Join(filepath.Dir(git), relative), env); bashErr == nil {
return bash, nil
}
}
}
return found, err
}
+56
View File
@@ -0,0 +1,56 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package lookpath
import (
"io/fs"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLookPathWindowsFallsBackToGitBash(t *testing.T) {
root := "C:" + string(filepath.Separator)
system32 := filepath.Join(root, "Windows", "System32")
gitCmd, gitBin := filepath.Join(root, "Git", "cmd"), filepath.Join(root, "Git", "bin")
msys := filepath.Join(root, "msys64", "usr", "bin")
for _, test := range []struct {
name, command, want string
path []string
}{
{"wsl launcher first", "bash", filepath.Join(gitBin, "bash.exe"), []string{system32, gitCmd}},
{"no bash on path", "BASH.EXE", filepath.Join(gitBin, "bash.exe"), []string{gitCmd}},
{"own bash first", "bash", filepath.Join(msys, "bash.exe"), []string{msys, system32, gitCmd}},
{"no git", "bash", filepath.Join(system32, "bash.exe"), []string{system32}},
{"other command", "sh", filepath.Join(system32, "sh.exe"), []string{system32, gitCmd}},
} {
t.Run(test.name, func(t *testing.T) {
files := map[string]bool{
filepath.Join(system32, "bash.exe"): true, filepath.Join(system32, "sh.exe"): true,
filepath.Join(gitCmd, "git.exe"): true, filepath.Join(gitBin, "bash.exe"): true, filepath.Join(msys, "bash.exe"): true,
}
lookup := func(file string, env map[string]string) (string, error) {
candidates := []string{file}
if !strings.ContainsAny(file, `:\/`) {
candidates = nil
for directory := range strings.SplitSeq(env["PATH"], ";") {
candidates = append(candidates, filepath.Join(directory, strings.TrimSuffix(strings.ToLower(file), ".exe")+".exe"))
}
}
for _, candidate := range candidates {
if files[filepath.Clean(candidate)] {
return filepath.Clean(candidate), nil
}
}
return "", fs.ErrNotExist
}
got, err := lookPathWindows(test.command, map[string]string{"PATH": strings.Join(test.path, ";")}, lookup)
require.NoError(t, err)
assert.Equal(t, test.want, got)
})
}
}
+4 -3
View File
@@ -53,6 +53,10 @@ func findExecutable(file string, exts []string) (string, error) {
// a suitable candidate. // a suitable candidate.
// The result may be an absolute path or a path relative to the current directory. // The result may be an absolute path or a path relative to the current directory.
func LookPath2(file string, env map[string]string) (string, error) { func LookPath2(file string, env map[string]string) (string, error) {
return lookPathWindows(file, env, lookPath)
}
func lookPath(file string, env map[string]string) (string, error) {
var exts []string var exts []string
x := getenv(env, `PATHEXT`) x := getenv(env, `PATHEXT`)
if x != "" { if x != "" {
@@ -76,9 +80,6 @@ func LookPath2(file string, env map[string]string) (string, error) {
return "", &Error{file, err} return "", &Error{file, err}
} }
} }
if f, err := findExecutable(filepath.Join(".", file), exts); err == nil {
return f, nil
}
path := getenv(env, "path") path := getenv(env, "path")
for _, dir := range filepath.SplitList(path) { for _, dir := range filepath.SplitList(path) {
if f, err := findExecutable(filepath.Join(dir, file), exts); err == nil { if f, err := findExecutable(filepath.Join(dir, file), exts); err == nil {
+20
View File
@@ -0,0 +1,20 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package lookpath
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/require"
)
func TestLookPath2IgnoresTheCurrentDirectory(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "tool.exe"), nil, 0o755))
t.Chdir(dir)
_, err := LookPath2("tool", map[string]string{"PATH": t.TempDir()})
require.EqualError(t, err, ErrNotFound.Error())
}
+119 -54
View File
@@ -19,6 +19,7 @@ import (
"path/filepath" "path/filepath"
"regexp" "regexp"
"runtime" "runtime"
"slices"
"strings" "strings"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -74,18 +75,24 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
reader, closer, err = readFile("action.yaml") reader, closer, err = readFile("action.yaml")
addError("action.yaml", err) addError("action.yaml", err)
if os.IsNotExist(err) { if os.IsNotExist(err) {
_, closer, err := readFile("Dockerfile") dockerfile := "Dockerfile"
addError("Dockerfile", err) _, closer, err := readFile(dockerfile)
addError(dockerfile, err)
if os.IsNotExist(err) {
dockerfile = "dockerfile"
_, closer, err = readFile(dockerfile)
addError(dockerfile, err)
}
if err == nil { if err == nil {
closer.Close() closer.Close()
action := &model.Action{ action := &model.Action{
Name: "(Synthetic)", Name: "(Synthetic)",
Runs: model.ActionRuns{ Runs: model.ActionRuns{
Using: "docker", Using: "docker",
Image: "Dockerfile", Image: dockerfile,
}, },
} }
logger.Debugf("Using synthetic action %v for Dockerfile", action) logger.Debugf("Using synthetic action %v for %s", action, dockerfile)
return action, nil return action, nil
} }
if val, ok := step.With["args"]; ok { if val, ok := step.With["args"]; ok {
@@ -95,6 +102,7 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
action := &model.Action{ action := &model.Action{
Name: "(Synthetic)", Name: "(Synthetic)",
Inputs: map[string]model.Input{ Inputs: map[string]model.Input{
"args": {},
"cwd": { "cwd": {
Description: "(Actual working directory)", Description: "(Actual working directory)",
Required: false, Required: false,
@@ -146,12 +154,17 @@ func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actio
} }
defer unlock() defer unlock()
if !rc.Config.NoActionPatch { if err := rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", false, true)(ctx); err != nil { // an exported commit has no .git to apply gitignore with
// Other jobs share the directory, so patch under the copy's lock. return err
defer patchActions(ctx, actionScriptPaths(actionDir, filepath.Join(actionDir, actionPath), step.getActionModel()))()
} }
if rc.Config.NoActionPatch {
return rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", false, true)(ctx) // an exported commit has no .git to apply gitignore with return nil
}
patched := patchedScripts(ctx, actionDir, actionScriptPaths(actionDir, filepath.Join(actionDir, actionPath), step.getActionModel()))
if len(patched) == 0 {
return nil
}
return rc.JobContainer.Copy(containerActionDirCopy, patched...)(ctx)
} }
func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor { func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor {
@@ -167,11 +180,13 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
action := step.getActionModel() action := step.getActionModel()
rc.withGithubEnv(ctx, step.getGithubContext(ctx), *step.getEnv()) rc.withGithubEnv(ctx, getStepGithubContext(ctx, step), *step.getEnv())
populateEnvsFromSavedState(step.getEnv(), step, rc) populateEnvsFromSavedState(step.getEnv(), step, rc)
if err := populateEnvsFromInput(ctx, step.getEnv(), action, rc); err != nil { if err := populateEnvsFromInput(ctx, step, action); err != nil {
return err return err
} }
warnUnexpectedInputs(ctx, action, stepModel.With)
printRunActionHeader(ctx, stepModel, action, *step.getEnv(), rc)
actionName, containerActionDir := getContainerActionPaths(step) actionName, containerActionDir := getContainerActionPaths(step)
@@ -183,7 +198,7 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil { if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil {
return err return err
} }
containerArgs := nodeActionCommand(path.Join(containerActionDir, action.Runs.Main)) containerArgs := nodeActionCommand(container.JoinPath(containerActionDir, action.Runs.Main))
logger.Debugf("executing remote job container: %s", containerArgs) logger.Debugf("executing remote job container: %s", containerArgs)
rc.ApplyExtraPath(ctx, step.getEnv()) rc.ApplyExtraPath(ctx, step.getEnv())
@@ -211,7 +226,7 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
execFileName := action.Runs.Main + ".out" execFileName := action.Runs.Main + ".out"
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Main}, *step.getEnv(), "", containerActionDir), rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Main}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""), rc.JobContainer.Exec([]string{container.JoinPath(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx) )(ctx)
default: default:
return fmt.Errorf("the runs.using key must be one of: %v, got %s", []string{ return fmt.Errorf("the runs.using key must be one of: %v, got %s", []string{
@@ -268,7 +283,7 @@ func (rc *RunContext) dockerActionImage(ctx context.Context, step actionStep, ac
} }
image, ok := job.dockerActionImages[step.getStepModel()] image, ok := job.dockerActionImages[step.getStepModel()]
if !ok { if !ok {
image = dockerActionImageTag(step.getGithubContext(ctx).Repository, actionName, localAction) + "-" + strings.ToLower(rand.Text()) image = dockerActionImageTag(getStepGithubContext(ctx, step).Repository, actionName, localAction) + "-" + strings.ToLower(rand.Text())
if !common.Dryrun(ctx) { // a dry run builds nothing to remove if !common.Dryrun(ctx) { // a dry run builds nothing to remove
job.dockerActionImages[step.getStepModel()] = image job.dockerActionImages[step.getStepModel()] = image
} }
@@ -280,6 +295,9 @@ func (rc *RunContext) dockerActionImage(ctx context.Context, step actionStep, ac
func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, basedir string, localAction bool, stage stepStage) error { func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, basedir string, localAction bool, stage stepStage) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
rc := step.getRunContext() rc := step.getRunContext()
if rc.isWindowsHost() {
return errWindowsHostContainer
}
action := step.getActionModel() action := step.getActionModel()
if rc.kubernetes { if rc.kubernetes {
return errKubernetesDocker return errKubernetesDocker
@@ -301,9 +319,12 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
return err return err
} }
correctArchExists, err := ContainerImageExistsLocally(ctx, image, rc.Config.ContainerArchitecture) correctArchExists := anyArchExists
if err != nil { if anyArchExists && rc.Config.ContainerArchitecture != "" && rc.Config.ContainerArchitecture != "any" {
return err correctArchExists, err = ContainerImageExistsLocally(ctx, image, rc.Config.ContainerArchitecture)
if err != nil {
return err
}
} }
if anyArchExists && !correctArchExists { if anyArchExists && !correctArchExists {
@@ -318,46 +339,39 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
if !correctArchExists || rc.Config.ForceRebuild { if !correctArchExists || rc.Config.ForceRebuild {
logger.Debugf("image '%s' for architecture '%s' will be built from context '%s", image, rc.Config.ContainerArchitecture, contextDir) logger.Debugf("image '%s' for architecture '%s' will be built from context '%s", image, rc.Config.ContainerArchitecture, contextDir)
var buildContext io.ReadCloser prepImage = common.Executor(func(ctx context.Context) error {
if localAction { input := container.NewDockerBuildExecutorInput{
buildContext, err = rc.JobContainer.GetContainerArchive(ctx, contextDir+"/.") ContextDir: contextDir,
if err != nil { Dockerfile: fileName,
return err ImageTag: image,
Platform: rc.Config.ContainerArchitecture,
BuildArgs: rc.proxyBuildArgs(),
} }
defer buildContext.Close() if localAction {
} buildContext, err := rc.JobContainer.GetContainerArchive(ctx, contextDir+"/.")
prepImage = ContainerNewDockerBuildExecutor(container.NewDockerBuildExecutorInput{ if err != nil {
ContextDir: contextDir, return err
Dockerfile: fileName, }
ImageTag: image, defer buildContext.Close()
BuildContext: buildContext, input.BuildContext = buildContext
Platform: rc.Config.ContainerArchitecture, } else {
BuildArgs: rc.proxyBuildArgs(), unlock, err := git.AcquireCloneLock(ctx, actionDir) // held across the whole build, the daemon drains contextDir lazily
})
if buildContext == nil {
// Held across the whole build: the daemon drains contextDir lazily.
inner := prepImage
prepImage = func(ctx context.Context) error {
unlock, err := git.AcquireCloneLock(ctx, actionDir)
if err != nil { if err != nil {
return err return err
} }
defer unlock() defer unlock()
return inner(ctx)
} }
} return ContainerNewDockerBuildExecutor(input)(ctx)
}).Retry(3)
} else { } else {
logger.Debugf("image '%s' for architecture '%s' already exists", image, rc.Config.ContainerArchitecture) logger.Debugf("image '%s' for architecture '%s' already exists", image, rc.Config.ContainerArchitecture)
} }
} }
cmd, err := shellquote.Split(step.getStepModel().With["args"])
if err != nil {
return err
}
ee, err := evalDockerEnv(ctx, step, action) ee, err := evalDockerEnv(ctx, step, action)
if err != nil { if err != nil {
return err return err
} }
var cmd []string
if action.Runs.Args != nil { if action.Runs.Args != nil {
// a fresh slice, the manifest is evaluated again for every stage // a fresh slice, the manifest is evaluated again for every stage
cmd = make([]string, len(action.Runs.Args)) cmd = make([]string, len(action.Runs.Args))
@@ -366,6 +380,11 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
return fmt.Errorf("unable to interpolate runs.args: %w", err) return fmt.Errorf("unable to interpolate runs.args: %w", err)
} }
} }
} else {
cmd, err = shellquote.Split(stepInput(step.getStepModel(), "args"))
if err != nil {
return err
}
} }
entrypoint, err := dockerEntrypoint(step, stage) entrypoint, err := dockerEntrypoint(step, stage)
if err != nil { if err != nil {
@@ -395,7 +414,7 @@ func dockerEntrypoint(step actionStep, stage stepStage) ([]string, error) {
default: default:
entrypoint = runs.Entrypoint entrypoint = runs.Entrypoint
if entrypoint == "" { if entrypoint == "" {
if fields := strings.Fields(step.getStepModel().With["entrypoint"]); len(fields) > 0 { if fields := strings.Fields(stepInput(step.getStepModel(), "entrypoint")); len(fields) > 0 {
return fields, nil return fields, nil
} }
} }
@@ -413,9 +432,15 @@ func evalDockerEnv(ctx context.Context, step step, action *model.Action) (*expre
var err error var err error
inputs := make(map[string]string) inputs := make(map[string]string)
eval := rc.NewExpressionEvaluator(ctx) providedInputs := make(map[string]bool, len(step.getStepModel().With))
// Set Defaults for key := range step.getStepModel().With {
providedInputs[strings.ToLower(key)] = true
}
eval := rc.NewStepExpressionEvaluator(ctx, step)
for k, input := range action.Inputs { for k, input := range action.Inputs {
if providedInputs[strings.ToLower(k)] {
continue
}
if inputs[k], err = eval.Interpolate(ctx, input.Default); err != nil { if inputs[k], err = eval.Interpolate(ctx, input.Default); err != nil {
return nil, fmt.Errorf("unable to interpolate the default of input %s: %w", k, err) return nil, fmt.Errorf("unable to interpolate the default of input %s: %w", k, err)
} }
@@ -430,6 +455,7 @@ func evalDockerEnv(ctx context.Context, step step, action *model.Action) (*expre
} }
} }
mergeIntoMap(step, step.getEnv(), runsEnv, maps.Clone(*step.getEnv())) mergeIntoMap(step, step.getEnv(), runsEnv, maps.Clone(*step.getEnv()))
aliasGiteaEnv(*step.getEnv())
return ee, nil return ee, nil
} }
@@ -479,10 +505,15 @@ func populateEnvsFromSavedState(env *map[string]string, step actionStep, rc *Run
} }
} }
func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *model.Action, rc *RunContext) error { func inputEnvKey(inputID string) string {
eval := rc.NewExpressionEvaluator(ctx) return "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_")
}
func populateEnvsFromInput(ctx context.Context, step step, action *model.Action) error {
env := step.getEnv()
eval := step.getRunContext().NewStepExpressionEvaluator(ctx, step)
for inputID, input := range action.Inputs { for inputID, input := range action.Inputs {
envKey := "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_") envKey := inputEnvKey(inputID)
if _, ok := (*env)[envKey]; !ok { if _, ok := (*env)[envKey]; !ok {
var err error var err error
if (*env)[envKey], err = eval.Interpolate(ctx, input.Default); err != nil { if (*env)[envKey], err = eval.Interpolate(ctx, input.Default); err != nil {
@@ -493,6 +524,33 @@ func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *
return nil return nil
} }
func stepInput(step *model.Step, name string) string {
value, _ := common.LookupFold(step.With, name, true)
return value
}
func warnUnexpectedInputs(ctx context.Context, action *model.Action, inputs map[string]string) {
docker := action.Runs.Using.IsDocker()
var unexpectedInputs []string
for input := range inputs {
_, declared := common.LookupFold(action.Inputs, input, true)
containerInput := docker && (strings.EqualFold(input, "entryPoint") || strings.EqualFold(input, "args"))
if !declared && !containerInput {
unexpectedInputs = append(unexpectedInputs, input)
}
}
if len(unexpectedInputs) == 0 {
return
}
validInputs := slices.Sorted(maps.Keys(action.Inputs))
if docker {
validInputs = append([]string{"entryPoint", "args"}, validInputs...)
}
slices.Sort(unexpectedInputs)
common.Logger(ctx).With(rawOutputField, true).Warnf("##[warning]%s", EscapeCommandData(fmt.Sprintf(
"Unexpected input(s) '%s', valid inputs are ['%s']", strings.Join(unexpectedInputs, "', '"), strings.Join(validInputs, "', '"))))
}
func getContainerActionPaths(step actionStep) (string, string) { func getContainerActionPaths(step actionStep) (string, string) {
rc := step.getRunContext() rc := step.getRunContext()
if sar, ok := step.(*stepActionRemote); ok { // a copy per action, as its repository's directory is shared if sar, ok := step.(*stepActionRemote); ok { // a copy per action, as its repository's directory is shared
@@ -581,17 +639,18 @@ func runPreStep(step actionStep) common.Executor {
x := action.Runs.Using x := action.Runs.Using
if !x.IsComposite() { if !x.IsComposite() {
// defaults in pre steps were missing, however provided inputs are available // defaults in pre steps were missing, however provided inputs are available
if err := populateEnvsFromInput(ctx, step.getEnv(), action, rc); err != nil { if err := populateEnvsFromInput(ctx, step, action); err != nil {
return err return err
} }
} }
printRunActionHeader(ctx, step.getStepModel(), action, *step.getEnv(), rc)
switch { switch {
case x.IsNode(): case x.IsNode():
if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil { if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil {
return err return err
} }
containerArgs := nodeActionCommand(path.Join(containerActionDir, action.Runs.Pre)) containerArgs := nodeActionCommand(container.JoinPath(containerActionDir, action.Runs.Pre))
logger.Debugf("executing remote job container: %s", containerArgs) logger.Debugf("executing remote job container: %s", containerArgs)
rc.ApplyExtraPath(ctx, step.getEnv()) rc.ApplyExtraPath(ctx, step.getEnv())
@@ -623,7 +682,7 @@ func runPreStep(step actionStep) common.Executor {
execFileName := action.Runs.Pre + ".out" execFileName := action.Runs.Pre + ".out"
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Pre}, *step.getEnv(), "", containerActionDir), rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Pre}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""), rc.JobContainer.Exec([]string{container.JoinPath(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx) )(ctx)
default: default:
return nil return nil
@@ -633,6 +692,9 @@ func runPreStep(step actionStep) common.Executor {
func shouldRunPostStep(step actionStep) common.Conditional { func shouldRunPostStep(step actionStep) common.Conditional {
return func(ctx context.Context) bool { return func(ctx context.Context) bool {
if remote, ok := step.(*stepActionRemote); ok && remote.preStepStarted && !remote.action.Runs.Using.IsComposite() {
return true
}
log := common.Logger(ctx) log := common.Logger(ctx)
stepResults := step.getRunContext().StepResults stepResults := step.getRunContext().StepResults
stepResult := stepResults[step.getStepModel().ID] stepResult := stepResults[step.getStepModel().ID]
@@ -680,12 +742,15 @@ func runPostStep(step actionStep) common.Executor {
actionDir, actionPath, _, containerActionDir := actionStagePaths(step) actionDir, actionPath, _, containerActionDir := actionStagePaths(step)
x := action.Runs.Using x := action.Runs.Using
if !x.IsComposite() { // a composite's children print their own
logger.With(rawOutputField, true).Info("Post job cleanup.")
}
switch { switch {
case x.IsNode(): case x.IsNode():
populateEnvsFromSavedState(step.getEnv(), step, rc) populateEnvsFromSavedState(step.getEnv(), step, rc)
containerArgs := nodeActionCommand(path.Join(containerActionDir, action.Runs.Post)) containerArgs := nodeActionCommand(container.JoinPath(containerActionDir, action.Runs.Post))
logger.Debugf("executing remote job container: %s", containerArgs) logger.Debugf("executing remote job container: %s", containerArgs)
rc.ApplyExtraPath(ctx, step.getEnv()) rc.ApplyExtraPath(ctx, step.getEnv())
@@ -714,7 +779,7 @@ func runPostStep(step actionStep) common.Executor {
execFileName := action.Runs.Post + ".out" execFileName := action.Runs.Post + ".out"
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Post}, *step.getEnv(), "", containerActionDir), rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Post}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""), rc.JobContainer.Exec([]string{container.JoinPath(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx) )(ctx)
default: default:
+7 -4
View File
@@ -31,7 +31,7 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
ee := parent.NewStepExpressionEvaluator(ctx, step) ee := parent.NewStepExpressionEvaluator(ctx, step)
for inputID, input := range step.getActionModel().Inputs { for inputID, input := range step.getActionModel().Inputs {
envKey := "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_") envKey := inputEnvKey(inputID)
// lookup if key is defined in the step but the already // lookup if key is defined in the step but the already
// evaluated value from the environment // evaluated value from the environment
@@ -52,9 +52,11 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
} }
} }
} }
gh := step.getGithubContext(ctx) for envKey, value := range (&model.Step{With: step.getStepModel().With}).GetEnv() {
env["GITHUB_ACTION_REPOSITORY"] = gh.ActionRepository if _, ok := env[envKey]; !ok {
env["GITHUB_ACTION_REF"] = gh.ActionRef env[envKey] = value
}
}
return env, nil return env, nil
} }
@@ -105,6 +107,7 @@ func newCompositeRunContext(ctx context.Context, parent *RunContext, step action
kubernetes: parent.kubernetes, kubernetes: parent.kubernetes,
jobIndex: parent.jobIndex, jobIndex: parent.jobIndex,
jobTotal: parent.jobTotal, jobTotal: parent.jobTotal,
runtimeToken: parent.runtimeToken,
} }
compositerc.setCompositeActionEnv(env) compositerc.setCompositeActionEnv(env)
compositerc.ExprEval = compositerc.NewExpressionEvaluator(ctx) compositerc.ExprEval = compositerc.NewExpressionEvaluator(ctx)
+48 -10
View File
@@ -7,7 +7,10 @@ import (
"context" "context"
"testing" "testing"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git" "gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -16,11 +19,41 @@ import (
) )
func TestCompositeActionParity(t *testing.T) { func TestCompositeActionParity(t *testing.T) {
t.Run("input defaults use the step identity and docker env keeps the GITEA aliases", func(t *testing.T) {
for _, uses := range []string{"./action", "own/action@v2"} {
t.Run(uses, func(t *testing.T) {
rc := &RunContext{
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
Env: map[string]string{"GITHUB_ACTION_REPOSITORY": "other/composite", "GITHUB_ACTION_REF": "v1"},
}
var currentStep step = &stepActionLocal{RunContext: rc, Step: &model.Step{Uses: uses}, env: map[string]string{}}
want := "|"
if uses == "own/action@v2" {
currentStep = &stepActionRemote{RunContext: rc, Step: &model.Step{Uses: uses}, env: map[string]string{}, remoteAction: mustNewRemoteAction(t, uses)}
want = "own/action|v2"
}
action := &model.Action{
Inputs: map[string]model.Input{"identity": {Default: "${{ github.action_repository }}|${{ github.action_ref }}"}},
Runs: model.ActionRuns{Env: map[string]string{"GITHUB_WORKFLOW_SHA": "runs-sha"}},
}
require.NoError(t, setupEnv(t.Context(), currentStep))
require.NoError(t, populateEnvsFromInput(t.Context(), currentStep, action))
assert.Equal(t, want, (*currentStep.getEnv())["INPUT_IDENTITY"])
_, err := evalDockerEnv(t.Context(), currentStep, action)
require.NoError(t, err)
assert.Equal(t, want, (*currentStep.getEnv())["identity"])
assert.Equal(t, "runs-sha", (*currentStep.getEnv())["GITEA_WORKFLOW_SHA"])
})
}
})
t.Run("inherits contexts without leaking inputs", func(t *testing.T) { t.Run("inherits contexts without leaking inputs", func(t *testing.T) {
ctx := t.Context() hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
strategy := &model.Strategy{MaxParallelString: "3", RawMatrix: yaml.Node{Kind: yaml.MappingNode}} strategy := &model.Strategy{MaxParallelString: "3", RawMatrix: yaml.Node{Kind: yaml.MappingNode}}
parent := &RunContext{ parent := &RunContext{
Config: &Config{}, Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Matrix: map[string]any{"os": "linux"}, Matrix: map[string]any{"os": "linux"},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Name: "workflow", Jobs: map[string]*model.Job{"job": {Strategy: strategy}}}}, Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Name: "workflow", Jobs: map[string]*model.Job{"job": {Strategy: strategy}}}},
JobContainer: &jobContainerMock{}, JobContainer: &jobContainerMock{},
@@ -28,24 +61,29 @@ func TestCompositeActionParity(t *testing.T) {
jobIndex: 1, jobIndex: 1,
jobTotal: 4, jobTotal: 4,
} }
composite, err := newCompositeRunContext(ctx, parent, &stepActionRemote{ step := &stepActionLocal{
Step: &model.Step{With: map[string]string{"SHARED": "outer"}}, Step: &model.Step{Uses: "./action", With: map[string]string{"SHARED": "outer", "extra": "literal${{ github.job }}"}},
RunContext: parent, RunContext: parent,
action: &model.Action{Inputs: map[string]model.Input{"shared": {Default: "outer-default"}}}, action: &model.Action{Inputs: map[string]model.Input{"shared": {Default: "outer-default"}}, Runs: model.ActionRuns{Using: "composite"}},
env: map[string]string{"INPUT_SHARED": "outer"}, env: map[string]string{"INPUT_SHARED": "outer"},
}, "/action") }
require.NoError(t, runActionImpl(step, "/action", nil)(ctx))
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, "##[warning]Unexpected input(s) 'extra', valid inputs are ['shared']", hook.LastRecord().Message)
composite, err := step.getCompositeRunContext(ctx)
require.NoError(t, err) require.NoError(t, err)
assert.Same(t, strategy, composite.Run.Job().Strategy) assert.Same(t, strategy, composite.Run.Job().Strategy)
assert.True(t, composite.IsHostEnv()) assert.True(t, composite.IsHostEnv())
interpolated, err := composite.NewExpressionEvaluator(ctx).Interpolate(ctx, interpolated, err := composite.NewExpressionEvaluator(ctx).Interpolate(ctx,
"${{ matrix.os }}|${{ strategy.max-parallel }}|${{ strategy.job-index }}|${{ inputs.shared }}") "${{ matrix.os }}|${{ strategy.max-parallel }}|${{ strategy.job-index }}|${{ inputs.shared }}|${{ inputs.extra }}")
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "linux|3|1|outer", interpolated) assert.Equal(t, "linux|3|1|outer|literal${{ github.job }}", interpolated)
assert.NotContains(t, composite.Env, "INPUT_SHARED") assert.NotContains(t, composite.Env, "INPUT_SHARED")
assert.NotContains(t, composite.Env, "INPUT_EXTRA")
nestedEnv := composite.GetEnv() nestedEnv := composite.GetEnv()
require.NoError(t, populateEnvsFromInput(ctx, &nestedEnv, &model.Action{Inputs: map[string]model.Input{"shared": {Default: "inner-default"}}}, composite)) require.NoError(t, populateEnvsFromInput(ctx, &stepActionLocal{RunContext: composite, env: nestedEnv}, &model.Action{Inputs: map[string]model.Input{"shared": {Default: "inner-default"}}}))
assert.Equal(t, "inner-default", nestedEnv["INPUT_SHARED"]) assert.Equal(t, "inner-default", nestedEnv["INPUT_SHARED"])
}) })
@@ -54,7 +92,7 @@ func TestCompositeActionParity(t *testing.T) {
return "", "", assert.AnError return "", "", assert.AnError
}) })
rc := &RunContext{ rc := &RunContext{
Config: &Config{GitHubInstance: "github.com", ActionCacheDir: t.TempDir()}, Config: &Config{GitHubInstance: "github.com", PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}}, Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{}, JobContainer: &jobContainerMock{},
} }
+204 -16
View File
@@ -8,6 +8,7 @@ import (
"context" "context"
"io" "io"
"io/fs" "io/fs"
"net/http"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
@@ -19,6 +20,8 @@ import (
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git" "gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container" "gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
@@ -85,6 +88,19 @@ runs:
}, },
}, },
}, },
{
name: "readLowercaseDockerfile",
step: &model.Step{},
filename: "dockerfile",
fileContent: "FROM ubuntu:20.04",
expected: &model.Action{
Name: "(Synthetic)",
Runs: model.ActionRuns{
Using: "docker",
Image: "dockerfile",
},
},
},
{ {
name: "readWithArgs", name: "readWithArgs",
step: &model.Step{ step: &model.Step{
@@ -95,10 +111,11 @@ runs:
expected: &model.Action{ expected: &model.Action{
Name: "(Synthetic)", Name: "(Synthetic)",
Inputs: map[string]model.Input{ Inputs: map[string]model.Input{
"args": {},
"cwd": { "cwd": {
Description: "(Actual working directory)", Description: "(Actual working directory)",
Required: false, Required: false,
Default: "actionDir/actionPath", Default: filepath.Join("actionDir", "actionPath"),
}, },
"command": { "command": {
Description: "(Actual program)", Description: "(Actual program)",
@@ -127,7 +144,7 @@ runs:
} }
writeFile := func(filename string, _data []byte, perm fs.FileMode) error { writeFile := func(filename string, _data []byte, perm fs.FileMode) error {
assert.Equal(t, "actionDir/actionPath/trampoline.js", filename) assert.Equal(t, filepath.Join("actionDir", "actionPath", "trampoline.js"), filename)
assert.Equal(t, fs.FileMode(0o644), perm) assert.Equal(t, fs.FileMode(0o644), perm)
return nil return nil
} }
@@ -147,14 +164,33 @@ func TestActionRunner(t *testing.T) {
for _, tt := range []struct { for _, tt := range []struct {
name string name string
inputs map[string]model.Input inputs map[string]model.Input
with map[string]string
state map[string]string state map[string]string
expectedEnv map[string]string expectedEnv map[string]string
warning string
}{ }{
{ {
name: "with-input", name: "with-input",
inputs: map[string]model.Input{"key": {Default: "default value"}}, inputs: map[string]model.Input{"key": {Default: "default value"}},
expectedEnv: map[string]string{"INPUT_KEY": "default value"}, expectedEnv: map[string]string{"INPUT_KEY": "default value"},
}, },
{
name: "declared input ignores case",
inputs: map[string]model.Input{"key": {Default: "default value"}},
with: map[string]string{"KEY": "provided value"},
expectedEnv: map[string]string{"INPUT_KEY": "provided value"},
},
{
name: "unexpected inputs warn once across stages",
inputs: map[string]model.Input{"key": {}},
with: map[string]string{"KEY": "provided value", "UNKNOWN": "value", "other%0A\r\n": "value"},
warning: "##[warning]Unexpected input(s) 'UNKNOWN', 'other%250A%0D%0A', valid inputs are ['key']",
},
{
name: "no declared inputs",
with: map[string]string{"unknown": "value"},
warning: "##[warning]Unexpected input(s) 'unknown', valid inputs are ['']",
},
{ {
name: "restore-saved-state", name: "restore-saved-state",
state: map[string]string{"name": "state value"}, state: map[string]string{"name": "state value"},
@@ -162,6 +198,8 @@ func TestActionRunner(t *testing.T) {
}, },
} { } {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
cm := &containerMock{} cm := &containerMock{}
containerActionDir := "/var/run/act/actions/" + model.UsesHash("/org/repo/path@ref") + "/" containerActionDir := "/var/run/act/actions/" + model.UsesHash("/org/repo/path@ref") + "/"
cm.On("CopyDir", containerActionDir, "dir/", false, true).Return(func(_ctx context.Context) error { return nil }) cm.On("CopyDir", containerActionDir, "dir/", false, true).Return(func(_ctx context.Context) error { return nil })
@@ -175,21 +213,70 @@ func TestActionRunner(t *testing.T) {
return true return true
}) })
cm.On("Exec", []string{"node", "--preserve-symlinks-main", containerActionDir + "path"}, envMatcher, "", "").Return(func(_ctx context.Context) error { return nil }) cm.On("Exec", []string{"node", "--preserve-symlinks-main", containerActionDir + "path"}, envMatcher, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", nodeActionCommand(containerActionDir+"path/pre.js"), mock.Anything, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", nodeActionCommand(containerActionDir+"path/post.js"), envMatcher, "", "").Return(func(_ctx context.Context) error { return nil })
remoteAction := mustNewRemoteAction(t, "org/repo/path@ref") remoteAction := mustNewRemoteAction(t, "org/repo/path@ref")
step := &stepActionRemote{ step := &stepActionRemote{
Step: &model.Step{ID: "step", Uses: "org/repo/path@ref"}, Step: &model.Step{ID: "step", Uses: "org/repo/path@ref", With: tt.with},
RunContext: &RunContext{ RunContext: &RunContext{
Config: &Config{}, Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}}, Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
IntraActionState: map[string]map[string]string{"step": tt.state}, IntraActionState: map[string]map[string]string{"step": tt.state},
JobContainer: cm, JobContainer: cm,
}, },
action: &model.Action{Inputs: tt.inputs, Runs: model.ActionRuns{Using: "node16"}}, action: &model.Action{Inputs: tt.inputs, Runs: model.ActionRuns{Using: "node16", Pre: "pre.js", Post: "post.js"}},
env: map[string]string{}, actionDir: "dir",
env: (&model.Step{With: tt.with}).GetEnv(),
remoteAction: remoteAction, remoteAction: remoteAction,
} }
require.NoError(t, runActionImpl(step, "dir", remoteAction)(t.Context())) require.NoError(t, runPreStep(step)(ctx))
assert.Empty(t, hook.AllRecords())
require.NoError(t, runActionImpl(step, "dir", remoteAction)(ctx))
require.NoError(t, runPostStep(step)(ctx))
if tt.warning == "" {
assert.Empty(t, hook.AllRecords())
} else {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, tt.warning, hook.LastRecord().Message)
rawOutput, ok := hook.LastRecord().Attr(rawOutputField)
require.True(t, ok)
assert.Equal(t, true, rawOutput)
}
cm.AssertExpectations(t)
})
}
}
func TestGoActionUsesLinuxExecutablePaths(t *testing.T) {
actionRoot := t.TempDir()
for _, stage := range []string{"main", "pre", "post"} {
t.Run(stage, func(t *testing.T) {
cm := &containerMock{}
containerActionDir := "/var/run/act/actions/" + model.UsesHash("/org/repo@ref")
cm.On("CopyDir", containerActionDir+"/", mock.Anything, false, true).Return(noopExecutor).Maybe()
cm.On("Exec", []string{"go", "build", "-o", stage + ".go.out", stage + ".go"}, mock.Anything, "", containerActionDir).Return(noopExecutor).Once()
cm.On("Exec", []string{containerActionDir + "/" + stage + ".go.out"}, mock.Anything, "", "").Return(noopExecutor).Once()
step := &stepActionRemote{
Step: &model.Step{ID: "step", Uses: "org/repo@ref"},
RunContext: &RunContext{
Config: &Config{NoActionPatch: true, PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: cm,
},
action: &model.Action{Runs: model.ActionRuns{Using: model.ActionRunsUsingGo, Main: "main.go", Pre: "pre.go", Post: "post.go"}},
actionDir: filepath.Join(actionRoot, stage),
env: map[string]string{},
remoteAction: mustNewRemoteAction(t, "org/repo@ref"),
}
switch stage {
case "main":
require.NoError(t, runActionImpl(step, step.actionDir, step.remoteAction)(t.Context()))
case "pre":
require.NoError(t, runPreStep(step)(t.Context()))
case "post":
require.NoError(t, runPostStep(step)(t.Context()))
}
cm.AssertExpectations(t) cm.AssertExpectations(t)
}) })
} }
@@ -376,7 +463,7 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
step := &stepActionRemote{ step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "remote/action@v1", With: map[string]string{}}, Step: &model.Step{ID: "1", Uses: "remote/action@v1", With: map[string]string{}},
RunContext: &RunContext{ RunContext: &RunContext{
Config: &Config{}, Config: &Config{Env: testCheckoutEnv()},
Run: &model.Run{ Run: &model.Run{
JobID: "1", JobID: "1",
Workflow: &model.Workflow{ Workflow: &model.Workflow{
@@ -414,7 +501,71 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
cancel() cancel()
releaseOnce() releaseOnce()
<-done assert.ErrorIs(t, <-done, context.Canceled)
}
func TestExecAsDockerImageChecks(t *testing.T) {
actionRoot := t.TempDir()
originalExists, originalBuild, originalContainer := ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer
t.Cleanup(func() {
ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer = originalExists, originalBuild, originalContainer
})
ContainerNewContainer = func(*container.NewContainerInput) container.ExecutionsEnvironment { return fakeContainer{} }
var removed bool
fakeDockerDaemon(t, func(writer http.ResponseWriter, request *http.Request) {
if request.Method == http.MethodDelete {
removed = true
_, _ = io.WriteString(writer, `[]`)
} else {
_, _ = io.WriteString(writer, `{"Architecture":"aarch64"}`)
}
})
for _, testcase := range []struct {
name, platform string
exists, matches, rebuild bool
wantBuild, wantRemove bool
wantChecks []string
}{
{name: "missing image", platform: "linux/arm64", wantBuild: true, wantChecks: []string{"any"}},
{name: "unspecified platform", exists: true, matches: true, wantChecks: []string{"any"}},
{name: "any platform", platform: "any", exists: true, matches: true, wantChecks: []string{"any"}},
{name: "matching platform", platform: "linux/arm64", exists: true, matches: true, wantChecks: []string{"any", "linux/arm64"}},
{name: "different platform", platform: "linux/arm64", exists: true, wantBuild: true, wantRemove: true, wantChecks: []string{"any", "linux/arm64"}},
{name: "forced rebuild", exists: true, matches: true, rebuild: true, wantBuild: true, wantChecks: []string{"any"}},
} {
t.Run(testcase.name, func(t *testing.T) {
var checks []string
ContainerImageExistsLocally = func(_ctx context.Context, _image, platform string) (bool, error) {
checks = append(checks, platform)
return testcase.exists && (platform == "any" || testcase.matches), nil
}
var built bool
removed = false
ContainerNewDockerBuildExecutor = func(container.NewDockerBuildExecutorInput) common.Executor {
return func(context.Context) error { built = true; return nil }
}
step := &stepActionRemote{
Step: &model.Step{ID: "action"},
RunContext: &RunContext{
Config: &Config{
PresetGitHubContext: &model.GithubContext{},
ContainerArchitecture: testcase.platform,
ForceRebuild: testcase.rebuild,
},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{},
},
action: &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "Dockerfile"}},
env: map[string]string{},
}
dir := filepath.Join(actionRoot, testcase.name)
require.NoError(t, execAsDocker(t.Context(), step, "action", dir, dir, false, stepStageMain))
assert.Equal(t, testcase.wantChecks, checks)
assert.Equal(t, testcase.wantBuild, built)
assert.Equal(t, testcase.wantRemove, removed)
})
}
} }
func TestDockerActionImageTag(t *testing.T) { func TestDockerActionImageTag(t *testing.T) {
@@ -468,15 +619,20 @@ func TestDockerActionImageTag(t *testing.T) {
func TestExecAsDockerStageEntrypoint(t *testing.T) { func TestExecAsDockerStageEntrypoint(t *testing.T) {
orig := ContainerNewContainer orig := ContainerNewContainer
defer func() { ContainerNewContainer = orig }() defer func() { ContainerNewContainer = orig }()
fakeDockerDaemon(t, func(writer http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(writer, `{"Architecture":"x86_64"}`)
})
for _, tc := range []struct { for _, tc := range []struct {
name string name string
stage stepStage stage stepStage
with map[string]string with map[string]string
inputs map[string]model.Input
runs model.ActionRuns runs model.ActionRuns
env map[string]string env map[string]string
wantCmd []string wantCmd []string
wantEntrypoint []string wantEntrypoint []string
warning string
}{ }{
{ {
name: "main stage prefers manifest values", name: "main stage prefers manifest values",
@@ -491,17 +647,36 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
wantEntrypoint: []string{"main.sh"}, wantEntrypoint: []string{"main.sh"},
}, },
{ {
name: "main stage uses caller fallbacks", name: "main stage uses caller fallbacks in any case and warns about undeclared inputs",
stage: stepStageMain, stage: stepStageMain,
with: map[string]string{"args": "caller --flag", "entrypoint": "input.sh --verbose"}, with: map[string]string{"ARGS": "caller --flag", "entryPoint": "input.sh --verbose", "unknown": "value"},
inputs: map[string]model.Input{"message": {}},
runs: model.ActionRuns{Env: map[string]string{"ACTION_ONLY": "manifest"}}, runs: model.ActionRuns{Env: map[string]string{"ACTION_ONLY": "manifest"}},
wantCmd: []string{"caller", "--flag"}, wantCmd: []string{"caller", "--flag"},
wantEntrypoint: []string{"input.sh", "--verbose"}, wantEntrypoint: []string{"input.sh", "--verbose"},
warning: "##[warning]Unexpected input(s) 'unknown', valid inputs are ['entryPoint', 'args', 'message']",
},
{
name: "manifest args keep caller input quotes literal",
stage: stepStageMain,
with: map[string]string{"args": "it's literal"},
runs: model.ActionRuns{Args: []string{"${{ inputs.args }}"}},
env: map[string]string{"INPUT_ARGS": "it's literal"},
wantCmd: []string{"it's literal"},
},
{
name: "caller input skips its default expression",
stage: stepStageMain,
with: map[string]string{"MESSAGE": "provided"},
inputs: map[string]model.Input{"message": {Default: "${{ fromJSON('') }}"}},
runs: model.ActionRuns{Args: []string{"${{ inputs.message }}"}},
env: map[string]string{"INPUT_MESSAGE": "provided"},
wantCmd: []string{"provided"},
}, },
{ {
name: "explicit empty manifest args suppress caller args", name: "explicit empty manifest args suppress caller args",
stage: stepStageMain, stage: stepStageMain,
with: map[string]string{"args": "caller"}, with: map[string]string{"args": "it's ignored"},
runs: model.ActionRuns{Args: []string{}}, runs: model.ActionRuns{Args: []string{}},
wantCmd: []string{}, wantCmd: []string{},
}, },
@@ -527,6 +702,8 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
tc.runs.Using, tc.runs.Image = "docker", "docker://node:14" tc.runs.Using, tc.runs.Image = "docker", "docker://node:14"
cm := &containerMock{} cm := &containerMock{}
var input *container.NewContainerInput var input *container.NewContainerInput
@@ -538,12 +715,13 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
step := &stepActionRemote{ step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "org/action@v1", With: tc.with}, Step: &model.Step{ID: "1", Uses: "org/action@v1", With: tc.with},
RunContext: &RunContext{ RunContext: &RunContext{
Config: &Config{}, Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}}, Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: cm, JobContainer: cm,
}, },
action: &model.Action{Runs: tc.runs}, action: &model.Action{Inputs: tc.inputs, Runs: tc.runs},
env: mergeMaps(tc.env), env: mergeMaps(tc.env),
remoteAction: mustNewRemoteAction(t, "org/action@v1"),
} }
cm.On("Pull", false).Return(func(context.Context) error { return nil }) cm.On("Pull", false).Return(func(context.Context) error { return nil })
@@ -552,7 +730,17 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
cm.On("Start", true).Return(func(context.Context) error { return nil }) cm.On("Start", true).Return(func(context.Context) error { return nil })
cm.On("Close").Return(func(context.Context) error { return nil }) cm.On("Close").Return(func(context.Context) error { return nil })
require.NoError(t, execAsDocker(t.Context(), step, "action", t.TempDir(), t.TempDir(), false, tc.stage)) if tc.stage == stepStageMain {
require.NoError(t, runActionImpl(step, "", step.remoteAction)(ctx))
} else {
require.NoError(t, execAsDocker(ctx, step, "action", "", "", false, tc.stage))
}
if tc.warning == "" {
assert.Empty(t, hook.AllRecords())
} else {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, tc.warning, hook.LastRecord().Message)
}
require.NotNil(t, input) require.NotNil(t, input)
assert.Equal(t, tc.wantCmd, input.Cmd) assert.Equal(t, tc.wantCmd, input.Cmd)
assert.Equal(t, tc.wantEntrypoint, input.Entrypoint) assert.Equal(t, tc.wantEntrypoint, input.Entrypoint)
+4
View File
@@ -101,9 +101,13 @@ func TestMainStepsExecutorRunsAlwaysStepsAfterInterruption(t *testing.T) {
assert.Equal(t, []string{"step1", "always-step"}, ran) assert.Equal(t, []string{"step1", "always-step"}, ran)
require.NoError(t, laterStepCtxErr) require.NoError(t, laterStepCtxErr)
assert.Equal(t, wantError == context.Canceled, rc.jobCancelled) assert.Equal(t, wantError == context.Canceled, rc.jobCancelled)
setJobResult(t.Context(), rc, rc, true)
if wantError == context.DeadlineExceeded { if wantError == context.DeadlineExceeded {
assert.True(t, rc.jobFailed) assert.True(t, rc.jobFailed)
assert.Equal(t, "failure", rc.getJobContext().Status) assert.Equal(t, "failure", rc.getJobContext().Status)
assert.Equal(t, "failure", rc.Run.Job().Result)
} else {
assert.Equal(t, "cancelled", rc.Run.Job().Result)
} }
}) })
} }
+24 -3
View File
@@ -8,6 +8,8 @@ import (
"cmp" "cmp"
"context" "context"
"fmt" "fmt"
"maps"
"os"
"strings" "strings"
"unicode" "unicode"
@@ -158,11 +160,18 @@ const allowUnsecureCommandsVar = "ACTIONS_ALLOW_UNSECURE_COMMANDS"
// not run, recording the error that fails the step. GitHub disabled both because a step that // not run, recording the error that fails the step. GitHub disabled both because a step that
// echoes untrusted content can use them to set NODE_OPTIONS or PATH for every later step. // echoes untrusted content can use them to set NODE_OPTIONS or PATH for every later step.
func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string) bool { func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string) bool {
if isTruthyEnv(rc.currentStepEnv()[allowUnsecureCommandsVar]) || if isTruthyEnv(os.Getenv(allowUnsecureCommandsVar)) { // the runner's own opt-in wins, as on GitHub
isTruthyEnv(rc.Env[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.GlobalEnv[allowUnsecureCommandsVar]) {
return false return false
} }
caseInsensitive := rc.JobContainer != nil && rc.JobContainer.IsEnvironmentCaseInsensitive()
for _, env := range []map[string]string{rc.currentStepEnv(), rc.Env, rc.GlobalEnv} {
if value, ok := common.LookupFold(env, allowUnsecureCommandsVar, caseInsensitive); ok {
if isTruthyEnv(value) {
return false
}
break
}
}
// The step executor logs the failure itself, so keep this line's wording distinct. // The step executor logs the failure itself, so keep this line's wording distinct.
common.Logger(ctx).With(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(fmt.Sprintf( common.Logger(ctx).With(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(fmt.Sprintf(
@@ -192,6 +201,18 @@ func (rc *RunContext) takeUnsecureCommandError() error {
return err return err
} }
// runWithCommands runs executor judging its workflow commands by env, failing it on a refused one.
func (rc *RunContext) runWithCommands(ctx context.Context, env map[string]string, executor common.Executor) error {
rc.setCurrentStepEnv(maps.Clone(env)) // the executor keeps writing to env while the command handler reads it
defer rc.setCurrentStepEnv(nil)
_ = rc.takeUnsecureCommandError() // a refusal from before belongs to no step
err := executor(ctx)
if refusal := rc.takeUnsecureCommandError(); err == nil {
return refusal
}
return err
}
func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string, fromCommand bool) { func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string, fromCommand bool) {
name := kvPairs["name"] name := kvPairs["name"]
if strings.EqualFold(name, "NODE_OPTIONS") { if strings.EqualFold(name, "NODE_OPTIONS") {
+18 -7
View File
@@ -136,21 +136,28 @@ func TestEscapeCommandData(t *testing.T) {
func TestUnsecureCommands(t *testing.T) { func TestUnsecureCommands(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
jobEnv map[string]string jobEnv map[string]string
stepEnv map[string]string globalEnv map[string]string
optedIn bool stepEnv map[string]string
processEnv string
caseInsensitive bool
optedIn bool
}{ }{
{name: "refused with no opt-in"}, {name: "refused with no opt-in"},
// GitHub reads the opt-in with bool.TryParse, so "1" is not one.
{name: "refused for a value bool.TryParse rejects", jobEnv: map[string]string{allowUnsecureCommandsVar: "1"}}, {name: "refused for a value bool.TryParse rejects", jobEnv: map[string]string{allowUnsecureCommandsVar: "1"}},
{name: "opted in through the step environment", stepEnv: map[string]string{allowUnsecureCommandsVar: "true"}, optedIn: true}, {name: "opted in through the step environment", stepEnv: map[string]string{allowUnsecureCommandsVar: "true"}, optedIn: true},
{name: "opted in through the job environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "TRUE"}, optedIn: true}, {name: "opted in through the job environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "TRUE"}, optedIn: true},
{name: "step opt-out overrides the job and global environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "true"}, globalEnv: map[string]string{allowUnsecureCommandsVar: "true"}, stepEnv: map[string]string{allowUnsecureCommandsVar: "false"}},
{name: "the runner's own opt-in overrides a step opt-out", stepEnv: map[string]string{allowUnsecureCommandsVar: "false"}, processEnv: "true", optedIn: true},
{name: "a differently cased opt-in counts where env names are case-insensitive", jobEnv: map[string]string{"Actions_Allow_Unsecure_Commands": "true"}, caseInsensitive: true, optedIn: true},
{name: "a differently cased opt-in is ignored where env names are case-sensitive", jobEnv: map[string]string{"Actions_Allow_Unsecure_Commands": "true"}},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Setenv(allowUnsecureCommandsVar, tt.processEnv)
a := assert.New(t) a := assert.New(t)
rc := &RunContext{Config: &Config{}, Env: tt.jobEnv} rc := &RunContext{Config: &Config{}, Env: tt.jobEnv, GlobalEnv: tt.globalEnv, JobContainer: fakeContainer{caseInsensitive: tt.caseInsensitive}}
rc.setCurrentStepEnv(tt.stepEnv) rc.setCurrentStepEnv(tt.stepEnv)
handler := rc.commandHandler(t.Context()) handler := rc.commandHandler(t.Context())
@@ -160,7 +167,6 @@ func TestUnsecureCommands(t *testing.T) {
if !tt.optedIn { if !tt.optedIn {
a.Empty(rc.Env["x"]) a.Empty(rc.Env["x"])
a.Empty(rc.ExtraPath) a.Empty(rc.ExtraPath)
// The refusal fails the step that produced it, once.
require.ErrorContains(t, rc.takeUnsecureCommandError(), "set-env") require.ErrorContains(t, rc.takeUnsecureCommandError(), "set-env")
a.NoError(rc.takeUnsecureCommandError()) a.NoError(rc.takeUnsecureCommandError())
return return
@@ -187,6 +193,11 @@ func TestCommandLogWriter(t *testing.T) {
log []string log []string
masks []string masks []string
}{ }{
{
name: "mask commands decode modern and legacy escapes once",
steps: [][]string{{"::add-mask::%25", "##[add-mask]alpha beta%3Bgamma%5Ddelta%250A"}},
masks: []string{"%", "alpha beta;gamma]delta%0A"},
},
{ {
name: "applied commands are hidden unless echoed", name: "applied commands are hidden unless echoed",
steps: [][]string{{ steps: [][]string{{
+9 -1
View File
@@ -16,10 +16,18 @@ import (
var noopExecutor = func(context.Context) error { return nil } var noopExecutor = func(context.Context) error { return nil }
type linuxEnvironment struct {
container.LinuxContainerEnvironmentExtensions
}
func (linuxEnvironment) GetRunnerContext(context.Context) map[string]any {
return map[string]any{"os": "Linux", "arch": "X64", "temp": "/tmp", "tool_cache": container.DefaultToolCache}
}
type containerMock struct { type containerMock struct {
mock.Mock mock.Mock
container.Container container.Container
container.LinuxContainerEnvironmentExtensions linuxEnvironment
} }
func (cm *containerMock) Create(capAdd, capDrop []string) common.Executor { func (cm *containerMock) Create(capAdd, capDrop []string) common.Executor {
+2 -1
View File
@@ -1,7 +1,7 @@
// Copyright 2026 The Gitea Authors. All rights reserved. // Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT // SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd)) //go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd || freebsd))
package runner package runner
@@ -49,6 +49,7 @@ func TestDockerProxyMountedJob(t *testing.T) {
ContainerNamePrefix: resourceName, ContainerNamePrefix: resourceName,
ContainerDaemonSocket: dockerClient.DaemonHost(), ContainerDaemonSocket: dockerClient.DaemonHost(),
ContainerMaxLifetime: 2 * time.Minute, ContainerMaxLifetime: 2 * time.Minute,
Sweep: true,
ContainerOptions: "-v /usr/local/bin/docker:/usr/local/bin/docker:ro -v /usr/local/libexec/docker/cli-plugins:/usr/local/libexec/docker/cli-plugins:ro", ContainerOptions: "-v /usr/local/bin/docker:/usr/local/bin/docker:ro -v /usr/local/libexec/docker/cli-plugins:/usr/local/libexec/docker/cli-plugins:ro",
ValidVolumes: []string{"/usr/local/bin/docker", "/usr/local/libexec/docker/cli-plugins"}, ValidVolumes: []string{"/usr/local/bin/docker", "/usr/local/libexec/docker/cli-plugins"},
Env: map[string]string{"PROXY_TEST_RESOURCE": resourceName, "PROXY_TEST_MODE": mode, "PROXY_TEST_IMAGE": baseImage, "COMPOSE_PROJECT_NAME": resourceName}, Env: map[string]string{"PROXY_TEST_RESOURCE": resourceName, "PROXY_TEST_MODE": mode, "PROXY_TEST_IMAGE": baseImage, "COMPOSE_PROJECT_NAME": resourceName},
+39 -19
View File
@@ -7,9 +7,9 @@ package runner
import ( import (
"bytes" "bytes"
"context" "context"
"errors"
"fmt" "fmt"
"maps" "maps"
"path"
"reflect" "reflect"
"regexp" "regexp"
"strings" "strings"
@@ -89,18 +89,18 @@ var hashfiles string
// NewStepExpressionEvaluator creates a new evaluator with the `inputs` of the enclosing workflow or composite action // NewStepExpressionEvaluator creates a new evaluator with the `inputs` of the enclosing workflow or composite action
func (rc *RunContext) NewStepExpressionEvaluator(ctx context.Context, step step) *ExpressionEvaluator { func (rc *RunContext) NewStepExpressionEvaluator(ctx context.Context, step step) *ExpressionEvaluator {
return rc.newStepExpressionEvaluator(ctx, step, rc.actionInputs, rc.getJobContext()) return rc.newStepExpressionEvaluator(ctx, step, rc.actionInputs, rc.getJobContext(), getStepGithubContext(ctx, step))
} }
// NewActionInputsExpressionEvaluator creates a new evaluator with the step's own with: values as `inputs` // NewActionInputsExpressionEvaluator creates a new evaluator with the step's own with: values as `inputs`
func (rc *RunContext) NewActionInputsExpressionEvaluator(ctx context.Context, step step) *ExpressionEvaluator { func (rc *RunContext) NewActionInputsExpressionEvaluator(ctx context.Context, step step) *ExpressionEvaluator {
return rc.newStepExpressionEvaluator(ctx, step, inputsFromEnv(*step.getEnv()), rc.getJobContext()) return rc.newStepExpressionEvaluator(ctx, step, inputsFromEnv(*step.getEnv()), rc.getJobContext(), getStepGithubContext(ctx, step))
} }
func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step, stepInputs map[string]any, jobContext *model.JobContext) *ExpressionEvaluator { func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step, stepInputs map[string]any, jobContext *model.JobContext, github *model.GithubContext) *ExpressionEvaluator {
// todo: cleanup EvaluationEnvironment creation // todo: cleanup EvaluationEnvironment creation
ee := &exprparser.EvaluationEnvironment{ ee := &exprparser.EvaluationEnvironment{
Github: step.getGithubContext(ctx), Github: github,
Env: *step.getEnv(), Env: *step.getEnv(),
Job: jobContext, Job: jobContext,
Steps: rc.StepResults, Steps: rc.StepResults,
@@ -111,7 +111,7 @@ func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step,
Needs: exprparser.NeedsContext(rc.Run), Needs: exprparser.NeedsContext(rc.Run),
// todo: should be unavailable // todo: should be unavailable
// but required to interpolate/evaluate the inputs in actions/composite // but required to interpolate/evaluate the inputs in actions/composite
Inputs: getEvaluatorInputs(rc, stepInputs, rc.getGithubContext(ctx)), Inputs: getEvaluatorInputs(rc, stepInputs, github),
HashFiles: getHashFilesFunction(ctx, rc), HashFiles: getHashFilesFunction(ctx, rc),
} }
ee.Runner = rc.getRunnerContext(ctx) ee.Runner = rc.getRunnerContext(ctx)
@@ -127,7 +127,7 @@ func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step,
func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.Value) (any, error) { func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.Value) (any, error) {
return func(v []reflect.Value) (any, error) { return func(v []reflect.Value) (any, error) {
if rc.JobContainer != nil { if rc.JobContainer != nil {
timeed, cancel := context.WithTimeout(ctx, time.Minute) timeoutCtx, cancel := context.WithTimeout(ctx, 120*time.Second)
defer cancel() defer cancel()
name := "workflow/hashfiles/index.js" name := "workflow/hashfiles/index.js"
hout := &bytes.Buffer{} hout := &bytes.Buffer{}
@@ -154,20 +154,30 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
} }
stdout, stderr := rc.JobContainer.ReplaceLogWriter(hout, herr) stdout, stderr := rc.JobContainer.ReplaceLogWriter(hout, herr)
_ = rc.JobContainer.Copy(rc.JobContainer.GetActPath(), &container.FileEntry{ defer rc.JobContainer.ReplaceLogWriter(stdout, stderr)
err := rc.JobContainer.Copy(rc.JobContainer.GetActPath(), &container.FileEntry{
Name: name, Name: name,
Mode: 0o644, Mode: 0o644,
Body: hashfiles, Body: hashfiles,
}). }).Then(rc.JobContainer.Exec([]string{"node", container.JoinPath(rc.JobContainer.GetActPath(), name)}, env, "", ""))(timeoutCtx)
Then(rc.JobContainer.Exec([]string{"node", path.Join(rc.JobContainer.GetActPath(), name)}, quotedPatterns := expressionStringEscape(strings.Join(patterns, ", "))
env, "", "")). switch {
Finally(func(context.Context) error { case err == nil:
rc.JobContainer.ReplaceLogWriter(stdout, stderr) case ctx.Err() != nil:
return nil return "", ctx.Err()
})(timeed) case timeoutCtx.Err() != nil || errors.Is(err, context.DeadlineExceeded):
if _, output, found := strings.Cut(hout.String()+"\n"+herr.String(), "__OUTPUT__"); found { return "", fmt.Errorf("hashFiles('%s') couldn't finish within 120 seconds", quotedPatterns)
if hash, _, found := strings.Cut(output, "__OUTPUT__"); found { case hashFilesNodeMissing(err):
return hash, nil common.Logger(ctx).With(rawOutputField, true).Warnf("##[warning]hashFiles could not run because node is unavailable in the job environment. Returning an empty string.")
return "", nil
default:
return "", fmt.Errorf("hashFiles('%s') failed. Fail to hash files under directory '%s'", quotedPatterns, rc.githubWorkspace())
}
for line := range strings.Lines(hout.String() + "\n" + herr.String()) {
if hash, ok := strings.CutPrefix(strings.TrimRight(line, "\r\n"), "__OUTPUT__"); ok {
if hash, ok = strings.CutSuffix(hash, "__OUTPUT__"); ok {
return hash, nil
}
} }
} }
} }
@@ -175,6 +185,11 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
} }
} }
func hashFilesNodeMissing(err error) bool {
exitErr, ok := errors.AsType[container.ExitCodeError](err)
return ok && (exitErr == 126 || exitErr == 127) || err.Error() == "Cannot find: node in PATH"
}
type expressionEvaluator struct { type expressionEvaluator struct {
interpreter exprparser.Interpreter interpreter exprparser.Interpreter
} }
@@ -316,5 +331,10 @@ func getWorkflowSecrets(rc *RunContext) map[string]string {
if rc.caller != nil { if rc.caller != nil {
return rc.workflowCallSecrets return rc.workflowCallSecrets
} }
return rc.Config.Secrets if rc.runtimeToken == "" {
return rc.Config.Secrets
}
secrets := map[string]string{"ACTIONS_RUNTIME_TOKEN": rc.runtimeToken} // as the daemon exposes its token
maps.Copy(secrets, rc.Config.Secrets)
return secrets
} }
+113 -10
View File
@@ -5,9 +5,20 @@
package runner package runner
import ( import (
"bytes"
"context"
"errors"
"fmt"
"io"
"reflect"
"strings" "strings"
"sync" "sync"
"testing" "testing"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/exprparser" "gitea.dev/actionslib/pkg/exprparser"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
@@ -22,10 +33,11 @@ func createRunContext(t *testing.T) *RunContext {
"os": {"Linux", "Windows"}, "os": {"Linux", "Windows"},
"foo": {"bar", "baz"}, "foo": {"bar", "baz"},
}) })
assert.NoError(t, err) require.NoError(t, err)
return &RunContext{ return &RunContext{
Config: &Config{ Config: &Config{
Env: testCheckoutEnv(),
Workdir: ".", Workdir: ".",
Secrets: map[string]string{ Secrets: map[string]string{
"CASE_INSENSITIVE_SECRET": "value", "CASE_INSENSITIVE_SECRET": "value",
@@ -144,6 +156,92 @@ func TestEvaluateRunContext(t *testing.T) {
} }
} }
type hashFilesContainer struct {
fakeContainer
stdout, stderr io.Writer
copyErr error
execute func([]string, map[string]string, string, string) common.Executor
}
func (c *hashFilesContainer) Copy(_destPath string, _files ...*container.FileEntry) common.Executor {
return common.NewErrorExecutor(c.copyErr)
}
func (c *hashFilesContainer) Exec(command []string, env map[string]string, user, workdir string) common.Executor {
return c.execute(command, env, user, workdir)
}
func (c *hashFilesContainer) ReplaceLogWriter(stdout, stderr io.Writer) (io.Writer, io.Writer) {
previousOut, previousErr := c.stdout, c.stderr
c.stdout, c.stderr = stdout, stderr
return previousOut, previousErr
}
func TestGetHashFilesFunction(t *testing.T) {
const failureMessage = "hashFiles('dir''file, *.sum') failed. Fail to hash files under directory '/workspace'"
for _, test := range []struct {
name string
output string
execErr error
copyErr error
hash string
err string
warning bool
expired bool
}{
{name: "success", output: "patterns __OUTPUT__wrong__OUTPUT__ echoed\n__OUTPUT__digest__OUTPUT__", hash: "digest"},
{name: "no matches", output: "__OUTPUT____OUTPUT__"},
{name: "script failure", output: "Cannot find: node in PATH\n__OUTPUT__digest__OUTPUT__", execErr: container.ExitCodeError(1), err: failureMessage},
{name: "timeout", execErr: context.DeadlineExceeded, err: "hashFiles('dir''file, *.sum') couldn't finish within 120 seconds"},
{name: "job deadline", execErr: context.DeadlineExceeded, expired: true, err: context.DeadlineExceeded.Error()},
{name: "node missing", execErr: fmt.Errorf("wrapped: %w", container.ExitCodeError(127)), warning: true},
{name: "node not executable", execErr: container.ExitCodeError(126), warning: true},
{name: "host node missing", execErr: errors.New("Cannot find: node in PATH"), warning: true},
{name: "copy failure", copyErr: errors.New("open /runner/cache: permission denied"), err: failureMessage},
} {
t.Run(test.name, func(t *testing.T) {
logger, hook := logtest.NewLogger()
stdout, stderr := &bytes.Buffer{}, &bytes.Buffer{}
jobContainer := &hashFilesContainer{stdout: stdout, stderr: stderr, copyErr: test.copyErr}
jobContainer.execute = func(command []string, env map[string]string, user, workdir string) common.Executor {
assert.Equal(t, []string{"node", "/var/run/act/workflow/hashfiles/index.js"}, command)
assert.Equal(t, map[string]string{"patterns": "dir'file\n*.sum", "followSymbolicLinks": "true"}, env)
assert.Empty(t, user)
assert.Empty(t, workdir)
return func(_ctx context.Context) error {
_, err := io.WriteString(jobContainer.stderr, test.output)
require.NoError(t, err)
return test.execErr
}
}
ctx := common.WithLogger(t.Context(), logger)
if test.expired {
var cancel context.CancelFunc
ctx, cancel = context.WithDeadline(ctx, time.Now())
defer cancel()
}
result, err := getHashFilesFunction(ctx, &RunContext{
Config: &Config{Workdir: "/workspace"},
JobContainer: jobContainer,
})([]reflect.Value{reflect.ValueOf("--follow-symbolic-links"), reflect.ValueOf("dir'file"), reflect.ValueOf("*.sum")})
if test.err == "" {
require.NoError(t, err)
} else {
require.EqualError(t, err, test.err)
}
assert.Equal(t, test.hash, result)
assert.Same(t, stdout, jobContainer.stdout)
assert.Same(t, stderr, jobContainer.stderr)
if test.warning {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, "##[warning]hashFiles could not run because node is unavailable in the job environment. Returning an empty string.", hook.LastRecord().Message)
} else {
assert.Empty(t, hook.AllRecords())
}
})
}
}
func TestEvaluateStep(t *testing.T) { func TestEvaluateStep(t *testing.T) {
rc := createRunContext(t) rc := createRunContext(t)
rc.Env["INPUT_FORGED"] = "leaked" rc.Env["INPUT_FORGED"] = "leaked"
@@ -182,6 +280,7 @@ func TestEvaluateStep(t *testing.T) {
func TestInterpolate(t *testing.T) { func TestInterpolate(t *testing.T) {
rc := &RunContext{ rc := &RunContext{
Config: &Config{ Config: &Config{
Env: testCheckoutEnv(),
Workdir: ".", Workdir: ".",
Secrets: map[string]string{ Secrets: map[string]string{
"CASE_INSENSITIVE_SECRET": "value", "CASE_INSENSITIVE_SECRET": "value",
@@ -313,11 +412,10 @@ func TestGetEvaluatorInputsBoolean(t *testing.T) {
} }
for _, eventName := range []string{"workflow_call", "workflow_dispatch"} { for _, eventName := range []string{"workflow_call", "workflow_dispatch"} {
wf, err := model.ReadWorkflow(strings.NewReader(workflow(eventName)))
require.NoError(t, err)
for _, table := range tables { for _, table := range tables {
t.Run(eventName+"/"+table.name, func(t *testing.T) { t.Run(eventName+"/"+table.name, func(t *testing.T) {
wf, err := model.ReadWorkflow(strings.NewReader(workflow(eventName)))
require.NoError(t, err)
rc := &RunContext{ rc := &RunContext{
Config: &Config{Workdir: "."}, Config: &Config{Workdir: "."},
Run: &model.Run{JobID: "job1", Workflow: wf}, Run: &model.Run{JobID: "job1", Workflow: wf},
@@ -347,9 +445,9 @@ jobs:
for _, value := range []string{"true", "false"} { for _, value := range []string{"true", "false"} {
t.Run(value, func(t *testing.T) { t.Run(value, func(t *testing.T) {
t.Parallel() t.Parallel()
parent, err := (&runnerImpl{config: &Config{}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "call"}, map[string]any{"args": `{"flag":` + value + `,"name":"${{ 'runner' }}"}`}) parent, err := (&runnerImpl{config: &Config{PresetGitHubContext: &model.GithubContext{}}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "call"}, map[string]any{"args": `{"flag":` + value + `,"name":"${{ 'runner' }}"}`})
require.NoError(t, err) require.NoError(t, err)
child, err := (&runnerImpl{config: &Config{}, caller: &caller{runContext: parent}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "test"}, nil) child, err := (&runnerImpl{config: &Config{PresetGitHubContext: &model.GithubContext{}}, caller: &caller{runContext: parent}}).newRunContext(t.Context(), &model.Run{Workflow: workflow, JobID: "test"}, nil)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, map[string]any{"flag": value == "true", "name": "${{ 'runner' }}", "unset": false}, child.workflowCallInputs) assert.Equal(t, map[string]any{"flag": value == "true", "name": "${{ 'runner' }}", "unset": false}, child.workflowCallInputs)
assert.Equal(t, rawWith, job.RawWith) assert.Equal(t, rawWith, job.RawWith)
@@ -405,7 +503,7 @@ jobs:
assert.Equal(t, "workflow", env["INPUT_FALLBACK"]) assert.Equal(t, "workflow", env["INPUT_FALLBACK"])
} }
func TestResolveWorkflowCallToleratesMismatchesAndPassesInheritedSecretsVerbatim(t *testing.T) { func TestResolveWorkflowCallToleratesMismatchesAndPassesInheritedSecretsAndRuntimeToken(t *testing.T) {
callerWorkflow, err := model.ReadWorkflow(strings.NewReader(` callerWorkflow, err := model.ReadWorkflow(strings.NewReader(`
jobs: jobs:
call: call:
@@ -428,7 +526,11 @@ jobs:
require.NoError(t, err) require.NoError(t, err)
notCallable, err := model.ReadWorkflow(strings.NewReader("on: push\njobs:\n test: {}\n")) notCallable, err := model.ReadWorkflow(strings.NewReader("on: push\njobs:\n test: {}\n"))
require.NoError(t, err) require.NoError(t, err)
config := &Config{Secrets: map[string]string{"token": "${{ github.token }}"}} config := &Config{
Secrets: map[string]string{"token": "${{ github.token }}"},
PresetGitHubContext: &model.GithubContext{},
RuntimeTokens: map[*model.Workflow]string{callerWorkflow: "caller-run"},
}
resolve := func(callerJobID string, workflow *model.Workflow) *RunContext { resolve := func(callerJobID string, workflow *model.Workflow) *RunContext {
parent, err := (&runnerImpl{config: config}).newRunContext(t.Context(), &model.Run{Workflow: callerWorkflow, JobID: callerJobID}, nil) parent, err := (&runnerImpl{config: config}).newRunContext(t.Context(), &model.Run{Workflow: callerWorkflow, JobID: callerJobID}, nil)
require.NoError(t, err) require.NoError(t, err)
@@ -440,9 +542,10 @@ jobs:
called := resolve("call", callee) called := resolve("call", callee)
assert.Equal(t, map[string]any{"flag": true}, called.workflowCallInputs) assert.Equal(t, map[string]any{"flag": true}, called.workflowCallInputs)
assert.Equal(t, map[string]string{"undeclared": "${{ github.token }}"}, called.workflowCallSecrets) assert.Equal(t, map[string]string{"undeclared": "${{ github.token }}"}, called.workflowCallSecrets)
assert.Equal(t, "caller-run", called.GetEnv()["ACTIONS_RUNTIME_TOKEN"])
inherited := resolve("inherit", notCallable) inherited := resolve("inherit", notCallable)
assert.Empty(t, inherited.workflowCallInputs) assert.Empty(t, inherited.workflowCallInputs)
assert.Equal(t, config.Secrets, inherited.workflowCallSecrets) assert.Equal(t, map[string]string{"token": "${{ github.token }}", "ACTIONS_RUNTIME_TOKEN": "caller-run"}, inherited.workflowCallSecrets)
} }
func TestJobNameMasksSecrets(t *testing.T) { func TestJobNameMasksSecrets(t *testing.T) {
@@ -455,7 +558,7 @@ jobs:
`)) `))
require.NoError(t, err) require.NoError(t, err)
runner := &runnerImpl{config: &Config{Secrets: map[string]string{"A": "s3cr3t-a", "B": "s3cr3t-b"}}} runner := &runnerImpl{config: &Config{Secrets: map[string]string{"A": "s3cr3t-a", "B": "s3cr3t-b"}, PresetGitHubContext: &model.GithubContext{}}}
containerName := func(jobID string) string { containerName := func(jobID string) string {
rc, err := runner.newRunContext(t.Context(), &model.Run{JobID: jobID, Workflow: workflow}, nil) rc, err := runner.newRunContext(t.Context(), &model.Run{JobID: jobID, Workflow: workflow}, nil)
require.NoError(t, err) require.NoError(t, err)
+4
View File
@@ -60,3 +60,7 @@ func requireHostTools(t *testing.T, tools ...string) {
} }
} }
} }
func testCheckoutEnv() map[string]string {
return map[string]string{"GITHUB_REPOSITORY": "owner/repo", "GITHUB_REF": "refs/heads/main", "SHA_REF": "1234567890"}
}
+44 -20
View File
@@ -23,6 +23,7 @@ import (
"sync" "sync"
"time" "time"
"unicode" "unicode"
"unicode/utf8"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging" "gitea.com/gitea/runner/act/common/logging"
@@ -30,6 +31,8 @@ import (
"gitea.dev/actionslib/pkg/exprparser" "gitea.dev/actionslib/pkg/exprparser"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
unicodeencoding "golang.org/x/text/encoding/unicode"
"golang.org/x/text/transform"
) )
const ( const (
@@ -111,7 +114,7 @@ func printPrepareActions(rc *RunContext, preparers []actionPreparer) common.Exec
if !ok || strings.Contains(action.Step.Uses, "${{") || action.action != nil { if !ok || strings.Contains(action.Step.Uses, "${{") || action.action != nil {
continue continue
} }
github := action.getGithubContext(common.WithLogger(ctx, logging.New(logging.DiscardHandler))) github := action.RunContext.getGithubContext(common.WithLogger(ctx, logging.New(logging.DiscardHandler)))
remote, err := newRemoteAction(action.Step.Uses, github, action.RunContext.Config.GitHubInstance) remote, err := newRemoteAction(action.Step.Uses, github, action.RunContext.Config.GitHubInstance)
if err != nil || action.skipsLocalCheckout(remote, github) { if err != nil || action.skipsLocalCheckout(remote, github) {
continue continue
@@ -285,7 +288,7 @@ func newJobExecutor(info jobInfo, sf stepFactory, rc *RunContext) common.Executo
jobError := common.JobError(ctx) jobError := common.JobError(ctx)
var err error var err error
// always allow 1 min for stopping and removing the runner, even if we were cancelled // always allow 1 min for stopping and removing the runner, even if we were cancelled
ctx, cancel := context.WithTimeout(common.WithLogger(context.Background(), common.Logger(ctx)), jobCleanupTimeout) ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), jobCleanupTimeout)
defer cancel() defer cancel()
logger := common.Logger(ctx) logger := common.Logger(ctx)
@@ -400,6 +403,7 @@ func runMainStepsAfterInterrupt(ctx context.Context, rc *RunContext, steps []com
func setJobResult(ctx context.Context, info jobInfo, rc *RunContext, success bool) { func setJobResult(ctx context.Context, info jobInfo, rc *RunContext, success bool) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
success = success && !rc.jobFailed
// Matrix combinations share one *model.Job and run in parallel; serialize the // Matrix combinations share one *model.Job and run in parallel; serialize the
// read-modify-write of the job result so a failing combination is not lost-updated by a // read-modify-write of the job result so a failing combination is not lost-updated by a
@@ -460,7 +464,7 @@ func evaluateJobEnv(ctx context.Context, rc *RunContext) (map[string]string, err
} }
} }
if workflowEnv != nil { if workflowEnv != nil {
evaluated = mergeMaps(workflowEnv, env) evaluated = rc.mergeEnv(workflowEnv, env)
evaluator = rc.NewExpressionEvaluatorWithEnv(ctx, evaluated) evaluator = rc.NewExpressionEvaluatorWithEnv(ctx, evaluated)
} }
var err error var err error
@@ -568,14 +572,34 @@ func tryUploadJobSummary(ctx context.Context, rc *RunContext) {
actPath := rc.JobContainer.GetActPath() actPath := rc.JobContainer.GetActPath()
// Reuse a single client across all step uploads so connections can be pooled. // Reuse a single client across all step uploads so connections can be pooled.
client := &http.Client{Timeout: jobSummaryUploadRequestTimeout, Transport: rc.Config.instanceTransportFor(base)} client := &http.Client{Timeout: jobSummaryUploadRequestTimeout, Transport: rc.Config.instanceTransportFor(base)}
maskers := rc.secretMaskers()
longestMask := 0
for i := 0; i < len(maskers); i += 2 {
longestMask = max(longestMask, len(maskers[i]))
}
masker := NewSecretReplacer(maskers)
limit := maxJobSummaryBytes + longestMask
for i := range rc.Run.Job().Steps { for i := range rc.Run.Job().Steps {
summaryPath := path.Join(actPath, "workflow", "step-summary-"+strconv.Itoa(i)+".md") summaryPath := container.JoinPath(actPath, "workflow", "step-summary-"+strconv.Itoa(i)+".md")
body, ok := readSingleFileFromContainerArchive(ctx, rc.JobContainer, summaryPath, maxJobSummaryBytes) body, ok := readSingleFileFromContainerArchive(ctx, rc.JobContainer, summaryPath, int64(limit+1))
if !ok || len(body) == 0 { if !ok || len(body) == 0 {
continue continue
} }
// Gitea renders summaries on the run page, so mask before the upload. masked := masker.Replace(string(body))
uploadJobSummary(ctx, client, base+strconv.Itoa(i)+"/summary", runtimeToken, []byte(rc.maskSecrets(string(body)))) split := len(body) > limit
if split { // the read can end inside a secret, so keep only what masks alike without the lookahead
unsplit := masker.Replace(string(body[:maxJobSummaryBytes]))
kept := 0
for kept < len(unsplit) && kept < len(masked) && unsplit[kept] == masked[kept] {
kept++
}
for kept > 0 && kept < len(masked) && !utf8.RuneStart(masked[kept]) {
kept--
}
masked = masked[:kept]
}
body = truncateJobSummary(ctx, []byte(masked), summaryPath, maxJobSummaryBytes, split)
uploadJobSummary(ctx, client, base+strconv.Itoa(i)+"/summary", runtimeToken, body)
} }
} }
@@ -689,26 +713,26 @@ func readSingleFileFromContainerArchive(ctx context.Context, env container.Execu
if entryName != requestedPath && entryName != path.Base(requestedPath) { if entryName != requestedPath && entryName != path.Base(requestedPath) {
continue continue
} }
// Summaries larger than the limit are truncated rather than dropped, so the b, err := io.ReadAll(io.LimitReader(transform.NewReader(tr, unicodeencoding.BOMOverride(unicodeencoding.UTF8.NewDecoder())), maxBytes))
// user still gets the leading content (mirroring how GitHub caps oversized
// step summaries instead of discarding them). Read one extra byte so an
// over-limit file is detected from the actual stream rather than trusting
// header.Size, then cap the returned content at maxBytes.
b, err := io.ReadAll(io.LimitReader(tr, maxBytes+1))
if err != nil { if err != nil {
return nil, false return nil, false
} }
if int64(len(b)) > maxBytes {
// Reserve room for the marker so the marked-up result still fits in maxBytes.
marker := []byte(jobSummaryTruncationMarker)
keep := max(maxBytes-int64(len(marker)), 0)
b = append(b[:keep], marker...)
common.Logger(ctx).Warnf("job summary truncated: path=%s max=%d", p, maxBytes)
}
return b, true return b, true
} }
} }
func truncateJobSummary(ctx context.Context, body []byte, summaryPath string, maxBytes int, split bool) []byte {
if !split && len(body) <= maxBytes {
return body
}
keep := max(min(len(body), maxBytes-len(jobSummaryTruncationMarker)), 0)
for keep > 0 && keep < len(body) && !utf8.RuneStart(body[keep]) {
keep--
}
common.Logger(ctx).Warnf("job summary truncated: path=%s max=%d", summaryPath, maxBytes)
return append(body[:keep], jobSummaryTruncationMarker...)
}
// useJobStageLogger tags the log entries of a job level executor with the stage they belong to. // useJobStageLogger tags the log entries of a job level executor with the stage they belong to.
func useJobStageLogger(stage string, executor common.Executor) common.Executor { func useJobStageLogger(stage string, executor common.Executor) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
+108 -68
View File
@@ -34,6 +34,8 @@ import (
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
yaml "go.yaml.in/yaml/v4" yaml "go.yaml.in/yaml/v4"
"golang.org/x/text/encoding"
"golang.org/x/text/encoding/unicode"
) )
func TestJobExecutor(t *testing.T) { func TestJobExecutor(t *testing.T) {
@@ -102,7 +104,7 @@ func (jim *jobInfoMock) result(result string) {
type jobContainerMock struct { type jobContainerMock struct {
container.Container container.Container
container.LinuxContainerEnvironmentExtensions linuxEnvironment
} }
func (jcm *jobContainerMock) ReplaceLogWriter(_, _ io.Writer) (io.Writer, io.Writer) { func (jcm *jobContainerMock) ReplaceLogWriter(_, _ io.Writer) (io.Writer, io.Writer) {
@@ -290,7 +292,7 @@ func TestNewJobExecutorDownloadsAllActionsBeforeTheFirstStep(t *testing.T) {
Jobs: map[string]*model.Job{"test": {}}, Jobs: map[string]*model.Job{"test": {}},
}, },
}, },
Config: &Config{}, Config: &Config{PresetGitHubContext: &model.GithubContext{}},
} }
rc.ExprEval = rc.NewExpressionEvaluator(ctx) rc.ExprEval = rc.NewExpressionEvaluator(ctx)
@@ -550,7 +552,7 @@ func TestNewJobExecutor(t *testing.T) {
}, },
}, },
}, },
Config: &Config{}, Config: &Config{PresetGitHubContext: &model.GithubContext{}},
} }
if tt.output != "" { if tt.output != "" {
require.NoError(t, rc.Run.Job().RawOutputs.Encode(map[string]string{"bad": tt.output})) require.NoError(t, rc.Run.Job().RawOutputs.Encode(map[string]string{"bad": tt.output}))
@@ -695,6 +697,7 @@ func TestSetJobResultMatrixContinueOnError(t *testing.T) {
workflow := &model.Workflow{Name: "workflow1", Jobs: map[string]*model.Job{"job1": job}} workflow := &model.Workflow{Name: "workflow1", Jobs: map[string]*model.Job{"job1": job}}
for _, experimental := range []bool{true, tolerated} { for _, experimental := range []bool{true, tolerated} {
rc := newTestRC(workflow, map[string]any{"experimental": experimental}) rc := newTestRC(workflow, map[string]any{"experimental": experimental})
rc.Config.PresetGitHubContext = &model.GithubContext{}
setJobResult(ctx, rc, rc, false) setJobResult(ctx, rc, rc, false)
} }
@@ -931,49 +934,42 @@ func TestExtractJobIDFromRuntimeToken(t *testing.T) {
assert.Equal(t, int64(0), extractJobIDFromRuntimeToken("")) assert.Equal(t, int64(0), extractJobIDFromRuntimeToken(""))
} }
func TestReadSingleFileFromContainerArchiveFindsMatchingRegularFile(t *testing.T) { func TestReadSingleFileFromContainerArchive(t *testing.T) {
ctx := t.Context() retainedBytes := 200 - len(jobSummaryTruncationMarker)
cm := &containerMock{}
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t,
tarEntry{name: "workflow", typeflag: tar.TypeDir},
tarEntry{name: "other.md", body: "wrong"},
tarEntry{name: "SUMMARY.md", body: "right"},
))),
nil,
).Once()
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", 1024)
assert.True(t, ok)
assert.Equal(t, []byte("right"), body)
cm.AssertExpectations(t)
}
func TestReadSingleFileFromContainerArchiveSizeLimit(t *testing.T) {
for name, tt := range map[string]struct { for name, tt := range map[string]struct {
content string content string
maxBytes int64 maxBytes int64
want string want string
}{ }{
"truncates oversized summary": {strings.Repeat("a", 300), 200, strings.Repeat("a", 200-len(jobSummaryTruncationMarker)) + jobSummaryTruncationMarker}, "truncates oversized summary without reading entire file": {strings.Repeat("a", 100000), 200, strings.Repeat("a", 200-len(jobSummaryTruncationMarker)) + jobSummaryTruncationMarker},
"keeps exact limit": {"abc", 3, "abc"}, "keeps exact limit": {"abc", 3, "abc"},
"strips UTF-8 BOM before limit": {"\xef\xbb\xbfabc", 3, "abc"},
"decodes UTF-16 before limit": {"\xff\xfea\x00b\x00c\x00", 3, "abc"},
"limits expanded UTF-16": {"\xff\xfe" + strings.Repeat("\x2d\x4e", 150), 200, strings.Repeat("中", retainedBytes/3) + jobSummaryTruncationMarker},
"keeps complete UTF-8 characters": {strings.Repeat("a", retainedBytes-1) + "€" + strings.Repeat("a", 200), 200, strings.Repeat("a", retainedBytes-1) + jobSummaryTruncationMarker},
} { } {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
logger, hook := logtest.NewLogger() logger, hook := logtest.NewLogger()
ctx := common.WithLogger(t.Context(), logger) ctx := common.WithLogger(t.Context(), logger)
cm := &containerMock{} cm := &containerMock{}
archive := bytes.NewReader(tarArchive(t,
tarEntry{name: "workflow", typeflag: tar.TypeDir},
tarEntry{name: "other.md", body: "wrong"},
tarEntry{name: "SUMMARY.md", body: tt.content},
))
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return( cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "SUMMARY.md", body: tt.content}))), io.NopCloser(archive),
nil, nil,
).Once() ).Once()
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", tt.maxBytes) body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", tt.maxBytes+1)
assert.LessOrEqual(t, int64(len(body)), tt.maxBytes+1)
body = truncateJobSummary(ctx, body, "/var/run/act/workflow/SUMMARY.md", int(tt.maxBytes), false)
assert.True(t, ok) assert.True(t, ok)
assert.Equal(t, []byte(tt.want), body) assert.Equal(t, []byte(tt.want), body)
assert.LessOrEqual(t, int64(len(body)), tt.maxBytes) assert.LessOrEqual(t, int64(len(body)), tt.maxBytes)
if int64(len(tt.content)) > tt.maxBytes { if strings.HasSuffix(tt.want, jobSummaryTruncationMarker) {
if assert.Len(t, hook.AllRecords(), 1) { if assert.Len(t, hook.AllRecords(), 1) {
assert.Contains(t, hook.LastRecord().Message, "job summary truncated") assert.Contains(t, hook.LastRecord().Message, "job summary truncated")
} }
@@ -997,6 +993,7 @@ func tarArchive(t *testing.T, entries ...tarEntry) []byte {
buf := &bytes.Buffer{} buf := &bytes.Buffer{}
tw := tar.NewWriter(buf) tw := tar.NewWriter(buf)
for _, entry := range entries { for _, entry := range entries {
buf.Grow(len(entry.body) + 3*512)
typeflag := entry.typeflag typeflag := entry.typeflag
if typeflag == 0 { if typeflag == 0 {
typeflag = tar.TypeReg typeflag = tar.TypeReg
@@ -1013,7 +1010,7 @@ func tarArchive(t *testing.T, entries ...tarEntry) []byte {
} }
require.NoError(t, tw.WriteHeader(header)) require.NoError(t, tw.WriteHeader(header))
if typeflag == tar.TypeReg { if typeflag == tar.TypeReg {
_, err := tw.Write([]byte(entry.body)) _, err := io.CopyN(tw, strings.NewReader(entry.body), int64(len(entry.body)))
require.NoError(t, err) require.NoError(t, err)
} }
} }
@@ -1023,7 +1020,7 @@ func tarArchive(t *testing.T, entries ...tarEntry) []byte {
func newTestRC(wf *model.Workflow, matrix map[string]any) *RunContext { func newTestRC(wf *model.Workflow, matrix map[string]any) *RunContext {
return &RunContext{ return &RunContext{
Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }}, Config: &Config{Workdir: ".", Env: testCheckoutEnv(), PlatformPicker: func([]string) string { return "ubuntu-latest" }},
StepResults: map[string]*model.StepResult{}, StepResults: map[string]*model.StepResult{},
Env: map[string]string{}, Env: map[string]string{},
Matrix: matrix, Matrix: matrix,
@@ -1039,29 +1036,32 @@ func makeTestRC(t *testing.T, jobYAML string) *RunContext {
Name: "workflow1", Name: "workflow1",
Jobs: map[string]*model.Job{"job1": job}, Jobs: map[string]*model.Job{"job1": job},
}, nil) }, nil)
rc.Config.PresetGitHubContext = &model.GithubContext{}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context()) rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
return rc return rc
} }
func TestApplyJobTimeout(t *testing.T) { func TestApplyJobTimeout(t *testing.T) {
cases := []struct { cases := []struct {
name string name string
yaml string yaml string
wantTimeout bool want time.Duration
}{ }{
{"empty", "runs-on: ubuntu-latest", false}, {"empty", "runs-on: ubuntu-latest", 0},
{"integer", "timeout-minutes: 5\nruns-on: ubuntu-latest", true}, {"integer", "timeout-minutes: 5\nruns-on: ubuntu-latest", 5 * time.Minute},
{"zero ignored", "timeout-minutes: 0\nruns-on: ubuntu-latest", false}, {"zero ignored", "timeout-minutes: 0\nruns-on: ubuntu-latest", 0},
{"non-numeric ignored", "timeout-minutes: abc\nruns-on: ubuntu-latest", false}, {"non-numeric ignored", "timeout-minutes: abc\nruns-on: ubuntu-latest", 0},
} }
for _, tc := range cases { for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
rc := makeTestRC(t, tc.yaml) rc := makeTestRC(t, tc.yaml)
ctx := t.Context() newCtx, cancel := applyJobTimeout(t.Context(), rc, rc.Run.Job())
newCtx, cancel := applyJobTimeout(ctx, rc, rc.Run.Job())
defer cancel() defer cancel()
_, hasDeadline := newCtx.Deadline() deadline, hasDeadline := newCtx.Deadline()
assert.Equal(t, tc.wantTimeout, hasDeadline) assert.Equal(t, tc.want != 0, hasDeadline)
if hasDeadline {
assert.WithinDuration(t, time.Now().Add(tc.want), deadline, time.Minute)
}
}) })
} }
} }
@@ -1109,35 +1109,75 @@ func TestJobSetContinueOnError(t *testing.T) {
} }
func TestTryUploadJobSummaryMasksSecrets(t *testing.T) { func TestTryUploadJobSummaryMasksSecrets(t *testing.T) {
var got string const prefix = "deployed true with s3cr3t and runtime-added via pr0xypw 😀"
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { const maskedPrefix = "deployed true with *** and *** via *** 😀"
body, err := io.ReadAll(r.Body) padding := strings.Repeat("a", maxJobSummaryBytes-len(jobSummaryTruncationMarker)-len(prefix)-2)
assert.NoError(t, err) content := prefix + padding + "s3cr3t" + strings.Repeat("中", 100)
got = string(body) want := maskedPrefix + padding + "***" + strings.Repeat("中", (len(prefix)-len(maskedPrefix)-1)/3) + jobSummaryTruncationMarker
w.WriteHeader(http.StatusNoContent) longSecret := "PREFIX" + strings.Repeat("xy", 100) + "Z"
})) readLimit := maxJobSummaryBytes + base64.StdEncoding.EncodedLen(len(longSecret)) + 1
defer server.Close() shrinkingPrefix := strings.Repeat(longSecret, (readLimit-len(longSecret)+1)/len(longSecret))
for name, testcase := range map[string]struct {
encoding encoding.Encoding
content, want string
masks []string
}{
"UTF-8": {unicode.UTF8, content, want, nil},
"UTF-8 BOM": {unicode.UTF8BOM, content, want, nil},
"UTF-16LE BOM": {unicode.UTF16(unicode.LittleEndian, unicode.UseBOM), content, want, nil},
"UTF-16BE BOM": {unicode.UTF16(unicode.BigEndian, unicode.UseBOM), content, want, nil},
"secret split by the read": {unicode.UTF8, strings.Repeat("s3cr3t", maxJobSummaryBytes/3), "", nil},
"split read keeps every masked secret before the limit": {
unicode.UTF8,
"# public\n" + strings.Repeat(longSecret, readLimit/len(longSecret)+1),
"# public\n" + strings.Repeat("***", (maxJobSummaryBytes-len("# public\n"))/len(longSecret)) + jobSummaryTruncationMarker,
[]string{longSecret},
},
"nested secret expands the split tail": {
unicode.UTF8,
shrinkingPrefix + strings.Repeat("*", readLimit-len(shrinkingPrefix)-len(longSecret)+1) + longSecret,
"",
[]string{"xy", longSecret},
},
} {
t.Run(name, func(t *testing.T) {
var got string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
got = string(body)
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
cm := &containerMock{} content, err := testcase.encoding.NewEncoder().String(testcase.content)
cm.On("GetContainerArchive", mock.Anything, "/var/run/act/workflow/step-summary-0.md").Return( require.NoError(t, err)
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{ cm := &containerMock{}
name: "step-summary-0.md", body: "deployed true with s3cr3t and runtime-added via pr0xypw", cm.On("GetContainerArchive", mock.Anything, "/var/run/act/workflow/step-summary-0.md").Return(
}))), io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{
nil, name: "step-summary-0.md", body: content,
).Once() }))),
nil,
).Once()
rc := newJobSummaryRC(map[string]string{ rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": "job-summary", "GITEA_ACTIONS_CAPABILITIES": "job-summary",
"ACTIONS_RUNTIME_URL": server.URL, "ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": fakeRuntimeToken(34), "ACTIONS_RUNTIME_TOKEN": fakeRuntimeToken(34),
"GITEA_RUN_ID": "12", "GITEA_RUN_ID": "12",
}, cm, 1) }, cm, 1)
rc.Config.Secrets = map[string]string{"TOK": "s3cr3t", "ACTIONS_STEP_DEBUG": "true"} rc.Config.Secrets = map[string]string{"TOK": "s3cr3t", "ACTIONS_STEP_DEBUG": "true"}
rc.Config.ExtraMasks = []string{"pr0xypw"} rc.Config.ExtraMasks = append([]string{"pr0xypw"}, testcase.masks...)
rc.Masks = []string{"runtime-added"} rc.Masks = []string{"runtime-added"}
tryUploadJobSummary(t.Context(), rc) tryUploadJobSummary(t.Context(), rc)
assert.Equal(t, "deployed true with *** and *** via ***", got) if testcase.want == "" {
cm.AssertExpectations(t) assert.Equal(t, jobSummaryTruncationMarker, strings.TrimLeft(got, "*"))
} else {
assert.Equal(t, testcase.want, got)
}
cm.AssertExpectations(t)
})
}
} }
+17 -15
View File
@@ -40,28 +40,31 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
if hookPath == "" { if hookPath == "" {
return nil return nil
} }
previousStep := rc.CurrentStep
rc.CurrentStep = ""
defer func() { rc.CurrentStep = previousStep }()
cmd, shell := hookCommand(hookPath) cmd, shell := hookCommand(hookPath)
rawLogger := common.Logger(ctx).With(rawOutputField, true) rawLogger := common.Logger(ctx).With(rawOutputField, true)
defer rawLogger.Infof("::endgroup::")
rawLogger.Infof("::group::Run '%s'", EscapeCommandData(hookPath))
rawLogger.Infof("A %s hook has been configured by the runner administrator", name) rawLogger.Infof("A %s hook has been configured by the runner administrator", name)
rawLogger.Infof("::group::Run '%s'", EscapeCommandData(hookPath))
if shell != "" { if shell != "" {
rawLogger.Infof("shell: %s", shell) rawLogger.Infof("shell: %s", shell)
} }
rawLogger.Infof("::endgroup::")
env := mergeMaps(rc.containerSpec.Env, rc.GetEnv()) env := rc.mergeEnv(rc.caEnv, rc.containerSpec.Env, rc.GetEnv())
rc.withGithubEnv(ctx, rc.getGithubContext(ctx), env) rc.withGithubEnv(ctx, rc.getGithubContext(ctx), env)
rc.ApplyExtraPath(ctx, &env) rc.ApplyExtraPath(ctx, &env)
err := rc.setupHookFileCommands(ctx, env) err := rc.setupHookFileCommands(ctx, env)
if err == nil { if err == nil {
err = rc.withCommandLogWriter(ctx, rc.JobContainer.Exec(cmd, env, "", "")) err = rc.runWithCommands(ctx, env, func(ctx context.Context) error {
} return rc.withCommandLogWriter(ctx, rc.JobContainer.Exec(cmd, env, "", ""))
// Processed even on failure, so a hook that exports what it managed to set up before })
// failing still hands it to the job. if processErr := rc.processHookFileCommands(ctx); err == nil { // a failed hook still exports what it set up
if processErr := rc.processHookFileCommands(ctx); err == nil { err = processErr
err = processErr }
} }
if err == nil { if err == nil {
return nil return nil
@@ -79,10 +82,9 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
// the first one wrote. // the first one wrote.
func (rc *RunContext) setupHookFileCommands(ctx context.Context, env map[string]string) error { func (rc *RunContext) setupHookFileCommands(ctx context.Context, env map[string]string) error {
actPath := rc.JobContainer.GetActPath() actPath := rc.JobContainer.GetActPath()
env["GITHUB_ENV"] = path.Join(actPath, hookEnvFileCommand) env["GITHUB_ENV"] = container.JoinPath(actPath, hookEnvFileCommand)
env["GITHUB_PATH"] = path.Join(actPath, hookPathFileCommand) env["GITHUB_PATH"] = container.JoinPath(actPath, hookPathFileCommand)
env["GITEA_ENV"] = env["GITHUB_ENV"] aliasGiteaEnv(env)
env["GITEA_PATH"] = env["GITHUB_PATH"]
return rc.JobContainer.Copy(actPath, return rc.JobContainer.Copy(actPath,
&container.FileEntry{Name: hookEnvFileCommand, Mode: 0o666}, &container.FileEntry{Name: hookEnvFileCommand, Mode: 0o666},
@@ -92,7 +94,7 @@ func (rc *RunContext) setupHookFileCommands(ctx context.Context, env map[string]
func (rc *RunContext) processHookFileCommands(ctx context.Context) error { func (rc *RunContext) processHookFileCommands(ctx context.Context) error {
err := processRunnerEnvFileCommand(ctx, hookEnvFileCommand, rc, rc.setEnvFile) err := processRunnerEnvFileCommand(ctx, hookEnvFileCommand, rc, rc.setEnvFile)
if pathErr := rc.UpdateExtraPath(ctx, path.Join(rc.JobContainer.GetActPath(), hookPathFileCommand)); pathErr != nil && err == nil { if pathErr := rc.UpdateExtraPath(ctx, container.JoinPath(rc.JobContainer.GetActPath(), hookPathFileCommand)); pathErr != nil && err == nil {
err = pathErr err = pathErr
} }
return err return err
@@ -107,7 +109,7 @@ func hookCommand(hookPath string) (cmd []string, shell string) {
case ".sh": case ".sh":
return []string{"bash", "-e", hookPath}, "bash -e {0}" return []string{"bash", "-e", hookPath}, "bash -e {0}"
case ".ps1": case ".ps1":
return []string{"pwsh", "-command", ". '" + hookPath + "'"}, `pwsh -command ". '{0}'"` return []string{"pwsh", "-command", ". '" + expressionStringEscape(hookPath) + "'"}, `pwsh -command ". '{0}'"`
default: default:
return []string{hookPath}, "" return []string{hookPath}, ""
} }
+75 -6
View File
@@ -14,6 +14,7 @@ import (
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging" "gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest" "gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -27,6 +28,7 @@ type hookContainer struct {
cmd []string cmd []string
env map[string]string env map[string]string
err error err error
copyErr error
envFile map[string]string envFile map[string]string
pathTar []byte pathTar []byte
output string output string
@@ -58,6 +60,10 @@ func (c *hookContainer) ReplaceLogWriter(stdout, _ io.Writer) (io.Writer, io.Wri
return old, nil return old, nil
} }
func (c *hookContainer) Copy(_destPath string, _files ...*container.FileEntry) common.Executor {
return common.NewErrorExecutor(c.copyErr)
}
func (c *hookContainer) UpdateFromEnv(_ string, env *map[string]string) common.Executor { func (c *hookContainer) UpdateFromEnv(_ string, env *map[string]string) common.Executor {
return func(context.Context) error { return func(context.Context) error {
maps.Copy(*env, c.envFile) maps.Copy(*env, c.envFile)
@@ -72,7 +78,7 @@ func (c *hookContainer) GetContainerArchive(context.Context, string) (io.ReadClo
// newHookRunContext returns a RunContext and the context to run a hook with, whose logger is // newHookRunContext returns a RunContext and the context to run a hook with, whose logger is
// silenced so the hook's job-log output does not reach the test output. // silenced so the hook's job-log output does not reach the test output.
func newHookRunContext(jobContainer *hookContainer, config *Config) (*RunContext, context.Context) { func newHookRunContext(jobContainer *hookContainer, config *Config) (*RunContext, context.Context) {
// Env is left nil so that it is built from the config, as it is for a real job. config.PresetGitHubContext = &model.GithubContext{}
rc := &RunContext{ rc := &RunContext{
Config: config, Config: config,
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}}, Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
@@ -85,15 +91,29 @@ func newHookRunContext(jobContainer *hookContainer, config *Config) (*RunContext
} }
func TestRunJobHook(t *testing.T) { func TestRunJobHook(t *testing.T) {
t.Run("runs the hook with the job environment", func(t *testing.T) { t.Run("runs the hook with the job environment after its closed header group", func(t *testing.T) {
jobContainer := &hookContainer{} jobContainer := &hookContainer{output: "from the hook\n"}
rc, ctx := newHookRunContext(jobContainer, &Config{ rc, ctx := newHookRunContext(jobContainer, &Config{
JobStartedHook: "/hooks/started.sh", JobStartedHook: "/hooks/started.sh",
Env: map[string]string{"A_VAR": "value", jobStartedHookEnv: "/from/env.sh"}, Env: map[string]string{"A_VAR": "value", jobStartedHookEnv: "/from/env.sh"},
}) })
records := logtest.New()
ctx = common.WithLogger(ctx, logging.New(records))
require.NoError(t, rc.runJobStartedHook(ctx)) require.NoError(t, rc.runJobStartedHook(ctx))
var lines []string
for _, record := range records.AllRecords() {
lines = append(lines, record.Message)
}
assert.Equal(t, []string{
"A job started hook has been configured by the runner administrator",
"::group::Run '/hooks/started.sh'",
"shell: bash -e {0}",
"::endgroup::",
"from the hook\n",
}, lines)
// The setting wins over the environment variable. // The setting wins over the environment variable.
assert.Equal(t, []string{"bash", "-e", "/hooks/started.sh"}, jobContainer.cmd) assert.Equal(t, []string{"bash", "-e", "/hooks/started.sh"}, jobContainer.cmd)
assert.Equal(t, "value", jobContainer.env["A_VAR"]) assert.Equal(t, "value", jobContainer.env["A_VAR"])
@@ -101,6 +121,7 @@ func TestRunJobHook(t *testing.T) {
assert.Equal(t, "job", jobContainer.env["GITHUB_JOB"]) assert.Equal(t, "job", jobContainer.env["GITHUB_JOB"])
assert.Equal(t, "/var/run/act/workflow/hook-envs.txt", jobContainer.env["GITHUB_ENV"]) assert.Equal(t, "/var/run/act/workflow/hook-envs.txt", jobContainer.env["GITHUB_ENV"])
assert.Equal(t, "/var/run/act/workflow/hook-path.txt", jobContainer.env["GITHUB_PATH"]) assert.Equal(t, "/var/run/act/workflow/hook-path.txt", jobContainer.env["GITHUB_PATH"])
assert.Equal(t, jobContainer.env["GITHUB_PATH"], jobContainer.env["GITEA_PATH"])
}) })
t.Run("each hook has its own command state", func(t *testing.T) { t.Run("each hook has its own command state", func(t *testing.T) {
@@ -116,6 +137,44 @@ func TestRunJobHook(t *testing.T) {
assert.Same(t, jobWriter, jobContainer.stdout) assert.Same(t, jobWriter, jobContainer.stdout)
}) })
t.Run("fails on refused commands and honors the container opt-in", func(t *testing.T) {
for _, allow := range []bool{false, true} {
jobContainer := &hookContainer{output: "::set-env name=FROM_HOOK::1\n::add-path::/opt/hook/bin"}
rc, ctx := newHookRunContext(jobContainer, &Config{JobStartedHook: "/started.sh"})
if allow {
rc.containerSpec.Env = map[string]string{allowUnsecureCommandsVar: "true"}
}
err := rc.runJobStartedHook(ctx)
if allow {
require.NoError(t, err)
assert.Equal(t, "1", rc.Env["FROM_HOOK"])
assert.Equal(t, []string{"/opt/hook/bin"}, rc.ExtraPath)
} else {
require.ErrorContains(t, err, "workflow command is disabled")
assert.Equal(t, "failure", rc.getJobContext().Status)
require.ErrorContains(t, common.JobError(ctx), "workflow command is disabled")
assert.NotContains(t, rc.Env, "FROM_HOOK")
assert.Empty(t, rc.ExtraPath)
}
assert.Nil(t, rc.currentStepEnv())
require.NoError(t, rc.takeUnsecureCommandError())
}
})
t.Run("the completed hook has no preceding step identity", func(t *testing.T) {
jobContainer := &hookContainer{output: "::set-output name=value::after\n::save-state name=value::after"}
rc, ctx := newHookRunContext(jobContainer, &Config{JobCompletedHook: "/completed.sh"})
rc.CurrentStep = "last-step"
rc.StepResults = map[string]*model.StepResult{"last-step": {Outputs: map[string]string{"value": "before"}}}
require.NoError(t, rc.runJobCompletedHook(ctx))
assert.Empty(t, jobContainer.env["GITHUB_ACTION"])
assert.Equal(t, "before", rc.StepResults["last-step"].Outputs["value"])
assert.Empty(t, rc.IntraActionState)
assert.Equal(t, "last-step", rc.CurrentStep)
})
// Each hook reads its own variable, so a swapped constant cannot pass. // Each hook reads its own variable, so a swapped constant cannot pass.
t.Run("falls back to the GitHub environment variables", func(t *testing.T) { t.Run("falls back to the GitHub environment variables", func(t *testing.T) {
for name, hook := range map[string]struct { for name, hook := range map[string]struct {
@@ -161,6 +220,15 @@ func TestRunJobHook(t *testing.T) {
require.ErrorContains(t, common.JobError(ctx), "boom") require.ErrorContains(t, common.JobError(ctx), "boom")
}) })
t.Run("a hook whose file commands cannot be reset neither runs nor exports stale ones", func(t *testing.T) {
jobContainer := &hookContainer{copyErr: errors.New("boom"), envFile: map[string]string{"STALE": "1"}}
rc, ctx := newHookRunContext(jobContainer, &Config{JobStartedHook: "/hooks/started.sh"})
require.ErrorContains(t, rc.runJobStartedHook(ctx), "boom")
assert.Nil(t, jobContainer.cmd)
assert.NotContains(t, rc.Env, "STALE")
})
t.Run("is a no-op without a hook", func(t *testing.T) { t.Run("is a no-op without a hook", func(t *testing.T) {
jobContainer := &hookContainer{} jobContainer := &hookContainer{}
rc, ctx := newHookRunContext(jobContainer, &Config{}) rc, ctx := newHookRunContext(jobContainer, &Config{})
@@ -178,9 +246,10 @@ func TestHookCommand(t *testing.T) {
cmd []string cmd []string
shell string shell string
}{ }{
"/hooks/started.sh": {[]string{"bash", "-e", "/hooks/started.sh"}, "bash -e {0}"}, "/hooks/started.sh": {[]string{"bash", "-e", "/hooks/started.sh"}, "bash -e {0}"},
"/hooks/started.PS1": {[]string{"pwsh", "-command", ". '/hooks/started.PS1'"}, `pwsh -command ". '{0}'"`}, "/hooks/started.PS1": {[]string{"pwsh", "-command", ". '/hooks/started.PS1'"}, `pwsh -command ". '{0}'"`},
"/hooks/started": {[]string{"/hooks/started"}, ""}, "/hooks/started": {[]string{"/hooks/started"}, ""},
`C:\hooks\O'Brien\started.ps1`: {[]string{"pwsh", "-command", `. 'C:\hooks\O''Brien\started.ps1'`}, `pwsh -command ". '{0}'"`},
} { } {
cmd, shell := hookCommand(hookPath) cmd, shell := hookCommand(hookPath)
assert.Equal(t, want.cmd, cmd, hookPath) assert.Equal(t, want.cmd, cmd, hookPath)
+50 -11
View File
@@ -60,6 +60,8 @@ type masksContextKey string
const masksContextKeyVal = masksContextKey("logrus.FieldLogger") const masksContextKeyVal = masksContextKey("logrus.FieldLogger")
type valueMaskerContextKey struct{}
// Logger returns the appropriate logger for current context // Logger returns the appropriate logger for current context
func Masks(ctx context.Context) *[]string { func Masks(ctx context.Context) *[]string {
if masks, ok := ctx.Value(masksContextKeyVal).(*[]string); ok { if masks, ok := ctx.Value(masksContextKeyVal).(*[]string); ok {
@@ -88,6 +90,8 @@ func WithJobLoggerFactory(ctx context.Context, factory JobLoggerFactory) context
// WithJobLogger attaches a new logger to context that is aware of steps // WithJobLogger attaches a new logger to context that is aware of steps
func WithJobLogger(ctx context.Context, jobID, jobName string, config *Config, masks *[]string, matrix map[string]any) context.Context { func WithJobLogger(ctx context.Context, jobID, jobName string, config *Config, masks *[]string, matrix map[string]any) context.Context {
masker := valueMasker(config.InsecureSecrets, config.maskers())
ctx = context.WithValue(ctx, valueMaskerContextKey{}, masker)
ctx = WithMasks(ctx, masks) ctx = WithMasks(ctx, masks)
var output logging.Handler var output logging.Handler
@@ -109,7 +113,7 @@ func WithJobLogger(ctx context.Context, jobID, jobName string, config *Config, m
} }
handlers = append(handlers, &maskedHandler{ handlers = append(handlers, &maskedHandler{
Handler: output, Handler: output,
masker: valueMasker(config.InsecureSecrets, config.maskers()), masker: masker,
}) })
level := logging.LevelTrace level := logging.LevelTrace
@@ -140,6 +144,9 @@ func WithCompositeStepLogger(ctx context.Context, stepID string) context.Context
} }
func withStepLogger(ctx context.Context, stepNumber int, stepID, stepName, stageName string) context.Context { func withStepLogger(ctx context.Context, stepNumber int, stepID, stepName, stageName string) context.Context {
if masker, ok := ctx.Value(valueMaskerContextKey{}).(entryProcessor); ok {
stepName = masker(ctx, logging.Record{Message: stepName}).Message
}
stepIDs := []string{stepID} stepIDs := []string{stepID}
ctx = common.WithStepIDs(ctx, stepIDs) ctx = common.WithStepIDs(ctx, stepIDs)
@@ -256,29 +263,27 @@ func AppendSecretMaskers(oldnew []string, secrets map[string]string) []string {
return oldnew return oldnew
} }
// AppendSecretMasker registers v and each of its lines, as GitHub does. // AppendSecretMasker skips one-byte line fragments to avoid masking punctuation from structured secrets.
func AppendSecretMasker(oldnew []string, v string) []string { func AppendSecretMasker(oldnew []string, v string) []string {
ret := appendMaskedValue(oldnew, v) ret := appendMaskedValue(oldnew, v)
for l := range strings.FieldsFuncSeq(v, func(r rune) bool { return r == '\r' || r == '\n' }) { for l := range strings.FieldsFuncSeq(v, func(r rune) bool { return r == '\r' || r == '\n' }) {
ret = appendMaskedValue(ret, strings.TrimSpace(l)) if line := strings.TrimSpace(l); len(line) > 1 {
ret = appendMaskedValue(ret, line)
}
} }
return ret return ret
} }
// appendMaskedValue registers one value and every shape it takes on its way into a log.
func appendMaskedValue(ret []string, v string) []string { func appendMaskedValue(ret []string, v string) []string {
// formatted JSON secrets could otherwise mask {,[,],} everywhere if strings.TrimSpace(v) == "" || slices.Contains(ret, v) {
if len(strings.TrimSpace(v)) <= 1 || slices.Contains(ret, v) {
return ret return ret
} }
ret = append(ret, v, "***") ret = append(ret, v, "***")
// command data reaches the log escaped, so "pass%word" also arrives as "pass%25word"
if strings.ContainsAny(v, "%\r\n") { if strings.ContainsAny(v, "%\r\n") {
ret = append(ret, EscapeCommandData(v), "***") ret = append(ret, EscapeCommandData(v), "***")
} }
for _, encode := range secretValueEncoders { for _, encode := range secretValueEncoders {
encoded := encode(v) encoded := encode(v)
// An encoding that leaves the value unchanged is already masked above.
if encoded == v || len(encoded) <= 1 || slices.Contains(ret, encoded) { if encoded == v || len(encoded) <= 1 || slices.Contains(ret, encoded) {
continue continue
} }
@@ -312,7 +317,7 @@ func valueMasker(insecureSecrets bool, oldnew []string) entryProcessor {
masks := Masks(ctx) masks := Masks(ctx)
if len(*masks) == 0 { if len(*masks) == 0 {
r.Message = defReplacer.Replace(r.Message) r.Message = MaskLog(r.Message, defReplacer)
return r return r
} }
@@ -332,12 +337,42 @@ func valueMasker(insecureSecrets bool, oldnew []string) entryProcessor {
cmasker := replacer cmasker := replacer
mu.Unlock() mu.Unlock()
r.Message = cmasker.Replace(r.Message) r.Message = MaskLog(r.Message, cmasker)
return r return r
} }
} }
// MaskLog decodes command data before masking to cover partially escaped values.
func MaskLog(message string, replacer *strings.Replacer) string {
var command, data string
var found bool
unescape := UnescapeCommandData
if strings.HasPrefix(message, "##[") {
command, data, found = strings.Cut(message[3:], "]")
unescape = UnescapeLegacyCommand
} else if strings.HasPrefix(message, "::") {
command, data, found = strings.Cut(message[2:], "::")
if name, _, properties := strings.Cut(command, " "); properties {
switch name {
case "error", "warning", "notice", "debug":
command = name
}
}
}
switch command {
case "error", "warning", "notice", "debug", "group", "endgroup", "add-matcher":
if found {
trimmedData := strings.TrimRight(data, "\r\n")
normalized := message[:len(message)-len(data)] + EscapeCommandData(unescape(trimmedData)) + data[len(trimmedData):]
if masked := replacer.Replace(normalized); masked != normalized {
return masked
}
}
}
return replacer.Replace(message)
}
type maskedHandler struct { type maskedHandler struct {
logging.Handler logging.Handler
masker entryProcessor masker entryProcessor
@@ -472,11 +507,15 @@ func (f textFormatter) isColored() bool {
// such as a container name or a job summary. Masks added at runtime count, so a summary written // such as a container name or a job summary. Masks added at runtime count, so a summary written
// after ::add-mask:: is covered too. // after ::add-mask:: is covered too.
func (rc *RunContext) maskSecrets(value string) string { func (rc *RunContext) maskSecrets(value string) string {
return NewSecretReplacer(rc.secretMaskers()).Replace(value)
}
func (rc *RunContext) secretMaskers() []string {
oldnew := rc.Config.maskers() oldnew := rc.Config.maskers()
for _, mask := range rc.Masks { for _, mask := range rc.Masks {
oldnew = AppendSecretMasker(oldnew, mask) oldnew = AppendSecretMasker(oldnew, mask)
} }
return NewSecretReplacer(oldnew).Replace(value) return oldnew
} }
// maskers is every value this job's configuration says to hide, whatever the sink. // maskers is every value this job's configuration says to hide, whatever the sink.
+74 -5
View File
@@ -28,6 +28,7 @@ func TestValueMasker(t *testing.T) {
secrets map[string]string secrets map[string]string
masks []string masks []string
disallowed []string disallowed []string
want string
}{ }{
{ {
name: "Multiline Private Key", name: "Multiline Private Key",
@@ -49,6 +50,60 @@ func TestValueMasker(t *testing.T) {
secrets: map[string]string{"TOKEN": "pass%word"}, secrets: map[string]string{"TOKEN": "pass%word"},
disallowed: []string{"pass%25word"}, disallowed: []string{"pass%25word"},
}, },
{
name: "one-character mask",
lines: "%",
masks: []string{"%"},
want: "***",
},
{
name: "fully escaped legacy secret",
lines: "##[group]alpha beta%3Bgamma%5Ddelta",
secrets: map[string]string{"TOKEN": "alpha beta;gamma]delta"},
want: "##[group]***",
},
{
name: "partially escaped legacy mask",
lines: "##[group]alpha beta%3Bgamma]delta",
masks: []string{"alpha beta;gamma]delta"},
want: "##[group]***",
},
{
name: "partially escaped modern mask preserves unrelated escapes",
lines: "::group::alpha%25beta%gamma tail%250A%0A",
masks: []string{"alpha%beta%gamma"},
want: "::group::*** tail%250A%0A",
},
{
name: "command escapes are decoded once",
lines: "##[group]alpha%253Bbeta%5Dgamma",
masks: []string{"alpha;beta]gamma"},
want: "##[group]alpha%253Bbeta%5Dgamma",
},
{
name: "plain output preserves escapes",
lines: "alpha beta%3Bgamma%5Ddelta",
masks: []string{"alpha beta;gamma]delta"},
want: "alpha beta%3Bgamma%5Ddelta",
},
{
name: "command display preserves literal escapes around a mask",
lines: "##[command]alpha tail%3Bvalue%5D",
masks: []string{"alpha"},
want: "##[command]*** tail%3Bvalue%5D",
},
{
name: "unknown modern commands preserve literal escapes",
lines: "::unknown::alpha tail%25value",
masks: []string{"alpha"},
want: "::unknown::*** tail%25value",
},
{
name: "masks can span command headers and data",
lines: "::group::hidden other",
masks: []string{"group::hidden", "other"},
want: "::*** ***",
},
} }
for _, entry := range table { for _, entry := range table {
t.Run(entry.name, func(t *testing.T) { t.Run(entry.name, func(t *testing.T) {
@@ -56,6 +111,9 @@ func TestValueMasker(t *testing.T) {
masker := valueMasker(false, AppendSecretMaskers(nil, entry.secrets)) masker := valueMasker(false, AppendSecretMaskers(nil, entry.secrets))
for line := range strings.SplitSeq(entry.lines, "\n") { for line := range strings.SplitSeq(entry.lines, "\n") {
lentry := masker(ctx, logging.Record{Message: line}) lentry := masker(ctx, logging.Record{Message: line})
if entry.want != "" {
assert.Equal(t, entry.want, lentry.Message)
}
for _, line := range entry.disallowed { for _, line := range entry.disallowed {
assert.NotContains(t, lentry.Message, line) assert.NotContains(t, lentry.Message, line)
} }
@@ -139,6 +197,7 @@ func TestValueMaskerEncodedMasks(t *testing.T) {
{"encoded: " + base64.StdEncoding.EncodeToString([]byte(" s3cr3t ")), "encoded: ***"}, {"encoded: " + base64.StdEncoding.EncodeToString([]byte(" s3cr3t ")), "encoded: ***"},
{"encoded: " + base64.StdEncoding.EncodeToString([]byte("s3cr3t")), "encoded: ***"}, {"encoded: " + base64.StdEncoding.EncodeToString([]byte("s3cr3t")), "encoded: ***"},
{"encoded: " + base64.StdEncoding.EncodeToString([]byte("first")), "encoded: ***"}, {"encoded: " + base64.StdEncoding.EncodeToString([]byte("first")), "encoded: ***"},
{"##[group]s3cr3t value\r\n", "##[group]***\r\n"},
} { } {
entry := masker(WithMasks(t.Context(), &masks), logging.Record{Message: tc.line}) entry := masker(WithMasks(t.Context(), &masks), logging.Record{Message: tc.line})
assert.Equal(t, tc.want, entry.Message, tc.line) assert.Equal(t, tc.want, entry.Message, tc.line)
@@ -194,17 +253,15 @@ func TestValueMaskerCachedReplacerSeesNewMasks(t *testing.T) {
} }
func TestAppendSecretMaskerSkipsUselessEncodings(t *testing.T) { func TestAppendSecretMaskerSkipsUselessEncodings(t *testing.T) {
// A token with no character an escape would touch only gains its base64 forms:
// JSON, query and path escaping all leave it unchanged.
pairs := AppendSecretMasker(nil, "plaintoken") pairs := AppendSecretMasker(nil, "plaintoken")
assert.Equal(t, []string{ assert.Equal(t, []string{
"plaintoken", "***", "cGxhaW50b2tlbg==", "***", "bGFpbnRva2Vu", "***", "YWludG9rZW4=", "***", "plaintoken", "***", "cGxhaW50b2tlbg==", "***", "bGFpbnRva2Vu", "***", "YWludG9rZW4=", "***",
"aW50b2tl", "***", "YWludG9r", "***", "bGFpbnRv", "***", "aW50b2tl", "***", "YWludG9r", "***", "bGFpbnRv", "***",
}, pairs) }, pairs)
// Too short to mask. assert.Contains(t, AppendSecretMasker(nil, "x"), "x")
assert.Empty(t, AppendSecretMasker(nil, "x"))
assert.Empty(t, AppendSecretMasker(nil, " \t")) assert.Empty(t, AppendSecretMasker(nil, " \t"))
assert.NotContains(t, AppendSecretMasker(nil, "{\nvalue\n}"), "{")
assert.NotContains(t, AppendSecretMasker(nil, `"123456"`), "123456") assert.NotContains(t, AppendSecretMasker(nil, `"123456"`), "123456")
} }
@@ -255,7 +312,11 @@ func TestWithJobLoggerFansOutAndMasksOnlyOutput(t *testing.T) {
func TestJobLogJSONHandler(t *testing.T) { func TestJobLogJSONHandler(t *testing.T) {
buf := &bytes.Buffer{} buf := &bytes.Buffer{}
logger := logging.New(newJobLogJSONHandler(buf)).With("job", "build", "msg", "clash").WithError(errors.New("boom")) rc := &RunContext{Config: &Config{Secrets: map[string]string{"TOKEN": "s3cr3t-value"}}}
ctx := WithJobLoggerFactory(t.Context(), factoryFunc(func() logging.Handler { return newJobLogJSONHandler(buf) }))
ctx = WithJobLogger(ctx, "j1", "build", rc.Config, &rc.Masks, nil)
logger := common.Logger(withStepLogger(ctx, 1, "s1", "deploy s3cr3t-value", StageMain)).
With("msg", "clash").WithError(errors.New("boom"))
logger.Warnf("careful %d", 1) logger.Warnf("careful %d", 1)
var got map[string]any var got map[string]any
@@ -263,7 +324,15 @@ func TestJobLogJSONHandler(t *testing.T) {
assert.Equal(t, "careful 1", got["msg"]) assert.Equal(t, "careful 1", got["msg"])
assert.Equal(t, "warning", got["level"]) assert.Equal(t, "warning", got["level"])
assert.Equal(t, "build", got["job"]) assert.Equal(t, "build", got["job"])
assert.Equal(t, "deploy ***", got["step"])
assert.Equal(t, "boom", got["error"]) assert.Equal(t, "boom", got["error"])
assert.Equal(t, "clash", got["fields.msg"]) assert.Equal(t, "clash", got["fields.msg"])
assert.NotEmpty(t, got["time"]) assert.NotEmpty(t, got["time"])
rc.AddMask(ctx, "runtime-value")
buf.Reset()
common.Logger(withStepLogger(ctx, 2, "s2", "deploy s3cr3t-value runtime-value", StageMain)).Info("done")
require.NoError(t, json.Unmarshal(buf.Bytes(), &got))
assert.Equal(t, "deploy *** ***", got["step"])
} }
+2 -2
View File
@@ -47,7 +47,7 @@ jobs:
require.NoError(t, err) require.NoError(t, err)
job := workflow.Jobs["test"] job := workflow.Jobs["test"]
rawStrategy := model.CloneYamlNode(job.RawStrategy) rawStrategy := model.CloneYamlNode(job.RawStrategy)
require.NoError(t, (&runnerImpl{config: &Config{}}).NewPlanExecutor(&model.Plan{Stages: []*model.Stage{{Runs: []*model.Run{{Workflow: workflow, JobID: "test"}}}}})(t.Context())) require.NoError(t, (&runnerImpl{config: &Config{Env: testCheckoutEnv()}}).NewPlanExecutor(&model.Plan{Stages: []*model.Stage{{Runs: []*model.Run{{Workflow: workflow, JobID: "test"}}}}})(t.Context()))
require.NotNil(t, job.Strategy) require.NotNil(t, job.Strategy)
assert.False(t, job.Strategy.GetFailFast()) assert.False(t, job.Strategy.GetFailFast())
assert.Equal(t, 2, maxParallelFor(job.Strategy, 5)) assert.Equal(t, 2, maxParallelFor(job.Strategy, 5))
@@ -70,7 +70,7 @@ func TestNewPlanExecutorInvalidMatrix(t *testing.T) {
}}, }},
JobID: "test", JobID: "test",
}}}}} }}}}}
runner := &runnerImpl{config: &Config{}} runner := &runnerImpl{config: &Config{Env: testCheckoutEnv()}}
require.ErrorContains(t, runner.NewPlanExecutor(plan)(t.Context()), "could not get job matrix:") require.ErrorContains(t, runner.NewPlanExecutor(plan)(t.Context()), "could not get job matrix:")
Loaded 100 of 238 files, more files were not shown because too many files have changed in this diff. Show more