Compare commits

...
22 Commits
Author SHA1 Message Date
Zettat123andsilverwind ff9965e940 fix: stop the job's docker socket from becoming a directory (#1215)
Fixes https://gitea.com/gitea/runner/issues/1213

Fix the DooD regression that mounts `/var/run/docker.sock` as a directory. Keep the Docker proxy available through job and post steps. Clean stale resources before opening it, then remove containers before their networks and volumes during teardown.

Use a unique filesystem probe and preserve socket ownership. Fall back to direct access when proxying is unsupported. Preserve exec output and clean up active streams and failed starts.

Add a real Docker job test for mounted socket access, post steps and resource cleanup.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1215
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-09-08 04:16:02 +00:00
Renovate Botandsilverwind ba4d3c5b4f chore(deps): update dependencies and tweak make checks (#1212)
1. Dependency updates: gobwas/glob v1.0.0, which replaced its `Glob` interface with `*Pattern`, logrus v1.10.2, prometheus/client_model v0.6.3, golangci-lint v2.13.2, Go 1.27.1, docker/setup-qemu-action and nginx digests.
2. `make fmt-check` and `make tidy-check` report only what the formatter or `go mod tidy` would change, so unrelated uncommitted work no longer fails `make checks`.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1212
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-09-07 08:20:15 +00:00
silverwindandbircni 9e3647395a feat: add GITEA_DOCKER_WORKSPACE and container cleanup (#1204)
1. Add `GITEA_DOCKER_WORKSPACE` which holds the workspace path as the daemon sees it, enabling `${GITEA_DOCKER_WORKSPACE:-.}/data:/app/data` in a compose file without having to resort to `bind_workdir` (which causes much more problems like breaking `actions/cache` because of unstable workspace paths).
2. Add container/network/volume cleanup for containers started within jobs, for example via `docker compose` inside a job. It works by running a lightweight docker socket proxy and injecting a `com.gitea.runner.job` label into every container creation and that label is used to remove containers started by that job at the end. Perf impact of this is near-zero.

Docs: https://gitea.com/gitea/docs/pulls/535

Assisted by Claude (Fable 5.1).
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1204
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-09-05 08:58:23 +00:00
silverwind c158ac5472 perf: speed up action downloads (#1209)
An action pinned to a full commit hash cloned the whole repository, and a cached one hit the network on every run. Now only the pinned commit is fetched at depth 1, a cached commit is reused with no network at all, and the action's `.git` directory no longer ships into the job container, matching GitHub.

```
repo                  cold ms         warm ms       cache KiB
actions/checkout      5807 → 787    1064 → 12    11492 → 2349
actions/setup-node   15257 → 1013   1365 → 21    64303 → 9924
actions/cache        18782 → 760    1589 → 16    60890 → 12507
actions/setup-go      5513 → 764     589 → 20    16600 → 9104
docker/login-action  25809 → 1225   1288 → 10    81709 → 12503
```

Reviewed-on: https://gitea.com/gitea/runner/pulls/1209
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-09-05 08:52:10 +00:00
1745c7c841 fix: fail the step or job whose expression cannot be interpolated (#1199)
An expression that failed to evaluate was logged and replaced by an empty string, so a `run:` step executed an empty script and reported success. The same swallow covered `shell:`, `working-directory:`, step `env:`, `with:`, `uses:`, the job's `env:`, `container:`, `services:`, `runs-on:` and outputs, and a called workflow's `with:` and `secrets:`. Every interpolation now propagates its error as actions/runner does: step-level values fail the step, job-level values fail the job at setup, `timeout-minutes` logs the error and runs unbounded, and a job or step name keeps its source text.

`defaults.run` and a called workflow's inputs and secrets are resolved once at job setup with the job context rather than per step, and the job's image is resolved once, so host mode and `ImageOS` derive from the image the job started with.

Closes https://gitea.com/gitea/runner/issues/392
Closes https://gitea.com/gitea/runner/issues/555

---------

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1199
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-09-03 10:43:04 +00:00
Renovate Bot 7f53eca716 fix(deps): update module golang.org/x/crypto to v0.56.0 [security] (#1205)
This PR contains the following updates:

| Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| [golang.org/x/crypto](https://pkg.go.dev/golang.org/x/crypto) | [`v0.55.0` → `v0.56.0`](https://cs.opensource.google/go/x/crypto/+/refs/tags/v0.55.0...refs/tags/v0.56.0) | ![age](https://developer.mend.io/api/mc/badges/age/go/golang.org%2fx%2fcrypto/v0.56.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/go/golang.org%2fx%2fcrypto/v0.55.0/v0.56.0?slim=true) |

---

### Prevent DoS on deadlocked undecided channel in golang.org/x/crypto/ssh
[CVE-2026-78662](https://nvd.nist.gov/vuln/detail/CVE-2026-78662) / [GO-2026-6354](https://pkg.go.dev/vuln/GO-2026-6354)

<details>
<summary>More information</summary>

#### Details
Previously, a channel registered in the mux's chanList is not usable until it is established. A malicious peer was able flood the channel's incomingRequests, deadlocking the entire connection.

Now, we add an atomic established state, set when a channel becomes usable. Until such a time, handlePacket drops every packet other than the open confirmation/failure, without blocking and without tearing down the connection.

#### Severity
Unknown

#### References
- [https://go.dev/issue/81316](https://go.dev/issue/81316)
- [https://go.dev/cl/826504](https://go.dev/cl/826504)
- [https://groups.google.com/g/golang-announce/c/1y3fb2np35U](https://groups.google.com/g/golang-announce/c/1y3fb2np35U)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6354) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Prevent DoS on deadlocked established channel in golang.org/x/crypto/ssh
[CVE-2026-56855](https://nvd.nist.gov/vuln/detail/CVE-2026-56855) / [GO-2026-6355](https://pkg.go.dev/vuln/GO-2026-6355)

<details>
<summary>More information</summary>

#### Details
Previously, after a channel has been established, a malicious peer could send crafted messages that would deadlock the entire connection.

Now, we handle all RFC 4254 channel messages; global requests are handled explicitly. Then, treat all other messages as a protocol error and tear the connection down instead of buffering and blocking.

#### Severity
Unknown

#### References
- [https://go.dev/issue/81317](https://go.dev/issue/81317)
- [https://go.dev/cl/826524](https://go.dev/cl/826524)
- [https://groups.google.com/g/golang-announce/c/1y3fb2np35U](https://groups.google.com/g/golang-announce/c/1y3fb2np35U)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6355) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xOTEuMiIsInVwZGF0ZWRJblZlciI6IjQzLjE5MS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://gitea.com/gitea/runner/pulls/1205
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-09-03 06:40:31 +00:00
silverwind fca6b44c60 fix: automatically add the workspace mount to allowed volumes (#1203)
Automatically allow workspace mounts in `valid_volumes` for the mounts done via `bind_workdir`, this obsoletes the need for `/workspace/**` or other insecure configurations which would expose workspaces between tasks.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1203
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-09-02 20:32:41 +00:00
bircniandsilverwind b9018aca31 fix: preserve symlinked Node action entrypoints (#1202)
Node resolves an ESM main to its realpath but leaves `process.argv[1]` as passed. `/var/run/act` reaches the action through the `/var/run` -> `/run` symlink most images ship, so the two disagree and actions comparing them skip their own `run()`. Passing `--preserve-symlinks-main` makes them match.

Trade-off: an action whose `runs.main` is a symlink now resolves dependencies from the link's directory rather than the target's.

Fixes https://gitea.com/gitea/runner/issues/1201
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1202
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-08-31 13:06:49 +00:00
Renovate Bot 6c77065295 chore(deps): update dependencies (#1198)
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [node](https://github.com/nodejs/node) | digest | `934240a` → `be23f54` |
| [node](https://github.com/nodejs/node) | digest | `3638d9a` → `ba849c6` |

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xOTEuMiIsInVwZGF0ZWRJblZlciI6IjQzLjE5MS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://gitea.com/gitea/runner/pulls/1198
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-08-29 02:00:21 +00:00
235c035003 feat: add input and input-file flags for exec command (#1173)
Add `input` and `input-file` flags for exec command and refactor common code from LoadVars and LoadEnvs.
Closes (https://gitea.com/gitea/runner/issues/1022)

---------

Co-authored-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1173
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Minjie Fang <[email protected]>
2026-08-27 16:38:01 +00:00
silverwind 12dc9d26a2 fix: improve behaviour across masking, commands and status (#1194)
Fixes 51 bugs discovered via comparison with `actions/runner`. Every fix has test coverage.

### Secrets

- A short secret registered no shifted-base64 form, so `base64("user:$TOKEN")` printed in the clear
- Encoded forms came only from the whole trimmed value, missing padded and per-line spellings
- Masks split only on `\n`, so `::add-mask::a%0Db` registered neither half
- Adds XML, expression-string and quote-trimming encoders

### Workflow commands

- Split at the last `::` or `]` rather than the first, so `::add-mask::a::b` registered no mask
- A command on the last line without a newline was ignored, and `::ADD-MASK::` did nothing
- `##[...]` did not decode `%3B`/`%5D`, properties lost anything after a second `=`
- `$GITHUB_ENV` and `::set-env::` now refuse `NODE_OPTIONS`

### Status

- `continue-on-error` reported failed, a cancelled job reported success, an `if:` error reported cancelled
- File commands ran after `continue-on-error`, failing the job while the step stayed green
- A bad job output aborted the whole run instead of that job

### Steps and actions

- `${{ matrix.* }}` and `${{ strategy.* }}` were empty inside composite actions
- Composite inputs leaked into nested actions as `INPUT_*`, `with:` matched case-sensitively, `pre` failures were dropped
- Docker actions dropped `runs.env` when the caller passed `with: args:`, and caller `args`/`entrypoint` beat the manifest
- An implicit shell ran with `pipefail`, a `shell:` without `{0}` passed without running
- `container.env` overrode job env and every `$GITHUB_ENV` write, heredocs lost leading blank lines, `$GITHUB_PATH` was not BOM-decoded

Written by Claude Opus 5.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1194
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-08-27 09:36:27 +00:00
silverwindandbircni d9f4d65545 fix: honor volumes declared on service containers (#1186)
Service containers were built without a volume policy, so every bind and mount they declared was dropped, whatever `valid_volumes` allowed. GitHub passes a service's declared volumes straight to `docker create`, so a workflow that mounts into a service silently did nothing here. Services now get the configured policy, but not `validVolumes()`, which would also hand them the docker daemon socket that GitHub mounts only into the job container.

### What changes for users

On the default `valid_volumes: []` nothing changes: a service's volumes are still dropped, now with a warning rather than in silence. Once `valid_volumes` is configured, a service's declared volumes are honored under it instead of discarded, which is what that setting already documents. No workflow that worked before stops working, and a service can reach no volume the policy does not already allow the job container, so this is not a breaking change.

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1186
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-08-26 15:55:27 +00:00
ABiscuittttandsilverwind 212909db7b fix: keep a step's own with: values out of its inputs context (#1192)
A step's `if:`, its `continue-on-error:` and its `run:` script resolved `inputs.*` from the step's own `INPUT_*` env. A `with:` key colliding with a workflow input flipped conditions, and any `INPUT_`-shaped variable from `env:` or a `GITHUB_ENV` write forged an input that never existed.

GitHub evaluates all three in the enclosing scope: the workflow inputs, or for a composite action's steps that action's inputs. Action-input interpolation is the one place that legitimately sees a step's own `with:`, so it keeps its own evaluator.

Fixes https://gitea.com/gitea/runner/issues/1191, ports https://github.com/nektos/act/pull/2473 and extends it to the pre and post stages.

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1192
Reviewed-by: silverwind <[email protected]>
Co-authored-by: ABiscuitttt <[email protected]>
2026-08-26 14:45:05 +00:00
silverwind 0712b2a7a1 fix: mask secrets on every path they leave a job (#1188)
A secret in a matrix value reached the log in the clear:

```yaml
strategy:
  matrix:
    include: "${{ github.token }}"
```

Chasing that one route is pointless, so this masks every sink a secret leaves a job by: the uploaded log rows and the on-disk `job.log`, both through one choke point in `appendLogRow`; the runner's own log, which is where planning errors like that one land with no job logger in reach; the job logger's stdout under debug logging; job summaries; job outputs; and the job name that becomes a container name.

Values the runner knows but the job never declared, the proxy password and the task token, are hidden the same way. Masks apply longest first, since `strings.Replacer` matches in argument order and one secret prefixing another would otherwise mask the prefix and print the rest.

### What changes for users

An output whose value carries a secret is skipped with a warning instead of sent, matching GitHub. Output that showed a secret now shows `***`. `ACTIONS_STEP_DEBUG` and `ACTIONS_RUNNER_DEBUG` are never masked, also matching GitHub, so an output of `true` still reaches the jobs that need it.

Each fix has a test that fails without it.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1188
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-08-25 20:36:05 +00:00
34df4887af fix: report step log ranges with the log flush (#1189)
Gitea slices a task's single log stream into steps by the LogIndex/LogLength that UpdateTask carries. The reporter's daemon flushed log rows without those counters, which only left on the separate state ticker, so rows the server acked between two state reports belonged to no step. The web UI attributes them to no step while the job runs, and a runner that stops reporting in that window leaves them under "Complete job" for good once Gitea finalizes the task.

The three log-flush paths now report the state describing the rows the server just took, so the gap is at most one RPC. Live output also stops waiting up to `state_report_interval` to become visible. An idle job adds no requests.

This does not explain why the runner in that job stopped reporting for 13m40s until Gitea reaped the task as a zombie, that needs the runner host's own log.

Fixes https://gitea.com/gitea/runner/issues/1184

---------

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1189
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-08-25 12:13:18 +00:00
silverwind 745a1e70e6 fix: keep the runner's own container.options when privileged is off (#1151)
The runner's own `container.options` and the workflow's were joined into one string before parsing, so the host-escape filter added in https://gitea.com/gitea/runner/pulls/1058 dropped the administrator's options along with the workflow's. Setups that need `--device` or `--security-opt` from the config file had no way left to get them short of enabling privileged mode.

`NewContainerInput` now carries the two sources apart, as `RunnerOptions` and `WorkflowOptions`, down to the point where the filter runs. With privileged mode off, the host-escape fields are reset to what the runner's own options parse to on their own, so only the workflow's contribution is dropped.

Three further ways a workflow's options reached past its container, all resolved on the runner before anything reaches the daemon:

1. `--env-file` and `--label-file` name files that are read on the runner, so any file it could read became container environment or labels. Both are refused from a workflow now, and still serve the runner's own options.
2. A bare `--env NAME` was resolved from the runner's own environment by docker's validator. That lookup is gone, for every source. Use `runner.envs` or `runner.env_file` to pass a variable on.
3. A volume driver decides for itself what it mounts, and the local driver's `device=` option turns a name `valid_volumes` allows into a bind of any host path. A workflow's mounts may no longer carry one.

`--isolation`, `--volume-driver` and the two paths `--security-opt systempaths=unconfined` lands in were also missing from the fields a workflow may not set.

Last, the `--network and --net in the options will be ignored.` warning fired for every container, because the runner's own network mode is fed into the parsed options before the check runs.

Fixes https://gitea.com/gitea/runner/issues/1142

Reviewed-on: https://gitea.com/gitea/runner/pulls/1151
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-08-24 22:13:28 +00:00
Renovate Botandsilverwind e30c2fed62 chore: update deps, adapt lint, use json v2 (#1185)
- Raised go to 1.27
- Adopted json v2
- Sync lint config from gitea
- Fixed all issues

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1185
Reviewed-by: bircni <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-08-24 19:53:17 +00:00
silverwind 7b4356c746 fix: fail the run when matrix expansion fails (#1187)
A `GetMatrixes` error was logged and discarded, leaving a nil matrix list. That collapsed `maxParallel` to zero, so no executor was built and the parallel executor returned nil for an empty list: the job reported success without running anything.

It now fails the run. Every error it returns is a workflow validation failure that GitHub rejects too, so nothing that runs there starts failing here.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1187
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-08-24 17:03:45 +00:00
Renovate Bot e78123cee3 fix(deps): update go toolchain directive to v1.26.6 [security] (#1183)
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [go](https://go.dev/) ([source](https://github.com/golang/go)) | toolchain | patch | `1.26.5` → `1.26.6` |

---

### Invoking failure to reject ASCII-only Punycode-encoded labels in golang.org/x/net/idna
[CVE-2026-39821](https://nvd.nist.gov/vuln/detail/CVE-2026-39821) / [GO-2026-5026](https://pkg.go.dev/vuln/GO-2026-5026)

<details>
<summary>More information</summary>

#### Details
The ToASCII and ToUnicode functions incorrectly accept Punycode-encoded labels that decode to an ASCII-only label. For example, ToUnicode("xn--example-.com") incorrectly returns the name "example.com" rather than an error.

This behavior can lead to privilege escalation in programs using the idna package. For example, a program which performs privilege checks on the ASCII hostname may reject "example.com" but permit "xn--example-.com". If that program subsequently converts the ASCII hostname to Unicode, it will inadvertently permits access to the Unicode name "example.com".

#### Severity
Unknown

#### References
- [https://go.dev/cl/767220](https://go.dev/cl/767220)
- [https://go.dev/issue/78760](https://go.dev/issue/78760)
- [https://groups.google.com/g/golang-announce/c/iI-mYSI0lu8](https://groups.google.com/g/golang-announce/c/iI-mYSI0lu8)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-5026) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Parsing an invalid SVCB or HTTPS RR can panic in golang.org/x/net/dns/dnsmessage
BIT-golang-2026-46600 / [CVE-2026-46600](https://nvd.nist.gov/vuln/detail/CVE-2026-46600) / [GO-2026-5942](https://pkg.go.dev/vuln/GO-2026-5942)

<details>
<summary>More information</summary>

#### Details
Parsing an invalid SVCB or HTTPS RR can panic when the size of a parameter value overflows the message buffer.

#### Severity
Unknown

#### References
- [https://go.dev/cl/786345](https://go.dev/cl/786345)
- [https://go.dev/issue/79795](https://go.dev/issue/79795)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-5942) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Enforce maximum recursion depth in encoding/asn1
BIT-golang-2026-33818 / [CVE-2026-33818](https://nvd.nist.gov/vuln/detail/CVE-2026-33818) / [GO-2026-5972](https://pkg.go.dev/vuln/GO-2026-5972)

<details>
<summary>More information</summary>

#### Details
Enforce a recursion limit in Unmarshal to prevent stack exhaustion when parsing deeply-nested, recursive structures.

#### Severity
Unknown

#### References
- [https://go.dev/issue/80405](https://go.dev/issue/80405)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)
- [https://go.dev/cl/814980](https://go.dev/cl/814980)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-5972) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Add recursion depth guard during decode in encoding/xml
BIT-golang-2026-56859 / [CVE-2026-56859](https://nvd.nist.gov/vuln/detail/CVE-2026-56859) / [GO-2026-6088](https://pkg.go.dev/vuln/GO-2026-6088)

<details>
<summary>More information</summary>

#### Details
Previously, DecodeElement would reset the depth counter causing it to never fire; this could lead to stack exhaustion.

#### Severity
Unknown

#### References
- [https://go.dev/issue/80481](https://go.dev/issue/80481)
- [https://go.dev/cl/803320](https://go.dev/cl/803320)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6088) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Apply ReadHeaderTimeout when doing unencrypted HTTP/2 check in net/http
BIT-golang-2026-56853 / [CVE-2026-56853](https://nvd.nist.gov/vuln/detail/CVE-2026-56853) / [GO-2026-6089](https://pkg.go.dev/vuln/GO-2026-6089)

<details>
<summary>More information</summary>

#### Details
When a server is configured to support unencrypted HTTP/2, it reads a few bytes from each new connection to see if they contain the HTTP/2 client preface. ReadHeaderTimeout is unexpectedly not being applied when doing this.

#### Severity
Unknown

#### References
- [https://go.dev/issue/80205](https://go.dev/issue/80205)
- [https://go.dev/cl/795540](https://go.dev/cl/795540)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6089) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Limit handshake messages we are willing to accept post-handshake in crypto/tls
BIT-golang-2026-56862 / [CVE-2026-56862](https://nvd.nist.gov/vuln/detail/CVE-2026-56862) / [GO-2026-6090](https://pkg.go.dev/vuln/GO-2026-6090)

<details>
<summary>More information</summary>

#### Details
Handshake messages, such as KeyUpdate, are always considered as state-advancing, regardless of whether a handshake has been completed or not. As a result, a malicious client can keep sending KeyUpdate messages to force the server to keep performing key derivation operations indefinitely.

#### Severity
Unknown

#### References
- [https://go.dev/issue/80528](https://go.dev/issue/80528)
- [https://go.dev/cl/804261](https://go.dev/cl/804261)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6090) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Fix Javascript regexp context tracking in html/template
BIT-golang-2026-56858 / [CVE-2026-56858](https://nvd.nist.gov/vuln/detail/CVE-2026-56858) / [GO-2026-6091](https://pkg.go.dev/vuln/GO-2026-6091)

<details>
<summary>More information</summary>

#### Details
Previously, pathological inputs could close an unescaped '/' early, allowing for attack-controlled data to inject arbitrary content, potentially leading to XSS.

#### Severity
Unknown

#### References
- [https://go.dev/issue/80435](https://go.dev/issue/80435)
- [https://go.dev/cl/807100](https://go.dev/cl/807100)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6091) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Avoid quadratic complexity in resolvePath in net/url
BIT-golang-2026-56860 / [CVE-2026-56860](https://nvd.nist.gov/vuln/detail/CVE-2026-56860) / [GO-2026-6218](https://pkg.go.dev/vuln/GO-2026-6218)

<details>
<summary>More information</summary>

#### Details
Previously, resolving relative paths containing parent directory ('..') segments performed string conversions and buffer rewrites on each step, resulting in quadratic time complexity and high memory allocation overhead.

Now, path resolution operates on a byte buffer using index-based backtracking for '..' segments, eliminating the quadratic time complexity and significantly reducing memory allocations.

#### Severity
Unknown

#### References
- [https://go.dev/cl/803681](https://go.dev/cl/803681)
- [https://go.dev/issue/80494](https://go.dev/issue/80494)
- [https://groups.google.com/g/golang-announce/c/94pEornpRlI](https://groups.google.com/g/golang-announce/c/94pEornpRlI)

This data is provided by [OSV](https://osv.dev/vulnerability/GO-2026-6218) and the [Go Vulnerability Database](https://github.com/golang/vulndb) ([CC-BY 4.0](https://github.com/golang/vulndb#license)).
</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - ""
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xOTEuMiIsInVwZGF0ZWRJblZlciI6IjQzLjE5MS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://gitea.com/gitea/runner/pulls/1183
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Renovate Bot <[email protected]>
2026-08-23 12:46:12 +00:00
bircniandsilverwind 8260e2def2 test: add end-to-end Gitea compatibility suite (#1180)
Adds a real-Gitea compatibility suite against stable and nightly to catch runner and API drift before release.

The suite shares one Gitea and regular runner, using repository runners only for cache v1/v2 and ephemeral behavior. It covers registration, payload and log encoding, secrets, variables, services, artifacts, outputs, matrices, cache, dispatch, live logs, cancellation, and ephemeral teardown.

CI runs both images in parallel. Warm local timings:

| Image | Before | After |
| --- | ---: | ---: |
| `gitea/gitea:latest` | ~70s | 26.54s |
| `gitea/gitea:main-nightly` | ~54s | 22.53s |

The former serial matrix took about 2 minutes. Parallel suite execution is now bounded by the slower ~26.5-second variant.

Shared Renovate matcher: https://gitea.com/gitea/renovate-config/pulls/552

---------

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1180
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-08-23 12:32:22 +00:00
silverwind b97c61aa14 test: speed up tests (#1181)
Parallelize isolated workflow tests, consolidate redundant fixtures, and replace fixed waits with deterministic synchronization. Keep the readable curl service probe and use `getent` for hostname resolution.

Measured on the same machine with `make test`:

1. Wall time: 170.50s to 136.12s, down 34.38s or 20.2%.
1. `act/runner`: 138.417s to 124.291s, down 14.126s or 10.2%.
1. `act/runner` coverage: unchanged at 85.2%.
1. `TestDockerExecAbort`: 2.514s to 0.012s package time.

Stability checks:

1. Cancellation and deadline tests: 100 race-enabled repetitions.
1. Host runner suite: 10 race-enabled repetitions.
1. Changed Docker fixtures: 3 consecutive repetitions.

Full race suite, Go and Windows lint, source checks, and security scan pass.

Assisted-by: Codex:GPT-5
Reviewed-on: https://gitea.com/gitea/runner/pulls/1181
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-08-22 20:53:55 +00:00
silverwind 546eca312e refactor: remove unreachable runner code (#1179)
Remove inherited act APIs, configuration branches, and test seams that neither the daemon nor exec can reach. Constant-fold settings both entry points already enforce and consolidate duplicate runner paths.

Major removals:

- Unwired custom action-cache and local-repository-cache implementations.
- Legacy matrix, platform, input, container-reuse, logging, Git remote, and action-replacement configuration paths.
- Unused Docker socket, container network, tar-copy, and platform PTY wrappers.
- Single-implementation filesystem, environment, runner, and expression abstractions.
- Duplicated step-container, command-logging, credential, reusable-workflow, and execution paths.
- Generated client mock boilerplate, obsolete fixtures, test-only seams, stale wrappers, and commented-out code.

This removes 2844 net Go lines while retaining Gitea RPC, event, matrix, input, cache, artifact, action, reusable workflow, Docker, host, exec, and release behavior.

Assisted by Codex (GPT-5).

Reviewed-on: https://gitea.com/gitea/runner/pulls/1179
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-08-22 16:54:41 +00:00
162 changed files with 6550 additions and 5213 deletions

No files matched your search

+2 -2
View File
@@ -74,10 +74,10 @@ jobs:
fetch-depth: 0 # all history for all branches and tags fetch-depth: 0 # all history for all branches and tags
- name: Set up QEMU - name: Set up QEMU
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4 uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v4
- name: Set up Docker BuildX - name: Set up Docker BuildX
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4 uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
- name: Login to DockerHub - name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
+2 -2
View File
@@ -77,10 +77,10 @@ jobs:
fetch-depth: 0 # all history for all branches and tags fetch-depth: 0 # all history for all branches and tags
- name: Set up QEMU - name: Set up QEMU
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4 uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v4
- name: Set up Docker BuildX - name: Set up Docker BuildX
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4 uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
- name: Login to DockerHub - name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
+37 -10
View File
@@ -5,17 +5,18 @@ on:
- main - main
pull_request: pull_request:
env:
# The runner image ships a stale docker.io login; point docker at an empty config so
# image pulls go straight to anonymous instead of attempting (and failing) that auth
# first. The path must be a literal: the `runner` context is unavailable in workflow-level
# env, so `${{ runner.temp }}` would resolve to empty and config.Dir() would fall back
# to ~/.docker with the stale credentials.
DOCKER_CONFIG: /tmp/docker-noauth
jobs: jobs:
lint: lint:
name: check and test name: check and test
runs-on: ubuntu-latest runs-on: ubuntu-latest
env:
# The runner image ships a stale docker.io login; point docker at an empty config so
# image pulls go straight to anonymous instead of attempting (and failing) that auth
# first. The path must be a literal: the `runner` context is unavailable in job-level
# env, so `${{ runner.temp }}` would resolve to empty and config.Dir() would fall back
# to ~/.docker with the stale credentials.
DOCKER_CONFIG: /tmp/docker-noauth
steps: steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
@@ -28,9 +29,16 @@ jobs:
# the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host # the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host
# daemon retains them between runs, so this is usually a fast manifest re-check. # daemon retains them between runs, so this is usually a fast manifest re-check.
- name: pre-pull test images - name: pre-pull test images
env:
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:ba849c60be29959425b8734d57b8b4b7d56f98edd9504c9af091d5281095a71e # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:72ba65eb42c10344912a84ff42408db7d34f2feb642204570ab8fc5ffd29f1d3 # renovate: datasource=docker
run: | run: |
for img in node:24-bookworm-slim nginx:alpine; do for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do
for try in 1 2 3; do docker pull "$img" && break || sleep 5; done for attempt in 1 2 3; do
docker pull "$image" && break
[ "$attempt" = 3 ] || sleep 5
done
docker tag "$image" "${image%@*}"
done done
- name: lint - name: lint
run: make lint run: make lint
@@ -48,4 +56,23 @@ jobs:
- name: coverage report - name: coverage report
run: | run: |
make coverage-report make coverage-report
cat .tmp/coverage.md >> "$GITHUB_STEP_SUMMARY" cat .tmp/coverage.md >> "$GITHUB_STEP_SUMMARY"
e2e:
name: gitea compatibility (${{ matrix.gitea_image }})
runs-on: ubuntu-latest
strategy:
matrix:
gitea_image:
- gitea/gitea:latest
- gitea/gitea:main-nightly
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: 'go.mod'
check-latest: true
- name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
- name: e2e compatibility
run: make test-e2e E2E_GITEA_IMAGE=${{ matrix.gitea_image }}
+8 -6
View File
@@ -46,8 +46,7 @@ linters:
gocritic: gocritic:
enabled-checks: enabled-checks:
- equalFold - equalFold
disabled-checks: disabled-checks: []
- ifElseChain
revive: revive:
severity: error severity: error
rules: rules:
@@ -71,10 +70,14 @@ linters:
- name: unexported-return - name: unexported-return
- name: var-declaration - name: var-declaration
- name: var-naming - name: var-naming
arguments:
- [] # AllowList - do not remove as args for the rule are positional and won't work without lists first
- [] # DenyList
- - skip-initialism-name-checks: true
staticcheck: staticcheck:
checks: checks:
- all - all
- -ST1005 testifylint: {}
usetesting: usetesting:
os-temp-dir: true os-temp-dir: true
perfsprint: perfsprint:
@@ -92,8 +95,6 @@ linters:
generated: lax generated: lax
presets: presets:
- comments - comments
- common-false-positives
- legacy
- std-error-handling - std-error-handling
rules: rules:
- linters: - linters:
@@ -118,7 +119,8 @@ formatters:
- blank - blank
- default - default
gofumpt: gofumpt:
extra-rules: true extra:
group-params: true
exclusions: exclusions:
generated: lax generated: lax
run: run:
+9
View File
@@ -30,3 +30,12 @@ depending on the prefix:
encoded forms too. encoded forms too.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the reporter - Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the reporter
decodes exactly those two when folding a location into an annotation. decodes exactly those two when folding a location into an annotation.
## End-to-end compatibility tests
`make test-e2e` runs the runner against `E2E_GITEA_IMAGE`. It defaults to the nightly image.
It requires Docker and is excluded from `make test`. CI runs stable and nightly variants in
parallel.
The suite shares one Gitea and regular runner. Cache and ephemeral scenarios use isolated
repository runners. A run path is `<workflow>@<ref>` and a log row is `<timestamp>Z <payload>`.
+3 -3
View File
@@ -1,7 +1,7 @@
### BUILDER STAGE ### BUILDER STAGE
# #
# #
FROM golang:1.26-alpine3.23 AS builder FROM golang:1.27-alpine3.23 AS builder
# Do not remove `git` here, it is required for getting runner version when executing `make build` # Do not remove `git` here, it is required for getting runner version when executing `make build`
RUN apk add --no-cache make git RUN apk add --no-cache make git
@@ -17,7 +17,7 @@ RUN make clean && make build
### DIND VARIANT ### DIND VARIANT
# #
# #
FROM docker:29.7.1-dind AS dind FROM docker:29.7.2-dind AS dind
ARG VERSION=dev ARG VERSION=dev
@@ -37,7 +37,7 @@ ENTRYPOINT ["s6-svscan","/etc/s6"]
### DIND-ROOTLESS VARIANT ### DIND-ROOTLESS VARIANT
# #
# #
FROM docker:29.7.1-dind-rootless AS dind-rootless FROM docker:29.7.2-dind-rootless AS dind-rootless
ARG VERSION=dev ARG VERSION=dev
+17 -18
View File
@@ -5,7 +5,7 @@ GO ?= go
SHASUM ?= shasum -a 256 SHASUM ?= shasum -a 256
HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" ) HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" )
XGO_PACKAGE ?= src.techknowlogick.com/[email protected] # renovate: datasource=go XGO_PACKAGE ?= src.techknowlogick.com/[email protected] # renovate: datasource=go
XGO_VERSION := go-1.26.x XGO_VERSION := go-1.27.x
GXZ_PACKAGE ?= github.com/ulikunitz/xz/cmd/[email protected] # renovate: datasource=go GXZ_PACKAGE ?= github.com/ulikunitz/xz/cmd/[email protected] # renovate: datasource=go
LINUX_ARCHS ?= linux/amd64,linux/arm64 LINUX_ARCHS ?= linux/amd64,linux/arm64
@@ -18,8 +18,8 @@ DOCKER_TAG ?= nightly
DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG) DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)
DOCKER_ROOTLESS_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)-dind-rootless DOCKER_ROOTLESS_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)-dind-rootless
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]2.2 # renovate: datasource=go GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]3.2 # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.6.0 # renovate: datasource=go GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.7.0 # renovate: datasource=go
GOTEST_FLAGS ?= -race -timeout 20m -parallel 8 GOTEST_FLAGS ?= -race -timeout 20m -parallel 8
@@ -96,13 +96,8 @@ go-check:
fi fi
.PHONY: fmt-check .PHONY: fmt-check
fmt-check: fmt fmt-check: ## check the Go code formatting
@diff=$$(git diff --color=always -- '*.go'); \ $(GO) run $(GOLANGCI_LINT_PACKAGE) fmt --diff-colored
if [ -n "$$diff" ]; then \
echo "Please run 'make fmt' and commit the result:"; \
printf "%s" "$${diff}"; \
exit 1; \
fi
.PHONY: deps-tools .PHONY: deps-tools
deps-tools: ## install tool dependencies deps-tools: ## install tool dependencies
@@ -137,7 +132,7 @@ lint-pr-title: ## lint PR title against Conventional Commits (set PR_TITLE=...)
.PHONY: security-check .PHONY: security-check
security-check: security-check:
GOEXPERIMENT= $(GO) run $(GOVULNCHECK_PACKAGE) -show color ./... || true $(GO) run $(GOVULNCHECK_PACKAGE) -show color ./... || true
.PHONY: tidy .PHONY: tidy
tidy: ## run go mod tidy tidy: ## run go mod tidy
@@ -149,13 +144,8 @@ tidy: ## run go mod tidy
fi fi
.PHONY: tidy-check .PHONY: tidy-check
tidy-check: tidy tidy-check: ## check that go.mod and go.sum are tidy
@diff=$$(git diff --color=always -- go.mod go.sum); \ $(GO) mod tidy -diff
if [ -n "$$diff" ]; then \
echo "Please run 'make tidy' and commit the result:"; \
printf "%s" "$${diff}"; \
exit 1; \
fi
.PHONY: test .PHONY: test
test: ## test everything (integration tests self-skip without docker/network) test: ## test everything (integration tests self-skip without docker/network)
@@ -171,6 +161,15 @@ coverage-report: ## turn coverage.txt from `make test` into .tmp/coverage.md
test-dind: ## run the daemon-facing tests against the built dind image (TARGET=dind|dind-rootless) test-dind: ## run the daemon-facing tests against the built dind image (TARGET=dind|dind-rootless)
@./scripts/test-dind.sh $(TARGET) @./scripts/test-dind.sh $(TARGET)
E2E_JOB_IMAGE ?= node:24-bookworm@sha256:be23f54a88d34e8824c741b19b91064094f92c1c97b194144bfc8b50d67258e2 # renovate: datasource=docker
SERVICE_IMAGE ?= nginx:1.31.4-alpine@sha256:db35bfc6b2951e7f8a72db5db120288c127ffaeeb4a6d4b95a26fead017d5913 # renovate: datasource=docker
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
E2E_CONCURRENCY ?= 8
.PHONY: test-e2e
test-e2e: ## run Gitea compatibility tests against E2E_GITEA_IMAGE
@E2E_CONCURRENCY=$(E2E_CONCURRENCY) E2E_GITEA_IMAGE=$(E2E_GITEA_IMAGE) E2E_JOB_IMAGE=$(E2E_JOB_IMAGE) GO=$(GO) SERVICE_IMAGE=$(SERVICE_IMAGE) ./tools/test-e2e.sh
.PHONY: install .PHONY: install
install: $(GOFILES) ## install the runner binary via `go install` install: $(GOFILES) ## install the runner binary via `go install`
$(GO) install -v -tags '$(TAGS)' -ldflags '-s -w $(EXTLDFLAGS) $(LDFLAGS)' $(GO) install -v -tags '$(TAGS)' -ldflags '-s -w $(EXTLDFLAGS) $(LDFLAGS)'
+15
View File
@@ -266,6 +266,17 @@ A job in a container reaches a service by its id on the job network, on the port
Unlike GitHub, a job whose steps run on the host (a `host` label without `container:`) starts no service containers, so `job.services` and `job.container` stay empty. Give such a job a `container:` when it needs services. Unlike GitHub, a job whose steps run on the host (a `host` label without `container:`) starts no service containers, so `job.services` and `job.container` stay empty. Give such a job a `container:` when it needs services.
#### Docker from a job (`GITEA_DOCKER_WORKSPACE`)
A container a job starts through the Docker socket cannot bind-mount the workspace by the job's own path, the daemon does not have it. `GITEA_DOCKER_WORKSPACE` holds the path the daemon sees. Use it as the prefix of workspace binds, with `.` as the fallback for local use, here in a `docker-compose.yaml`:
```yaml
volumes:
- ${GITEA_DOCKER_WORKSPACE:-.}/data:/app/data
```
Linux container jobs use a Docker proxy when socket sharing and permissions allow it. Other setups use the daemon socket directly. The proxy removes containers, networks and volumes after post steps and the completed hook. Named volumes created through it are job-scoped. To retain resources, mount the daemon socket explicitly in `container.options`. Host jobs use their existing Docker access, where `.` works.
#### Proxy #### Proxy
Set these variables in the runner's environment, with systemd `Environment=`, `docker run -e`, or Kubernetes `env:`: Set these variables in the runner's environment, with systemd `Environment=`, `docker run -e`, or Kubernetes `env:`:
@@ -390,6 +401,10 @@ See **[docs/job-hooks.md](docs/job-hooks.md)** for the execution order, environm
Set `log.job.dir` to a path and the runner writes a copy of every task's log there as `<start time>-task-<id>.log`: the rows exactly as Gitea received them, with the same secrets masked and the job's result on the last line. Off by default, and what Gitea shows does not change. Set `log.job.dir` to a path and the runner writes a copy of every task's log there as `<start time>-task-<id>.log`: the rows exactly as Gitea received them, with the same secrets masked and the job's result on the last line. Off by default, and what Gitea shows does not change.
#### Secret masking
A job's secrets and its `::add-mask::` values are hidden from what the runner writes and uploads: the job log, the local copy above, job summaries, and the names of the containers it creates. A job output carrying one is skipped with a warning rather than sent masked, as GitHub does, so a downstream `needs.<job>.outputs.<name>` reading it is empty.
`log.job.retention` (default `168h`) is how long a log is kept, expired ones being deleted as new tasks start, and `log.job.max_size` (default `1GB`) caps one log. Keep `retention` above `runner.timeout` so a long job cannot outlive its own log, and prefer local disk, the file is written while the job runs. Only the runner's own user can read it. `log.job.retention` (default `168h`) is how long a log is kept, expired ones being deleted as new tasks start, and `log.job.max_size` (default `1GB`) caps one log. Keep `retention` above `runner.timeout` so a long job cannot outlive its own log, and prefer local disk, the file is written while the job runs. Only the runner's own user can read it.
### Example Deployments ### Example Deployments
+10 -8
View File
@@ -11,7 +11,7 @@ import (
"crypto/rand" "crypto/rand"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json/v2"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -357,8 +357,8 @@ func (h *Handler) Close() error {
func (h *Handler) openDB() (*bolthold.Store, error) { func (h *Handler) openDB() (*bolthold.Store, error) {
return bolthold.Open(filepath.Join(h.dir, "bolt.db"), 0o644, &bolthold.Options{ return bolthold.Open(filepath.Join(h.dir, "bolt.db"), 0o644, &bolthold.Options{
Encoder: json.Marshal, Encoder: func(value any) ([]byte, error) { return json.Marshal(value) },
Decoder: json.Unmarshal, Decoder: func(data []byte, value any) error { return json.Unmarshal(data, value) },
Options: &bbolt.Options{ Options: &bbolt.Options{
Timeout: 5 * time.Second, Timeout: 5 * time.Second,
NoGrowSync: bbolt.DefaultOptions.NoGrowSync, NoGrowSync: bbolt.DefaultOptions.NoGrowSync,
@@ -422,13 +422,15 @@ func (h *Handler) lookupCache(db *bolthold.Store, repo string, keys []string, ve
func (h *Handler) reserve(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { func (h *Handler) reserve(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
cred := credFromContext(r.Context()) cred := credFromContext(r.Context())
api := &Request{} api := &Request{}
if err := json.NewDecoder(r.Body).Decode(api); err != nil { if err := json.UnmarshalRead(r.Body, api); err != nil {
h.responseJSON(w, r, 400, err) h.responseJSON(w, r, 400, err)
return return
} }
cache := api.ToCache() cache := &Cache{Repo: cred.Repo, Key: api.Key, Version: api.Version, Size: api.Size}
cache.Repo = cred.Repo if cache.Size == 0 {
cache.Size = -1
}
db, err := h.openDB() db, err := h.openDB()
if err != nil { if err != nil {
h.responseJSON(w, r, 500, err) h.responseJSON(w, r, 500, err)
@@ -690,7 +692,7 @@ func (h *Handler) ResultsURL(cred JobCredential) string {
func (h *Handler) internalRegister(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { func (h *Handler) internalRegister(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
var body internalRegisterBody var body internalRegisterBody
if err := json.NewDecoder(r.Body).Decode(&body); err != nil { if err := json.UnmarshalRead(r.Body, &body); err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err) h.responseJSON(w, r, http.StatusBadRequest, err)
return return
} }
@@ -706,7 +708,7 @@ func (h *Handler) internalRegister(w http.ResponseWriter, r *http.Request, _ htt
// POST /_internal/revoke // POST /_internal/revoke
func (h *Handler) internalRevoke(w http.ResponseWriter, r *http.Request, _ httprouter.Params) { func (h *Handler) internalRevoke(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
var body internalRevokeBody var body internalRevokeBody
if err := json.NewDecoder(r.Body).Decode(&body); err != nil { if err := json.UnmarshalRead(r.Body, &body); err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err) h.responseJSON(w, r, http.StatusBadRequest, err)
return return
} }
+16 -16
View File
@@ -7,7 +7,7 @@ package artifactcache
import ( import (
"bytes" "bytes"
"crypto/rand" "crypto/rand"
"encoding/json" "encoding/json/v2"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -136,7 +136,7 @@ func TestHandler(t *testing.T) {
defer resp.Body.Close() defer resp.Body.Close()
assert.Equal(t, 200, resp.StatusCode) assert.Equal(t, 200, resp.StatusCode)
require.NoError(t, json.NewDecoder(resp.Body).Decode(&first)) require.NoError(t, json.UnmarshalRead(resp.Body, &first))
assert.NotZero(t, first.CacheID) assert.NotZero(t, first.CacheID)
} }
{ {
@@ -151,7 +151,7 @@ func TestHandler(t *testing.T) {
defer resp.Body.Close() defer resp.Body.Close()
assert.Equal(t, 200, resp.StatusCode) assert.Equal(t, 200, resp.StatusCode)
require.NoError(t, json.NewDecoder(resp.Body).Decode(&second)) require.NoError(t, json.UnmarshalRead(resp.Body, &second))
assert.NotZero(t, second.CacheID) assert.NotZero(t, second.CacheID)
} }
@@ -204,7 +204,7 @@ func TestHandler(t *testing.T) {
got := struct { got := struct {
CacheID uint64 `json:"cacheId"` CacheID uint64 `json:"cacheId"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
id = got.CacheID id = got.CacheID
} }
{ {
@@ -259,7 +259,7 @@ func TestHandler(t *testing.T) {
got := struct { got := struct {
CacheID uint64 `json:"cacheId"` CacheID uint64 `json:"cacheId"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
id = got.CacheID id = got.CacheID
} }
{ {
@@ -315,7 +315,7 @@ func TestHandler(t *testing.T) {
got := struct { got := struct {
CacheID uint64 `json:"cacheId"` CacheID uint64 `json:"cacheId"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
id = got.CacheID id = got.CacheID
} }
{ {
@@ -362,7 +362,7 @@ func TestHandler(t *testing.T) {
got := struct { got := struct {
CacheID uint64 `json:"cacheId"` CacheID uint64 `json:"cacheId"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
id = got.CacheID id = got.CacheID
} }
@@ -413,7 +413,7 @@ func TestHandler(t *testing.T) {
got := struct { got := struct {
CacheID uint64 `json:"cacheId"` CacheID uint64 `json:"cacheId"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
id = got.CacheID id = got.CacheID
} }
{ {
@@ -493,7 +493,7 @@ func TestHandler(t *testing.T) {
ArchiveLocation string `json:"archiveLocation"` ArchiveLocation string `json:"archiveLocation"`
CacheKey string `json:"cacheKey"` CacheKey string `json:"cacheKey"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
assert.Equal(t, "hit", got.Result) assert.Equal(t, "hit", got.Result)
assert.Equal(t, keys[except], got.CacheKey) assert.Equal(t, keys[except], got.CacheKey)
@@ -528,7 +528,7 @@ func TestHandler(t *testing.T) {
ArchiveLocation string `json:"archiveLocation"` ArchiveLocation string `json:"archiveLocation"`
CacheKey string `json:"cacheKey"` CacheKey string `json:"cacheKey"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
assert.Equal(t, "hit", got.Result) assert.Equal(t, "hit", got.Result)
assert.Equal(t, key, got.CacheKey) assert.Equal(t, key, got.CacheKey)
assert.NotEqual(t, strings.ToLower(key), got.CacheKey) assert.NotEqual(t, strings.ToLower(key), got.CacheKey)
@@ -577,7 +577,7 @@ func TestHandler(t *testing.T) {
ArchiveLocation string `json:"archiveLocation"` ArchiveLocation string `json:"archiveLocation"`
CacheKey string `json:"cacheKey"` CacheKey string `json:"cacheKey"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
assert.Equal(t, keys[expect], got.CacheKey) assert.Equal(t, keys[expect], got.CacheKey)
contentResp, err := testClient.Get(got.ArchiveLocation) contentResp, err := testClient.Get(got.ArchiveLocation)
@@ -633,7 +633,7 @@ func TestHandler(t *testing.T) {
ArchiveLocation string `json:"archiveLocation"` ArchiveLocation string `json:"archiveLocation"`
CacheKey string `json:"cacheKey"` CacheKey string `json:"cacheKey"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
assert.Equal(t, keys[expect], got.CacheKey) assert.Equal(t, keys[expect], got.CacheKey)
contentResp, err := testClient.Get(got.ArchiveLocation) contentResp, err := testClient.Get(got.ArchiveLocation)
@@ -677,7 +677,7 @@ func uploadCacheNormally(t *testing.T, base, key, version string, content []byte
got := struct { got := struct {
CacheID uint64 `json:"cacheId"` CacheID uint64 `json:"cacheId"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
id = got.CacheID id = got.CacheID
} }
{ {
@@ -708,7 +708,7 @@ func uploadCacheNormally(t *testing.T, base, key, version string, content []byte
ArchiveLocation string `json:"archiveLocation"` ArchiveLocation string `json:"archiveLocation"`
CacheKey string `json:"cacheKey"` CacheKey string `json:"cacheKey"`
}{} }{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
assert.Equal(t, "hit", got.Result) assert.Equal(t, "hit", got.Result)
assert.Equal(t, key, got.CacheKey) assert.Equal(t, key, got.CacheKey)
archiveLocation = got.ArchiveLocation archiveLocation = got.ArchiveLocation
@@ -1197,7 +1197,7 @@ func TestHandler_CrossRepoIsolation(t *testing.T) {
var reserved struct { var reserved struct {
CacheID uint64 `json:"cacheId"` CacheID uint64 `json:"cacheId"`
} }
require.NoError(t, json.NewDecoder(resp.Body).Decode(&reserved)) require.NoError(t, json.UnmarshalRead(resp.Body, &reserved))
resp.Body.Close() resp.Body.Close()
require.NotZero(t, reserved.CacheID) require.NotZero(t, reserved.CacheID)
@@ -1331,7 +1331,7 @@ func TestHandler_ArtifactSignatureDownload(t *testing.T) {
var hit struct { var hit struct {
ArchiveLocation string `json:"archiveLocation"` ArchiveLocation string `json:"archiveLocation"`
} }
require.NoError(t, json.NewDecoder(resp.Body).Decode(&hit)) require.NoError(t, json.UnmarshalRead(resp.Body, &hit))
resp.Body.Close() resp.Body.Close()
require.Contains(t, hit.ArchiveLocation, "sig=") require.Contains(t, hit.ArchiveLocation, "sig=")
+33 -7
View File
@@ -5,7 +5,8 @@ package artifactcache
import ( import (
"cmp" "cmp"
"encoding/json" "encoding/json/jsontext"
"encoding/json/v2"
"encoding/xml" "encoding/xml"
"errors" "errors"
"fmt" "fmt"
@@ -128,7 +129,7 @@ func (h *Handler) v2FinalizeCacheEntryUpload(w http.ResponseWriter, r *http.Requ
} }
db.Close() // commitCache needs the store closed db.Close() // commitCache needs the store closed
cache.Size, _ = cmp.Or(req.SizeBytes, req.SizeBytesCamel).Int64() cache.Size = int64(cmp.Or(req.SizeBytes, req.SizeBytesCamel))
if err := h.commitCache(cache); err != nil { if err := h.commitCache(cache); err != nil {
h.logger.Errorf("finalize cache %d (%s): %v", cache.ID, cache.Key, err) h.logger.Errorf("finalize cache %d (%s): %v", cache.ID, cache.Key, err)
h.twirpNotOK(w, r) h.twirpNotOK(w, r)
@@ -245,10 +246,10 @@ type (
} }
v2FinalizeRequest struct { v2FinalizeRequest struct {
Key string `json:"key"` Key string `json:"key"`
Version string `json:"version"` Version string `json:"version"`
SizeBytes json.Number `json:"size_bytes"` SizeBytes twirpInt64 `json:"size_bytes"`
SizeBytesCamel json.Number `json:"sizeBytes"` SizeBytesCamel twirpInt64 `json:"sizeBytes"`
} }
v2DownloadRequest struct { v2DownloadRequest struct {
@@ -259,6 +260,31 @@ type (
} }
) )
// twirpInt64 accepts its value as the JSON string the mapping prescribes or as a bare number.
type twirpInt64 int64
func (n *twirpInt64) UnmarshalJSONFrom(dec *jsontext.Decoder) error {
val, err := dec.ReadValue()
if err != nil {
return err
}
digits := []byte(val)
switch val.Kind() {
case 'n': // absent, keep the zero value
return nil
case '"':
if digits, err = jsontext.AppendUnquote(nil, val); err != nil {
return err
}
}
parsed, err := strconv.ParseInt(string(digits), 10, 64)
if err != nil {
return err
}
*n = twirpInt64(parsed)
return nil
}
func (d v2DownloadRequest) keys() []string { func (d v2DownloadRequest) keys() []string {
restoreKeys := d.RestoreKeys restoreKeys := d.RestoreKeys
if len(restoreKeys) == 0 { if len(restoreKeys) == 0 {
@@ -269,6 +295,6 @@ func (d v2DownloadRequest) keys() []string {
func decodeTwirpRequest[T any](r *http.Request) (T, error) { func decodeTwirpRequest[T any](r *http.Request) (T, error) {
var req T var req T
err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&req) err := json.UnmarshalRead(io.LimitReader(r.Body, 1<<20), &req)
return req, err return req, err
} }
+3 -3
View File
@@ -6,7 +6,7 @@ package artifactcache
import ( import (
"bytes" "bytes"
"encoding/base64" "encoding/base64"
"encoding/json" "encoding/json/v2"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -32,7 +32,7 @@ func v2Call(t *testing.T, handler *Handler, client *http.Client, method string,
require.Equal(t, http.StatusOK, resp.StatusCode) require.Equal(t, http.StatusOK, resp.StatusCode)
got := map[string]any{} got := map[string]any{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
return got return got
} }
@@ -227,7 +227,7 @@ func TestCacheServiceV2Lookups(t *testing.T) {
require.Equal(t, http.StatusOK, resp.StatusCode) require.Equal(t, http.StatusOK, resp.StatusCode)
got := map[string]any{} got := map[string]any{}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&got)) require.NoError(t, json.UnmarshalRead(resp.Body, &got))
assert.Equal(t, "deps-abc", got["cacheKey"]) assert.Equal(t, "deps-abc", got["cacheKey"])
assert.NotEmpty(t, got["archiveLocation"]) assert.NotEmpty(t, got["archiveLocation"])
}) })
-17
View File
@@ -10,23 +10,6 @@ type Request struct {
Size int64 `json:"cacheSize"` Size int64 `json:"cacheSize"`
} }
func (c *Request) ToCache() *Cache {
if c == nil {
return nil
}
ret := &Cache{
Key: c.Key,
Version: c.Version,
Size: c.Size,
}
if c.Size == 0 {
// So the request comes from old versions of actions, like `actions/cache@v2`.
// It doesn't send cache size. Set it to -1 to indicate that.
ret.Size = -1
}
return ret
}
type Cache struct { type Cache struct {
ID uint64 `json:"id" boltholdKey:"ID"` ID uint64 `json:"id" boltholdKey:"ID"`
Repo string `json:"repo" boltholdIndex:"Repo"` Repo string `json:"repo" boltholdIndex:"Repo"`
+36 -108
View File
@@ -6,7 +6,7 @@ package artifacts
import ( import (
"context" "context"
"encoding/json" "encoding/json/v2"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -50,65 +50,29 @@ type ResponseMessage struct {
Message string `json:"message"` Message string `json:"message"`
} }
type WritableFile interface {
io.WriteCloser
}
type WriteFS interface {
OpenWritable(name string) (WritableFile, error)
OpenAppendable(name string) (WritableFile, error)
}
type readWriteFSImpl struct{}
func (fwfs readWriteFSImpl) Open(name string) (fs.File, error) {
return os.Open(name)
}
func (fwfs readWriteFSImpl) OpenWritable(name string) (WritableFile, error) {
if err := os.MkdirAll(filepath.Dir(name), os.ModePerm); err != nil {
return nil, err
}
return os.OpenFile(name, os.O_CREATE|os.O_RDWR|os.O_TRUNC, 0o644)
}
func (fwfs readWriteFSImpl) OpenAppendable(name string) (WritableFile, error) {
if err := os.MkdirAll(filepath.Dir(name), os.ModePerm); err != nil {
return nil, err
}
file, err := os.OpenFile(name, os.O_CREATE|os.O_RDWR, 0o644)
if err != nil {
return nil, err
}
_, err = file.Seek(0, io.SeekEnd)
if err != nil {
return nil, err
}
return file, nil
}
var gzipExtension = ".gz__" var gzipExtension = ".gz__"
func safeResolve(baseDir, relPath string) string { func safeResolve(baseDir, relPath string) string {
return filepath.Join(baseDir, filepath.Clean(filepath.Join(string(os.PathSeparator), relPath))) return filepath.Join(baseDir, filepath.Clean(filepath.Join(string(os.PathSeparator), relPath)))
} }
func uploads(router *httprouter.Router, baseDir string, fsys WriteFS) { func writeJSON(w http.ResponseWriter, value any) {
data, err := json.Marshal(value)
if err != nil {
panic(err)
}
if _, err := w.Write(data); err != nil {
panic(err)
}
}
func uploads(router *httprouter.Router, baseDir string) {
router.POST("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.POST("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
runID := params.ByName("runId") runID := params.ByName("runId")
json, err := json.Marshal(FileContainerResourceURL{ writeJSON(w, FileContainerResourceURL{
FileContainerResourceURL: fmt.Sprintf("http://%s/upload/%s", req.Host, runID), FileContainerResourceURL: fmt.Sprintf("http://%s/upload/%s", req.Host, runID),
}) })
if err != nil {
panic(err)
}
_, err = w.Write(json)
if err != nil {
panic(err)
}
}) })
router.PUT("/upload/:runId", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.PUT("/upload/:runId", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
@@ -122,67 +86,47 @@ func uploads(router *httprouter.Router, baseDir string, fsys WriteFS) {
safeRunPath := safeResolve(baseDir, runID) safeRunPath := safeResolve(baseDir, runID)
safePath := safeResolve(safeRunPath, itemPath) safePath := safeResolve(safeRunPath, itemPath)
file, err := func() (WritableFile, error) { if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil {
contentRange := req.Header.Get("Content-Range") panic(err)
if contentRange != "" && !strings.HasPrefix(contentRange, "bytes 0-") { }
return fsys.OpenAppendable(safePath) flags := os.O_CREATE | os.O_WRONLY | os.O_TRUNC
} appendUpload := req.Header.Get("Content-Range")
return fsys.OpenWritable(safePath) if appendUpload != "" && !strings.HasPrefix(appendUpload, "bytes 0-") {
}() flags = os.O_CREATE | os.O_WRONLY | os.O_APPEND
}
file, err := os.OpenFile(safePath, flags, 0o644)
if err != nil { if err != nil {
panic(err) panic(err)
} }
defer file.Close() defer file.Close()
writer, ok := file.(io.Writer)
if !ok {
panic(errors.New("File is not writable"))
}
if req.Body == nil { if req.Body == nil {
panic(errors.New("No body given")) panic(errors.New("no body given"))
} }
_, err = io.Copy(writer, req.Body) _, err = io.Copy(file, req.Body)
if err != nil { if err != nil {
panic(err) panic(err)
} }
json, err := json.Marshal(ResponseMessage{ writeJSON(w, ResponseMessage{
Message: "success", Message: "success",
}) })
if err != nil {
panic(err)
}
_, err = w.Write(json)
if err != nil {
panic(err)
}
}) })
router.PATCH("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.PATCH("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
json, err := json.Marshal(ResponseMessage{ writeJSON(w, ResponseMessage{
Message: "success", Message: "success",
}) })
if err != nil {
panic(err)
}
_, err = w.Write(json)
if err != nil {
panic(err)
}
}) })
} }
func downloads(router *httprouter.Router, baseDir string, fsys fs.FS) { func downloads(router *httprouter.Router, baseDir string) {
router.GET("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.GET("/_apis/pipelines/workflows/:runId/artifacts", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
runID := params.ByName("runId") runID := params.ByName("runId")
safePath := safeResolve(baseDir, runID) safePath := safeResolve(baseDir, runID)
entries, err := fs.ReadDir(fsys, safePath) entries, err := os.ReadDir(safePath)
if err != nil { if err != nil {
panic(err) panic(err)
} }
@@ -195,18 +139,10 @@ func downloads(router *httprouter.Router, baseDir string, fsys fs.FS) {
}) })
} }
json, err := json.Marshal(NamedFileContainerResourceURLResponse{ writeJSON(w, NamedFileContainerResourceURLResponse{
Count: len(list), Count: len(list),
Value: list, Value: list,
}) })
if err != nil {
panic(err)
}
_, err = w.Write(json)
if err != nil {
panic(err)
}
}) })
router.GET("/download/:container", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.GET("/download/:container", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
@@ -215,7 +151,7 @@ func downloads(router *httprouter.Router, baseDir string, fsys fs.FS) {
safePath := safeResolve(baseDir, filepath.Join(container, itemPath)) safePath := safeResolve(baseDir, filepath.Join(container, itemPath))
var files []ContainerItem var files []ContainerItem
err := fs.WalkDir(fsys, safePath, func(path string, entry fs.DirEntry, err error) error { err := filepath.WalkDir(safePath, func(path string, entry fs.DirEntry, err error) error {
if !entry.IsDir() { if !entry.IsDir() {
rel, err := filepath.Rel(safePath, path) rel, err := filepath.Rel(safePath, path)
if err != nil { if err != nil {
@@ -241,17 +177,9 @@ func downloads(router *httprouter.Router, baseDir string, fsys fs.FS) {
panic(err) panic(err)
} }
json, err := json.Marshal(ContainerItemResponse{ writeJSON(w, ContainerItemResponse{
Value: files, Value: files,
}) })
if err != nil {
panic(err)
}
_, err = w.Write(json)
if err != nil {
panic(err)
}
}) })
router.GET("/artifact/*path", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) { router.GET("/artifact/*path", func(w http.ResponseWriter, req *http.Request, params httprouter.Params) {
@@ -259,15 +187,16 @@ func downloads(router *httprouter.Router, baseDir string, fsys fs.FS) {
safePath := safeResolve(baseDir, path) safePath := safeResolve(baseDir, path)
file, err := fsys.Open(safePath) file, err := os.Open(safePath)
if err != nil { if err != nil {
// try gzip file // try gzip file
file, err = fsys.Open(safePath + gzipExtension) file, err = os.Open(safePath + gzipExtension)
if err != nil { if err != nil {
panic(err) panic(err)
} }
w.Header().Add("Content-Encoding", "gzip") w.Header().Add("Content-Encoding", "gzip")
} }
defer file.Close()
_, err = io.Copy(w, file) _, err = io.Copy(w, file)
if err != nil { if err != nil {
@@ -287,9 +216,8 @@ func Serve(ctx context.Context, artifactPath, addr, port string) context.CancelF
router := httprouter.New() router := httprouter.New()
logger.Debugf("Artifacts base path '%s'", artifactPath) logger.Debugf("Artifacts base path '%s'", artifactPath)
fsys := readWriteFSImpl{} uploads(router, artifactPath)
uploads(router, artifactPath, fsys) downloads(router, artifactPath)
downloads(router, artifactPath, fsys)
server := &http.Server{ server := &http.Server{
Addr: fmt.Sprintf("%s:%s", addr, port), Addr: fmt.Sprintf("%s:%s", addr, port),
+22 -314
View File
@@ -7,8 +7,7 @@ package artifacts
import ( import (
"bytes" "bytes"
"compress/gzip" "compress/gzip"
"encoding/json" "encoding/json/v2"
"fmt"
"io" "io"
"maps" "maps"
"net/http" "net/http"
@@ -18,238 +17,18 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"testing/fstest"
"time" "time"
"github.com/julienschmidt/httprouter" "github.com/julienschmidt/httprouter"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
type writableMapFile struct {
fstest.MapFile
}
func (f *writableMapFile) Write(data []byte) (int, error) {
f.Data = data
return len(data), nil
}
func (f *writableMapFile) Close() error {
return nil
}
type writeMapFS struct {
fstest.MapFS
}
func (fsys writeMapFS) OpenWritable(name string) (WritableFile, error) {
file := &writableMapFile{
MapFile: fstest.MapFile{
Data: []byte("content2"),
},
}
fsys.MapFS[name] = &file.MapFile
return file, nil
}
func (fsys writeMapFS) OpenAppendable(name string) (WritableFile, error) {
file := &writableMapFile{
MapFile: fstest.MapFile{
Data: []byte("content2"),
},
}
fsys.MapFS[name] = &file.MapFile
return file, nil
}
func TestNewArtifactUploadPrepare(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{})
router := httprouter.New()
uploads(router, "artifact/server/path", writeMapFS{memfs})
req, _ := http.NewRequest(http.MethodPost, "http://localhost/_apis/pipelines/workflows/1/artifacts", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.Fail("Wrong status")
}
response := FileContainerResourceURL{}
err := json.Unmarshal(rr.Body.Bytes(), &response)
if err != nil {
panic(err)
}
assert.Equal("http://localhost/upload/1", response.FileContainerResourceURL)
}
func TestArtifactUploadBlob(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{})
router := httprouter.New()
uploads(router, "artifact/server/path", writeMapFS{memfs})
req, _ := http.NewRequest(http.MethodPut, "http://localhost/upload/1?itemPath=some/file", strings.NewReader("content"))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.Fail("Wrong status")
}
response := ResponseMessage{}
err := json.Unmarshal(rr.Body.Bytes(), &response)
if err != nil {
panic(err)
}
assert.Equal("success", response.Message)
assert.Equal("content", string(memfs["artifact/server/path/1/some/file"].Data))
}
func TestFinalizeArtifactUpload(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{})
router := httprouter.New()
uploads(router, "artifact/server/path", writeMapFS{memfs})
req, _ := http.NewRequest(http.MethodPatch, "http://localhost/_apis/pipelines/workflows/1/artifacts", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.Fail("Wrong status")
}
response := ResponseMessage{}
err := json.Unmarshal(rr.Body.Bytes(), &response)
if err != nil {
panic(err)
}
assert.Equal("success", response.Message)
}
func TestListArtifacts(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{
"artifact/server/path/1/file.txt": {
Data: []byte(""),
},
})
router := httprouter.New()
downloads(router, "artifact/server/path", memfs)
req, _ := http.NewRequest(http.MethodGet, "http://localhost/_apis/pipelines/workflows/1/artifacts", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.FailNow(fmt.Sprintf("Wrong status: %d", status))
}
response := NamedFileContainerResourceURLResponse{}
err := json.Unmarshal(rr.Body.Bytes(), &response)
if err != nil {
panic(err)
}
assert.Equal(1, response.Count)
assert.Equal("file.txt", response.Value[0].Name)
assert.Equal("http://localhost/download/1", response.Value[0].FileContainerResourceURL)
}
func TestListArtifactContainer(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{
"artifact/server/path/1/some/file": {
Data: []byte(""),
},
})
router := httprouter.New()
downloads(router, "artifact/server/path", memfs)
req, _ := http.NewRequest(http.MethodGet, "http://localhost/download/1?itemPath=some/file", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.FailNow(fmt.Sprintf("Wrong status: %d", status))
}
response := ContainerItemResponse{}
err := json.Unmarshal(rr.Body.Bytes(), &response)
if err != nil {
panic(err)
}
assert.Len(response.Value, 1)
assert.Equal("some/file", response.Value[0].Path)
assert.Equal("file", response.Value[0].ItemType)
assert.Equal("http://localhost/artifact/1/some/file/.", response.Value[0].ContentLocation)
}
func TestDownloadArtifactFile(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{
"artifact/server/path/1/some/file": {
Data: []byte("content"),
},
})
router := httprouter.New()
downloads(router, "artifact/server/path", memfs)
req, _ := http.NewRequest(http.MethodGet, "http://localhost/artifact/1/some/file", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.FailNow(fmt.Sprintf("Wrong status: %d", status))
}
data := rr.Body.Bytes()
assert.Equal("content", string(data))
}
// TestArtifactFlow drives the real Serve() artifact server over a loopback socket, exercising
// the same upload -> finalize -> list -> download protocol the upload-artifact/download-artifact
// actions speak. Running it in-process (rather than from a job container) keeps it network-free
// and reachable everywhere, including when the CI job is itself a container.
func TestArtifactFlow(t *testing.T) { func TestArtifactFlow(t *testing.T) {
artifactPath := t.TempDir() artifactPath := t.TempDir()
// Serve the exact routes Serve() wires up, on a real loopback socket via httptest. httptest
// picks a free port and Close() tears the server down synchronously — avoiding both the
// port-rebind race and Serve()'s detached ListenAndServe goroutine, which logger.Fatal()s
// (process exit) on a bind error and can outlive the test's temp-dir cleanup.
router := httprouter.New() router := httprouter.New()
fsys := readWriteFSImpl{} uploads(router, artifactPath)
uploads(router, artifactPath, fsys) downloads(router, artifactPath)
downloads(router, artifactPath, fsys)
server := httptest.NewServer(router) server := httptest.NewServer(router)
defer server.Close() defer server.Close()
@@ -257,8 +36,6 @@ func TestArtifactFlow(t *testing.T) {
client := server.Client() client := server.Client()
client.Timeout = 5 * time.Second client.Timeout = 5 * time.Second
// request performs one HTTP call and returns the status and body. The default transport adds
// Accept-Encoding: gzip and transparently decompresses, so gzipped downloads come back plain.
request := func(t *testing.T, method, rawURL string, body io.Reader, header http.Header) (int, []byte) { request := func(t *testing.T, method, rawURL string, body io.Reader, header http.Header) (int, []byte) {
t.Helper() t.Helper()
req, err := http.NewRequest(method, rawURL, body) req, err := http.NewRequest(method, rawURL, body)
@@ -289,6 +66,8 @@ func TestArtifactFlow(t *testing.T) {
status, data = request(t, http.MethodPatch, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil) status, data = request(t, http.MethodPatch, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data)) require.Equal(t, http.StatusOK, status, string(data))
require.NoError(t, json.Unmarshal(data, &msg))
require.Equal(t, "success", msg.Message)
status, data = request(t, http.MethodGet, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil) status, data = request(t, http.MethodGet, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data)) require.Equal(t, http.StatusOK, status, string(data))
@@ -314,6 +93,21 @@ func TestArtifactFlow(t *testing.T) {
require.Equal(t, content, string(stored)) require.Equal(t, content, string(stored))
}) })
t.Run("content-range", func(t *testing.T) {
const rawURL = "/upload/4?itemPath=chunks.txt"
status, data := request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("first"),
http.Header{"Content-Range": []string{"bytes 0-4/11"}})
require.Equal(t, http.StatusOK, status, string(data))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"),
http.Header{"Content-Range": []string{"bytes 5-11/11"}})
require.Equal(t, http.StatusOK, status, string(data))
stored, err := os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err)
require.Equal(t, "first-second", string(stored))
})
t.Run("gzip-roundtrip", func(t *testing.T) { t.Run("gzip-roundtrip", func(t *testing.T) {
const runID, item, content = "2", "logs/app.log", "compressed payload\n" const runID, item, content = "2", "logs/app.log", "compressed payload\n"
@@ -365,9 +159,7 @@ func TestArtifactFlow(t *testing.T) {
}) })
} }
func TestMkdirFsImplSafeResolve(t *testing.T) { func TestSafeResolve(t *testing.T) {
assert := assert.New(t)
baseDir := "/foo/bar" baseDir := "/foo/bar"
tests := map[string]struct { tests := map[string]struct {
@@ -385,97 +177,13 @@ func TestMkdirFsImplSafeResolve(t *testing.T) {
for name, tc := range tests { for name, tc := range tests {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
assert.Equal(tc.want, safeResolve(baseDir, tc.input)) require.Equal(t, tc.want, safeResolve(baseDir, tc.input))
}) })
} }
} }
func TestReadWriteFSWritableAndAppendable(t *testing.T) {
fsys := readWriteFSImpl{}
name := filepath.Join(t.TempDir(), "nested", "artifact.txt")
w, err := fsys.OpenWritable(name)
require.NoError(t, err)
_, err = w.Write([]byte("first"))
require.NoError(t, err)
require.NoError(t, w.Close())
w, err = fsys.OpenAppendable(name)
require.NoError(t, err)
_, err = w.Write([]byte("-second"))
require.NoError(t, err)
require.NoError(t, w.Close())
got, err := os.ReadFile(name)
require.NoError(t, err)
require.Equal(t, "first-second", string(got))
w, err = fsys.OpenWritable(name)
require.NoError(t, err)
_, err = w.Write([]byte("replaced"))
require.NoError(t, err)
require.NoError(t, w.Close())
got, err = os.ReadFile(name)
require.NoError(t, err)
require.Equal(t, "replaced", string(got))
}
func TestServeEmptyArtifactPathReturnsCancelableNoop(t *testing.T) { func TestServeEmptyArtifactPathReturnsCancelableNoop(t *testing.T) {
cancel := Serve(t.Context(), "", "127.0.0.1", "0") cancel := Serve(t.Context(), "", "127.0.0.1", "0")
require.NotNil(t, cancel) require.NotNil(t, cancel)
cancel() cancel()
} }
func TestDownloadArtifactFileUnsafePath(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{
"artifact/server/path/some/file": {
Data: []byte("content"),
},
})
router := httprouter.New()
downloads(router, "artifact/server/path", memfs)
req, _ := http.NewRequest(http.MethodGet, "http://localhost/artifact/2/../../some/file", nil)
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.FailNow(fmt.Sprintf("Wrong status: %d", status))
}
data := rr.Body.Bytes()
assert.Equal("content", string(data))
}
func TestArtifactUploadBlobUnsafePath(t *testing.T) {
assert := assert.New(t)
memfs := fstest.MapFS(map[string]*fstest.MapFile{})
router := httprouter.New()
uploads(router, "artifact/server/path", writeMapFS{memfs})
req, _ := http.NewRequest(http.MethodPut, "http://localhost/upload/1?itemPath=../../some/file", strings.NewReader("content"))
rr := httptest.NewRecorder()
router.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
assert.Fail("Wrong status")
}
response := ResponseMessage{}
err := json.Unmarshal(rr.Body.Bytes(), &response)
if err != nil {
panic(err)
}
assert.Equal("success", response.Message)
assert.Equal("content", string(memfs["artifact/server/path/1/some/file"].Data))
}
+1 -24
View File
@@ -54,22 +54,6 @@ func NewPipelineExecutor(executors ...Executor) Executor {
return rtn return rtn
} }
// NewConditionalExecutor creates a new executor based on conditions
func NewConditionalExecutor(conditional Conditional, trueExecutor, falseExecutor Executor) Executor {
return func(ctx context.Context) error {
if conditional(ctx) {
if trueExecutor != nil {
return trueExecutor(ctx)
}
} else {
if falseExecutor != nil {
return falseExecutor(ctx)
}
}
return nil
}
}
// NewErrorExecutor creates a new executor that always errors out // NewErrorExecutor creates a new executor that always errors out
func NewErrorExecutor(err error) Executor { func NewErrorExecutor(err error) Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
@@ -187,15 +171,8 @@ func (e Executor) Finally(finally Executor) Executor {
err := e(ctx) err := e(ctx)
err2 := finally(ctx) err2 := finally(ctx)
if err2 != nil { if err2 != nil {
return fmt.Errorf("Error occurred running finally: %v (original error: %v)", err2, err) return fmt.Errorf("error occurred running finally: %v (original error: %v)", err2, err)
} }
return err return err
} }
} }
// Not return an inverted conditional
func (c Conditional) Not() Conditional {
return func(ctx context.Context) bool {
return !c(ctx)
}
}
-44
View File
@@ -45,43 +45,6 @@ func TestNewWorkflow(t *testing.T) {
assert.Equal(2, runcount) assert.Equal(2, runcount)
} }
func TestNewConditionalExecutor(t *testing.T) {
assert := assert.New(t)
ctx := context.Background()
trueCount := 0
falseCount := 0
err := NewConditionalExecutor(func(ctx context.Context) bool {
return false
}, func(ctx context.Context) error {
trueCount++
return nil
}, func(ctx context.Context) error {
falseCount++
return nil
})(ctx)
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(0, trueCount)
assert.Equal(1, falseCount)
err = NewConditionalExecutor(func(ctx context.Context) bool {
return true
}, func(ctx context.Context) error {
trueCount++
return nil
}, func(ctx context.Context) error {
falseCount++
return nil
})(ctx)
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(1, trueCount)
assert.Equal(1, falseCount)
}
// concurrencyProbe returns an executor recording the peak number of concurrent copies. Copies // concurrencyProbe returns an executor recording the peak number of concurrent copies. Copies
// block until wantActive are in flight so the peak is exact without sleeping, and later copies // block until wantActive are in flight so the peak is exact without sleeping, and later copies
// find the gate already open so the last one still finishes with no partner left. // find the gate already open so the last one still finishes with no partner left.
@@ -223,10 +186,3 @@ func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
t.Fatalf("finally error = %q, want both cleanup and original error", err) t.Fatalf("finally error = %q, want both cleanup and original error", err)
} }
} }
func TestConditionalNot(t *testing.T) {
cond := Conditional(func(context.Context) bool { return false })
if !cond.Not()(context.Background()) {
t.Fatal("inverted conditional should be true")
}
}
+106 -56
View File
@@ -36,7 +36,6 @@ var (
cloneLocks lock.Keyed[string] // key: clone target directory cloneLocks lock.Keyed[string] // key: clone target directory
ErrShortRef = errors.New("short SHA references are not supported") ErrShortRef = errors.New("short SHA references are not supported")
ErrNoRepo = errors.New("unable to find git repo")
) )
// AcquireCloneLock returns an unlock function after locking the per-directory mutex for dir. // AcquireCloneLock returns an unlock function after locking the per-directory mutex for dir.
@@ -187,19 +186,16 @@ func FindGitRef(ctx context.Context, file string) (string, error) {
} }
// FindGithubRepo get the repo // FindGithubRepo get the repo
func FindGithubRepo(ctx context.Context, file, githubInstance, remoteName string) (string, error) { func FindGithubRepo(ctx context.Context, file, githubInstance string) (string, error) {
goGitMu.Lock() goGitMu.Lock()
defer goGitMu.Unlock() defer goGitMu.Unlock()
if remoteName == "" {
remoteName = "origin"
}
url, err := findGitRemoteURL(ctx, file, remoteName) url, err := findGitRemoteURL(ctx, file, "origin")
if err != nil { if err != nil {
return "", err return "", err
} }
_, slug, err := findGitSlug(url, githubInstance) _, slug := findGitSlug(url, githubInstance)
return slug, err return slug, nil
} }
func findGitRemoteURL(_ context.Context, file, remoteName string) (string, error) { func findGitRemoteURL(_ context.Context, file, remoteName string) (string, error) {
@@ -226,25 +222,25 @@ func findGitRemoteURL(_ context.Context, file, remoteName string) (string, error
return remote.Config().URLs[0], nil return remote.Config().URLs[0], nil
} }
func findGitSlug(url, githubInstance string) (string, string, error) { //nolint:unparam // pre-existing issue from nektos/act func findGitSlug(url, githubInstance string) (string, string) {
if matches := codeCommitHTTPRegex.FindStringSubmatch(url); matches != nil { if matches := codeCommitHTTPRegex.FindStringSubmatch(url); matches != nil {
return "CodeCommit", matches[2], nil return "CodeCommit", matches[2]
} else if matches := codeCommitSSHRegex.FindStringSubmatch(url); matches != nil { } else if matches := codeCommitSSHRegex.FindStringSubmatch(url); matches != nil {
return "CodeCommit", matches[2], nil return "CodeCommit", matches[2]
} else if matches := githubHTTPRegex.FindStringSubmatch(url); matches != nil { } else if matches := githubHTTPRegex.FindStringSubmatch(url); matches != nil {
return "GitHub", fmt.Sprintf("%s/%s", matches[1], matches[2]), nil return "GitHub", fmt.Sprintf("%s/%s", matches[1], matches[2])
} else if matches := githubSSHRegex.FindStringSubmatch(url); matches != nil { } else if matches := githubSSHRegex.FindStringSubmatch(url); matches != nil {
return "GitHub", fmt.Sprintf("%s/%s", matches[1], matches[2]), nil return "GitHub", fmt.Sprintf("%s/%s", matches[1], matches[2])
} else if githubInstance != "github.com" { } else if githubInstance != "github.com" {
gheHTTPRegex := regexp.MustCompile(fmt.Sprintf(`^https?://%s/(.+)/(.+?)(?:.git)?$`, githubInstance)) gheHTTPRegex := regexp.MustCompile(fmt.Sprintf(`^https?://%s/(.+)/(.+?)(?:.git)?$`, githubInstance))
gheSSHRegex := regexp.MustCompile(githubInstance + "[:/](.+)/(.+?)(?:.git)?$") gheSSHRegex := regexp.MustCompile(githubInstance + "[:/](.+)/(.+?)(?:.git)?$")
if matches := gheHTTPRegex.FindStringSubmatch(url); matches != nil { if matches := gheHTTPRegex.FindStringSubmatch(url); matches != nil {
return "GitHubEnterprise", fmt.Sprintf("%s/%s", matches[1], matches[2]), nil return "GitHubEnterprise", fmt.Sprintf("%s/%s", matches[1], matches[2])
} else if matches := gheSSHRegex.FindStringSubmatch(url); matches != nil { } else if matches := gheSSHRegex.FindStringSubmatch(url); matches != nil {
return "GitHubEnterprise", fmt.Sprintf("%s/%s", matches[1], matches[2]), nil return "GitHubEnterprise", fmt.Sprintf("%s/%s", matches[1], matches[2])
} }
} }
return "", url, nil return "", url
} }
// NewGitCloneExecutorInput the input for the NewGitCloneExecutor // NewGitCloneExecutorInput the input for the NewGitCloneExecutor
@@ -278,11 +274,12 @@ func CloneIfRequired(ctx context.Context, refName plumbing.ReferenceName, input
return r, true, nil return r, true, nil
} }
if err != nil { switch {
case err != nil:
logger.Debugf("Removing cached clone at %s because origin cannot be read: %v", input.Dir, err) logger.Debugf("Removing cached clone at %s because origin cannot be read: %v", input.Dir, err)
} else if len(remote.Config().URLs) == 0 { case len(remote.Config().URLs) == 0:
logger.Debugf("Removing cached clone at %s because origin has no URL", input.Dir) logger.Debugf("Removing cached clone at %s because origin has no URL", input.Dir)
} else { default:
logger.Debugf("Removing cached clone at %s because origin URL changed from %s to %s", input.Dir, remote.Config().URLs[0], input.URL) logger.Debugf("Removing cached clone at %s because origin URL changed from %s to %s", input.Dir, remote.Config().URLs[0], input.URL)
} }
if err := os.RemoveAll(input.Dir); err != nil { if err := os.RemoveAll(input.Dir); err != nil {
@@ -394,7 +391,11 @@ func NewGitCloneExecutor(input NewGitCloneExecutorInput) common.Executor {
} }
} }
if !isOfflineMode { // A just-cloned ref is as current as a fetch would make it, and a commit hash never moves.
_, _, present := refRevision(r, input.Ref)
refresh := !isOfflineMode && (!present || (reused && !plumbing.IsHash(input.Ref)))
if refresh {
err = r.FetchContext(ctx, &fetchOptions) err = r.FetchContext(ctx, &fetchOptions)
if err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) { if err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) {
return err return err
@@ -414,23 +415,7 @@ func NewGitCloneExecutor(input NewGitCloneExecutorInput) common.Executor {
} }
} }
// At this point we need to know if it's a tag or a branch rev, refType, _ := refRevision(r, input.Ref)
// And the easiest way to do it is duck typing
//
// If err is nil, it's a tag so let's proceed with that hash like we would if
// it was a sha
refType := "tag"
rev = plumbing.Revision(path.Join("refs", "tags", input.Ref))
if _, err := r.Tag(input.Ref); errors.Is(err, git.ErrTagNotFound) {
rName := plumbing.ReferenceName(path.Join("refs", "remotes", "origin", input.Ref))
if _, err := r.Reference(rName, false); errors.Is(err, plumbing.ErrReferenceNotFound) {
refType = "sha"
rev = plumbing.Revision(input.Ref)
} else {
refType = "branch"
rev = plumbing.Revision(rName)
}
}
if hash, err = r.ResolveRevision(rev); err != nil { if hash, err = r.ResolveRevision(rev); err != nil {
logger.Errorf("Unable to resolve %s: %v", input.Ref, err) logger.Errorf("Unable to resolve %s: %v", input.Ref, err)
@@ -462,7 +447,7 @@ func NewGitCloneExecutor(input NewGitCloneExecutorInput) common.Executor {
reusedMsg := "" reusedMsg := ""
switch { switch {
case !isOfflineMode && !shallow: case refresh && !shallow:
// In shallow mode the depth-limited fetch above already advanced the ref. // In shallow mode the depth-limited fetch above already advanced the ref.
if err = w.PullContext(ctx, &pullOptions); err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) { if err = w.PullContext(ctx, &pullOptions); err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) {
logger.Debugf("Unable to pull %s: %v", refName, err) logger.Debugf("Unable to pull %s: %v", refName, err)
@@ -504,27 +489,37 @@ func NewGitCloneExecutor(input NewGitCloneExecutorInput) common.Executor {
} }
} }
// cloneAtDepth clones input.URL into input.Dir using opts. // cloneAtDepth falls back to a full clone when the shallow attempt fails.
// With input.Depth > 0 it first tries a shallow, single-branch clone of input.Ref, falling back when error.
func cloneAtDepth(ctx context.Context, input NewGitCloneExecutorInput, opts git.CloneOptions, logger log.FieldLogger) (*git.Repository, error) { func cloneAtDepth(ctx context.Context, input NewGitCloneExecutorInput, opts git.CloneOptions, logger log.FieldLogger) (*git.Repository, error) {
if input.Depth > 0 { if input.Depth > 0 {
for _, refName := range []plumbing.ReferenceName{ if plumbing.IsHash(input.Ref) {
plumbing.NewBranchReferenceName(input.Ref), r, err := fetchPinnedSHA(ctx, input, opts)
plumbing.NewTagReferenceName(input.Ref),
} {
shallowOpts := opts
shallowOpts.Depth = input.Depth
shallowOpts.SingleBranch = true
shallowOpts.ReferenceName = refName
shallowOpts.Tags = git.NoTags
r, err := git.PlainCloneContext(ctx, input.Dir, false, &shallowOpts)
if err == nil { if err == nil {
return r, nil return r, nil
} }
logger.Debugf("Shallow clone of %s as %s failed: %v", input.URL, refName, err) logger.Debugf("Shallow fetch of %s at %s failed: %v", input.URL, input.Ref, err)
if rmErr := os.RemoveAll(input.Dir); rmErr != nil { if err := removePartialClone(input.Dir); err != nil {
return nil, fmt.Errorf("remove partial clone %s: %w", input.Dir, rmErr) return nil, err
}
} else {
for _, refName := range []plumbing.ReferenceName{
plumbing.NewBranchReferenceName(input.Ref),
plumbing.NewTagReferenceName(input.Ref),
} {
shallowOpts := opts
shallowOpts.Depth = input.Depth
shallowOpts.SingleBranch = true
shallowOpts.ReferenceName = refName
shallowOpts.Tags = git.NoTags
r, err := git.PlainCloneContext(ctx, input.Dir, false, &shallowOpts)
if err == nil {
return r, nil
}
logger.Debugf("Shallow clone of %s as %s failed: %v", input.URL, refName, err)
if err := removePartialClone(input.Dir); err != nil {
return nil, err
}
} }
} }
logger.Debugf("Falling back to a full clone of %s for ref %q", input.URL, input.Ref) logger.Debugf("Falling back to a full clone of %s for ref %q", input.URL, input.Ref)
@@ -533,14 +528,65 @@ func cloneAtDepth(ctx context.Context, input NewGitCloneExecutorInput, opts git.
return git.PlainCloneContext(ctx, input.Dir, false, &opts) return git.PlainCloneContext(ctx, input.Dir, false, &opts)
} }
func removePartialClone(dir string) error {
if err := os.RemoveAll(dir); err != nil {
return fmt.Errorf("remove partial clone %s: %w", dir, err)
}
return nil
}
func fetchPinnedSHA(ctx context.Context, input NewGitCloneExecutorInput, opts git.CloneOptions) (*git.Repository, error) {
r, err := git.PlainInit(input.Dir, false)
if err != nil {
return nil, err
}
if _, err := r.CreateRemote(&config.RemoteConfig{Name: "origin", URLs: []string{input.URL}}); err != nil {
return nil, err
}
if err := r.FetchContext(ctx, &git.FetchOptions{
RefSpecs: []config.RefSpec{pinnedRefSpec(input.Ref)},
Depth: input.Depth,
Tags: git.NoTags,
Auth: opts.Auth,
Progress: opts.Progress,
InsecureSkipTLS: opts.InsecureSkipTLS,
}); err != nil {
return nil, err
}
return r, nil
}
// pinnedRefSpec keeps a hash-shaped name out of refs/heads and refs/tags.
func pinnedRefSpec(sha string) config.RefSpec {
return config.RefSpec(fmt.Sprintf("+%s:refs/pinned/%s", sha, sha))
}
// refRevision picks the revision to check out and reports whether it resolves locally.
func refRevision(r *git.Repository, ref string) (plumbing.Revision, string, bool) {
if plumbing.IsHash(ref) {
// git ignores a ref named as 40 hex digits, so a full hash always denotes the commit itself.
_, err := r.CommitObject(plumbing.NewHash(ref))
return plumbing.Revision(ref), "sha", err == nil
}
if _, err := r.Tag(ref); err == nil {
return plumbing.Revision(path.Join("refs", "tags", ref)), "tag", true
}
remoteRef := plumbing.ReferenceName(path.Join("refs", "remotes", "origin", ref))
if _, err := r.Reference(remoteRef, false); err == nil {
return plumbing.Revision(remoteRef), "branch", true
}
rev := plumbing.Revision(ref)
_, err := r.ResolveRevision(rev)
return rev, "sha", err == nil
}
// isShallow reports whether the local repository was cloned with a limited depth. // isShallow reports whether the local repository was cloned with a limited depth.
func isShallow(r *git.Repository) bool { func isShallow(r *git.Repository) bool {
shallows, err := r.Storer.Shallow() shallows, err := r.Storer.Shallow()
return err == nil && len(shallows) > 0 return err == nil && len(shallows) > 0
} }
// shallowFetchRefSpec returns the single refspec that updates only input.Ref, keeping a shallow clone from re-downloading every branch's history. // shallowFetchRefSpec limits a shallow update to the requested ref, falling back to the broad refspec when it is not present locally.
// ok is false when the ref is not present locally as a tag or remote-tracking branch, in which case the broad default refspec is used.
func shallowFetchRefSpec(r *git.Repository, ref string) (config.RefSpec, bool) { func shallowFetchRefSpec(r *git.Repository, ref string) (config.RefSpec, bool) {
tagRef := plumbing.NewTagReferenceName(ref) tagRef := plumbing.NewTagReferenceName(ref)
if _, err := r.Reference(tagRef, false); err == nil { if _, err := r.Reference(tagRef, false); err == nil {
@@ -551,5 +597,9 @@ func shallowFetchRefSpec(r *git.Repository, ref string) (config.RefSpec, bool) {
branchRef := plumbing.NewBranchReferenceName(ref) branchRef := plumbing.NewBranchReferenceName(ref)
return config.RefSpec(fmt.Sprintf("+%s:%s", branchRef, remoteRef)), true return config.RefSpec(fmt.Sprintf("+%s:%s", branchRef, remoteRef)), true
} }
if plumbing.IsHash(ref) {
// The broad refspec carries only advertised tips, never a pinned commit.
return pinnedRefSpec(ref), true
}
return "", false return "", false
} }
+119 -42
View File
@@ -16,6 +16,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"syscall" "syscall"
"testing" "testing"
"time" "time"
@@ -24,6 +25,9 @@ import (
gogit "github.com/go-git/go-git/v5" gogit "github.com/go-git/go-git/v5"
gogitconfig "github.com/go-git/go-git/v5/config" gogitconfig "github.com/go-git/go-git/v5/config"
"github.com/go-git/go-git/v5/plumbing/transport"
gogitclient "github.com/go-git/go-git/v5/plumbing/transport/client"
gogitfile "github.com/go-git/go-git/v5/plumbing/transport/file"
log "github.com/sirupsen/logrus" log "github.com/sirupsen/logrus"
logrustest "github.com/sirupsen/logrus/hooks/test" logrustest "github.com/sirupsen/logrus/hooks/test"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -51,9 +55,7 @@ func TestFindGitSlug(t *testing.T) {
} }
for _, tt := range slugTests { for _, tt := range slugTests {
provider, slug, err := findGitSlug(tt.url, "github.com") provider, slug := findGitSlug(tt.url, "github.com")
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(tt.provider, provider) assert.Equal(tt.provider, provider)
assert.Equal(tt.slug, slug) assert.Equal(tt.slug, slug)
} }
@@ -87,45 +89,20 @@ func cleanGitHooks(dir string) error {
return nil return nil
} }
func TestFindGitRemoteURL(t *testing.T) { func TestFindGithubRepoUsesOrigin(t *testing.T) {
assert := assert.New(t)
basedir := t.TempDir()
err := gitCmd("init", basedir)
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
err = cleanGitHooks(basedir)
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
remoteURL := "https://git-codecommit.us-east-1.amazonaws.com/v1/repos/my-repo-name"
err = gitCmd("-C", basedir, "remote", "add", "origin", remoteURL)
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
u, err := findGitRemoteURL(context.Background(), basedir, "origin")
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(remoteURL, u)
remoteURL = "[email protected]/AwesomeOwner/MyAwesomeRepo.git"
err = gitCmd("-C", basedir, "remote", "add", "upstream", remoteURL)
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
u, err = findGitRemoteURL(context.Background(), basedir, "upstream")
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(remoteURL, u)
}
func TestFindGithubRepoUsesOriginAndCustomRemote(t *testing.T) {
basedir := t.TempDir() basedir := t.TempDir()
const remoteURL = "https://github.com/owner/repo.git"
require.NoError(t, gitCmd("init", basedir)) require.NoError(t, gitCmd("init", basedir))
require.NoError(t, cleanGitHooks(basedir)) require.NoError(t, cleanGitHooks(basedir))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", "https://github.com/owner/repo.git")) require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "ghe", "[email protected]:team/project.git"))
slug, err := FindGithubRepo(context.Background(), basedir, "github.com", "") url, err := findGitRemoteURL(context.Background(), basedir, "origin")
require.NoError(t, err)
require.Equal(t, remoteURL, url)
slug, err := FindGithubRepo(context.Background(), basedir, "github.com")
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, "owner/repo", slug) require.Equal(t, "owner/repo", slug)
slug, err = FindGithubRepo(context.Background(), basedir, "git.example.com", "ghe")
require.NoError(t, err)
require.Equal(t, "team/project", slug)
} }
func TestGitFindRef(t *testing.T) { func TestGitFindRef(t *testing.T) {
@@ -462,7 +439,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", m)) require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", m))
} }
require.NoError(t, gitCmd("-C", workDir, "tag", "v1")) require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
sha := gitRevParse(t, workDir, "HEAD~1") // c2, a SHA that go-git cannot shallow-clone unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main")) require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v1")) require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v1"))
@@ -488,14 +465,43 @@ func TestGitCloneExecutorShallow(t *testing.T) {
assert.Equal(t, "c3", gitHeadSubject(t, dir)) assert.Equal(t, "c3", gitHeadSubject(t, dir))
}) })
t.Run("SHA falls back to a full clone", func(t *testing.T) { unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string {
remote := filepath.Join(t.TempDir(), "remote.git")
require.NoError(t, gitCmd("clone", "--bare", remoteDir, remote))
require.NoError(t, gitCmd("-C", remote, "config", "uploadpack.allowAnySHA1InWant", strconv.FormatBool(allowUnadvertised)))
return remote
}
t.Run("commit hash falls back to a full clone when the remote refuses unadvertised objects", func(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{ require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: sha, Dir: dir, Depth: 1, URL: unadvertisedRemote(t, false), Ref: unadvertisedSHA, Dir: dir, Depth: 1,
})(t.Context())) })(t.Context()))
// go-git cannot shallow-clone a raw SHA, so it falls back to a full clone; the absence of a shallow marker proves the fallback happened. assert.NoFileExists(t, shallowMarker(dir))
assert.NoFileExists(t, shallowMarker(dir), "a SHA ref must not produce a shallow clone") assert.Equal(t, unadvertisedSHA, gitRevParse(t, dir, "HEAD"))
assert.Equal(t, sha, gitRevParse(t, dir, "HEAD")) })
t.Run("commit hash is fetched shallowly, moved to another hash, then reused without reaching the remote", func(t *testing.T) {
remote := unadvertisedRemote(t, true)
dir := t.TempDir()
cloneAt := func(ref string) error {
return NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remote, Ref: ref, Dir: dir, Depth: 1,
})(t.Context())
}
require.NoError(t, cloneAt(unadvertisedSHA))
assert.FileExists(t, shallowMarker(dir))
assert.Equal(t, 1, gitRevCount(t, dir))
assert.Equal(t, unadvertisedSHA, gitRevParse(t, dir, "HEAD"))
olderSHA := gitRevParse(t, workDir, "HEAD~2")
require.NoError(t, cloneAt(olderSHA))
assert.Equal(t, olderSHA, gitRevParse(t, dir, "HEAD"))
require.NoError(t, os.RemoveAll(remote))
require.NoError(t, cloneAt(olderSHA))
assert.Equal(t, olderSHA, gitRevParse(t, dir, "HEAD"))
}) })
t.Run("moving branch updates while staying shallow", func(t *testing.T) { t.Run("moving branch updates while staying shallow", func(t *testing.T) {
@@ -518,6 +524,77 @@ func TestGitCloneExecutorShallow(t *testing.T) {
}) })
} }
func TestGitCloneExecutorColdCloneSkipsRefresh(t *testing.T) {
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "c1"))
require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v1"))
for name, tt := range map[string]struct {
Ref string
Depth int
}{
"shallow branch": {"main", 1},
"full clone tag": {"v1", 0},
} {
t.Run(name, func(t *testing.T) {
counter := installCountingTransport(t)
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: tt.Ref, Dir: t.TempDir(), Depth: tt.Depth,
})(t.Context()))
assert.Equal(t, int64(1), counter.sessions.Load())
})
}
}
func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "pinned"))
pinned := gitRevParse(t, workDir, "HEAD")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy"))
require.NoError(t, gitCmd("-C", workDir, "tag", pinned))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", pinned))
for name, depth := range map[string]int{"shallow": 1, "full clone": 0} {
t.Run(name, func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: pinned, Dir: dir, Depth: depth,
})(t.Context()))
assert.Equal(t, pinned, gitRevParse(t, dir, "HEAD"))
})
}
}
type countingTransport struct {
transport.Transport
sessions atomic.Int64
}
func (c *countingTransport) NewUploadPackSession(ep *transport.Endpoint, auth transport.AuthMethod) (transport.UploadPackSession, error) {
c.sessions.Add(1)
return c.Transport.NewUploadPackSession(ep, auth)
}
func installCountingTransport(t *testing.T) *countingTransport {
t.Helper()
counter := &countingTransport{Transport: gogitfile.DefaultClient}
gogitclient.InstallProtocol("file", counter)
t.Cleanup(func() { gogitclient.InstallProtocol("file", gogitfile.DefaultClient) })
return counter
}
func gitRevParse(t *testing.T, dir, rev string) string { func gitRevParse(t *testing.T, dir, rev string) string {
t.Helper() t.Helper()
out, err := exec.Command("git", "-C", dir, "rev-parse", rev).Output() out, err := exec.Command("git", "-C", dir, "rev-parse", rev).Output()
+6 -6
View File
@@ -46,14 +46,14 @@ func (lw *lineWriter) Write(p []byte) (n int, err error) {
line, err := pBuf.ReadString('\n') line, err := pBuf.ReadString('\n')
w, _ := lw.buffer.WriteString(line) w, _ := lw.buffer.WriteString(line)
written += w written += w
if err == nil { if err != nil {
lw.handleLine(lw.buffer.String()) if err == io.EOF {
lw.buffer.Reset() break
} else if err == io.EOF { }
break
} else {
return written, err return written, err
} }
lw.handleLine(lw.buffer.String())
lw.buffer.Reset()
} }
return written, nil return written, nil
+37 -23
View File
@@ -9,6 +9,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"sync"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -25,26 +26,27 @@ func (e ExitCodeError) Error() string {
// NewContainerInput the input for the New function // NewContainerInput the input for the New function
type NewContainerInput struct { type NewContainerInput struct {
Image string Image string
Username string Username string
Password string Password string
Entrypoint []string Entrypoint []string
Cmd []string Cmd []string
WorkingDir string WorkingDir string
Env []string Env []string
Binds []string Binds []string
Mounts map[string]string Mounts map[string]string
Name string Name string
Stdout io.Writer Stdout io.Writer
Stderr io.Writer Stderr io.Writer
NetworkMode string NetworkMode string
Privileged bool Privileged bool
UsernsMode string UsernsMode string
Platform string Platform string
Options string RunnerOptions string // container options the runner was configured with, trusted
NetworkAliases []string WorkflowOptions string // container options the workflow asked for, untrusted
ExposedPorts nat.PortSet NetworkAliases []string
PortBindings nat.PortMap ExposedPorts nat.PortSet
PortBindings nat.PortMap
// Gitea specific // Gitea specific
AutoRemove bool AutoRemove bool
@@ -74,6 +76,19 @@ const (
// fragment, missingContainerError composes it into the message every operation shares. // fragment, missingContainerError composes it into the message every operation shares.
var ErrContainerNotFound = errors.New("does not exist") var ErrContainerNotFound = errors.New("does not exist")
// DockerProxy is a job's docker socket, fronting the daemon's for the job's lifetime.
type DockerProxy struct {
Socket string
close func(context.Context) error
closeOnce sync.Once
closeErr error
}
func (p *DockerProxy) Close(ctx context.Context) error {
p.closeOnce.Do(func() { p.closeErr = p.close(ctx) })
return p.closeErr
}
// Info is a snapshot of a container, as of one inspect. // Info is a snapshot of a container, as of one inspect.
type Info struct { type Info struct {
ID string ID string
@@ -83,15 +98,14 @@ type Info struct {
// HealthOutput is the last healthcheck probe's output. // HealthOutput is the last healthcheck probe's output.
HealthOutput string HealthOutput string
Ports map[string]string // container port ("5432") to the host port it is published on Ports map[string]string // container port ("5432") to the host port it is published on
Mounts map[string]string // container path to its source on the daemon
} }
// Container for managing docker run containers // Container for managing docker run containers
type Container interface { type Container interface {
Create(capAdd, capDrop []string) common.Executor Create(capAdd, capDrop []string) common.Executor
ConnectToNetwork(name string) common.Executor
Copy(destPath string, files ...*FileEntry) common.Executor Copy(destPath string, files ...*FileEntry) common.Executor
CopyTarStream(ctx context.Context, destPath string, tarStream io.Reader) error CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor
CopyDir(destPath, srcPath string, useGitIgnore bool) common.Executor
GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error)
Inspect(ctx context.Context) (*Info, error) Inspect(ctx context.Context) (*Info, error)
DumpLogs(ctx context.Context) error DumpLogs(ctx context.Context) error
+5 -6
View File
@@ -17,8 +17,7 @@
package container package container
import ( import (
"bytes" "encoding/json/jsontext"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"net" "net"
@@ -351,7 +350,7 @@ type containerConfig struct {
// parse parses the args for the specified command and generates a Config, // parse parses the args for the specified command and generates a Config,
// a HostConfig and returns them with the specified command. // a HostConfig and returns them with the specified command.
// If the specified args are not valid, it will return an error. // If the specified args are not valid, it will return an error.
func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*containerConfig, error) { //nolint:gocyclo // verbatim copy from docker/cli func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*containerConfig, error) { //nolint:gocyclo,unparam // verbatim copy from docker/cli
var ( var (
attachStdin = copts.attach.Get("stdin") attachStdin = copts.attach.Get("stdin")
attachStdout = copts.attach.Get("stdout") attachStdout = copts.attach.Get("stdout")
@@ -959,11 +958,11 @@ func parseSecurityOpts(securityOpts []string) ([]string, error) {
if err != nil { if err != nil {
return securityOpts, fmt.Errorf("opening seccomp profile (%s) failed: %w", v, err) return securityOpts, fmt.Errorf("opening seccomp profile (%s) failed: %w", v, err)
} }
var b bytes.Buffer profile := jsontext.Value(f)
if err := json.Compact(&b, f); err != nil { if err := profile.Compact(); err != nil {
return securityOpts, fmt.Errorf("compacting json for seccomp profile (%s) failed: %w", v, err) return securityOpts, fmt.Errorf("compacting json for seccomp profile (%s) failed: %w", v, err)
} }
securityOpts[key] = "seccomp=" + b.String() securityOpts[key] = "seccomp=" + string(profile)
} }
} }
} }
+29 -2
View File
@@ -10,7 +10,9 @@ import (
"fmt" "fmt"
"io" "io"
"slices" "slices"
"strings"
"github.com/docker/cli/opts"
"github.com/kballard/go-shellquote" "github.com/kballard/go-shellquote"
"github.com/spf13/pflag" "github.com/spf13/pflag"
) )
@@ -51,15 +53,16 @@ func parseContainerOptions(options string) (*pflag.FlagSet, *containerOptions, *
flags := pflag.NewFlagSet("container_flags", pflag.ContinueOnError) flags := pflag.NewFlagSet("container_flags", pflag.ContinueOnError)
flags.SetOutput(io.Discard) flags.SetOutput(io.Discard)
copts := addFlags(flags) copts := addFlags(flags)
copts.env = opts.NewListOpts(validateEnv) // addFlags registered this field's address, so the swap takes effect
cf := registerCreateFlags(flags) cf := registerCreateFlags(flags)
args, err := shellquote.Split(options) args, err := shellquote.Split(options)
if err != nil { if err != nil {
return flags, copts, cf, fmt.Errorf("Cannot split container options: '%s': '%w'", options, err) return flags, copts, cf, fmt.Errorf("cannot split container options: '%s': '%w'", options, err)
} }
if err := flags.Parse(args); err != nil { if err := flags.Parse(args); err != nil {
return flags, copts, cf, fmt.Errorf("Cannot parse container options: '%s': '%w'", options, err) return flags, copts, cf, fmt.Errorf("cannot parse container options: '%s': '%w'", options, err)
} }
return flags, copts, cf, nil return flags, copts, cf, nil
@@ -73,6 +76,30 @@ func createFlagsFromOptions(options string) *createFlags {
return cf return cf
} }
// validateEnv is opts.ValidateEnv without its lookup of a bare name in the runner's environment.
func validateEnv(val string) (string, error) {
if name, _, _ := strings.Cut(val, "="); name == "" {
return "", errors.New("invalid environment variable: " + val)
}
return val, nil
}
// rejectHostReadingOptions refuses the flags naming files that are read here, on the
// runner, rather than in the container.
func rejectHostReadingOptions(options string) error {
flags, _, _, err := parseContainerOptions(options)
if err != nil {
return err
}
for _, name := range []string{"env-file", "label-file"} {
if flags.Changed(name) {
return fmt.Errorf("container option --%s reads files from the runner and is not allowed in a workflow", name)
}
}
return nil
}
func (cf *createFlags) validate() error { func (cf *createFlags) validate() error {
if !slices.Contains(pullPolicies, cf.pull) { if !slices.Contains(pullPolicies, cf.pull) {
return fmt.Errorf("invalid --pull option %q: must be one of %q", cf.pull, pullPolicies) return fmt.Errorf("invalid --pull option %q: must be one of %q", cf.pull, pullPolicies)
+2 -2
View File
@@ -50,13 +50,13 @@ func TestCreateFlagsValidate(t *testing.T) {
} }
func TestNewContainerAppliesCreateFlags(t *testing.T) { func TestNewContainerAppliesCreateFlags(t *testing.T) {
input := &NewContainerInput{Platform: "linux/amd64", Options: "--platform linux/arm64 --pull never"} input := &NewContainerInput{Platform: "linux/amd64", RunnerOptions: "--pull never", WorkflowOptions: "--platform linux/arm64"}
cr, ok := NewContainer(input).(*containerReference) cr, ok := NewContainer(input).(*containerReference)
require.True(t, ok) require.True(t, ok)
assert.Equal(t, "linux/arm64", input.Platform) assert.Equal(t, "linux/arm64", input.Platform)
assert.Equal(t, pullPolicyNever, cr.pullPolicy) assert.Equal(t, pullPolicyNever, cr.pullPolicy)
kept := &NewContainerInput{Platform: "linux/amd64", Options: "--privileged"} kept := &NewContainerInput{Platform: "linux/amd64", RunnerOptions: "--privileged"}
NewContainer(kept) NewContainer(kept)
assert.Equal(t, "linux/amd64", kept.Platform) assert.Equal(t, "linux/amd64", kept.Platform)
} }
+7 -6
View File
@@ -8,7 +8,7 @@ package container
import ( import (
"bufio" "bufio"
"encoding/json" "encoding/json/v2"
"errors" "errors"
"io" "io"
@@ -20,8 +20,8 @@ type dockerMessage struct {
Stream string `json:"stream"` Stream string `json:"stream"`
Error string `json:"error"` Error string `json:"error"`
ErrorDetail struct { ErrorDetail struct {
Message string Message string `json:"message"`
} } `json:"errorDetail"`
Status string `json:"status"` Status string `json:"status"`
Progress string `json:"progress"` Progress string `json:"progress"`
} }
@@ -60,15 +60,16 @@ func logDockerResponse(logger logrus.FieldLogger, dockerResponse io.ReadCloser,
return errors.New(msg.ErrorDetail.Message) return errors.New(msg.ErrorDetail.Message)
} }
if msg.Status != "" { switch {
case msg.Status != "":
if msg.Progress != "" { if msg.Progress != "" {
writeLog(logger, isError, "%s :: %s :: %s\n", msg.Status, msg.ID, msg.Progress) writeLog(logger, isError, "%s :: %s :: %s\n", msg.Status, msg.ID, msg.Progress)
} else { } else {
writeLog(logger, isError, "%s :: %s\n", msg.Status, msg.ID) writeLog(logger, isError, "%s :: %s\n", msg.Status, msg.ID)
} }
} else if msg.Stream != "" { case msg.Stream != "":
writeLog(logger, isError, "%s", msg.Stream) writeLog(logger, isError, "%s", msg.Stream)
} else { default:
writeLog(logger, false, "Unable to handle line: %s", string(line)) writeLog(logger, false, "Unable to handle line: %s", string(line))
} }
} }
+3 -3
View File
@@ -32,9 +32,9 @@ func TestRemoveOrphanNetworks(t *testing.T) {
client.On("NetworkList", ctx, mobyclient.NetworkListOptions{ client.On("NetworkList", ctx, mobyclient.NetworkListOptions{
Filters: make(mobyclient.Filters).Add("label", runnerUUIDLabel+"=runner-1"), Filters: make(mobyclient.Filters).Add("label", runnerUUIDLabel+"=runner-1"),
}).Return(mobyclient.NetworkListResult{Items: []network.Summary{ }).Return(mobyclient.NetworkListResult{Items: []network.Summary{
{Network: network.Network{ID: "orphan"}}, {ID: "orphan"},
{Network: network.Network{ID: "busy"}}, {ID: "busy"},
{Network: network.Network{ID: "starting"}}, {ID: "starting"},
}}, nil) }}, nil)
client.On("NetworkInspect", ctx, "orphan", mobyclient.NetworkInspectOptions{}). client.On("NetworkInspect", ctx, "orphan", mobyclient.NetworkInspectOptions{}).
Return(mobyclient.NetworkInspectResult{}, nil) Return(mobyclient.NetworkInspectResult{}, nil)
+395
View File
@@ -0,0 +1,395 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
package container
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"maps"
"mime"
"net"
"net/http"
"net/http/httputil"
"os"
"path/filepath"
"regexp"
"runtime"
"strings"
"sync"
"time"
"gitea.com/gitea/runner/act/common"
cerrdefs "github.com/containerd/errdefs"
"github.com/moby/moby/api/types/container"
"github.com/moby/moby/api/types/mount"
"github.com/moby/moby/client"
)
const (
jobLabel = "com.gitea.runner.job"
maxCreateBody = 8 << 20
dockerProxyProbeTimeout = 5 * time.Second
)
var (
createPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers|networks|volumes)/create$`)
rawStreamPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers/[^/]+/attach|exec/[^/]+/start)$`)
)
func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if host := os.Getenv("DOCKER_HOST"); runtime.GOOS != "linux" || host != "" && !strings.HasPrefix(host, "unix://") {
return nil
}
probeCtx, cancel := context.WithTimeout(ctx, dockerProxyProbeTimeout)
defer cancel()
cli, err := GetDockerClient(probeCtx)
if err != nil {
return nil
}
defer cli.Close()
daemonSocket, ok := strings.CutPrefix(cli.DaemonHost(), "unix://")
if !ok {
return nil
}
if info, err := os.Stat(daemonSocket); err != nil || info.Mode()&os.ModeSocket == 0 {
return nil
}
dir, err := filepath.Abs(os.TempDir())
if err != nil {
common.Logger(ctx).Infof("docker proxy probe failed, jobs get the daemon socket directly: %v", err)
return nil
}
seen, err := daemonSeesDir(probeCtx, cli, dir)
if err != nil {
common.Logger(ctx).Infof("docker proxy probe failed, jobs get the daemon socket directly: %v", err)
return nil
}
if !seen {
common.Logger(ctx).Infof("the docker daemon cannot reach the runner's temporary filesystem, jobs get the daemon socket directly")
return nil
}
if ctx.Err() != nil {
return nil
}
proxy, err := StartDockerProxy(daemonSocket, dir, job)
if err != nil {
common.Logger(ctx).Warnf("docker proxy not started, the job gets the daemon socket directly: %v", err)
}
return proxy
}
// daemonSeesDir reports whether the daemon opens the files the runner writes in dir,
// which is what a job's proxy socket mounted from there needs.
func daemonSeesDir(ctx context.Context, cli client.APIClient, dir string) (bool, error) {
marker, err := os.CreateTemp(dir, "gitea-runner-probe-")
if err != nil {
return false, err
}
defer func() {
if err := os.Remove(marker.Name()); err != nil {
common.Logger(ctx).Warnf("removing the docker proxy probe marker failed: %v", err)
}
}()
if err := marker.Close(); err != nil {
return false, err
}
images, err := cli.ImageList(ctx, client.ImageListOptions{})
if err != nil {
return false, err
}
if len(images.Items) == 0 {
return false, errors.New("no image available for the docker proxy probe")
}
created, err := cli.ContainerCreate(ctx, client.ContainerCreateOptions{
Config: &container.Config{Image: images.Items[0].ID, Cmd: []string{"true"}},
HostConfig: &container.HostConfig{Mounts: []mount.Mount{
{Type: mount.TypeBind, Source: marker.Name(), Target: "/gitea-runner-probe", ReadOnly: true},
}},
})
if cerrdefs.IsInvalidArgument(err) {
return false, nil
}
if err != nil {
return false, err
}
cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), dockerProxyProbeTimeout)
defer cancel()
if _, err := cli.ContainerRemove(cleanupCtx, created.ID, client.ContainerRemoveOptions{Force: true, RemoveVolumes: true}); err != nil {
return false, fmt.Errorf("removing the docker proxy probe container failed: %w", err)
}
return true, nil
}
// StartDockerProxy serves a job's docker socket in dir, labelling what the job creates through it.
func StartDockerProxy(daemonSocket, dir, job string) (*DockerProxy, error) {
info, err := os.Stat(daemonSocket)
if err != nil {
return nil, err
}
if info.Mode()&os.ModeSocket == 0 {
return nil, errors.New("docker daemon path is not a Unix socket")
}
if err := os.MkdirAll(dir, 0o700); err != nil {
return nil, err
}
instance, err := os.MkdirTemp(dir, "p-")
if err != nil {
return nil, err
}
socket := filepath.Join(instance, "docker.sock")
listener, err := net.Listen("unix", socket)
if err != nil {
return nil, errors.Join(err, os.RemoveAll(instance))
}
if err := copyDockerSocketPermissions(socket, info); err != nil {
return nil, errors.Join(err, listener.Close(), os.RemoveAll(instance))
}
dial := func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", daemonSocket)
}
transport := &http.Transport{DialContext: dial}
forward := &httputil.ReverseProxy{
Rewrite: func(r *httputil.ProxyRequest) {
r.Out.URL.Scheme = "http"
r.Out.URL.Host = "docker"
},
Transport: transport,
}
streams, cancelStreams := context.WithCancel(context.Background())
creates, cancelCreates := context.WithCancel(context.Background())
var admission sync.Mutex
var handlers sync.WaitGroup
server := &http.Server{ReadHeaderTimeout: 30 * time.Second, ConnContext: func(ctx context.Context, conn net.Conn) context.Context {
return context.WithValue(ctx, dockerProxyConnKey{}, conn)
}, Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
admission.Lock()
if streams.Err() != nil {
admission.Unlock()
http.Error(w, "docker proxy is closing", http.StatusServiceUnavailable)
return
}
handlers.Add(1)
admission.Unlock()
defer handlers.Done()
creating := r.Method == http.MethodPost && createPath.MatchString(r.URL.Path)
parent, lifetime := r.Context(), streams
if creating {
parent, lifetime = context.WithoutCancel(parent), creates
}
ctx, cancel := context.WithCancel(parent)
defer cancel()
stop := context.AfterFunc(lifetime, func() {
cancel()
if !creating {
if conn, ok := parent.Value(dockerProxyConnKey{}).(net.Conn); ok {
_ = conn.Close()
}
}
})
defer stop()
r = r.WithContext(ctx)
if creating {
r.Body = http.MaxBytesReader(w, r.Body, maxCreateBody)
if err := addLabel(r, job); err != nil {
status := http.StatusBadRequest
if _, ok := errors.AsType[*http.MaxBytesError](err); ok {
status = http.StatusRequestEntityTooLarge
}
http.Error(w, err.Error(), status)
return
}
} else if r.Method == http.MethodPost && rawStreamPath.MatchString(r.URL.Path) {
tunnel(w, r, dial, forward)
return
}
forward.ServeHTTP(w, r)
})}
served := make(chan struct{})
go func() {
defer close(served)
_ = server.Serve(listener)
}()
return &DockerProxy{Socket: socket, close: func(ctx context.Context) error {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
admission.Lock()
listenerErr := listener.Close()
cancelStreams()
admission.Unlock()
<-served
shutdownErr := server.Shutdown(ctx)
cancelCreates()
serverErr := server.Close()
handlers.Wait()
transport.CloseIdleConnections()
return errors.Join(ctx.Err(), listenerErr, shutdownErr, serverErr, os.RemoveAll(instance))
}}, nil
}
func addLabel(r *http.Request, job string) error {
body, err := io.ReadAll(r.Body)
if err != nil {
return err
}
if len(bytes.TrimSpace(body)) == 0 {
body = []byte("{}")
}
var fields map[string]json.RawMessage
var config struct{ Labels map[string]string }
if err := json.Unmarshal(body, &fields); err != nil {
return fmt.Errorf("invalid create request: %w", err)
}
if err := json.Unmarshal(body, &config); err != nil {
return fmt.Errorf("invalid create labels: %w", err)
}
if fields == nil {
fields = make(map[string]json.RawMessage)
}
maps.DeleteFunc(fields, func(name string, _ json.RawMessage) bool {
return strings.EqualFold(name, "Labels")
})
if config.Labels == nil {
config.Labels = make(map[string]string)
}
config.Labels[jobLabel] = job
if fields["Labels"], err = json.Marshal(config.Labels); err != nil {
return err
}
if body, err = json.Marshal(fields); err != nil {
return err
}
r.Body = io.NopCloser(bytes.NewReader(body))
r.ContentLength = int64(len(body))
r.TransferEncoding = nil
return nil
}
type dockerProxyConnKey struct{}
type dockerProxyResponse struct {
response *http.Response
}
func (r dockerProxyResponse) RoundTrip(_ *http.Request) (*http.Response, error) {
return r.response, nil
}
// tunnel splices attach and exec streams, which the daemon hijacks with or without an HTTP upgrade
func tunnel(w http.ResponseWriter, r *http.Request, dial func(context.Context, string, string) (net.Conn, error), forward *httputil.ReverseProxy) {
upstream, err := dial(r.Context(), "", "")
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
defer upstream.Close()
stop := context.AfterFunc(r.Context(), func() { _ = upstream.Close() })
defer stop()
if err := r.Write(upstream); err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
reader := bufio.NewReader(upstream)
var response *http.Response
for {
response, err = http.ReadResponse(reader, r)
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
if response.StatusCode >= 200 || response.StatusCode == http.StatusSwitchingProtocols {
break
}
maps.Copy(w.Header(), response.Header)
w.WriteHeader(response.StatusCode)
clear(w.Header())
_ = response.Body.Close()
}
defer func() {
_ = upstream.Close()
_ = response.Body.Close()
}()
mediaType, _, _ := mime.ParseMediaType(response.Header.Get("Content-Type"))
if response.StatusCode != http.StatusSwitchingProtocols && (response.StatusCode != http.StatusOK || mediaType != "application/vnd.docker.raw-stream") {
ordinary := *forward
ordinary.Transport = dockerProxyResponse{response: response}
ordinary.ServeHTTP(w, r)
return
}
downstream, buffered, err := http.NewResponseController(w).Hijack()
if err != nil {
return
}
defer downstream.Close()
if _, err := fmt.Fprintf(buffered, "%s %s\r\n", response.Proto, response.Status); err != nil {
return
}
if err := response.Header.Write(buffered); err != nil {
return
}
if _, err := buffered.WriteString("\r\n"); err != nil {
return
}
if err := buffered.Flush(); err != nil {
return
}
done := make(chan struct{})
go func() {
defer close(done)
if _, err := io.Copy(upstream, io.MultiReader(io.LimitReader(buffered, int64(buffered.Reader.Buffered())), downstream)); err != nil { // Bypass net/http after the prefix so stdin EOF preserves output.
_ = upstream.Close()
} else if writer, ok := upstream.(interface{ CloseWrite() error }); ok {
_ = writer.CloseWrite()
} else {
_ = upstream.Close()
}
}()
_, _ = io.Copy(downstream, reader)
_ = downstream.Close()
_ = upstream.Close()
<-done
}
func RemoveDockerJobResources(ctx context.Context, job string) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
}
defer cli.Close()
return removeLabelled(ctx, cli, job)
}
func removeLabelled(ctx context.Context, cli client.APIClient, job string) error {
logger := common.Logger(ctx)
filters := make(client.Filters).Add("label", jobLabel+"="+job)
containers, err := cli.ContainerList(ctx, client.ContainerListOptions{All: true, Filters: filters})
errs := []error{err}
for _, c := range containers.Items {
logger.Infof("removing container %s the job left behind", strings.TrimPrefix(strings.Join(c.Names, ","), "/"))
errs = append(errs, (&containerReference{cli: cli, id: c.ID}).remove()(ctx))
}
networks, err := cli.NetworkList(ctx, client.NetworkListOptions{Filters: filters})
errs = append(errs, err)
for _, n := range networks.Items {
if _, err := cli.NetworkRemove(ctx, n.ID, client.NetworkRemoveOptions{}); err != nil && !cerrdefs.IsNotFound(err) {
errs = append(errs, fmt.Errorf("failed to remove network %s: %w", n.Name, err))
}
}
volumes, err := cli.VolumeList(ctx, client.VolumeListOptions{Filters: filters})
errs = append(errs, err)
for _, v := range volumes.Items {
if _, err := cli.VolumeRemove(ctx, v.Name, client.VolumeRemoveOptions{}); err != nil && !cerrdefs.IsNotFound(err) {
errs = append(errs, fmt.Errorf("failed to remove volume %s: %w", v.Name, err))
}
}
return errors.Join(errs...)
}
+469
View File
@@ -0,0 +1,469 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
package container
import (
"archive/tar"
"bufio"
"bytes"
"context"
"errors"
"fmt"
"io"
"net"
"net/http"
"os"
"path/filepath"
"runtime"
"strconv"
"strings"
"testing"
"time"
cerrdefs "github.com/containerd/errdefs"
"github.com/moby/moby/api/types/container"
"github.com/moby/moby/api/types/image"
"github.com/moby/moby/api/types/mount"
"github.com/moby/moby/api/types/volume"
mobyclient "github.com/moby/moby/client"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
)
func (m *mockDockerClient) VolumeList(ctx context.Context, opts mobyclient.VolumeListOptions) (mobyclient.VolumeListResult, error) {
args := m.Called(ctx, opts)
return args.Get(0).(mobyclient.VolumeListResult), args.Error(1)
}
func (m *mockDockerClient) VolumeRemove(ctx context.Context, id string, opts mobyclient.VolumeRemoveOptions) (mobyclient.VolumeRemoveResult, error) {
args := m.Called(ctx, id, opts)
return args.Get(0).(mobyclient.VolumeRemoveResult), args.Error(1)
}
// unix socket paths are limited to about a hundred bytes, which the default macOS TMPDIR exceeds
func shortTempDir(t testing.TB) string {
t.Helper()
t.Setenv("TMPDIR", "/tmp")
return t.TempDir()
}
func daemonSocketPath(t testing.TB, cli mobyclient.APIClient) string {
t.Helper()
host := cli.DaemonHost()
if !strings.HasPrefix(host, "unix://") {
t.Skipf("skipping: daemon at %s is not a unix socket", host)
}
return strings.TrimPrefix(host, "unix://")
}
func TestDockerProxy(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("Unix socket ownership is unavailable on Windows")
}
bodies := make(chan []byte, 8)
createStarted, releaseCreate := make(chan struct{}), make(chan struct{})
streamsDone := make(chan struct{}, 2)
daemonSocket := filepath.Join(shortTempDir(t), "d.sock")
listener, err := net.Listen("unix", daemonSocket)
require.NoError(t, err)
daemon := &http.Server{ReadHeaderTimeout: time.Second, Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/create"):
body, _ := io.ReadAll(r.Body)
if r.URL.RawQuery == "wait" {
close(createStarted)
<-releaseCreate
}
bodies <- body
w.WriteHeader(http.StatusCreated)
case r.URL.Path == "/exec/detached/start" || r.URL.Path == "/exec/error/start":
if r.URL.Path == "/exec/error/start" {
w.Header().Set("Content-Type", "application/vnd.docker.raw-stream")
w.WriteHeader(http.StatusBadRequest)
}
_, _ = w.Write([]byte("ordinary"))
case rawStreamPath.MatchString(r.URL.Path) || r.URL.Path == "/session":
_, _ = io.Copy(io.Discard, r.Body)
conn, buffered, err := w.(http.Hijacker).Hijack()
if err != nil {
return
}
defer conn.Close()
if r.Header.Get("Upgrade") != "" {
_, _ = conn.Write([]byte("HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade\r\nUpgrade: tcp\r\n\r\nready\n"))
} else {
_, _ = conn.Write([]byte("HTTP/1.1 200 OK\r\nContent-Type: application/vnd.docker.raw-stream\r\n\r\nready\n"))
}
input, _ := io.ReadAll(buffered)
_, _ = conn.Write(append([]byte("final:"), input...))
streamsDone <- struct{}{}
default:
w.Header().Set("Api-Version", "1.47")
_, _ = w.Write([]byte("OK " + r.Method + " " + r.URL.Path))
}
})}
go func() { _ = daemon.Serve(listener) }()
t.Cleanup(func() { _ = daemon.Close() })
proxy, err := StartDockerProxy(daemonSocket, shortTempDir(t), "job-1")
require.NoError(t, err)
t.Cleanup(func() { _ = proxy.Close(context.Background()) })
client := &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", proxy.Socket)
}}}
t.Cleanup(client.CloseIdleConnections)
dialProxy := func(t *testing.T) *net.UnixConn {
t.Helper()
conn, err := net.DialUnix("unix", nil, &net.UnixAddr{Name: proxy.Socket, Net: "unix"})
require.NoError(t, err)
t.Cleanup(func() { _ = conn.Close() })
require.NoError(t, conn.SetDeadline(time.Now().Add(5*time.Second)))
return conn
}
startStream := func(t *testing.T, path, upgrade string) (*net.UnixConn, *bufio.Reader) {
t.Helper()
conn := dialProxy(t)
_, err := fmt.Fprintf(conn, "POST %s HTTP/1.1\r\nHost: docker\r\n%sContent-Length: 2\r\n\r\n{}input", path, upgrade)
require.NoError(t, err)
reader := bufio.NewReader(conn)
resp, err := http.ReadResponse(reader, nil)
require.NoError(t, err)
t.Cleanup(func() {
_ = conn.Close()
_ = resp.Body.Close()
})
prefix, err := reader.ReadString('\n')
require.NoError(t, err)
require.Equal(t, "ready\n", prefix)
return conn, reader
}
t.Run("labels creates", func(t *testing.T) {
for _, testCase := range []struct{ path, body, want string }{
{
"/v1.47/containers/create", `{"Image":"alpine","Unknown":{"enabled":true},"Labels":{"own":"1","com.gitea.runner.job":"other"}}`,
`{"Image":"alpine","Unknown":{"enabled":true},"Labels":{"own":"1","com.gitea.runner.job":"job-1"}}`,
},
{
"/networks/create", `{"Name":"n"}`,
`{"Name":"n","Labels":{"com.gitea.runner.job":"job-1"}}`,
},
{
"/volumes/create", `{"Name":"v","labels":null}`,
`{"Name":"v","Labels":{"com.gitea.runner.job":"job-1"}}`,
},
{
"/volumes/create", "",
`{"Labels":{"com.gitea.runner.job":"job-1"}}`,
},
{
"/volumes/create", "null",
`{"Labels":{"com.gitea.runner.job":"job-1"}}`,
},
{
"/volumes/create", `{"Labels":{"discard":"1"},"labels":null,"LABELS":{"own":"1"},"LABELS":{"extra":"2"}}`,
`{"Labels":{"own":"1","extra":"2","com.gitea.runner.job":"job-1"}}`,
},
} {
resp, err := client.Post("http://docker"+testCase.path, "application/json", strings.NewReader(testCase.body))
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusCreated, resp.StatusCode)
assert.JSONEq(t, testCase.want, string(<-bodies), testCase.body)
}
})
t.Run("passes other requests through", func(t *testing.T) {
resp, err := client.Get("http://docker/v1.47/_ping")
require.NoError(t, err)
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
assert.Equal(t, "1.47", resp.Header.Get("Api-Version"))
assert.Equal(t, "OK GET /v1.47/_ping", string(body))
})
t.Run("tunnels raw streams through stdin EOF", func(t *testing.T) {
for _, stream := range []struct{ path, upgrade string }{
{path: "/v1.47/exec/abc/start"},
{path: "/containers/abc/attach", upgrade: "Connection: Upgrade\r\nUpgrade: tcp\r\n"},
} {
conn, reader := startStream(t, stream.path, stream.upgrade)
require.NoError(t, conn.CloseWrite())
output, err := io.ReadAll(reader)
require.NoError(t, err)
assert.Equal(t, "final:input", string(output))
<-streamsDone
}
})
t.Run("detached and error responses retain keepalive label injection", func(t *testing.T) {
conn := dialProxy(t)
reader := bufio.NewReader(conn)
for endpoint, status := range map[string]int{"detached": http.StatusOK, "error": http.StatusBadRequest} {
_, err := fmt.Fprintf(conn, "POST /exec/%s/start HTTP/1.1\r\nHost: docker\r\nContent-Length: 2\r\n\r\n{}", endpoint)
require.NoError(t, err)
resp, err := http.ReadResponse(reader, nil)
require.NoError(t, err)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, status, resp.StatusCode)
assert.Equal(t, "ordinary", string(body))
_, err = io.WriteString(conn, "POST /volumes/create HTTP/1.1\r\nHost: docker\r\nContent-Length: 0\r\n\r\n")
require.NoError(t, err)
resp, err = http.ReadResponse(reader, nil)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode)
assert.JSONEq(t, `{"Labels":{"com.gitea.runner.job":"job-1"}}`, string(<-bodies))
}
})
t.Run("close joins streams and admitted create", func(t *testing.T) {
t.Cleanup(func() { close(releaseCreate) })
_, raw := startStream(t, "/exec/live/start", "")
_, upgraded := startStream(t, "/session", "Connection: Upgrade\r\nUpgrade: tcp\r\n")
creator := dialProxy(t)
_, err := io.WriteString(creator, "POST /volumes/create?wait HTTP/1.1\r\nHost: docker\r\nContent-Length: 2\r\n\r\n{}")
require.NoError(t, err)
<-createStarted
require.NoError(t, creator.CloseWrite())
closed := make(chan error, 1)
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
defer cancel()
go func() { closed <- proxy.Close(ctx) }()
for _, reader := range []*bufio.Reader{raw, upgraded} {
_, err := reader.ReadByte()
require.ErrorIs(t, err, io.EOF)
<-streamsDone
}
select {
case err := <-closed:
t.Fatalf("Close returned before admitted create settled: %v", err)
default:
}
releaseCreate <- struct{}{}
resp, err := http.ReadResponse(bufio.NewReader(creator), nil)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode)
assert.JSONEq(t, `{"Labels":{"com.gitea.runner.job":"job-1"}}`, string(<-bodies))
require.NoError(t, <-closed)
})
}
func TestRemoveLabelledRemovesContainersNetworksAndVolumes(t *testing.T) {
containerFailure := errors.New("container removal failed")
listFailure := errors.New("listing failed")
volumeFailure := errors.New("volume removal failed")
ctx := context.Background()
filters := make(mobyclient.Filters).Add("label", jobLabel+"=job-1")
cli := &mockDockerClient{}
cli.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true, Filters: filters}).
Return(mobyclient.ContainerListResult{Items: []container.Summary{{ID: "c1", Names: []string{"/app"}}}}, nil).Once()
cli.On("ContainerKill", ctx, "c1", mock.Anything).Return(mobyclient.ContainerKillResult{}, nil).Once()
cli.On("ContainerRemove", ctx, "c1", mobyclient.ContainerRemoveOptions{RemoveVolumes: true, Force: true}).
Return(mobyclient.ContainerRemoveResult{}, containerFailure).Once()
cli.On("NetworkList", ctx, mobyclient.NetworkListOptions{Filters: filters}).Return(mobyclient.NetworkListResult{}, listFailure).Once()
cli.On("VolumeList", ctx, mobyclient.VolumeListOptions{Filters: filters}).
Return(mobyclient.VolumeListResult{Items: []volume.Volume{{Name: "app_data"}}}, nil).Once()
cli.On("VolumeRemove", ctx, "app_data", mobyclient.VolumeRemoveOptions{}).Return(mobyclient.VolumeRemoveResult{}, volumeFailure).Once()
err := removeLabelled(ctx, cli, "job-1")
require.ErrorIs(t, err, containerFailure)
require.ErrorIs(t, err, listFailure)
require.ErrorIs(t, err, volumeFailure)
require.ErrorContains(t, err, "failed to remove container c1")
require.ErrorContains(t, err, "failed to remove volume app_data")
cli.AssertExpectations(t)
}
func TestDockerProxyWithDaemon(t *testing.T) {
requireDocker(t)
ctx := context.Background()
require.NoError(t, NewDockerPullExecutor(NewDockerPullExecutorInput{Image: "alpine"})(ctx))
direct, err := GetDockerClient(ctx)
require.NoError(t, err)
defer direct.Close()
dir := shortTempDir(t)
seen, err := daemonSeesDir(ctx, direct, dir)
require.NoError(t, err)
t.Logf("daemon sees the runner's filesystem: %v", seen)
job := "proxy-test-" + strconv.FormatInt(time.Now().UnixNano(), 36)
proxy, err := StartDockerProxy(daemonSocketPath(t, direct), dir, job)
require.NoError(t, err)
t.Cleanup(func() { _ = proxy.Close(context.Background()) })
viaProxy, err := mobyclient.New(mobyclient.WithHost("unix://" + proxy.Socket))
require.NoError(t, err)
defer viaProxy.Close()
created, err := viaProxy.ContainerCreate(ctx, mobyclient.ContainerCreateOptions{
Config: &container.Config{Image: "alpine", Cmd: []string{"sleep", "300"}, Labels: map[string]string{"own": "1"}},
})
require.NoError(t, err)
t.Cleanup(func() { _, _ = direct.ContainerRemove(ctx, created.ID, mobyclient.ContainerRemoveOptions{Force: true}) })
_, err = viaProxy.ContainerStart(ctx, created.ID, mobyclient.ContainerStartOptions{})
require.NoError(t, err)
net, err := viaProxy.NetworkCreate(ctx, job, mobyclient.NetworkCreateOptions{})
require.NoError(t, err)
t.Cleanup(func() { _, _ = direct.NetworkRemove(ctx, net.ID, mobyclient.NetworkRemoveOptions{}) })
_, err = viaProxy.VolumeCreate(ctx, mobyclient.VolumeCreateOptions{Name: job})
require.NoError(t, err)
t.Cleanup(func() { _, _ = direct.VolumeRemove(ctx, job, mobyclient.VolumeRemoveOptions{Force: true}) })
inspected, err := direct.ContainerInspect(ctx, created.ID, mobyclient.ContainerInspectOptions{})
require.NoError(t, err)
assert.Equal(t, map[string]string{"own": "1", jobLabel: job}, inspected.Container.Config.Labels)
vol, err := direct.VolumeInspect(ctx, job, mobyclient.VolumeInspectOptions{})
require.NoError(t, err)
assert.Equal(t, job, vol.Volume.Labels[jobLabel])
exec, err := viaProxy.ExecCreate(ctx, created.ID, mobyclient.ExecCreateOptions{Cmd: []string{"cat"}, AttachStdin: true, AttachStdout: true, TTY: true})
require.NoError(t, err)
attached, err := viaProxy.ExecAttach(ctx, exec.ID, mobyclient.ExecAttachOptions{TTY: true})
require.NoError(t, err)
_, err = attached.Conn.Write([]byte("hello\n"))
require.NoError(t, err)
echoed, err := attached.Reader.ReadString('\n')
require.NoError(t, err)
assert.Equal(t, "hello\r\n", echoed)
attached.Close()
require.NoError(t, proxy.Close(ctx))
require.NoError(t, RemoveDockerJobResources(ctx, job))
_, err = direct.ContainerInspect(ctx, created.ID, mobyclient.ContainerInspectOptions{})
assert.True(t, cerrdefs.IsNotFound(err))
_, err = direct.NetworkInspect(ctx, net.ID, mobyclient.NetworkInspectOptions{})
assert.True(t, cerrdefs.IsNotFound(err))
_, err = direct.VolumeInspect(ctx, job, mobyclient.VolumeInspectOptions{})
assert.True(t, cerrdefs.IsNotFound(err))
}
func BenchmarkDockerProxy(b *testing.B) {
ctx := context.Background()
direct, err := GetDockerClient(ctx)
require.NoError(b, err)
defer direct.Close()
if _, err := direct.Ping(ctx, mobyclient.PingOptions{}); err != nil {
b.Skipf("docker daemon unreachable: %v", err)
}
proxy, err := StartDockerProxy(daemonSocketPath(b, direct), shortTempDir(b), "bench")
require.NoError(b, err)
defer func() { _ = proxy.Close(ctx) }()
viaProxy, err := mobyclient.New(mobyclient.WithHost("unix://" + proxy.Socket))
require.NoError(b, err)
defer viaProxy.Close()
require.NoError(b, NewDockerPullExecutor(NewDockerPullExecutorInput{Image: "alpine"})(ctx))
created, err := direct.ContainerCreate(ctx, mobyclient.ContainerCreateOptions{Config: &container.Config{Image: "alpine", Cmd: []string{"sleep", "600"}}})
require.NoError(b, err)
defer func() { _, _ = direct.ContainerRemove(ctx, created.ID, mobyclient.ContainerRemoveOptions{Force: true}) }()
_, err = direct.ContainerStart(ctx, created.ID, mobyclient.ContainerStartOptions{})
require.NoError(b, err)
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
payload := make([]byte, 64<<20)
require.NoError(b, writer.WriteHeader(&tar.Header{Name: "blob", Mode: 0o600, Size: int64(len(payload))}))
_, _ = writer.Write(payload)
require.NoError(b, writer.Close())
for name, cli := range map[string]mobyclient.APIClient{"direct": direct, "proxy": viaProxy} {
b.Run("ping/"+name, func(b *testing.B) {
for b.Loop() {
if _, err := cli.Ping(ctx, mobyclient.PingOptions{}); err != nil {
b.Fatal(err)
}
}
})
b.Run("copy64MiB/"+name, func(b *testing.B) {
b.SetBytes(int64(archive.Len()))
for b.Loop() {
_, err := cli.CopyToContainer(ctx, created.ID, mobyclient.CopyToContainerOptions{DestinationPath: "/tmp", Content: bytes.NewReader(archive.Bytes())})
if err != nil {
b.Fatal(err)
}
}
})
}
}
type probeClient struct {
mobyclient.APIClient
create func(mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error)
remove func(context.Context, string, mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error)
}
func (c *probeClient) ImageList(context.Context, mobyclient.ImageListOptions) (mobyclient.ImageListResult, error) {
return mobyclient.ImageListResult{Items: []image.Summary{{ID: "probe-image"}}}, nil
}
func (c *probeClient) ContainerCreate(_ context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
return c.create(opts)
}
func (c *probeClient) ContainerRemove(ctx context.Context, id string, opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
return c.remove(ctx, id, opts)
}
func TestDaemonSeesDir(t *testing.T) {
dir := t.TempDir()
markers := make(map[string]bool)
for _, testCase := range []struct {
name string
private bool
removeErr error
}{
{name: "cleanup after cancellation"},
{name: "private filesystem with stale marker", private: true},
{name: "cleanup error after cancellation", removeErr: errors.New("cleanup failed")},
} {
t.Run(testCase.name, func(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
daemonDir := dir
if testCase.private {
daemonDir = t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(daemonDir, "probe"), []byte("stale"), 0o600))
}
removed := false
cli := &probeClient{
create: func(opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
require.Len(t, opts.HostConfig.Mounts, 1)
marker := opts.HostConfig.Mounts[0].Source
assert.Equal(t, mount.Mount{Type: mount.TypeBind, Source: marker, Target: "/gitea-runner-probe", ReadOnly: true}, opts.HostConfig.Mounts[0])
assert.False(t, markers[marker])
markers[marker] = true
info, err := os.Stat(marker)
require.NoError(t, err)
assert.True(t, info.Mode().IsRegular())
cancel()
if _, err := os.Stat(filepath.Join(daemonDir, filepath.Base(marker))); errors.Is(err, os.ErrNotExist) {
return mobyclient.ContainerCreateResult{}, cerrdefs.ErrInvalidArgument
}
return mobyclient.ContainerCreateResult{ID: "probe"}, nil
},
remove: func(ctx context.Context, id string, opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removed = true
assert.Equal(t, "probe", id)
assert.Equal(t, mobyclient.ContainerRemoveOptions{Force: true, RemoveVolumes: true}, opts)
require.NoError(t, ctx.Err())
_, bounded := ctx.Deadline()
assert.True(t, bounded)
return mobyclient.ContainerRemoveResult{}, testCase.removeErr
},
}
seen, err := daemonSeesDir(ctx, cli, dir)
require.ErrorIs(t, err, testCase.removeErr)
assert.Equal(t, !testCase.private && testCase.removeErr == nil, seen)
assert.Equal(t, !testCase.private, removed)
entries, err := os.ReadDir(dir)
require.NoError(t, err)
assert.Empty(t, entries)
})
}
}
+27
View File
@@ -0,0 +1,27 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER && (linux || darwin || netbsd)
package container
import (
"errors"
"os"
"path/filepath"
"syscall"
)
func copyDockerSocketPermissions(socket string, info os.FileInfo) error {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return errors.New("docker socket ownership is unavailable")
}
if err := os.Chown(socket, int(stat.Uid), int(stat.Gid)); err != nil {
return err
}
if err := os.Chown(filepath.Dir(socket), int(stat.Uid), -1); err != nil {
return err
}
return os.Chmod(socket, info.Mode().Perm())
}
+15
View File
@@ -0,0 +1,15 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !WITHOUT_DOCKER
package container
import (
"errors"
"os"
)
func copyDockerSocketPermissions(_ string, _ os.FileInfo) error {
return errors.New("docker socket ownership cannot be preserved on Windows")
}
+125 -173
View File
@@ -15,6 +15,7 @@ import (
"io" "io"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"regexp" "regexp"
"runtime" "runtime"
"slices" "slices"
@@ -57,7 +58,7 @@ func NewContainer(input *NewContainerInput) ExecutionsEnvironment {
cr := new(containerReference) cr := new(containerReference)
cr.input = input cr.input = input
// Resolved up front because the image pull runs before the container is created. // Resolved up front because the image pull runs before the container is created.
cf := createFlagsFromOptions(input.Options) cf := createFlagsFromOptions(input.allOptions())
if cf.platform != "" { if cf.platform != "" {
cr.input.Platform = cf.platform cr.input.Platform = cf.platform
} }
@@ -65,29 +66,6 @@ func NewContainer(input *NewContainerInput) ExecutionsEnvironment {
return cr return cr
} }
func (cr *containerReference) ConnectToNetwork(name string) common.Executor {
return common.
NewDebugExecutor("docker network connect %s %s", name, cr.input.Name).
Then(
common.NewPipelineExecutor(
cr.connect(),
cr.connectToNetwork(name, cr.input.NetworkAliases),
).IfNot(common.Dryrun),
)
}
func (cr *containerReference) connectToNetwork(name string, aliases []string) common.Executor {
return func(ctx context.Context) error {
_, err := cr.cli.NetworkConnect(ctx, name, client.NetworkConnectOptions{
Container: cr.input.Name,
EndpointConfig: &network.EndpointSettings{
Aliases: aliases,
},
})
return err
}
}
// supportsContainerImagePlatform reports whether the Docker server API version // supportsContainerImagePlatform reports whether the Docker server API version
// is 1.41 and beyond // is 1.41 and beyond
func supportsContainerImagePlatform(ctx context.Context, cli client.APIClient) (bool, error) { func supportsContainerImagePlatform(ctx context.Context, cli client.APIClient) (bool, error) {
@@ -160,12 +138,12 @@ func (cr *containerReference) Copy(destPath string, files ...*FileEntry) common.
).IfNot(common.Dryrun) ).IfNot(common.Dryrun)
} }
func (cr *containerReference) CopyDir(destPath, srcPath string, useGitIgnore bool) common.Executor { func (cr *containerReference) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
common.NewInfoExecutor("docker cp src=%s dst=%s", srcPath, destPath), common.NewInfoExecutor("docker cp src=%s dst=%s", srcPath, destPath),
cr.connect(), cr.connect(),
cr.find(), cr.find(),
cr.copyDir(destPath, srcPath, useGitIgnore), cr.copyDir(destPath, srcPath, useGitIgnore, skipGitDir),
func(ctx context.Context) error { func(ctx context.Context) error {
// If this fails, then folders have wrong permissions on non root container // If this fails, then folders have wrong permissions on non root container
if cr.UID != 0 || cr.GID != 0 { if cr.UID != 0 || cr.GID != 0 {
@@ -274,6 +252,10 @@ func containerInfoFromInspect(inspect container.InspectResponse) *Info {
ID: inspect.ID, ID: inspect.ID,
Health: HealthNone, Health: HealthNone,
Ports: map[string]string{}, // an empty map, never null, in the expression context Ports: map[string]string{}, // an empty map, never null, in the expression context
Mounts: make(map[string]string, len(inspect.Mounts)),
}
for _, mountPoint := range inspect.Mounts {
info.Mounts[mountPoint.Destination] = mountPoint.Source
} }
if state := inspect.State; state != nil { if state := inspect.State; state != nil {
@@ -506,14 +488,11 @@ func (cr *containerReference) remove() common.Executor {
RemoveVolumes: true, RemoveVolumes: true,
Force: true, Force: true,
}) })
switch { if cerrdefs.IsConflict(err) {
case cerrdefs.IsConflict(err): err = cr.waitForRemoval(ctx, idOrName)
// the daemon's own AutoRemove teardown is running, and it releases the volume }
// references and the network endpoint only once it finishes if err != nil && !cerrdefs.IsNotFound(err) {
cr.waitForRemoval(ctx, idOrName) return fmt.Errorf("failed to remove container %s: %w", idOrName, err)
case err != nil && !cerrdefs.IsNotFound(err):
logger.Error(fmt.Errorf("failed to remove container %s: %w", idOrName, err))
return nil // keep the id, the container is still there for a later Remove()
} }
logger.Debugf("Removed container: %v", idOrName) logger.Debugf("Removed container: %v", idOrName)
@@ -522,7 +501,7 @@ func (cr *containerReference) remove() common.Executor {
} }
} }
func (cr *containerReference) waitForRemoval(ctx context.Context, idOrName string) { func (cr *containerReference) waitForRemoval(ctx context.Context, idOrName string) error {
// per container, against the one minute the post-job executor allows for the whole // per container, against the one minute the post-job executor allows for the whole
// cleanup, so a job with several services can spend most of that budget here // cleanup, so a job with several services can spend most of that budget here
ctx, cancel := context.WithTimeout(ctx, 15*time.Second) ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
@@ -532,44 +511,49 @@ func (cr *containerReference) waitForRemoval(ctx context.Context, idOrName strin
Condition: container.WaitConditionRemoved, Condition: container.WaitConditionRemoved,
}) })
select { select {
case <-waitResult.Result: case result := <-waitResult.Result:
case <-waitResult.Error: if result.Error != nil {
case <-ctx.Done(): return errors.New(result.Error.Message)
// the client delivers the result over an unbuffered channel, so leave a receiver }
// behind or its goroutine parks on the send for the lifetime of the process return nil
go func() { case err := <-waitResult.Error:
select { return err
case <-waitResult.Result:
case <-waitResult.Error:
}
}()
common.Logger(ctx).Warnf("Timed out waiting for the daemon to remove container %s, its volumes and network may be left behind", idOrName)
} }
} }
// allOptions puts the runner's options first, so a flag both sources set ends up the workflow's.
func (input *NewContainerInput) allOptions() string {
return strings.TrimSpace(input.RunnerOptions + " " + input.WorkflowOptions)
}
func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) (*container.Config, *container.HostConfig, error) { func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) (*container.Config, *container.HostConfig, error) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
input := cr.input options := cr.input.allOptions()
if input.Options == "" { if options == "" {
return config, hostConfig, nil return config, hostConfig, nil
} }
// For Gitea, checked here because the parse below is what would read those files
if err := rejectHostReadingOptions(cr.input.WorkflowOptions); err != nil {
return nil, nil, err
}
// parse configuration from CLI container.options // parse configuration from CLI container.options
flags, copts, cf, err := parseContainerOptions(input.Options) flags, copts, cf, err := parseContainerOptions(options)
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
} }
if err := cf.validate(); err != nil { if err := cf.validate(); err != nil {
return nil, nil, fmt.Errorf("Cannot process container options: '%s': '%w'", input.Options, err) return nil, nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
} }
// FIXME: If everything is fine after gitea/act v0.260.0, remove the following comment. // FIXME: If everything is fine after gitea/act v0.260.0, remove the following comment.
// In the old fork version, the code is // In the old fork version, the code is
// if len(copts.netMode.Value()) == 0 { // if len(copts.netMode.Value()) == 0 {
// if err = copts.netMode.Set("host"); err != nil { // if err = copts.netMode.Set("host"); err != nil {
// return nil, nil, fmt.Errorf("Cannot parse networkmode=host. This is an internal error and should not happen: '%w'", err) // return nil, nil, fmt.Errorf("cannot parse networkmode=host. This is an internal error and should not happen: '%w'", err)
// } // }
// } // }
// And it has been commented with: // And it has been commented with:
@@ -581,7 +565,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
if len(copts.netMode.Value()) == 0 { if len(copts.netMode.Value()) == 0 {
if err = copts.netMode.Set(cr.input.NetworkMode); err != nil { if err = copts.netMode.Set(cr.input.NetworkMode); err != nil {
return nil, nil, fmt.Errorf("Cannot parse networkmode=%s. This is an internal error and should not happen: '%w'", cr.input.NetworkMode, err) return nil, nil, fmt.Errorf("cannot parse networkmode=%s. This is an internal error and should not happen: '%w'", cr.input.NetworkMode, err)
} }
} }
@@ -593,24 +577,31 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
containerConfig, err := parse(flags, copts, runtime.GOOS) containerConfig, err := parse(flags, copts, runtime.GOOS)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("Cannot process container options: '%s': '%w'", input.Options, err) return nil, nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
}
// workflow aliases join the runner's own, deduped so a re-create cannot grow the input
for _, endpoint := range containerConfig.NetworkingConfig.EndpointsConfig {
for _, alias := range endpoint.Aliases {
if !slices.Contains(cr.input.NetworkAliases, alias) {
cr.input.NetworkAliases = append(cr.input.NetworkAliases, alias)
}
}
} }
// For Gitea // For Gitea, forcing --privileged off is not enough, other options reach the host too
// When privileged mode is disabled, container.options is workflow-controlled
// untrusted input. Strip the HostConfig fields that would let a workflow break
// out of the container (host namespaces, capability expansion, security profile
// overrides, device and runtime access). Otherwise these survive into the final
// HostConfig even though --privileged is forced off.
if !hostConfig.Privileged { if !hostConfig.Privileged {
sanitizeOptionsHostConfig(logger, containerConfig.HostConfig) trusted, err := parseOptionsHostConfig(cr.input.RunnerOptions)
if err != nil {
return nil, nil, err
}
sanitizeOptionsHostConfig(logger, containerConfig.HostConfig, trusted)
} }
logger.Debugf("Custom container.Config from options ==> %+v", containerConfig.Config) logger.Debugf("Custom container.Config from options ==> %+v", containerConfig.Config)
err = mergo.Merge(config, containerConfig.Config, mergo.WithOverride, mergo.WithAppendSlice) err = mergo.Merge(config, containerConfig.Config, mergo.WithOverride, mergo.WithAppendSlice)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("Cannot merge container.Config options: '%s': '%w'", input.Options, err) return nil, nil, fmt.Errorf("cannot merge container.Config options: '%s': '%w'", options, err)
} }
logger.Debugf("Merged container.Config ==> %+v", config) logger.Debugf("Merged container.Config ==> %+v", config)
@@ -622,14 +613,15 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
networkMode := hostConfig.NetworkMode networkMode := hostConfig.NetworkMode
err = mergo.Merge(hostConfig, containerConfig.HostConfig, mergo.WithOverride) err = mergo.Merge(hostConfig, containerConfig.HostConfig, mergo.WithOverride)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("Cannot merge container.HostConfig options: '%s': '%w'", input.Options, err) return nil, nil, fmt.Errorf("cannot merge container.HostConfig options: '%s': '%w'", options, err)
} }
hostConfig.Binds = binds hostConfig.Binds = binds
hostConfig.Mounts = mounts hostConfig.Mounts = mounts
if cf.name != "" { if cf.name != "" {
logger.Warn("--name in the options will be ignored.") logger.Warn("--name in the options will be ignored.")
} }
if len(copts.netMode.Value()) > 0 { // the runner's own network mode was put into copts above, so ask the flags instead
if flags.Changed("network") || flags.Changed("net") {
logger.Warn("--network and --net in the options will be ignored.") logger.Warn("--network and --net in the options will be ignored.")
} }
hostConfig.NetworkMode = networkMode hostConfig.NetworkMode = networkMode
@@ -944,43 +936,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
} }
} }
func (cr *containerReference) CopyTarStream(ctx context.Context, destPath string, tarStream io.Reader) error { func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
if cr.id == "" {
return cr.missingContainerError("copy to %s", destPath)
}
// Mkdir, with a path relative to the DestinationPath ("/") below. Docker 29.5+
// rejects absolute tar entry names with "path escapes from parent".
buf := &bytes.Buffer{}
tw := tar.NewWriter(buf)
_ = tw.WriteHeader(&tar.Header{
Name: strings.TrimPrefix(destPath, "/"),
Mode: 0o777,
Typeflag: tar.TypeDir,
})
tw.Close()
_, err := cr.cli.CopyToContainer(ctx, cr.id, client.CopyToContainerOptions{
DestinationPath: "/",
Content: buf,
})
if err != nil {
return fmt.Errorf("failed to mkdir to copy content to container: %w", err)
}
// Copy Content
_, err = cr.cli.CopyToContainer(ctx, cr.id, client.CopyToContainerOptions{
DestinationPath: destPath,
Content: tarStream,
})
if err != nil {
return fmt.Errorf("failed to copy content to container: %w", err)
}
// If this fails, then folders have wrong permissions on non root container
if cr.UID != 0 || cr.GID != 0 {
_ = cr.Exec([]string{"chown", "-R", fmt.Sprintf("%d:%d", cr.UID, cr.GID), destPath}, nil, "0", "")(ctx)
}
return nil
}
func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore bool) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
if cr.id == "" { if cr.id == "" {
return cr.missingContainerError("copy directory to %s", dstPath) return cr.missingContainerError("copy directory to %s", dstPath)
@@ -1021,10 +977,10 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore bool
} }
fc := &filecollector.FileCollector{ fc := &filecollector.FileCollector{
Fs: &filecollector.DefaultFs{}, Ignorer: ignorer,
Ignorer: ignorer, SrcPath: srcPath,
SrcPath: srcPath, SrcPrefix: srcPrefix,
SrcPrefix: srcPrefix, SkipGitDir: skipGitDir,
Handler: &filecollector.TarCollector{ Handler: &filecollector.TarCollector{
TarWriter: tw, TarWriter: tw,
UID: cr.UID, UID: cr.UID,
@@ -1172,74 +1128,64 @@ func (cr *containerReference) wait() common.Executor {
} }
// For Gitea // For Gitea
// sanitizeOptionsHostConfig clears the HostConfig fields parsed from a // sanitizeOptionsHostConfig takes back everything a workflow could escape the container with,
// workflow-controlled container.options string that could be used to escape the // setting each field to trusted, which is what the runner's own options parse to on their own.
// container when privileged mode is disabled. It must only be called when the // Only for unprivileged mode, since privileged mode grants host access anyway.
// runner has privileged mode turned off; with privileged mode enabled the func sanitizeOptionsHostConfig(logger logrus.FieldLogger, hostConfig, trusted *container.HostConfig) {
// administrator has already opted into host access. resetOption(logger, "--pid", &hostConfig.PidMode, trusted.PidMode)
func sanitizeOptionsHostConfig(logger logrus.FieldLogger, hostConfig *container.HostConfig) { resetOption(logger, "--ipc", &hostConfig.IpcMode, trusted.IpcMode)
warn := func(option string) { resetOption(logger, "--uts", &hostConfig.UTSMode, trusted.UTSMode)
logger.Warnf("container option %q is not allowed when privileged mode is disabled and will be ignored", option) resetOption(logger, "--cgroupns", &hostConfig.CgroupnsMode, trusted.CgroupnsMode)
} resetOption(logger, "--userns", &hostConfig.UsernsMode, trusted.UsernsMode) // --userns=host would undo the remapping the runner asked for
resetOption(logger, "--cap-add", &hostConfig.CapAdd, trusted.CapAdd)
resetOption(logger, "--security-opt", &hostConfig.SecurityOpt, trusted.SecurityOpt)
resetOption(logger, "--device", &hostConfig.Devices, trusted.Devices)
resetOption(logger, "--device-cgroup-rule", &hostConfig.DeviceCgroupRules, trusted.DeviceCgroupRules)
resetOption(logger, "--gpus", &hostConfig.DeviceRequests, trusted.DeviceRequests)
resetOption(logger, "--volumes-from", &hostConfig.VolumesFrom, trusted.VolumesFrom)
resetOption(logger, "--runtime", &hostConfig.Runtime, trusted.Runtime)
resetOption(logger, "--cgroup-parent", &hostConfig.CgroupParent, trusted.CgroupParent)
resetOption(logger, "--sysctl", &hostConfig.Sysctls, trusted.Sysctls)
resetOption(logger, "--isolation", &hostConfig.Isolation, trusted.Isolation) // windows: process isolation drops the hyper-v boundary
resetOption(logger, "--volume-driver", &hostConfig.VolumeDriver, trusted.VolumeDriver)
// systempaths=unconfined lands in these two rather than in SecurityOpt
resetOption(logger, "--security-opt", &hostConfig.MaskedPaths, trusted.MaskedPaths)
resetOption(logger, "--security-opt", &hostConfig.ReadonlyPaths, trusted.ReadonlyPaths)
if hostConfig.PidMode != "" { // a driver mounts what it likes, e.g. local with device= binds any host path, which
warn("--pid") // valid_volumes never gets to see
hostConfig.PidMode = "" hostConfig.Mounts = slices.DeleteFunc(hostConfig.Mounts, func(mt mount.Mount) bool {
if mt.VolumeOptions == nil || mt.VolumeOptions.DriverConfig == nil ||
slices.ContainsFunc(trusted.Mounts, func(t mount.Mount) bool { return reflect.DeepEqual(t, mt) }) {
return false
}
logger.Warnf("volume driver of %q in the workflow is not allowed when privileged mode is disabled and will be ignored", mt.Source)
return true
})
}
// resetOption puts a field back to the runner's own value. It compares the values rather than
// the flags, so a field that more than one option feeds cannot slip through.
func resetOption[T any](logger logrus.FieldLogger, option string, field *T, trusted T) {
if reflect.DeepEqual(*field, trusted) {
return
} }
if hostConfig.IpcMode != "" { logger.Warnf("container option %q in the workflow is not allowed when privileged mode is disabled and will be ignored", option)
warn("--ipc") *field = trusted
hostConfig.IpcMode = "" }
// parseOptionsHostConfig parses one options string on its own, to see what it alone asks for.
// Even "" goes through the parser, or its empty slices and maps would differ from a real parse.
func parseOptionsHostConfig(options string) (*container.HostConfig, error) {
flags, copts, _, err := parseContainerOptions(options)
if err != nil {
return nil, err
} }
if hostConfig.UTSMode != "" { containerConfig, err := parse(flags, copts, runtime.GOOS)
warn("--uts") if err != nil {
hostConfig.UTSMode = "" return nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
}
if hostConfig.CgroupnsMode != "" {
warn("--cgroupns")
hostConfig.CgroupnsMode = ""
}
// UsernsMode is set from the runner-controlled input; never let options
// override it (e.g. --userns=host disables user namespace remapping).
if hostConfig.UsernsMode != "" {
warn("--userns")
hostConfig.UsernsMode = ""
}
if len(hostConfig.CapAdd) > 0 {
warn("--cap-add")
hostConfig.CapAdd = nil
}
if len(hostConfig.SecurityOpt) > 0 {
warn("--security-opt")
hostConfig.SecurityOpt = nil
}
if len(hostConfig.Devices) > 0 {
warn("--device")
hostConfig.Devices = nil
}
if len(hostConfig.DeviceCgroupRules) > 0 {
warn("--device-cgroup-rule")
hostConfig.DeviceCgroupRules = nil
}
if len(hostConfig.DeviceRequests) > 0 {
warn("--gpus")
hostConfig.DeviceRequests = nil
}
if len(hostConfig.VolumesFrom) > 0 {
warn("--volumes-from")
hostConfig.VolumesFrom = nil
}
if hostConfig.Runtime != "" {
warn("--runtime")
hostConfig.Runtime = ""
}
if hostConfig.CgroupParent != "" {
warn("--cgroup-parent")
hostConfig.CgroupParent = ""
}
if len(hostConfig.Sysctls) > 0 {
warn("--sysctl")
hostConfig.Sysctls = nil
} }
return containerConfig.HostConfig, nil
} }
// For Gitea // For Gitea
@@ -1280,6 +1226,12 @@ func (cr *containerReference) sanitizeConfig(ctx context.Context, config *contai
} }
hostConfig.Mounts = sanitizedMounts hostConfig.Mounts = sanitizedMounts
} else { } else {
for _, bind := range hostConfig.Binds {
logger.Warnf("[%s] is not a valid volume, will be ignored", bind)
}
for _, mt := range hostConfig.Mounts {
logger.Warnf("[%s] is not a valid volume, will be ignored", mt.Source)
}
hostConfig.Binds = []string{} hostConfig.Binds = []string{}
hostConfig.Mounts = []mount.Mount{} hostConfig.Mounts = []mount.Mount{}
} }
@@ -1317,15 +1269,15 @@ func overlayVolumes(dst, src *container.HostConfig) {
type validVolumeMatcher struct { type validVolumeMatcher struct {
allowAll bool allowAll bool
named []glob.Glob named []*glob.Pattern
host []glob.Glob host []*glob.Pattern
} }
func newValidVolumeMatcher(ctx context.Context, validVolumes []string) validVolumeMatcher { func newValidVolumeMatcher(ctx context.Context, validVolumes []string) validVolumeMatcher {
logger := common.Logger(ctx) logger := common.Logger(ctx)
ret := validVolumeMatcher{ ret := validVolumeMatcher{
named: make([]glob.Glob, 0, len(validVolumes)), named: make([]*glob.Pattern, 0, len(validVolumes)),
host: make([]glob.Glob, 0, len(validVolumes)), host: make([]*glob.Pattern, 0, len(validVolumes)),
} }
for _, v := range validVolumes { for _, v := range validVolumes {
+192 -263
View File
@@ -5,7 +5,6 @@
package container package container
import ( import (
"archive/tar"
"bufio" "bufio"
"bytes" "bytes"
"context" "context"
@@ -17,7 +16,6 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -149,12 +147,17 @@ func (m *mockDockerClient) NetworkRemove(ctx context.Context, id string, opts mo
return args.Get(0).(mobyclient.NetworkRemoveResult), args.Error(1) return args.Get(0).(mobyclient.NetworkRemoveResult), args.Error(1)
} }
type endlessReader struct { type interruptReader struct {
io.Reader started chan struct{}
interrupted chan struct{}
stopped chan struct{}
} }
func (r endlessReader) Read(_ []byte) (n int, err error) { func (r *interruptReader) Read(_ []byte) (int, error) {
return 1, nil close(r.started)
<-r.interrupted
close(r.stopped)
return 0, io.EOF
} }
type mockConn struct { type mockConn struct {
@@ -174,16 +177,17 @@ func (m *mockConn) Close() (err error) {
func TestDockerExecAbort(t *testing.T) { func TestDockerExecAbort(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
reader := &interruptReader{started: make(chan struct{}), interrupted: make(chan struct{}), stopped: make(chan struct{})}
conn := &mockConn{} conn := &mockConn{}
conn.On("Write", mock.AnythingOfType("[]uint8")).Return(1, nil) conn.On("Write", []byte{3}).
Run(func(mock.Arguments) { close(reader.interrupted) }).
Return(1, nil)
client := &mockDockerClient{} client := &mockDockerClient{}
client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).Return(mobyclient.ExecCreateResult{ID: "id"}, nil) client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).Return(mobyclient.ExecCreateResult{ID: "id"}, nil)
client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).Return(mobyclient.ExecAttachResult{ client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).Return(mobyclient.ExecAttachResult{
HijackedResponse: mobyclient.HijackedResponse{ Conn: conn,
Conn: conn, Reader: bufio.NewReader(reader),
Reader: bufio.NewReader(endlessReader{}),
},
}, nil) }, nil)
cr := &containerReference{ cr := &containerReference{
@@ -200,11 +204,11 @@ func TestDockerExecAbort(t *testing.T) {
channel <- cr.exec([]string{""}, map[string]string{}, "user", "workdir")(ctx) channel <- cr.exec([]string{""}, map[string]string{}, "user", "workdir")(ctx)
}() }()
time.Sleep(500 * time.Millisecond) <-reader.started
cancel() cancel()
err := <-channel err := <-channel
<-reader.stopped
assert.ErrorIs(t, err, context.Canceled) //nolint:testifylint // pre-existing issue from nektos/act assert.ErrorIs(t, err, context.Canceled) //nolint:testifylint // pre-existing issue from nektos/act
conn.AssertExpectations(t) conn.AssertExpectations(t)
@@ -219,10 +223,8 @@ func TestDockerExecFailure(t *testing.T) {
client := &mockDockerClient{} client := &mockDockerClient{}
client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).Return(mobyclient.ExecCreateResult{ID: "id"}, nil) client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).Return(mobyclient.ExecCreateResult{ID: "id"}, nil)
client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).Return(mobyclient.ExecAttachResult{ client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).Return(mobyclient.ExecAttachResult{
HijackedResponse: mobyclient.HijackedResponse{ Conn: conn,
Conn: conn, Reader: bufio.NewReader(strings.NewReader("output")),
Reader: bufio.NewReader(strings.NewReader("output")),
},
}, nil) }, nil)
client.On("ExecInspect", ctx, "id", mobyclient.ExecInspectOptions{}).Return(mobyclient.ExecInspectResult{ client.On("ExecInspect", ctx, "id", mobyclient.ExecInspectOptions{}).Return(mobyclient.ExecInspectResult{
ExitCode: 1, ExitCode: 1,
@@ -274,10 +276,8 @@ func TestDockerAttachFlushesTrailingLine(t *testing.T) {
client := &mockDockerClient{} client := &mockDockerClient{}
client.On("ContainerAttach", ctx, "123", mock.AnythingOfType("client.ContainerAttachOptions")). client.On("ContainerAttach", ctx, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{ Return(mobyclient.ContainerAttachResult{
HijackedResponse: mobyclient.HijackedResponse{ Conn: &mockConn{},
Conn: &mockConn{}, Reader: bufio.NewReader(framed),
Reader: bufio.NewReader(framed),
},
}, nil) }, nil)
statusCh := make(chan container.WaitResponse, 1) statusCh := make(chan container.WaitResponse, 1)
@@ -342,131 +342,25 @@ func TestDockerWaitFailure(t *testing.T) {
client.AssertExpectations(t) client.AssertExpectations(t)
} }
func TestDockerCopyTarStream(t *testing.T) {
ctx := context.Background()
client := &mockDockerClient{}
client.On("CopyToContainer", ctx, "123", mock.MatchedBy(func(opts mobyclient.CopyToContainerOptions) bool {
return opts.DestinationPath == "/" && opts.Content != nil
})).Return(mobyclient.CopyToContainerResult{}, nil)
client.On("CopyToContainer", ctx, "123", mock.MatchedBy(func(opts mobyclient.CopyToContainerOptions) bool {
return opts.DestinationPath == "/var/run/act" && opts.Content != nil
})).Return(mobyclient.CopyToContainerResult{}, nil)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
}
_ = cr.CopyTarStream(ctx, "/var/run/act", &bytes.Buffer{})
client.AssertExpectations(t)
}
// Docker 29.5+ rejects absolute names in the mkdir tarball with
// "path escapes from parent", since it is extracted relative to "/".
func TestDockerCopyTarStreamMkdirEntryIsRelative(t *testing.T) {
ctx := context.Background()
var mkdirNames []string
client := &mockDockerClient{}
client.On("CopyToContainer", ctx, "123", mock.MatchedBy(func(opts mobyclient.CopyToContainerOptions) bool {
if opts.DestinationPath != "/" || opts.Content == nil {
return false
}
tr := tar.NewReader(opts.Content)
for {
hdr, err := tr.Next()
if err != nil {
break
}
mkdirNames = append(mkdirNames, hdr.Name)
}
return true
})).Return(mobyclient.CopyToContainerResult{}, nil)
client.On("CopyToContainer", ctx, "123", mock.MatchedBy(func(opts mobyclient.CopyToContainerOptions) bool {
return opts.DestinationPath == "/var/run/act" && opts.Content != nil
})).Return(mobyclient.CopyToContainerResult{}, nil)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
}
require.NoError(t, cr.CopyTarStream(ctx, "/var/run/act", &bytes.Buffer{}))
assert.Equal(t, []string{"var/run/act"}, mkdirNames)
client.AssertExpectations(t)
}
func TestDockerCopyTarStreamErrorInCopyFiles(t *testing.T) {
ctx := context.Background()
merr := errors.New("Failure")
client := &mockDockerClient{}
client.On("CopyToContainer", ctx, "123", mock.MatchedBy(func(opts mobyclient.CopyToContainerOptions) bool {
return opts.DestinationPath == "/" && opts.Content != nil
})).Return(mobyclient.CopyToContainerResult{}, merr)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
}
err := cr.CopyTarStream(ctx, "/var/run/act", &bytes.Buffer{})
assert.ErrorIs(t, err, merr) //nolint:testifylint // pre-existing issue from nektos/act
client.AssertExpectations(t)
}
func TestDockerCopyTarStreamErrorInMkdir(t *testing.T) {
ctx := context.Background()
merr := errors.New("Failure")
client := &mockDockerClient{}
client.On("CopyToContainer", ctx, "123", mock.MatchedBy(func(opts mobyclient.CopyToContainerOptions) bool {
return opts.DestinationPath == "/" && opts.Content != nil
})).Return(mobyclient.CopyToContainerResult{}, nil)
client.On("CopyToContainer", ctx, "123", mock.MatchedBy(func(opts mobyclient.CopyToContainerOptions) bool {
return opts.DestinationPath == "/var/run/act" && opts.Content != nil
})).Return(mobyclient.CopyToContainerResult{}, merr)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
}
err := cr.CopyTarStream(ctx, "/var/run/act", &bytes.Buffer{})
assert.ErrorIs(t, err, merr) //nolint:testifylint // pre-existing issue from nektos/act
client.AssertExpectations(t)
}
// A remove that raced the daemon's AutoRemove teardown is not a failure and must not // A remove that raced the daemon's AutoRemove teardown is not a failure and must not
// be logged as one. // be logged as one.
func TestRemoveIgnoresAutoRemoveRace(t *testing.T) { func TestRemoveIgnoresAutoRemoveRace(t *testing.T) {
removeFailure := errors.New("driver failed to remove root filesystem")
removeOpts := mobyclient.ContainerRemoveOptions{RemoveVolumes: true, Force: true} removeOpts := mobyclient.ContainerRemoveOptions{RemoveVolumes: true, Force: true}
killOpts := mobyclient.ContainerKillOptions{Signal: "SIGKILL"} killOpts := mobyclient.ContainerKillOptions{Signal: "SIGKILL"}
for _, tc := range []struct { for _, tc := range []struct {
name string name string
err error err error
wantWait bool wantWait bool
wantFailure bool waitErr error
wantErr error
}{ }{
{name: "removal in progress", err: cerrdefs.ErrConflict.WithMessage("removal of container abc is already in progress"), wantWait: true}, {name: "removal in progress", err: cerrdefs.ErrConflict.WithMessage("removal of container abc is already in progress"), wantWait: true},
{name: "wait canceled", err: cerrdefs.ErrConflict, wantWait: true, waitErr: context.Canceled, wantErr: context.Canceled},
{name: "removed during wait", err: cerrdefs.ErrConflict, wantWait: true, waitErr: cerrdefs.ErrNotFound},
{name: "already removed", err: cerrdefs.ErrNotFound.WithMessage("No such container: abc")}, {name: "already removed", err: cerrdefs.ErrNotFound.WithMessage("No such container: abc")},
{name: "removed cleanly", err: nil}, {name: "removed cleanly", err: nil},
{name: "real failure", err: errors.New("driver failed to remove root filesystem"), wantFailure: true}, {name: "real failure", err: removeFailure, wantErr: removeFailure},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
logger, hook := test.NewNullLogger() logger, hook := test.NewNullLogger()
@@ -476,21 +370,24 @@ func TestRemoveIgnoresAutoRemoveRace(t *testing.T) {
client.On("ContainerRemove", ctx, "abc", removeOpts).Return(mobyclient.ContainerRemoveResult{}, tc.err) client.On("ContainerRemove", ctx, "abc", removeOpts).Return(mobyclient.ContainerRemoveResult{}, tc.err)
if tc.wantWait { if tc.wantWait {
removed := make(chan container.WaitResponse, 1) removed := make(chan container.WaitResponse, 1)
removed <- container.WaitResponse{} waitErrors := make(chan error, 1)
if tc.waitErr == nil {
removed <- container.WaitResponse{}
} else {
waitErrors <- tc.waitErr
}
client.On("ContainerWait", mock.Anything, "abc", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionRemoved}). client.On("ContainerWait", mock.Anything, "abc", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionRemoved}).
Return(mobyclient.ContainerWaitResult{Result: removed}) Return(mobyclient.ContainerWaitResult{Result: removed, Error: waitErrors})
} }
cr := &containerReference{id: "abc", cli: client} cr := &containerReference{id: "abc", cli: client}
require.NoError(t, cr.remove()(ctx)) require.ErrorIs(t, cr.remove()(ctx), tc.wantErr)
// a failure keeps the id, so a later Remove() can retry it if tc.wantErr != nil {
if tc.wantFailure {
assert.Equal(t, "abc", cr.id) assert.Equal(t, "abc", cr.id)
assert.Len(t, hook.AllEntries(), 1)
} else { } else {
assert.Empty(t, cr.id) assert.Empty(t, cr.id)
assert.Empty(t, hook.AllEntries())
} }
assert.Empty(t, hook.AllEntries())
client.AssertExpectations(t) client.AssertExpectations(t)
}) })
} }
@@ -581,8 +478,7 @@ func TestRejectsMissingContainer(t *testing.T) {
assert.Contains(t, err.Error(), `container "job-1" does not exist`, op) assert.Contains(t, err.Error(), `container "job-1" does not exist`, op)
} }
check("copyContent", cr.copyContent("/var/run/act", &FileEntry{Name: "x", Mode: 0o644})(ctx)) check("copyContent", cr.copyContent("/var/run/act", &FileEntry{Name: "x", Mode: 0o644})(ctx))
check("copyDir", cr.copyDir("/var/run/act", "/src", false)(ctx)) check("copyDir", cr.copyDir("/var/run/act", "/src", false, false)(ctx))
check("CopyTarStream", cr.CopyTarStream(ctx, "/var/run/act", &bytes.Buffer{}))
check("exec", cr.exec([]string{"echo"}, nil, "", "")(ctx)) check("exec", cr.exec([]string{"echo"}, nil, "", "")(ctx))
_, err := cr.GetContainerArchive(ctx, "/var/run/act/x") _, err := cr.GetContainerArchive(ctx, "/var/run/act/x")
check("GetContainerArchive", err) check("GetContainerArchive", err)
@@ -618,35 +514,6 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
client.AssertExpectations(t) client.AssertExpectations(t)
} }
// TestDockerCopyToSymlinkPath is a regression test for gitea/runner#981. Most base images
// symlink /var/run to /run, so copying into /var/run/act traverses that symlink. The broken
// docker 29.5.1 daemon fails the extraction with "mkdirat var/run: file exists" (fixed in
// 29.5.2). Running against the daemon shipped in the dind image, this catches a bad bump.
func TestDockerCopyToSymlinkPath(t *testing.T) {
requireDocker(t)
ctx := context.Background()
rc := NewContainer(&NewContainerInput{
Image: "alpine:latest",
Entrypoint: []string{"sleep", "30"},
Name: "act-test-symlink-" + time.Now().Format("20060102150405.000000"),
AutoRemove: true,
})
require.NoError(t, rc.Pull(false)(ctx))
require.NoError(t, rc.Create(nil, nil)(ctx))
require.NoError(t, rc.Start(false)(ctx))
t.Cleanup(func() {
_ = rc.Remove()(ctx)
_ = rc.Close()(ctx)
})
// CopyTarStream first creates the destination directory by extracting a tar at "/",
// which makes the daemon mkdir var, then var/run (the symlink), then act — the exact
// step that fails on the broken daemon.
err := rc.CopyTarStream(ctx, "/var/run/act/actions/", &bytes.Buffer{})
require.NoError(t, err)
}
// Type assert containerReference implements ExecutionsEnvironment // Type assert containerReference implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &containerReference{} var _ ExecutionsEnvironment = &containerReference{}
@@ -710,7 +577,7 @@ func TestCheckVolumes(t *testing.T) {
} }
for _, tc := range testCases { for _, tc := range testCases {
t.Run(tc.desc, func(t *testing.T) { t.Run(tc.desc, func(t *testing.T) {
logger, _ := test.NewNullLogger() logger, hook := test.NewNullLogger()
ctx := common.WithLogger(context.Background(), logger) ctx := common.WithLogger(context.Background(), logger)
cr := &containerReference{ cr := &containerReference{
input: &NewContainerInput{ input: &NewContainerInput{
@@ -719,112 +586,138 @@ func TestCheckVolumes(t *testing.T) {
} }
_, hostConf := cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{Binds: tc.binds}) _, hostConf := cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{Binds: tc.binds})
assert.Equal(t, tc.expectedBinds, hostConf.Binds) assert.Equal(t, tc.expectedBinds, hostConf.Binds)
assert.Len(t, hook.AllEntries(), len(tc.binds)-len(tc.expectedBinds)) // every drop is warned about
}) })
} }
} }
// A volume driver decides for itself what it mounts, e.g. the local driver with device= binds
// any host path, which valid_volumes never gets to see.
func TestMergeContainerConfigsDropsVolumeDriversFromWorkflows(t *testing.T) {
const escape = "--mount type=volume,src=job-escape,dst=/host,volume-driver=local,volume-opt=type=none,volume-opt=o=bind,volume-opt=device=/"
hostConfig, _ := mergeOptions(t, "", escape+" --mount type=volume,src=job-plain,dst=/cache", false)
require.Len(t, hostConfig.Mounts, 1)
assert.Equal(t, "job-plain", hostConfig.Mounts[0].Source)
// the same mount from the runner's own options is the administrator's to make
hostConfig, _ = mergeOptions(t, escape, "", false)
require.Len(t, hostConfig.Mounts, 1)
assert.Equal(t, "job-escape", hostConfig.Mounts[0].Source)
}
// Both of these are read here, on the runner, so a workflow could read the runner's files
// and environment with them.
func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
hostFile := filepath.Join(t.TempDir(), "host.env")
require.NoError(t, os.WriteFile(hostFile, []byte("STOLEN=from-the-host\n"), 0o600))
t.Setenv("RUNNER_SECRET", "s3cr3t")
for _, option := range []string{"--env-file " + hostFile, "--label-file " + hostFile} {
logger, _ := test.NewNullLogger()
cr := &containerReference{input: &NewContainerInput{NetworkMode: "bridge", WorkflowOptions: option}}
_, _, err := cr.mergeContainerConfigs(common.WithLogger(context.Background(), logger), &container.Config{}, &container.HostConfig{})
require.ErrorContains(t, err, "not allowed in a workflow")
// the runner reading its own files is what those options are for
cr = &containerReference{input: &NewContainerInput{NetworkMode: "bridge", RunnerOptions: option}}
_, _, err = cr.mergeContainerConfigs(common.WithLogger(context.Background(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
}
// a bare name is no longer resolved from the runner's environment, for either source
logger, _ := test.NewNullLogger()
cr := &containerReference{input: &NewContainerInput{
NetworkMode: "bridge",
RunnerOptions: "--env RUNNER_SECRET",
WorkflowOptions: "--env RUNNER_SECRET --env GIVEN=value",
}}
config, _, err := cr.mergeContainerConfigs(common.WithLogger(context.Background(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
assert.Equal(t, []string{"RUNNER_SECRET", "RUNNER_SECRET", "GIVEN=value"}, config.Env)
}
func TestSanitizeOptionsHostConfig(t *testing.T) { func TestSanitizeOptionsHostConfig(t *testing.T) {
logger, _ := test.NewNullLogger() logger, _ := test.NewNullLogger()
dangerous := func() *container.HostConfig { // every field the sanitizer resets, so a reset dropped in a refactor fails here
return &container.HostConfig{ hostConfig := &container.HostConfig{
PidMode: "host", PidMode: "host",
IpcMode: "host", IpcMode: "host",
UTSMode: "host", UTSMode: "host",
CgroupnsMode: "host", CgroupnsMode: "host",
UsernsMode: "host", UsernsMode: "host",
CapAdd: []string{"ALL"}, CapAdd: []string{"ALL"},
SecurityOpt: []string{"seccomp=unconfined", "apparmor=unconfined"}, SecurityOpt: []string{"seccomp=unconfined", "apparmor=unconfined"},
VolumesFrom: []string{"other"}, VolumesFrom: []string{"other"},
Runtime: "runc", Runtime: "runc",
Resources: container.Resources{ Isolation: "process",
CgroupParent: "/custom", VolumeDriver: "rogue",
Devices: []container.DeviceMapping{{PathOnHost: "/dev/sda", PathInContainer: "/dev/sda", CgroupPermissions: "rwm"}}, MaskedPaths: []string{},
DeviceCgroupRules: []string{"a *:* rwm"}, ReadonlyPaths: []string{},
}, CgroupParent: "/custom",
Sysctls: map[string]string{"net.ipv4.ip_forward": "1"}, Devices: []container.DeviceMapping{{PathOnHost: "/dev/sda", PathInContainer: "/dev/sda", CgroupPermissions: "rwm"}},
} DeviceCgroupRules: []string{"a *:* rwm"},
DeviceRequests: []container.DeviceRequest{{Count: -1, Capabilities: [][]string{{"gpu"}}}},
Sysctls: map[string]string{"net.ipv4.ip_forward": "1"},
} }
hostConfig := dangerous() sanitizeOptionsHostConfig(logger, hostConfig, &container.HostConfig{})
sanitizeOptionsHostConfig(logger, hostConfig)
assert.Empty(t, string(hostConfig.PidMode)) assert.Equal(t, &container.HostConfig{}, hostConfig)
assert.Empty(t, string(hostConfig.IpcMode)) }
assert.Empty(t, string(hostConfig.UTSMode))
assert.Empty(t, string(hostConfig.CgroupnsMode)) // mergeOptions merges both option sources into a bare container, returning the result and its log.
assert.Empty(t, string(hostConfig.UsernsMode)) func mergeOptions(t *testing.T, runnerOptions, workflowOptions string, privileged bool) (*container.HostConfig, *test.Hook) {
assert.Empty(t, hostConfig.CapAdd) t.Helper()
assert.Empty(t, hostConfig.SecurityOpt) logger, hook := test.NewNullLogger()
assert.Empty(t, hostConfig.Devices) cr := &containerReference{input: &NewContainerInput{
assert.Empty(t, hostConfig.DeviceCgroupRules) RunnerOptions: runnerOptions,
assert.Empty(t, hostConfig.VolumesFrom) WorkflowOptions: workflowOptions,
assert.Empty(t, hostConfig.Runtime) NetworkMode: "bridge",
assert.Empty(t, hostConfig.CgroupParent) UsernsMode: "private",
assert.Empty(t, hostConfig.Sysctls) }}
_, hostConfig, err := cr.mergeContainerConfigs(common.WithLogger(context.Background(), logger), &container.Config{}, &container.HostConfig{
Privileged: privileged,
UsernsMode: container.UsernsMode("private"),
NetworkMode: container.NetworkMode("bridge"),
})
require.NoError(t, err)
return hostConfig, hook
} }
func TestMergeContainerConfigsStripsDangerousOptionsWhenUnprivileged(t *testing.T) { func TestMergeContainerConfigsStripsDangerousOptionsWhenUnprivileged(t *testing.T) {
// OS-independent options only: --device parsing requires a linux/windows // OS-independent options only, --device and --gpus need a linux/windows server OS
// server OS, which is not guaranteed for the test host.
const dangerousOptions = "--pid=host --ipc=host --uts=host --cgroupns=host " + const dangerousOptions = "--pid=host --ipc=host --uts=host --cgroupns=host " +
"--userns=host --cap-add=ALL --security-opt seccomp=unconfined " + "--userns=host --cap-add=ALL --security-opt seccomp=unconfined " +
"--security-opt apparmor=unconfined --volumes-from other " + "--security-opt apparmor=unconfined --volumes-from other --isolation process " +
"--runtime runc --cgroup-parent /custom --sysctl net.ipv4.ip_forward=1" "--runtime runc --cgroup-parent /custom --sysctl net.ipv4.ip_forward=1"
t.Run("unprivileged strips host-escape options", func(t *testing.T) { // whatever the workflow adds, an unprivileged container comes out exactly as the runner's
logger, _ := test.NewNullLogger() // own options alone describe it, field for field
ctx := common.WithLogger(context.Background(), logger) for _, runnerOptions := range []string{"--shm-size 1g", dangerousOptions, "--cap-add SYS_ADMIN --security-opt seccomp=unconfined"} {
cr := &containerReference{ runnerOnly, _ := mergeOptions(t, runnerOptions, "", false)
input: &NewContainerInput{ withWorkflow, _ := mergeOptions(t, runnerOptions, dangerousOptions, false)
Options: dangerousOptions,
NetworkMode: "bridge",
UsernsMode: "private",
},
}
_, hostConfig, err := cr.mergeContainerConfigs(ctx, &container.Config{}, &container.HostConfig{ assert.Equal(t, runnerOnly, withWorkflow, "runner options: %q", runnerOptions)
Privileged: false, }
UsernsMode: container.UsernsMode("private"),
NetworkMode: container.NetworkMode("bridge"),
})
require.NoError(t, err)
assert.False(t, hostConfig.Privileged) // the same options from the runner reach the daemon, even --userns, which no workflow may set
assert.Empty(t, string(hostConfig.PidMode)) kept, _ := mergeOptions(t, dangerousOptions, "", false)
assert.Empty(t, string(hostConfig.IpcMode)) assert.Equal(t, "host", string(kept.PidMode))
assert.Empty(t, string(hostConfig.UTSMode)) assert.Equal(t, []string{"ALL"}, kept.CapAdd)
assert.Empty(t, string(hostConfig.CgroupnsMode)) assert.Equal(t, "runc", kept.Runtime)
// UsernsMode must keep the runner-controlled value, not the one from options. assert.Equal(t, "host", string(kept.UsernsMode))
assert.Equal(t, "private", string(hostConfig.UsernsMode)) assert.False(t, kept.Privileged)
assert.Empty(t, hostConfig.CapAdd)
assert.Empty(t, hostConfig.SecurityOpt)
assert.Empty(t, hostConfig.VolumesFrom)
assert.Empty(t, hostConfig.Runtime)
assert.Empty(t, hostConfig.CgroupParent)
assert.Empty(t, hostConfig.Sysctls)
})
t.Run("privileged preserves options", func(t *testing.T) { // privileged is the administrator opting in, so the workflow's options are honored
logger, _ := test.NewNullLogger() privileged, _ := mergeOptions(t, "", dangerousOptions, true)
ctx := common.WithLogger(context.Background(), logger) assert.Equal(t, "host", string(privileged.PidMode))
cr := &containerReference{ assert.Equal(t, []string{"ALL"}, privileged.CapAdd)
input: &NewContainerInput{ assert.Equal(t, []string{"seccomp=unconfined", "apparmor=unconfined"}, privileged.SecurityOpt)
Options: "--pid=host --cap-add=ALL --security-opt seccomp=unconfined",
NetworkMode: "bridge",
},
}
_, hostConfig, err := cr.mergeContainerConfigs(ctx, &container.Config{}, &container.HostConfig{
Privileged: true,
NetworkMode: container.NetworkMode("bridge"),
})
require.NoError(t, err)
assert.Equal(t, "host", string(hostConfig.PidMode))
assert.Equal(t, []string{"ALL"}, hostConfig.CapAdd)
assert.Equal(t, []string{"seccomp=unconfined"}, hostConfig.SecurityOpt)
})
} }
func TestCheckVolumesRejectsEscapingHostPaths(t *testing.T) { func TestCheckVolumesRejectsEscapingHostPaths(t *testing.T) {
@@ -915,6 +808,20 @@ func TestContainerInfoFromInspect(t *testing.T) {
assert.Equal(t, "abc123", info.ID) assert.Equal(t, "abc123", info.ID)
assert.Equal(t, HealthNone, info.Health) assert.Equal(t, HealthNone, info.Health)
}) })
t.Run("reports the mount sources by container path", func(t *testing.T) {
info := containerInfoFromInspect(container.InspectResponse{
Mounts: []container.MountPoint{
{Type: mount.TypeVolume, Name: "job", Source: "/var/lib/docker/volumes/job/_data", Destination: "/workspace/owner/repo"},
{Type: mount.TypeBind, Source: "/var/run/docker.sock", Destination: "/var/run/docker.sock"},
},
})
assert.Equal(t, map[string]string{
"/workspace/owner/repo": "/var/lib/docker/volumes/job/_data",
"/var/run/docker.sock": "/var/run/docker.sock",
}, info.Mounts)
})
} }
func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) { func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) {
@@ -922,8 +829,8 @@ func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) {
ctx := common.WithLogger(context.Background(), logger) ctx := common.WithLogger(context.Background(), logger)
cr := &containerReference{ cr := &containerReference{
input: &NewContainerInput{ input: &NewContainerInput{
NetworkMode: "bridge", NetworkMode: "bridge",
Options: "--volume /host/tools:/opt/hostedtoolcache", RunnerOptions: "--volume /host/tools:/opt/hostedtoolcache",
}, },
} }
@@ -936,6 +843,28 @@ func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) {
assert.Empty(t, hostConf.Mounts) assert.Empty(t, hostConf.Mounts)
} }
func TestMergeContainerConfigsWarnsOnlyAboutOptionsThatWereGiven(t *testing.T) {
warnings := func(runnerOptions, workflowOptions string) int {
_, hook := mergeOptions(t, runnerOptions, workflowOptions, false)
return len(hook.AllEntries())
}
assert.Zero(t, warnings("--volume /host/tools:/opt/hostedtoolcache", ""))
assert.Zero(t, warnings("", "--shm-size 1g"))
assert.Equal(t, 1, warnings("--network host", ""))
}
func TestMergeContainerConfigsKeepsNetworkAliasesFromOptions(t *testing.T) {
logger, _ := test.NewNullLogger()
cr := &containerReference{input: &NewContainerInput{
NetworkMode: "job-network", NetworkAliases: []string{"redis"}, WorkflowOptions: "--network-alias redis-primary",
}}
_, _, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
assert.Equal(t, []string{"redis", "redis-primary"}, cr.input.NetworkAliases)
}
// A dead daemon must fail the job, not panic through logrus and not silently // A dead daemon must fail the job, not panic through logrus and not silently
// drop the requested platform. // drop the requested platform.
func TestSupportsContainerImagePlatformDaemonError(t *testing.T) { func TestSupportsContainerImagePlatformDaemonError(t *testing.T) {
-138
View File
@@ -1,138 +0,0 @@
// Copyright 2024 The Gitea Authors. All rights reserved.
// Copyright 2024 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"fmt"
"os"
"path/filepath"
"strings"
log "github.com/sirupsen/logrus"
)
var CommonSocketLocations = []string{
"/var/run/docker.sock",
"/run/podman/podman.sock",
"$HOME/.colima/docker.sock",
"$XDG_RUNTIME_DIR/docker.sock",
"$XDG_RUNTIME_DIR/podman/podman.sock",
`\\.\pipe\docker_engine`,
"$HOME/.docker/run/docker.sock",
}
// returns socket URI or false if not found any
func socketLocation() (string, bool) {
if dockerHost, exists := os.LookupEnv("DOCKER_HOST"); exists {
return dockerHost, true
}
for _, p := range CommonSocketLocations {
if _, err := os.Lstat(os.ExpandEnv(p)); err == nil {
if strings.HasPrefix(p, `\\.\`) {
return "npipe://" + filepath.ToSlash(os.ExpandEnv(p)), true
}
return "unix://" + filepath.ToSlash(os.ExpandEnv(p)), true
}
}
return "", false
}
// This function, `isDockerHostURI`, takes a string argument `daemonPath`. It checks if the
// `daemonPath` is a valid Docker host URI. It does this by checking if the scheme of the URI (the
// part before "://") contains only alphabetic characters. If it does, the function returns true,
// indicating that the `daemonPath` is a Docker host URI. If it doesn't, or if the "://" delimiter
// is not found in the `daemonPath`, the function returns false.
func isDockerHostURI(daemonPath string) bool {
if before, _, ok := strings.Cut(daemonPath, "://"); ok {
scheme := before
if strings.IndexFunc(scheme, func(r rune) bool {
return (r < 'a' || r > 'z') && (r < 'A' || r > 'Z')
}) == -1 {
return true
}
}
return false
}
type SocketAndHost struct {
Socket string
Host string
}
func GetSocketAndHost(containerSocket string) (SocketAndHost, error) {
log.Debugf("Handling container host and socket")
// Prefer DOCKER_HOST, don't override it
dockerHost, hasDockerHost := socketLocation()
socketHost := SocketAndHost{Socket: containerSocket, Host: dockerHost}
// ** socketHost.Socket cases **
// Case 1: User does _not_ want to mount a daemon socket (passes a dash)
// Case 2: User passes a filepath to the socket; is that even valid?
// Case 3: User passes a valid socket; do nothing
// Case 4: User omitted the flag; set a sane default
// ** DOCKER_HOST cases **
// Case A: DOCKER_HOST is set; use it, i.e. do nothing
// Case B: DOCKER_HOST is empty; use sane defaults
// Set host for sanity's sake, when the socket isn't useful
if !hasDockerHost && (socketHost.Socket == "-" || !isDockerHostURI(socketHost.Socket) || socketHost.Socket == "") {
// Cases: 1B, 2B, 4B
socket, found := socketLocation()
socketHost.Host = socket
hasDockerHost = found
}
// A - (dash) in socketHost.Socket means don't mount, preserve this value
// otherwise if socketHost.Socket is a filepath don't use it as socket
// Exit early if we're in an invalid state (e.g. when no DOCKER_HOST and user supplied "-", a dash or omitted)
if !hasDockerHost && socketHost.Socket != "" && !isDockerHostURI(socketHost.Socket) {
// Cases: 1B, 2B
// Should we early-exit here, since there is no host nor socket to talk to?
return SocketAndHost{}, fmt.Errorf("DOCKER_HOST was not set, couldn't be found in the usual locations, and the container daemon socket ('%s') is invalid", socketHost.Socket)
}
// Default to DOCKER_HOST if set
if socketHost.Socket == "" && hasDockerHost {
// Cases: 4A
log.Debugf("Defaulting container socket to DOCKER_HOST")
socketHost.Socket = socketHost.Host
}
// Set sane default socket location if user omitted it
if socketHost.Socket == "" {
// Cases: 4B
socket, _ := socketLocation()
// socket is empty if it isn't found, so assignment here is at worst a no-op
log.Debugf("Defaulting container socket to default '%s'", socket)
socketHost.Socket = socket
}
// Exit if both the DOCKER_HOST and socket are fulfilled
if hasDockerHost {
// Cases: 1A, 2A, 3A, 4A
if !isDockerHostURI(socketHost.Socket) {
// Cases: 1A, 2A
log.Debugf("DOCKER_HOST is set, but socket is invalid '%s'", socketHost.Socket)
}
return socketHost, nil
}
// Set a sane DOCKER_HOST default if we can
if isDockerHostURI(socketHost.Socket) {
// Cases: 3B
log.Debugf("Setting DOCKER_HOST to container socket '%s'", socketHost.Socket)
socketHost.Host = socketHost.Socket
// Both DOCKER_HOST and container socket are valid; short-circuit exit
return socketHost, nil
}
// Here there is no DOCKER_HOST _and_ the supplied container socket is not a valid URI (either invalid or a file path)
// Cases: 2B <- but is already handled at the top
// I.e. this path should never be taken
return SocketAndHost{}, fmt.Errorf("no DOCKER_HOST and an invalid container socket '%s'", socketHost.Socket)
}
-167
View File
@@ -1,167 +0,0 @@
// Copyright 2024 The Gitea Authors. All rights reserved.
// Copyright 2024 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"os"
"testing"
log "github.com/sirupsen/logrus"
assert "github.com/stretchr/testify/assert"
)
func init() {
log.SetLevel(log.DebugLevel)
}
var originalCommonSocketLocations = CommonSocketLocations
func isolateSocketEnv(t *testing.T) {
t.Helper()
t.Cleanup(func() { CommonSocketLocations = originalCommonSocketLocations })
if host, ok := os.LookupEnv("DOCKER_HOST"); ok {
t.Setenv("DOCKER_HOST", host)
} else {
t.Cleanup(func() { os.Unsetenv("DOCKER_HOST") })
}
}
func TestGetSocketAndHostWithSocket(t *testing.T) {
// Arrange
isolateSocketEnv(t)
dockerHost := "unix:///my/docker/host.sock"
socketURI := "/path/to/my.socket"
t.Setenv("DOCKER_HOST", dockerHost)
// Act
ret, err := GetSocketAndHost(socketURI)
// Assert
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, SocketAndHost{socketURI, dockerHost}, ret)
}
func TestGetSocketAndHostNoSocket(t *testing.T) {
// Arrange
dockerHost := "unix:///my/docker/host.sock"
t.Setenv("DOCKER_HOST", dockerHost)
// Act
ret, err := GetSocketAndHost("")
// Assert
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, SocketAndHost{dockerHost, dockerHost}, ret)
}
func TestGetSocketAndHostOnlySocket(t *testing.T) {
// Arrange
isolateSocketEnv(t)
socketURI := "/path/to/my.socket"
os.Unsetenv("DOCKER_HOST")
defaultSocket, defaultSocketFound := socketLocation()
// Act
ret, err := GetSocketAndHost(socketURI)
// Assert
assert.NoError(t, err, "Expected no error from GetSocketAndHost") //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, defaultSocketFound, "Expected to find default socket")
assert.Equal(t, socketURI, ret.Socket, "Expected socket to match common location")
assert.Equal(t, defaultSocket, ret.Host, "Expected ret.Host to match default socket location")
}
func TestGetSocketAndHostDontMount(t *testing.T) {
// Arrange
isolateSocketEnv(t)
dockerHost := "unix:///my/docker/host.sock"
t.Setenv("DOCKER_HOST", dockerHost)
// Act
ret, err := GetSocketAndHost("-")
// Assert
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, SocketAndHost{"-", dockerHost}, ret)
}
func TestGetSocketAndHostNoHostNoSocket(t *testing.T) {
// Arrange
isolateSocketEnv(t)
os.Unsetenv("DOCKER_HOST")
defaultSocket, found := socketLocation()
// Act
ret, err := GetSocketAndHost("")
// Assert
assert.True(t, found, "Expected a default socket to be found")
assert.NoError(t, err, "Expected no error from GetSocketAndHost") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, SocketAndHost{defaultSocket, defaultSocket}, ret, "Expected to match default socket location")
}
// Catch
// > Your code breaks setting DOCKER_HOST if shouldMount is false.
// > This happens if neither DOCKER_HOST nor --container-daemon-socket has a value, but socketLocation() returns a URI
func TestGetSocketAndHostNoHostNoSocketDefaultLocation(t *testing.T) {
// Arrange
isolateSocketEnv(t)
mySocketFile, tmpErr := os.CreateTemp(t.TempDir(), "act-*.sock")
mySocket := mySocketFile.Name()
unixSocket := "unix://" + mySocket
defer os.RemoveAll(mySocket)
assert.NoError(t, tmpErr) //nolint:testifylint // pre-existing issue from nektos/act
os.Unsetenv("DOCKER_HOST")
CommonSocketLocations = []string{mySocket}
defaultSocket, found := socketLocation()
// Act
ret, err := GetSocketAndHost("")
// Assert
assert.Equal(t, unixSocket, defaultSocket, "Expected default socket to match common socket location")
assert.True(t, found, "Expected default socket to be found")
assert.NoError(t, err, "Expected no error from GetSocketAndHost") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, SocketAndHost{unixSocket, unixSocket}, ret, "Expected to match default socket location")
}
func TestGetSocketAndHostNoHostInvalidSocket(t *testing.T) {
// Arrange
isolateSocketEnv(t)
os.Unsetenv("DOCKER_HOST")
mySocket := "/my/socket/path.sock"
CommonSocketLocations = []string{"/unusual", "/socket", "/location"}
defaultSocket, found := socketLocation()
// Act
ret, err := GetSocketAndHost(mySocket)
// Assert
assert.False(t, found, "Expected no default socket to be found")
assert.Equal(t, "", defaultSocket, "Expected no default socket to be found") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, SocketAndHost{}, ret, "Expected to match default socket location")
assert.Error(t, err, "Expected an error in invalid state")
}
func TestGetSocketAndHostOnlySocketValidButUnusualLocation(t *testing.T) {
// Arrange
isolateSocketEnv(t)
socketURI := "unix:///path/to/my.socket"
CommonSocketLocations = []string{"/unusual", "/location"}
os.Unsetenv("DOCKER_HOST")
defaultSocket, found := socketLocation()
// Act
ret, err := GetSocketAndHost(socketURI)
// Assert
// Default socket locations
assert.Equal(t, "", defaultSocket, "Expect default socket location to be empty") //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, found, "Expected no default socket to be found")
// Sane default
assert.NoError(t, err, "Expect no error from GetSocketAndHost") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, socketURI, ret.Host, "Expect host to default to unusual socket")
}
+12
View File
@@ -29,6 +29,18 @@ func RemoveImage(ctx context.Context, imageName string, force, pruneChildren boo
return false, errors.New("Unsupported Operation") return false, errors.New("Unsupported Operation")
} }
func RemoveDockerJobResources(_ context.Context, _ string) error {
return nil
}
func NewDockerProxy(_ context.Context, _ string) *DockerProxy {
return nil
}
func StartDockerProxy(daemonSocket, dir, job string) (*DockerProxy, error) {
return nil, errors.New("Unsupported Operation")
}
// NewDockerBuildExecutor function to create a run executor for the container // NewDockerBuildExecutor function to create a run executor for the container
func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor { func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
+15 -60
View File
@@ -26,6 +26,7 @@ import (
"gitea.com/gitea/runner/act/lookpath" "gitea.com/gitea/runner/act/lookpath"
"gitea.com/gitea/runner/internal/pkg/process" "gitea.com/gitea/runner/internal/pkg/process"
"github.com/creack/pty"
"github.com/go-git/go-billy/v5/helper/polyfill" "github.com/go-git/go-billy/v5/helper/polyfill"
"github.com/go-git/go-billy/v5/osfs" "github.com/go-git/go-billy/v5/osfs"
"github.com/go-git/go-git/v5/plumbing/format/gitignore" "github.com/go-git/go-git/v5/plumbing/format/gitignore"
@@ -71,12 +72,6 @@ func (e *HostEnvironment) Create(_, _ []string) common.Executor {
} }
} }
func (e *HostEnvironment) ConnectToNetwork(name string) common.Executor {
return func(ctx context.Context) error {
return nil
}
}
func (e *HostEnvironment) Close() common.Executor { func (e *HostEnvironment) Close() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
return nil return nil
@@ -97,34 +92,7 @@ func (e *HostEnvironment) Copy(destPath string, files ...*FileEntry) common.Exec
} }
} }
func (e *HostEnvironment) CopyTarStream(ctx context.Context, destPath string, tarStream io.Reader) error { func (e *HostEnvironment) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
if err := os.RemoveAll(destPath); err != nil {
return err
}
tr := tar.NewReader(tarStream)
cp := &filecollector.CopyCollector{
DstDir: destPath,
}
for {
ti, err := tr.Next()
if errors.Is(err, io.EOF) {
return nil
} else if err != nil {
return err
}
if ti.FileInfo().IsDir() {
continue
}
if ctx.Err() != nil {
return errors.New("CopyTarStream has been cancelled")
}
if err := cp.WriteFile(ti.Name, ti.FileInfo(), ti.Linkname, tr); err != nil {
return err
}
}
}
func (e *HostEnvironment) CopyDir(destPath, srcPath string, useGitIgnore bool) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
srcPrefix := filepath.Dir(srcPath) srcPrefix := filepath.Dir(srcPath)
@@ -142,10 +110,10 @@ func (e *HostEnvironment) CopyDir(destPath, srcPath string, useGitIgnore bool) c
ignorer = gitignore.NewMatcher(ps) ignorer = gitignore.NewMatcher(ps)
} }
fc := &filecollector.FileCollector{ fc := &filecollector.FileCollector{
Fs: &filecollector.DefaultFs{}, Ignorer: ignorer,
Ignorer: ignorer, SrcPath: srcPath,
SrcPath: srcPath, SrcPrefix: srcPrefix,
SrcPrefix: srcPrefix, SkipGitDir: skipGitDir,
Handler: &filecollector.CopyCollector{ Handler: &filecollector.CopyCollector{
DstDir: destPath, DstDir: destPath,
}, },
@@ -180,7 +148,6 @@ func (e *HostEnvironment) GetContainerArchive(ctx context.Context, srcPath strin
srcPrefix += string(filepath.Separator) srcPrefix += string(filepath.Separator)
} }
fc := &filecollector.FileCollector{ fc := &filecollector.FileCollector{
Fs: &filecollector.DefaultFs{},
SrcPath: srcPath, SrcPath: srcPath,
SrcPrefix: srcPrefix, SrcPrefix: srcPrefix,
Handler: tc, Handler: tc,
@@ -246,24 +213,8 @@ func (w *ptyWriter) Write(buf []byte) (int, error) {
return w.Out.Write(buf) return w.Out.Write(buf)
} }
type localEnv struct {
env map[string]string
}
func (l *localEnv) Getenv(name string) string {
if runtime.GOOS == "windows" {
for k, v := range l.env {
if strings.EqualFold(name, k) {
return v
}
}
return ""
}
return l.env[name]
}
func lookupPathHost(cmd string, env map[string]string, writer io.Writer) (string, error) { func lookupPathHost(cmd string, env map[string]string, writer io.Writer) (string, error) {
f, err := lookpath.LookPath2(cmd, &localEnv{env: env}) f, err := lookpath.LookPath2(cmd, env)
if err != nil { if err != nil {
err := "Cannot find: " + cmd + " in PATH" err := "Cannot find: " + cmd + " in PATH"
if _, _err := writer.Write([]byte(err + "\n")); _err != nil { if _, _err := writer.Write([]byte(err + "\n")); _err != nil {
@@ -275,7 +226,7 @@ func lookupPathHost(cmd string, env map[string]string, writer io.Writer) (string
} }
func setupPty(cmd *exec.Cmd, cmdline string) (*os.File, *os.File, error) { func setupPty(cmd *exec.Cmd, cmdline string) (*os.File, *os.File, error) {
ppty, tty, err := openPty() ppty, tty, err := pty.Open()
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
} }
@@ -401,8 +352,7 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
} }
err = cmd.Wait() err = cmd.Wait()
if err != nil { if err != nil {
var exitErr *exec.ExitError if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
if errors.As(err, &exitErr) {
return ExitCodeError(exitErr.ExitCode()) return ExitCodeError(exitErr.ExitCode())
} }
return err return err
@@ -680,8 +630,11 @@ func (*HostEnvironment) JoinPathVariable(paths ...string) string {
func goArchToActionArch(arch string) string { func goArchToActionArch(arch string) string {
archMapper := map[string]string{ archMapper := map[string]string{
"x86_64": "X64", "x86_64": "X64",
"amd64": "X64",
"386": "X86", "386": "X86",
"arm": "ARM",
"aarch64": "ARM64", "aarch64": "ARM64",
"arm64": "ARM64",
} }
if arch, ok := archMapper[arch]; ok { if arch, ok := archMapper[arch]; ok {
return arch return arch
@@ -691,7 +644,9 @@ func goArchToActionArch(arch string) string {
func goOsToActionOs(os string) string { func goOsToActionOs(os string) string {
osMapper := map[string]string{ osMapper := map[string]string{
"darwin": "macOS", "linux": "Linux",
"darwin": "macOS",
"windows": "Windows",
} }
if os, ok := osMapper[os]; ok { if os, ok := osMapper[os]; ok {
return os return os
+28 -1
View File
@@ -27,6 +27,11 @@ import (
// Type assert HostEnvironment implements ExecutionsEnvironment // Type assert HostEnvironment implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &HostEnvironment{} var _ ExecutionsEnvironment = &HostEnvironment{}
func TestActionPlatformNames(t *testing.T) {
assert.Equal(t, []string{"Linux", "macOS", "Windows"}, []string{goOsToActionOs("linux"), goOsToActionOs("darwin"), goOsToActionOs("windows")})
assert.Equal(t, []string{"X86", "X64", "ARM", "ARM64"}, []string{goArchToActionArch("386"), goArchToActionArch("amd64"), goArchToActionArch("arm"), goArchToActionArch("arm64")})
}
func TestCopyDir(t *testing.T) { func TestCopyDir(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
ctx := context.Background() ctx := context.Background()
@@ -42,10 +47,32 @@ func TestCopyDir(t *testing.T) {
_ = os.MkdirAll(e.TmpDir, 0o700) _ = os.MkdirAll(e.TmpDir, 0o700)
_ = os.MkdirAll(e.ToolCache, 0o700) _ = os.MkdirAll(e.ToolCache, 0o700)
_ = os.MkdirAll(e.ActPath, 0o700) _ = os.MkdirAll(e.ActPath, 0o700)
err := e.CopyDir(e.Workdir, e.Path, true)(ctx) err := e.CopyDir(e.Workdir, e.Path, true, false)(ctx)
assert.NoError(t, err) assert.NoError(t, err)
} }
func TestCopyDirSkipGitDir(t *testing.T) {
for name, skipGitDir := range map[string]bool{"kept for a workspace copy": false, "skipped for an action copy": true} {
t.Run(name, func(t *testing.T) {
src := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.MkdirAll(filepath.Join(src, ".git", "objects"), 0o700))
require.NoError(t, os.WriteFile(filepath.Join(src, ".git", "objects", "pack"), []byte("x"), 0o600))
require.NoError(t, os.WriteFile(filepath.Join(src, "action.yml"), []byte("name: x\n"), 0o600))
dest := filepath.Join(t.TempDir(), "dest")
e := &HostEnvironment{StdOut: os.Stdout}
require.NoError(t, e.CopyDir(dest, src+string(filepath.Separator), false, skipGitDir)(context.Background()))
assert.FileExists(t, filepath.Join(dest, "action.yml"))
if skipGitDir {
assert.NoDirExists(t, filepath.Join(dest, ".git"))
} else {
assert.FileExists(t, filepath.Join(dest, ".git", "objects", "pack"))
}
})
}
}
func TestGetContainerArchive(t *testing.T) { func TestGetContainerArchive(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
ctx := context.Background() ctx := context.Background()
+7 -9
View File
@@ -46,10 +46,11 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
} }
singleLineEnv := strings.Index(line, "=") singleLineEnv := strings.Index(line, "=")
multiLineEnv := strings.Index(line, "<<") multiLineEnv := strings.Index(line, "<<")
if singleLineEnv != -1 && (multiLineEnv == -1 || singleLineEnv < multiLineEnv) { switch {
case singleLineEnv != -1 && (multiLineEnv == -1 || singleLineEnv < multiLineEnv):
localEnv[line[:singleLineEnv]] = line[singleLineEnv+1:] localEnv[line[:singleLineEnv]] = line[singleLineEnv+1:]
} else if multiLineEnv != -1 { case multiLineEnv != -1:
multiLineEnvContent := "" var multiLineEnvContent []string
multiLineEnvDelimiter := line[multiLineEnv+2:] multiLineEnvDelimiter := line[multiLineEnv+2:]
delimiterFound := false delimiterFound := false
for s.Scan() { for s.Scan() {
@@ -58,10 +59,7 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
delimiterFound = true delimiterFound = true
break break
} }
if multiLineEnvContent != "" { multiLineEnvContent = append(multiLineEnvContent, content)
multiLineEnvContent += "\n"
}
multiLineEnvContent += content
} }
if err := s.Err(); err != nil { if err := s.Err(); err != nil {
return fmt.Errorf("reading env file: %w", err) return fmt.Errorf("reading env file: %w", err)
@@ -69,8 +67,8 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
if !delimiterFound { if !delimiterFound {
return fmt.Errorf("invalid format delimiter '%v' not found before end of file", multiLineEnvDelimiter) return fmt.Errorf("invalid format delimiter '%v' not found before end of file", multiLineEnvDelimiter)
} }
localEnv[line[:multiLineEnv]] = multiLineEnvContent localEnv[line[:multiLineEnv]] = strings.Join(multiLineEnvContent, "\n")
} else { default:
return fmt.Errorf("invalid format '%v', expected a line with '=' or '<<'", line) return fmt.Errorf("invalid format '%v', expected a line with '=' or '<<'", line)
} }
} }
+5
View File
@@ -86,6 +86,11 @@ func TestParseEnvFileMultiLineKeepsBlankLines(t *testing.T) {
env := map[string]string{} env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(context.Background())) require.NoError(t, parseEnvFile(e, envPath, &env)(context.Background()))
assert.Equal(t, "line1\n\nline2", env["FOO"]) assert.Equal(t, "line1\n\nline2", env["FOO"])
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n\nline2\nEOF\n"), 0o600))
env = map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(context.Background()))
assert.Equal(t, "\nline2", env["FOO"])
} }
func TestParseEnvFileUTF8BOM(t *testing.T) { func TestParseEnvFileUTF8BOM(t *testing.T) {
-17
View File
@@ -1,17 +0,0 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build (!windows && !plan9 && !openbsd) || (!windows && !plan9 && !mips64)
package container
import (
"os"
"github.com/creack/pty"
)
func openPty() (*os.File, *os.File, error) {
return pty.Open()
}
-14
View File
@@ -1,14 +0,0 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"errors"
"os"
)
func openPty() (*os.File, *os.File, error) {
return nil, nil, errors.New("Unsupported")
}
-14
View File
@@ -1,14 +0,0 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"errors"
"os"
)
func openPty() (*os.File, *os.File, error) {
return nil, nil, errors.New("Unsupported")
}
-14
View File
@@ -1,14 +0,0 @@
// Copyright 2022 The Gitea Authors. All rights reserved.
// Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package container
import (
"errors"
"os"
)
func openPty() (*os.File, *os.File, error) {
return nil, nil, errors.New("Unsupported")
}
+22 -53
View File
@@ -94,58 +94,29 @@ func (cc *CopyCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string
} }
type FileCollector struct { type FileCollector struct {
Ignorer gitignore.Matcher Ignorer gitignore.Matcher
SrcPath string SrcPath string
SrcPrefix string SrcPrefix string
Fs Fs SkipGitDir bool
Handler Handler Handler Handler
} }
type Fs interface { func openGitIndex(path string) (*index.Index, error) {
Walk(root string, fn filepath.WalkFunc) error repo, err := git.PlainOpen(path)
OpenGitIndex(path string) (*index.Index, error)
Open(path string) (io.ReadCloser, error)
Readlink(path string) (string, error)
}
type DefaultFs struct{}
func (*DefaultFs) Walk(root string, fn filepath.WalkFunc) error {
return filepath.Walk(root, fn)
}
func (*DefaultFs) OpenGitIndex(path string) (*index.Index, error) {
r, err := git.PlainOpen(path)
if err != nil { if err != nil {
return nil, err return nil, err
} }
i, err := r.Storer.Index() return repo.Storer.Index()
if err != nil {
return nil, err
}
return i, nil
}
func (*DefaultFs) Open(path string) (io.ReadCloser, error) {
return os.Open(path)
}
func (*DefaultFs) Readlink(path string) (string, error) {
return os.Readlink(path)
} }
func (fc *FileCollector) CollectFiles(ctx context.Context, submodulePath []string) filepath.WalkFunc { func (fc *FileCollector) CollectFiles(ctx context.Context, submodulePath []string) filepath.WalkFunc {
i, _ := fc.Fs.OpenGitIndex(path.Join(fc.SrcPath, path.Join(submodulePath...))) i, _ := openGitIndex(path.Join(fc.SrcPath, path.Join(submodulePath...)))
return func(file string, fi os.FileInfo, err error) error { return func(file string, fi os.FileInfo, err error) error {
if err != nil { if err != nil {
return err return err
} }
if ctx != nil { if ctx != nil && ctx.Err() != nil {
select { return errors.New("copy cancelled")
case <-ctx.Done():
return errors.New("copy cancelled")
default:
}
} }
sansPrefix := strings.TrimPrefix(file, fc.SrcPrefix) sansPrefix := strings.TrimPrefix(file, fc.SrcPrefix)
@@ -154,6 +125,12 @@ func (fc *FileCollector) CollectFiles(ctx context.Context, submodulePath []strin
if fi.IsDir() && len(split) > 0 && split[len(split)-1] == "." { if fi.IsDir() && len(split) > 0 && split[len(split)-1] == "." {
return nil return nil
} }
if fc.SkipGitDir && split[len(split)-1] == ".git" {
if fi.IsDir() {
return filepath.SkipDir
}
return nil
}
var entry *index.Entry var entry *index.Entry
if i != nil { if i != nil {
entry, err = i.Entry(strings.Join(split[len(submodulePath):], "/")) entry, err = i.Entry(strings.Join(split[len(submodulePath):], "/"))
@@ -175,7 +152,7 @@ func (fc *FileCollector) CollectFiles(ctx context.Context, submodulePath []strin
} }
} }
if err == nil && entry.Mode == filemode.Submodule { if err == nil && entry.Mode == filemode.Submodule {
err = fc.Fs.Walk(file, fc.CollectFiles(ctx, split)) err = filepath.Walk(file, fc.CollectFiles(ctx, split))
if err != nil { if err != nil {
return err return err
} }
@@ -185,7 +162,7 @@ func (fc *FileCollector) CollectFiles(ctx context.Context, submodulePath []strin
// return on non-regular files (thanks to [kumo](https://medium.com/@komuw/just-like-you-did-fbdd7df829d3) for this suggested update) // return on non-regular files (thanks to [kumo](https://medium.com/@komuw/just-like-you-did-fbdd7df829d3) for this suggested update)
if fi.Mode()&os.ModeSymlink == os.ModeSymlink { if fi.Mode()&os.ModeSymlink == os.ModeSymlink {
linkName, err := fc.Fs.Readlink(file) linkName, err := os.Readlink(file)
if err != nil { if err != nil {
return fmt.Errorf("unable to readlink '%s': %w", file, err) return fmt.Errorf("unable to readlink '%s': %w", file, err)
} }
@@ -195,23 +172,15 @@ func (fc *FileCollector) CollectFiles(ctx context.Context, submodulePath []strin
} }
// open file // open file
f, err := fc.Fs.Open(file) f, err := os.Open(file)
if err != nil { if err != nil {
return err return err
} }
defer f.Close() defer f.Close()
if ctx != nil { if ctx != nil {
// make io.Copy cancellable by closing the file stop := context.AfterFunc(ctx, func() { _ = f.Close() })
cpctx, cpfinish := context.WithCancel(ctx) defer stop()
defer cpfinish()
go func() {
select {
case <-cpctx.Done():
case <-ctx.Done():
f.Close()
}
}()
} }
return fc.Handler.WriteFile(path, fi, "", f) return fc.Handler.WriteFile(path, fi, "", f)
+46 -177
View File
@@ -6,6 +6,7 @@ package filecollector
import ( import (
"archive/tar" "archive/tar"
"bytes"
"context" "context"
"io" "io"
"os" "os"
@@ -13,110 +14,41 @@ import (
"runtime" "runtime"
"strings" "strings"
"testing" "testing"
"time"
"github.com/go-git/go-billy/v5"
"github.com/go-git/go-billy/v5/memfs"
git "github.com/go-git/go-git/v5" git "github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing/cache"
"github.com/go-git/go-git/v5/plumbing/format/gitignore" "github.com/go-git/go-git/v5/plumbing/format/gitignore"
"github.com/go-git/go-git/v5/plumbing/format/index"
"github.com/go-git/go-git/v5/storage/filesystem"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
type memoryFs struct {
billy.Filesystem
}
func (mfs *memoryFs) walk(root string, fn filepath.WalkFunc) error {
dir, err := mfs.ReadDir(root)
if err != nil {
return err
}
for i := range dir {
filename := filepath.Join(root, dir[i].Name())
err = fn(filename, dir[i], nil)
if dir[i].IsDir() {
if err == filepath.SkipDir {
err = nil
} else if err := mfs.walk(filename, fn); err != nil {
return err
}
}
if err != nil {
return err
}
}
return nil
}
func (mfs *memoryFs) Walk(root string, fn filepath.WalkFunc) error {
stat, err := mfs.Lstat(root)
if err != nil {
return err
}
err = fn(strings.Join([]string{root, "."}, string(filepath.Separator)), stat, nil)
if err != nil {
return err
}
return mfs.walk(root, fn)
}
func (mfs *memoryFs) OpenGitIndex(path string) (*index.Index, error) {
f, _ := mfs.Filesystem.Chroot(filepath.Join(path, ".git")) //nolint:staticcheck // pre-existing issue from nektos/act
storage := filesystem.NewStorage(f, cache.NewObjectLRUDefault())
i, err := storage.Index()
if err != nil {
return nil, err
}
return i, nil
}
func (mfs *memoryFs) Open(path string) (io.ReadCloser, error) {
return mfs.Filesystem.Open(path)
}
func (mfs *memoryFs) Readlink(path string) (string, error) {
return mfs.Filesystem.Readlink(path)
}
func TestIgnoredTrackedfile(t *testing.T) { func TestIgnoredTrackedfile(t *testing.T) {
fs := memfs.New() repoDir := filepath.Join(t.TempDir(), "mygitrepo")
_ = fs.MkdirAll("mygitrepo/.git", 0o777) repo, err := git.PlainInit(repoDir, false)
dotgit, _ := fs.Chroot("mygitrepo/.git") require.NoError(t, err)
worktree, _ := fs.Chroot("mygitrepo") require.NoError(t, os.WriteFile(filepath.Join(repoDir, ".gitignore"), []byte(".*\n"), 0o644))
repo, _ := git.Init(filesystem.NewStorage(dotgit, cache.NewObjectLRUDefault()), worktree) require.NoError(t, os.WriteFile(filepath.Join(repoDir, ".env"), []byte("test=val1\n"), 0o644))
f, _ := worktree.Create(".gitignore") worktree, err := repo.Worktree()
_, _ = f.Write([]byte(".*\n")) require.NoError(t, err)
f.Close() _, err = worktree.Add(".gitignore")
// This file shouldn't be in the tar require.NoError(t, err)
f, _ = worktree.Create(".env")
_, _ = f.Write([]byte("test=val1\n"))
f.Close()
w, _ := repo.Worktree()
// .gitignore is in the tar after adding it to the index
_, _ = w.Add(".gitignore")
tmpTar, _ := fs.Create("temp.tar") var archive bytes.Buffer
tw := tar.NewWriter(tmpTar) tw := tar.NewWriter(&archive)
ps, _ := gitignore.ReadPatterns(worktree, []string{}) patterns, err := gitignore.ReadPatterns(worktree.Filesystem, nil)
ignorer := gitignore.NewMatcher(ps) require.NoError(t, err)
ignorer := gitignore.NewMatcher(patterns)
fc := &FileCollector{ fc := &FileCollector{
Fs: &memoryFs{Filesystem: fs},
Ignorer: ignorer, Ignorer: ignorer,
SrcPath: "mygitrepo", SrcPath: repoDir,
SrcPrefix: "mygitrepo" + string(filepath.Separator), SrcPrefix: repoDir + string(filepath.Separator),
Handler: &TarCollector{ Handler: &TarCollector{
TarWriter: tw, TarWriter: tw,
}, },
} }
err := fc.Fs.Walk("mygitrepo", fc.CollectFiles(context.Background(), []string{})) err = filepath.Walk(repoDir, fc.CollectFiles(context.Background(), nil))
assert.NoError(t, err, "successfully collect files") //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err, "successfully collect files")
tw.Close() require.NoError(t, tw.Close())
_, _ = tmpTar.Seek(0, io.SeekStart) tr := tar.NewReader(&archive)
tr := tar.NewReader(tmpTar)
h, err := tr.Next() h, err := tr.Next()
assert.NoError(t, err, "tar must not be empty") //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err, "tar must not be empty") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, ".gitignore", h.Name) assert.Equal(t, ".gitignore", h.Name)
@@ -125,47 +57,32 @@ func TestIgnoredTrackedfile(t *testing.T) {
} }
func TestSymlinks(t *testing.T) { func TestSymlinks(t *testing.T) {
fs := memfs.New() if runtime.GOOS == "windows" {
_ = fs.MkdirAll("mygitrepo/.git", 0o777) t.Skip("creating symlinks requires elevated privileges on Windows")
dotgit, _ := fs.Chroot("mygitrepo/.git") }
worktree, _ := fs.Chroot("mygitrepo") repoDir := filepath.Join(t.TempDir(), "mygitrepo")
repo, _ := git.Init(filesystem.NewStorage(dotgit, cache.NewObjectLRUDefault()), worktree) repo, err := git.PlainInit(repoDir, false)
// This file shouldn't be in the tar require.NoError(t, err)
f, err := worktree.Create(".env") require.NoError(t, os.WriteFile(filepath.Join(repoDir, ".env"), []byte("test=val1\n"), 0o644))
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act require.NoError(t, os.Symlink(".env", filepath.Join(repoDir, "test.env")))
_, err = f.Write([]byte("test=val1\n")) worktree, err := repo.Worktree()
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act require.NoError(t, err)
f.Close() _, err = worktree.Add("test.env")
err = worktree.Symlink(".env", "test.env") require.NoError(t, err)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
w, err := repo.Worktree() var archive bytes.Buffer
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act tw := tar.NewWriter(&archive)
// .gitignore is in the tar after adding it to the index
_, err = w.Add(".env")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
_, err = w.Add("test.env")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
tmpTar, _ := fs.Create("temp.tar")
tw := tar.NewWriter(tmpTar)
ps, _ := gitignore.ReadPatterns(worktree, []string{})
ignorer := gitignore.NewMatcher(ps)
fc := &FileCollector{ fc := &FileCollector{
Fs: &memoryFs{Filesystem: fs}, SrcPath: repoDir,
Ignorer: ignorer, SrcPrefix: repoDir + string(filepath.Separator),
SrcPath: "mygitrepo",
SrcPrefix: "mygitrepo" + string(filepath.Separator),
Handler: &TarCollector{ Handler: &TarCollector{
TarWriter: tw, TarWriter: tw,
}, },
} }
err = fc.Fs.Walk("mygitrepo", fc.CollectFiles(context.Background(), []string{})) err = filepath.Walk(repoDir, fc.CollectFiles(context.Background(), nil))
assert.NoError(t, err, "successfully collect files") //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err, "successfully collect files")
tw.Close() require.NoError(t, tw.Close())
_, _ = tmpTar.Seek(0, io.SeekStart) tr := tar.NewReader(&archive)
tr := tar.NewReader(tmpTar)
h, err := tr.Next() h, err := tr.Next()
files := map[string]tar.Header{} files := map[string]tar.Header{}
for err == nil { for err == nil {
@@ -223,62 +140,14 @@ func TestCopyCollectorWriteFileOverwritesFileWithSymlink(t *testing.T) {
assert.Equal(t, "target", resolved) assert.Equal(t, "target", resolved)
} }
func TestDefaultFsOpenReadlinkAndWalk(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("creating symlinks requires elevated privileges on Windows")
}
root := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(root, "file.txt"), []byte("content"), 0o644))
require.NoError(t, os.Symlink("file.txt", filepath.Join(root, "link.txt")))
fsys := &DefaultFs{}
var walked []string
require.NoError(t, fsys.Walk(root, func(path string, info os.FileInfo, err error) error {
require.NoError(t, err)
walked = append(walked, info.Name())
return nil
}))
require.Contains(t, walked, "file.txt")
require.Contains(t, walked, "link.txt")
file, err := fsys.Open(filepath.Join(root, "file.txt"))
require.NoError(t, err)
data, err := io.ReadAll(file)
require.NoError(t, err)
require.NoError(t, file.Close())
require.Equal(t, "content", string(data))
link, err := fsys.Readlink(filepath.Join(root, "link.txt"))
require.NoError(t, err)
require.Equal(t, "file.txt", link)
}
func TestFileCollectorCancellationAndWalkError(t *testing.T) { func TestFileCollectorCancellationAndWalkError(t *testing.T) {
fc := &FileCollector{Fs: &memoryFs{Filesystem: memfs.New()}} ctx, cancel := context.WithCancel(t.Context())
walk := fc.CollectFiles(cancelledContext(t), nil) cancel()
walk := (&FileCollector{}).CollectFiles(ctx, nil)
err := walk("file", fakeFileInfo{name: "file"}, nil) err := walk("file", nil, nil)
require.EqualError(t, err, "copy cancelled") require.EqualError(t, err, "copy cancelled")
err = walk("file", fakeFileInfo{name: "file"}, os.ErrPermission) err = walk("file", nil, os.ErrPermission)
require.ErrorIs(t, err, os.ErrPermission) require.ErrorIs(t, err, os.ErrPermission)
} }
func cancelledContext(t *testing.T) context.Context {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
cancel()
return ctx
}
type fakeFileInfo struct {
name string
}
func (f fakeFileInfo) Name() string { return f.name }
func (f fakeFileInfo) Size() int64 { return 0 }
func (f fakeFileInfo) Mode() os.FileMode { return 0o644 }
func (f fakeFileInfo) ModTime() time.Time { return time.Time{} }
func (f fakeFileInfo) IsDir() bool { return false }
func (f fakeFileInfo) Sys() any { return nil }
+13 -32
View File
@@ -25,32 +25,9 @@ var (
findGithubRepo = git.FindGithubRepo findGithubRepo = git.FindGithubRepo
) )
func withDefaultBranch(ctx context.Context, b string, event map[string]any) map[string]any {
repoI, ok := event["repository"]
if !ok {
repoI = make(map[string]any)
}
repo, ok := repoI.(map[string]any)
if !ok {
common.Logger(ctx).Warnf("unable to set default branch to %v", b)
return event
}
// if the branch is already there return with no changes
if _, ok = repo["default_branch"]; ok {
return event
}
repo["default_branch"] = b
event["repository"] = repo
return event
}
// SetRef resolves the ref of the context from its event payload, falling back // SetRef resolves the ref of the context from its event payload, falling back
// to the ref checked out in repoPath. // to the ref checked out in repoPath.
func SetRef(ctx context.Context, ghc *model.GithubContext, defaultBranch, repoPath string) { func SetRef(ctx context.Context, ghc *model.GithubContext, repoPath string) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
// https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows // https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows
@@ -82,11 +59,15 @@ func SetRef(ctx context.Context, ghc *model.GithubContext, defaultBranch, repoPa
ghc.Ref = ref ghc.Ref = ref
} }
// set the branch in the event data repository, exists := ghc.Event["repository"]
if defaultBranch != "" { if !exists {
ghc.Event = withDefaultBranch(ctx, defaultBranch, ghc.Event) repository = map[string]any{}
} else { }
ghc.Event = withDefaultBranch(ctx, "master", ghc.Event) if repository, ok := repository.(map[string]any); !ok {
logger.Warn("unable to set default branch to master")
} else if _, exists := repository["default_branch"]; !exists {
repository["default_branch"] = "master"
ghc.Event["repository"] = repository
} }
if ghc.Ref == "" { if ghc.Ref == "" {
@@ -125,11 +106,11 @@ func SetSha(ctx context.Context, ghc *model.GithubContext, repoPath string) {
// SetRepositoryAndOwner resolves the repository of the context from the git // SetRepositoryAndOwner resolves the repository of the context from the git
// remote in repoPath when it is not set yet, and derives its owner. // remote in repoPath when it is not set yet, and derives its owner.
func SetRepositoryAndOwner(ctx context.Context, ghc *model.GithubContext, githubInstance, remoteName, repoPath string) { func SetRepositoryAndOwner(ctx context.Context, ghc *model.GithubContext, githubInstance, repoPath string) {
if ghc.Repository == "" { if ghc.Repository == "" {
repo, err := findGithubRepo(ctx, repoPath, githubInstance, remoteName) repo, err := findGithubRepo(ctx, repoPath, githubInstance)
if err != nil { if err != nil {
common.Logger(ctx).Warningf("unable to get git repo (githubInstance: %v; remoteName: %v, repoPath: %v): %v", githubInstance, remoteName, repoPath, err) common.Logger(ctx).Warningf("unable to get git repo (githubInstance: %v, repoPath: %v): %v", githubInstance, repoPath, err)
return return
} }
ghc.Repository = repo ghc.Repository = repo
+2 -2
View File
@@ -104,7 +104,7 @@ func TestSetRef(t *testing.T) {
Event: table.event, Event: table.event,
} }
SetRef(context.Background(), ghc, "main", "/some/dir") SetRef(context.Background(), ghc, "/some/dir")
ghc.SetRefTypeAndName() ghc.SetRefTypeAndName()
assert.Equal(t, table.ref, ghc.Ref) assert.Equal(t, table.ref, ghc.Ref)
@@ -122,7 +122,7 @@ func TestSetRef(t *testing.T) {
Event: map[string]any{}, Event: map[string]any{},
} }
SetRef(context.Background(), ghc, "", "/some/dir") SetRef(context.Background(), ghc, "/some/dir")
assert.Equal(t, "refs/heads/master", ghc.Ref) assert.Equal(t, "refs/heads/master", ghc.Ref)
}) })
+14 -2
View File
@@ -4,6 +4,18 @@
package lookpath package lookpath
type Env interface { import (
Getenv(name string) string "runtime"
"strings"
)
func getenv(env map[string]string, name string) string {
if runtime.GOOS == "windows" {
for key, value := range env {
if strings.EqualFold(name, key) {
return value
}
}
}
return env[name]
} }
+1 -1
View File
@@ -18,7 +18,7 @@ var ErrNotFound = errors.New("executable file not found in $PATH")
// directories named by the PATH environment variable. // directories named by the PATH environment variable.
// If file contains a slash, it is tried directly and the PATH is not consulted. // If file contains a slash, it is tried directly and the PATH is not consulted.
// The result may be an absolute path or a path relative to the current directory. // The result may be an absolute path or a path relative to the current directory.
func LookPath2(file string, lenv Env) (string, error) { func LookPath2(file string, _ map[string]string) (string, error) {
// Wasm can not execute processes, so act as if there are no executables at all. // Wasm can not execute processes, so act as if there are no executables at all.
return "", &Error{file, ErrNotFound} return "", &Error{file, ErrNotFound}
} }
+2 -2
View File
@@ -32,7 +32,7 @@ func findExecutable(file string) error {
// If file begins with "/", "#", "./", or "../", it is tried // If file begins with "/", "#", "./", or "../", it is tried
// directly and the path is not consulted. // directly and the path is not consulted.
// The result may be an absolute path or a path relative to the current directory. // The result may be an absolute path or a path relative to the current directory.
func LookPath2(file string, lenv Env) (string, error) { func LookPath2(file string, env map[string]string) (string, error) {
// skip the path lookup for these prefixes // skip the path lookup for these prefixes
skip := []string{"/", "#", "./", "../"} skip := []string{"/", "#", "./", "../"}
@@ -46,7 +46,7 @@ func LookPath2(file string, lenv Env) (string, error) {
} }
} }
path := lenv.Getenv("path") path := getenv(env, "path")
for _, dir := range filepath.SplitList(path) { for _, dir := range filepath.SplitList(path) {
path := filepath.Join(dir, file) path := filepath.Join(dir, file)
if err := findExecutable(path); err == nil { if err := findExecutable(path); err == nil {
+2 -2
View File
@@ -33,7 +33,7 @@ func findExecutable(file string) error {
// directories named by the PATH environment variable. // directories named by the PATH environment variable.
// If file contains a slash, it is tried directly and the PATH is not consulted. // If file contains a slash, it is tried directly and the PATH is not consulted.
// The result may be an absolute path or a path relative to the current directory. // The result may be an absolute path or a path relative to the current directory.
func LookPath2(file string, lenv Env) (string, error) { func LookPath2(file string, env map[string]string) (string, error) {
// NOTE(rsc): I wish we could use the Plan 9 behavior here // NOTE(rsc): I wish we could use the Plan 9 behavior here
// (only bypass the path if file begins with / or ./ or ../) // (only bypass the path if file begins with / or ./ or ../)
// but that would not match all the Unix shells. // but that would not match all the Unix shells.
@@ -45,7 +45,7 @@ func LookPath2(file string, lenv Env) (string, error) {
} }
return "", &Error{file, err} return "", &Error{file, err}
} }
path := lenv.Getenv("PATH") path := getenv(env, "PATH")
for _, dir := range filepath.SplitList(path) { for _, dir := range filepath.SplitList(path) {
if dir == "" { if dir == "" {
// Unix shell semantics: path element "" means "." // Unix shell semantics: path element "" means "."
+4 -10
View File
@@ -13,12 +13,6 @@ import (
"testing" "testing"
) )
type testEnv map[string]string
func (e testEnv) Getenv(name string) string {
return e[name]
}
func TestLookPath2SearchesPathAndEmptyElement(t *testing.T) { func TestLookPath2SearchesPathAndEmptyElement(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
exe := filepath.Join(dir, "tool") exe := filepath.Join(dir, "tool")
@@ -26,7 +20,7 @@ func TestLookPath2SearchesPathAndEmptyElement(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
got, err := LookPath2("tool", testEnv{"PATH": string(filepath.ListSeparator) + dir}) got, err := LookPath2("tool", map[string]string{"PATH": string(filepath.ListSeparator) + dir})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -42,7 +36,7 @@ func TestLookPath2DirectPathDoesNotSearchPath(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
got, err := LookPath2(exe, testEnv{"PATH": ""}) got, err := LookPath2(exe, map[string]string{"PATH": ""})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -58,7 +52,7 @@ func TestLookPath2ReportsPermissionAndNotFound(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
_, err := LookPath2(file, testEnv{"PATH": dir}) _, err := LookPath2(file, map[string]string{"PATH": dir})
var pathErr *Error var pathErr *Error
if !errors.As(err, &pathErr) || !errors.Is(pathErr.Err, fs.ErrPermission) { if !errors.As(err, &pathErr) || !errors.Is(pathErr.Err, fs.ErrPermission) {
t.Fatalf("LookPath2(non-executable) error = %v, want fs.ErrPermission wrapped in *Error", err) t.Fatalf("LookPath2(non-executable) error = %v, want fs.ErrPermission wrapped in *Error", err)
@@ -67,7 +61,7 @@ func TestLookPath2ReportsPermissionAndNotFound(t *testing.T) {
t.Fatalf("Error() = %q, want %q", pathErr.Error(), fs.ErrPermission.Error()) t.Fatalf("Error() = %q, want %q", pathErr.Error(), fs.ErrPermission.Error())
} }
_, err = LookPath2("missing", testEnv{"PATH": dir}) _, err = LookPath2("missing", map[string]string{"PATH": dir})
if !errors.As(err, &pathErr) || !errors.Is(pathErr.Err, ErrNotFound) { if !errors.As(err, &pathErr) || !errors.Is(pathErr.Err, ErrNotFound) {
t.Fatalf("LookPath2(missing) error = %v, want ErrNotFound wrapped in *Error", err) t.Fatalf("LookPath2(missing) error = %v, want ErrNotFound wrapped in *Error", err)
} }
+3 -3
View File
@@ -58,9 +58,9 @@ func findExecutable(file string, exts []string) (string, error) {
// LookPath also uses PATHEXT environment variable to match // LookPath also uses PATHEXT environment variable to match
// a suitable candidate. // a suitable candidate.
// The result may be an absolute path or a path relative to the current directory. // The result may be an absolute path or a path relative to the current directory.
func LookPath2(file string, lenv Env) (string, error) { func LookPath2(file string, env map[string]string) (string, error) {
var exts []string var exts []string
x := lenv.Getenv(`PATHEXT`) x := getenv(env, `PATHEXT`)
if x != "" { if x != "" {
for e := range strings.SplitSeq(strings.ToLower(x), `;`) { for e := range strings.SplitSeq(strings.ToLower(x), `;`) {
if e == "" { if e == "" {
@@ -85,7 +85,7 @@ func LookPath2(file string, lenv Env) (string, error) {
if f, err := findExecutable(filepath.Join(".", file), exts); err == nil { if f, err := findExecutable(filepath.Join(".", file), exts); err == nil {
return f, nil return f, nil
} }
path := lenv.Getenv("path") path := getenv(env, "path")
for _, dir := range filepath.SplitList(path) { for _, dir := range filepath.SplitList(path) {
if f, err := findExecutable(filepath.Join(dir, file), exts); err == nil { if f, err := findExecutable(filepath.Join(dir, file), exts); err == nil {
return f, nil return f, nil
+114 -131
View File
@@ -32,7 +32,7 @@ type actionStep interface {
step step
getActionModel() *model.Action getActionModel() *model.Action
getCompositeRunContext(context.Context) *RunContext getCompositeRunContext(context.Context) (*RunContext, error)
getCompositeSteps() *compositeSteps getCompositeSteps() *compositeSteps
} }
@@ -124,21 +124,9 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
defer closer.Close() defer closer.Close()
action, err := model.ReadAction(reader) action, err := model.ReadAction(reader)
// For Gitea, reduce log noise
// logger.Debugf("Read action %v from '%s'", action, "Unknown")
return action, err return action, err
} }
// cachedActionTar returns the action's tree from the action cache, which only a remote action
// has an entry in.
func cachedActionTar(ctx context.Context, step actionStep, name, includePrefix string) (io.ReadCloser, error) {
remote, ok := step.(*stepActionRemote)
if !ok {
return nil, fmt.Errorf("action %q is a remote action but runs as %T", name, step)
}
return step.getRunContext().Config.ActionCache.GetTarArchive(ctx, remote.cacheDir, remote.resolvedSha, includePrefix)
}
func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actionPath, containerActionDir string) error { func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actionPath, containerActionDir string) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
rc := step.getRunContext() rc := step.getRunContext()
@@ -148,23 +136,13 @@ func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actio
return nil return nil
} }
var containerActionDirCopy string containerActionDirCopy := strings.TrimSuffix(containerActionDir, actionPath)
containerActionDirCopy = strings.TrimSuffix(containerActionDir, actionPath)
logger.Debug(containerActionDirCopy) logger.Debug(containerActionDirCopy)
if !strings.HasSuffix(containerActionDirCopy, `/`) { if !strings.HasSuffix(containerActionDirCopy, `/`) {
containerActionDirCopy += `/` containerActionDirCopy += `/`
} }
if rc.Config != nil && rc.Config.ActionCache != nil {
ta, err := cachedActionTar(ctx, step, stepModel.Uses, "")
if err != nil {
return err
}
defer ta.Close()
return rc.JobContainer.CopyTarStream(ctx, containerActionDirCopy, ta)
}
defer git.AcquireCloneLock(actionDir)() defer git.AcquireCloneLock(actionDir)()
if !rc.Config.NoActionPatch { if !rc.Config.NoActionPatch {
@@ -176,7 +154,7 @@ func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actio
return err return err
} }
return rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", rc.Config.UseGitIgnore)(ctx) return rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", rc.Config.UseGitIgnore, true)(ctx)
} }
func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor { func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor {
@@ -191,11 +169,10 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
} }
action := step.getActionModel() action := step.getActionModel()
// For Gitea, reduce log noise
// logger.Debugf("About to run action %v", action)
err := setupActionEnv(ctx, step, remoteAction) rc.withGithubEnv(ctx, step.getGithubContext(ctx), *step.getEnv())
if err != nil { populateEnvsFromSavedState(step.getEnv(), step, rc)
if err := populateEnvsFromInput(ctx, step.getEnv(), action, rc); err != nil {
return err return err
} }
@@ -210,12 +187,12 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil { if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil {
return err return err
} }
containerArgs := []string{"node", path.Join(containerActionDir, action.Runs.Main)} containerArgs := nodeActionCommand(path.Join(containerActionDir, action.Runs.Main))
logger.Debugf("executing remote job container: %s", containerArgs) logger.Debugf("executing remote job container: %s", containerArgs)
rc.ApplyExtraPath(ctx, step.getEnv()) rc.ApplyExtraPath(ctx, step.getEnv())
return rc.execJobContainer(containerArgs, *step.getEnv(), "", "")(ctx) return rc.JobContainer.Exec(containerArgs, *step.getEnv(), "", "")(ctx)
case x.IsDocker(): case x.IsDocker():
location := actionLocation location := actionLocation
if remoteAction == nil { if remoteAction == nil {
@@ -240,11 +217,11 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
execArgs := []string{filepath.Join(containerActionDir, execFileName)} execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.execJobContainer(buildArgs, *step.getEnv(), "", containerActionDir), rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.execJobContainer(execArgs, *step.getEnv(), "", ""), rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
)(ctx) )(ctx)
default: default:
return fmt.Errorf("The runs.using key must be one of: %v, got %s", []string{ return fmt.Errorf("the runs.using key must be one of: %v, got %s", []string{
model.ActionRunsUsingDocker, model.ActionRunsUsingDocker,
model.ActionRunsUsingNode12, model.ActionRunsUsingNode12,
model.ActionRunsUsingNode16, model.ActionRunsUsingNode16,
@@ -257,18 +234,9 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
} }
} }
func setupActionEnv(ctx context.Context, step actionStep, _ *remoteAction) error { // /var/run is a symlink, so without the flag node's import.meta.url differs from argv[1], which ESM actions compare.
rc := step.getRunContext() func nodeActionCommand(script string) []string {
return []string{"node", "--preserve-symlinks-main", script}
// A few fields in the environment (e.g. GITHUB_ACTION_REPOSITORY)
// are dependent on the action. That means we can complete the
// setup only after resolving the whole action model and cloning
// the action
rc.withGithubEnv(ctx, step.getGithubContext(ctx), *step.getEnv())
populateEnvsFromSavedState(step.getEnv(), step, rc)
populateEnvsFromInput(ctx, step.getEnv(), step.getActionModel(), rc)
return nil
} }
// https://github.com/nektos/act/issues/228#issuecomment-629709055 // https://github.com/nektos/act/issues/228#issuecomment-629709055
@@ -364,12 +332,6 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
return err return err
} }
defer buildContext.Close() defer buildContext.Close()
} else if rc.Config.ActionCache != nil {
buildContext, err = cachedActionTar(ctx, step, actionName, contextDir)
if err != nil {
return err
}
defer buildContext.Close()
} }
prepImage = ContainerNewDockerBuildExecutor(container.NewDockerBuildExecutorInput{ prepImage = ContainerNewDockerBuildExecutor(container.NewDockerBuildExecutorInput{
ContextDir: contextDir, ContextDir: contextDir,
@@ -391,34 +353,48 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
logger.Debugf("image '%s' for architecture '%s' already exists", image, rc.Config.ContainerArchitecture) logger.Debugf("image '%s' for architecture '%s' already exists", image, rc.Config.ContainerArchitecture)
} }
} }
eval := rc.NewStepExpressionEvaluator(ctx, step) eval := rc.NewActionInputsExpressionEvaluator(ctx, step)
cmd, err := shellquote.Split(eval.Interpolate(ctx, step.getStepModel().With["args"])) args, err := eval.Interpolate(ctx, step.getStepModel().With["args"])
if err != nil {
return fmt.Errorf("unable to interpolate with.args: %w", err)
}
cmd, err := shellquote.Split(args)
if err != nil { if err != nil {
return err return err
} }
if len(cmd) == 0 { ee, err := evalDockerEnv(ctx, step, action)
cmd = action.Runs.Args if err != nil {
evalDockerArgs(ctx, step, action, &cmd) return err
}
if action.Runs.Args != nil {
// a fresh slice, the manifest is evaluated again for every stage
cmd = make([]string, len(action.Runs.Args))
for i, v := range action.Runs.Args {
if cmd[i], err = ee.Interpolate(ctx, v); err != nil {
return fmt.Errorf("unable to interpolate runs.args: %w", err)
}
}
} }
entrypoint, err := dockerEntrypoint(ctx, step, eval, stage) entrypoint, err := dockerEntrypoint(ctx, step, eval, stage)
if err != nil { if err != nil {
return err return err
} }
stepContainer := newStepContainer(ctx, step, image, cmd, entrypoint) stepContainer, err := newStepContainer(ctx, step, image, cmd, entrypoint, rc.Config.ContainerOptions)
if err != nil {
return err
}
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
prepImage, prepImage,
stepContainer.Pull(forcePull), stepContainer.Pull(forcePull),
stepContainer.Remove().IfBool(!rc.Config.ReuseContainers), stepContainer.Remove(),
stepContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop), stepContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop),
stepContainer.Start(true), stepContainer.Start(true),
).Finally(
stepContainer.Remove().IfBool(!rc.Config.ReuseContainers && !rc.Config.AutoRemove),
).Finally(stepContainer.Close())(ctx) ).Finally(stepContainer.Close())(ctx)
} }
// dockerEntrypoint returns the entrypoint the action's image runs with for the given // dockerEntrypoint returns the entrypoint the action's image runs with for the given
// stage. Only the main stage honours the `entrypoint` input. // stage. Only the main stage honours the `entrypoint` input.
func dockerEntrypoint(ctx context.Context, step actionStep, eval ExpressionEvaluator, stage stepStage) ([]string, error) { func dockerEntrypoint(ctx context.Context, step actionStep, eval *expressionEvaluator, stage stepStage) ([]string, error) {
runs := step.getActionModel().Runs runs := step.getActionModel().Runs
var entrypoint string var entrypoint string
@@ -428,10 +404,16 @@ func dockerEntrypoint(ctx context.Context, step actionStep, eval ExpressionEvalu
case stepStagePost: case stepStagePost:
entrypoint = runs.PostEntrypoint entrypoint = runs.PostEntrypoint
default: default:
if fields := strings.Fields(eval.Interpolate(ctx, step.getStepModel().With["entrypoint"])); len(fields) > 0 {
return fields, nil
}
entrypoint = runs.Entrypoint entrypoint = runs.Entrypoint
if entrypoint == "" {
withEntrypoint, err := eval.Interpolate(ctx, step.getStepModel().With["entrypoint"])
if err != nil {
return nil, fmt.Errorf("unable to interpolate with.entrypoint: %w", err)
}
if fields := strings.Fields(withEntrypoint); len(fields) > 0 {
return fields, nil
}
}
} }
if entrypoint == "" { if entrypoint == "" {
@@ -440,47 +422,43 @@ func dockerEntrypoint(ctx context.Context, step actionStep, eval ExpressionEvalu
return shellquote.Split(entrypoint) return shellquote.Split(entrypoint)
} }
func evalDockerArgs(ctx context.Context, step step, action *model.Action, cmd *[]string) { // evalDockerEnv returns an evaluator bound to the environment it installed.
func evalDockerEnv(ctx context.Context, step step, action *model.Action) (*expressionEvaluator, error) {
rc := step.getRunContext() rc := step.getRunContext()
stepModel := step.getStepModel() stepModel := step.getStepModel()
var err error
inputs := make(map[string]string) inputs := make(map[string]string)
eval := rc.NewExpressionEvaluator(ctx) eval := rc.NewExpressionEvaluator(ctx)
// Set Defaults // Set Defaults
for k, input := range action.Inputs { for k, input := range action.Inputs {
inputs[k] = eval.Interpolate(ctx, input.Default) if inputs[k], err = eval.Interpolate(ctx, input.Default); err != nil {
return nil, fmt.Errorf("unable to interpolate the default of input %s: %w", k, err)
}
} }
if stepModel.With != nil { for k, v := range stepModel.With {
for k, v := range stepModel.With { if inputs[k], err = eval.Interpolate(ctx, v); err != nil {
inputs[k] = eval.Interpolate(ctx, v) return nil, fmt.Errorf("unable to interpolate with.%s: %w", k, err)
} }
} }
mergeIntoMap(step, step.getEnv(), inputs) mergeIntoMap(step, step.getEnv(), inputs)
stepEE := rc.NewStepExpressionEvaluator(ctx, step) env := make(map[string]string, len(action.Runs.Env)+len(*step.getEnv()))
for i, v := range *cmd { mergeIntoMap(step, &env, action.Runs.Env, *step.getEnv())
(*cmd)[i] = stepEE.Interpolate(ctx, v) *step.getEnv() = env
}
mergeIntoMap(step, step.getEnv(), action.Runs.Env)
ee := rc.NewStepExpressionEvaluator(ctx, step) ee := rc.NewActionInputsExpressionEvaluator(ctx, step)
for k, v := range *step.getEnv() { for k, v := range *step.getEnv() {
(*step.getEnv())[k] = ee.Interpolate(ctx, v) if (*step.getEnv())[k], err = ee.Interpolate(ctx, v); err != nil {
return nil, fmt.Errorf("unable to interpolate env %s: %w", k, err)
}
} }
return ee, nil
} }
func newStepContainer(ctx context.Context, step step, image string, cmd, entrypoint []string) container.Container { func newStepContainer(ctx context.Context, step step, image string, cmd, entrypoint []string, runnerOptions string) (container.Container, error) {
rc := step.getRunContext() rc := step.getRunContext()
stepModel := step.getStepModel() logWriter := rc.commandLogWriter(ctx)
rawLogger := common.Logger(ctx).WithField("raw_output", true)
logWriter := common.NewLineWriter(rc.commandHandler(ctx), func(s string) bool {
if rc.Config.LogOutput {
rawLogger.Infof("%s", s)
} else {
rawLogger.Debugf("%s", s)
}
return true
})
envList := make([]string, 0) envList := make([]string, 0)
for k, v := range *step.getEnv() { for k, v := range *step.getEnv() {
envList = append(envList, fmt.Sprintf("%s=%s", k, v)) envList = append(envList, fmt.Sprintf("%s=%s", k, v))
@@ -488,32 +466,34 @@ func newStepContainer(ctx context.Context, step step, image string, cmd, entrypo
envList = append(envList, rc.runnerEnv(ctx)...) envList = append(envList, rc.runnerEnv(ctx)...)
binds, mounts := rc.GetBindsAndMounts() binds, mounts, err := rc.GetBindsAndMounts()
if err != nil {
return nil, err
}
networkMode := "container:" + rc.jobContainerName() networkMode := "container:" + rc.jobContainerName()
if rc.IsHostEnv(ctx) { if rc.IsHostEnv() {
networkMode = "default" networkMode = "default"
} }
stepContainer := ContainerNewContainer(&container.NewContainerInput{ return ContainerNewContainer(&container.NewContainerInput{
Cmd: cmd, Cmd: cmd,
Entrypoint: entrypoint, Entrypoint: entrypoint,
WorkingDir: rc.JobContainer.ToContainerPath(rc.Config.Workdir), WorkingDir: rc.JobContainer.ToContainerPath(rc.Config.Workdir),
Image: image, Image: image,
Name: createContainerName(rc.jobContainerName(), "STEP-"+stepModel.ID), Name: createContainerName(rc.jobContainerName(), "STEP-"+step.getStepModel().ID),
Env: envList, Env: envList,
Mounts: mounts, Mounts: mounts,
NetworkMode: networkMode, NetworkMode: networkMode,
Binds: binds, Binds: binds,
Stdout: logWriter, Stdout: logWriter,
Stderr: logWriter, Stderr: logWriter,
Privileged: rc.Config.Privileged, Privileged: rc.Config.Privileged,
UsernsMode: rc.Config.UsernsMode, UsernsMode: rc.Config.UsernsMode,
Platform: rc.Config.ContainerArchitecture, Platform: rc.Config.ContainerArchitecture,
Options: rc.Config.ContainerOptions, RunnerOptions: runnerOptions,
AutoRemove: rc.Config.AutoRemove, AutoRemove: true,
ValidVolumes: rc.validVolumes(), ValidVolumes: rc.validVolumes(),
AllocatePTY: rc.Config.AllocatePTY, AllocatePTY: rc.Config.AllocatePTY,
}) }), nil
return stepContainer
} }
func populateEnvsFromSavedState(env *map[string]string, step actionStep, rc *RunContext) { func populateEnvsFromSavedState(env *map[string]string, step actionStep, rc *RunContext) {
@@ -526,15 +506,19 @@ func populateEnvsFromSavedState(env *map[string]string, step actionStep, rc *Run
} }
} }
func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *model.Action, rc *RunContext) { func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *model.Action, rc *RunContext) error {
eval := rc.NewExpressionEvaluator(ctx) eval := rc.NewExpressionEvaluator(ctx)
for inputID, input := range action.Inputs { for inputID, input := range action.Inputs {
envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_") envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_")
envKey = "INPUT_" + envKey envKey = "INPUT_" + envKey
if _, ok := (*env)[envKey]; !ok { if _, ok := (*env)[envKey]; !ok {
(*env)[envKey] = eval.Interpolate(ctx, input.Default) var err error
if (*env)[envKey], err = eval.Interpolate(ctx, input.Default); err != nil {
return fmt.Errorf("unable to interpolate the default of input %s: %w", inputID, err)
}
} }
} }
return nil
} }
func getContainerActionPaths(step *model.Step, actionDir string, rc *RunContext) (string, string) { func getContainerActionPaths(step *model.Step, actionDir string, rc *RunContext) (string, string) {
@@ -634,31 +618,33 @@ func runPreStep(step actionStep) common.Executor {
actionDir, actionPath, _, containerActionDir := actionStagePaths(step) actionDir, actionPath, _, containerActionDir := actionStagePaths(step)
x := action.Runs.Using x := action.Runs.Using
if !x.IsComposite() {
// defaults in pre steps were missing, however provided inputs are available
if err := populateEnvsFromInput(ctx, step.getEnv(), action, rc); err != nil {
return err
}
}
switch { switch {
case x.IsNode(): case x.IsNode():
// defaults in pre steps were missing, however provided inputs are available
populateEnvsFromInput(ctx, step.getEnv(), action, rc)
if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil { if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil {
return err return err
} }
containerArgs := []string{"node", path.Join(containerActionDir, action.Runs.Pre)} containerArgs := nodeActionCommand(path.Join(containerActionDir, action.Runs.Pre))
logger.Debugf("executing remote job container: %s", containerArgs) logger.Debugf("executing remote job container: %s", containerArgs)
rc.ApplyExtraPath(ctx, step.getEnv()) rc.ApplyExtraPath(ctx, step.getEnv())
return rc.execJobContainer(containerArgs, *step.getEnv(), "", "")(ctx) return rc.JobContainer.Exec(containerArgs, *step.getEnv(), "", "")(ctx)
case x.IsDocker(): case x.IsDocker():
// defaults in pre steps were missing, however provided inputs are available
populateEnvsFromInput(ctx, step.getEnv(), action, rc)
return execDockerActionStage(ctx, step, stepStagePre) return execDockerActionStage(ctx, step, stepStagePre)
case x.IsComposite(): case x.IsComposite():
if step.getCompositeSteps() == nil { if step.getCompositeSteps() == nil {
step.getCompositeRunContext(ctx) if _, err := step.getCompositeRunContext(ctx); err != nil {
return err
}
} }
if steps := step.getCompositeSteps(); steps != nil && steps.pre != nil { if steps := step.getCompositeSteps(); steps != nil && steps.pre != nil {
@@ -667,9 +653,6 @@ func runPreStep(step actionStep) common.Executor {
return errors.New("missing steps in composite action") return errors.New("missing steps in composite action")
case x == model.ActionRunsUsingGo: case x == model.ActionRunsUsingGo:
// defaults in pre steps were missing, however provided inputs are available
populateEnvsFromInput(ctx, step.getEnv(), action, rc)
if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil { if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil {
return err return err
} }
@@ -681,8 +664,8 @@ func runPreStep(step actionStep) common.Executor {
execArgs := []string{filepath.Join(containerActionDir, execFileName)} execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.execJobContainer(buildArgs, *step.getEnv(), "", containerActionDir), rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.execJobContainer(execArgs, *step.getEnv(), "", ""), rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
)(ctx) )(ctx)
default: default:
return nil return nil
@@ -744,12 +727,12 @@ func runPostStep(step actionStep) common.Executor {
populateEnvsFromSavedState(step.getEnv(), step, rc) populateEnvsFromSavedState(step.getEnv(), step, rc)
containerArgs := []string{"node", path.Join(containerActionDir, action.Runs.Post)} containerArgs := nodeActionCommand(path.Join(containerActionDir, action.Runs.Post))
logger.Debugf("executing remote job container: %s", containerArgs) logger.Debugf("executing remote job container: %s", containerArgs)
rc.ApplyExtraPath(ctx, step.getEnv()) rc.ApplyExtraPath(ctx, step.getEnv())
return rc.execJobContainer(containerArgs, *step.getEnv(), "", "")(ctx) return rc.JobContainer.Exec(containerArgs, *step.getEnv(), "", "")(ctx)
case x.IsDocker(): case x.IsDocker():
populateEnvsFromSavedState(step.getEnv(), step, rc) populateEnvsFromSavedState(step.getEnv(), step, rc)
@@ -775,8 +758,8 @@ func runPostStep(step actionStep) common.Executor {
execArgs := []string{filepath.Join(containerActionDir, execFileName)} execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.execJobContainer(buildArgs, *step.getEnv(), "", containerActionDir), rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.execJobContainer(execArgs, *step.getEnv(), "", ""), rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
)(ctx) )(ctx)
default: default:
-156
View File
@@ -1,156 +0,0 @@
// Copyright 2023 The Gitea Authors. All rights reserved.
// Copyright 2023 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package runner
import (
"archive/tar"
"context"
"crypto/rand"
"encoding/hex"
"errors"
"io"
"io/fs"
"path"
"strings"
git "github.com/go-git/go-git/v5"
config "github.com/go-git/go-git/v5/config"
"github.com/go-git/go-git/v5/plumbing"
"github.com/go-git/go-git/v5/plumbing/object"
"github.com/go-git/go-git/v5/plumbing/transport"
"github.com/go-git/go-git/v5/plumbing/transport/http"
)
type ActionCache interface {
Fetch(ctx context.Context, cacheDir, url, ref, token string) (string, error)
GetTarArchive(ctx context.Context, cacheDir, sha, includePrefix string) (io.ReadCloser, error)
}
type GoGitActionCache struct {
Path string
}
func (c GoGitActionCache) Fetch(ctx context.Context, cacheDir, url, ref, token string) (string, error) {
gitPath := path.Join(c.Path, safeFilename(cacheDir)+".git")
gogitrepo, err := git.PlainInit(gitPath, true)
if errors.Is(err, git.ErrRepositoryAlreadyExists) {
gogitrepo, err = git.PlainOpen(gitPath)
}
if err != nil {
return "", err
}
tmpBranch := make([]byte, 12)
if _, err := rand.Read(tmpBranch); err != nil {
return "", err
}
branchName := hex.EncodeToString(tmpBranch)
var auth transport.AuthMethod
if token != "" {
auth = &http.BasicAuth{
Username: "token",
Password: token,
}
}
remote, err := gogitrepo.CreateRemoteAnonymous(&config.RemoteConfig{
Name: "anonymous",
URLs: []string{
url,
},
})
if err != nil {
return "", err
}
defer func() {
_ = gogitrepo.DeleteBranch(branchName)
}()
if err := remote.FetchContext(ctx, &git.FetchOptions{
RefSpecs: []config.RefSpec{
config.RefSpec(ref + ":" + branchName),
},
Auth: auth,
Force: true,
}); err != nil {
return "", err
}
hash, err := gogitrepo.ResolveRevision(plumbing.Revision(branchName))
if err != nil {
return "", err
}
return hash.String(), nil
}
func (c GoGitActionCache) GetTarArchive(ctx context.Context, cacheDir, sha, includePrefix string) (io.ReadCloser, error) {
gitPath := path.Join(c.Path, safeFilename(cacheDir)+".git")
gogitrepo, err := git.PlainOpen(gitPath)
if err != nil {
return nil, err
}
commit, err := gogitrepo.CommitObject(plumbing.NewHash(sha))
if err != nil {
return nil, err
}
files, err := commit.Files()
if err != nil {
return nil, err
}
rpipe, wpipe := io.Pipe()
// Interrupt io.Copy using ctx
ch := make(chan int, 1)
go func() {
select {
case <-ctx.Done():
wpipe.CloseWithError(ctx.Err())
case <-ch:
}
}()
go func() {
defer wpipe.Close()
defer close(ch)
tw := tar.NewWriter(wpipe)
cleanIncludePrefix := path.Clean(includePrefix)
wpipe.CloseWithError(files.ForEach(func(f *object.File) error {
if err := ctx.Err(); err != nil {
return err
}
name := f.Name
if strings.HasPrefix(name, cleanIncludePrefix+"/") {
name = name[len(cleanIncludePrefix)+1:]
} else if cleanIncludePrefix != "." && name != cleanIncludePrefix {
return nil
}
fmode, err := f.Mode.ToOSFileMode()
if err != nil {
return err
}
if fmode&fs.ModeSymlink == fs.ModeSymlink {
content, err := f.Contents()
if err != nil {
return err
}
return tw.WriteHeader(&tar.Header{
Name: name,
Mode: int64(fmode),
Linkname: content,
})
}
err = tw.WriteHeader(&tar.Header{
Name: name,
Mode: int64(fmode),
Size: f.Size,
})
if err != nil {
return err
}
reader, err := f.Reader()
if err != nil {
return err
}
_, err = io.Copy(tw, reader)
return err
}))
}()
return rpipe, err
}
-157
View File
@@ -1,157 +0,0 @@
// Copyright 2023 The Gitea Authors. All rights reserved.
// Copyright 2023 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package runner
import (
"archive/tar"
"bytes"
"context"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func runGit(t *testing.T, dir string, args ...string) {
t.Helper()
if dir != "" {
args = append([]string{"-C", dir}, args...)
}
cmd := exec.Command("git", args...)
// Fixed identity and host-config isolation so commits succeed offline regardless of the
// host's git config (mirrors gitCmd in act/common/git).
cmd.Env = append(os.Environ(),
"GIT_AUTHOR_NAME=test", "[email protected]",
"GIT_COMMITTER_NAME=test", "[email protected]",
"GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null",
)
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
}
// TestShortShaActionRejected verifies a `uses` ref that is a shortened commit SHA is rejected
// with a clear error. The action is resolved from a local repo (via DefaultActionInstance) so
// this runs offline.
func TestShortShaActionRejected(t *testing.T) {
// a local "remote" action repo at <root>/actions/hello-world-docker-action
actionRoot := t.TempDir()
repo := filepath.Join(actionRoot, "actions", "hello-world-docker-action")
require.NoError(t, os.MkdirAll(repo, 0o755))
runGit(t, "", "init", "--initial-branch=main", repo)
require.NoError(t, os.WriteFile(filepath.Join(repo, "action.yml"),
[]byte("name: hello\nruns:\n using: node24\n main: index.js\n"), 0o644))
runGit(t, repo, "add", ".")
runGit(t, repo, "commit", "-m", "initial")
out, err := exec.Command("git", "-C", repo, "rev-parse", "HEAD").Output()
require.NoError(t, err)
shortSha := strings.TrimSpace(string(out))[:7]
// a workflow that uses the action at the short SHA
wfDir := filepath.Join(t.TempDir(), "wf")
require.NoError(t, os.MkdirAll(wfDir, 0o755))
wf := fmt.Sprintf("on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/hello-world-docker-action@%s\n", shortSha)
require.NoError(t, os.WriteFile(filepath.Join(wfDir, "push.yml"), []byte(wf), 0o644))
runner, err := New(&Config{
Workdir: wfDir,
EventName: "push",
Platforms: map[string]string{"ubuntu-latest": baseImage},
GitHubInstance: "github.com",
DefaultActionInstance: actionRoot,
ContainerMaxLifetime: time.Hour,
})
require.NoError(t, err)
planner, err := model.NewWorkflowPlanner(wfDir, true)
require.NoError(t, err)
plan, err := planner.PlanEvent("push")
require.NoError(t, err)
err = runner.NewPlanExecutor(plan)(common.WithDryrun(context.Background(), true))
require.Error(t, err)
assert.Contains(t, err.Error(), "shortened version of a commit SHA")
}
func TestActionCache(t *testing.T) {
a := assert.New(t)
ctx := context.Background()
// Build a local bare repo with a `js` action dir so this runs offline (formerly cloned
// github.com/nektos/act-test-actions over the network). allowAnySHA1InWant lets the
// "Fetch Sha" case fetch a commit hash directly.
remoteDir := t.TempDir()
runGit(t, "", "init", "--bare", "--initial-branch=main", remoteDir)
runGit(t, remoteDir, "config", "uploadpack.allowAnySHA1InWant", "true")
workDir := t.TempDir()
runGit(t, "", "clone", remoteDir, workDir)
require.NoError(t, os.MkdirAll(filepath.Join(workDir, "js"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(workDir, "js", "action.yml"),
[]byte("name: js\nruns:\n using: node24\n main: index.js\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(workDir, "js", "index.js"),
[]byte("console.log('hello');\n"), 0o644))
runGit(t, workDir, "add", ".")
runGit(t, workDir, "commit", "-m", "initial")
runGit(t, workDir, "push", "-u", "origin", "main")
out, err := exec.Command("git", "-C", workDir, "rev-parse", "main").Output()
require.NoError(t, err)
fullSha := strings.TrimSpace(string(out))
cache := &GoGitActionCache{
Path: t.TempDir(),
}
cacheDir := "local/act-test-actions"
refs := []struct {
Name string
Ref string
}{
{Name: "Fetch Branch Name", Ref: "main"},
{Name: "Fetch Branch Name Absolutely", Ref: "refs/heads/main"},
{Name: "Fetch HEAD", Ref: "HEAD"},
{Name: "Fetch Sha", Ref: fullSha},
}
for _, c := range refs {
t.Run(c.Name, func(t *testing.T) {
sha, err := cache.Fetch(ctx, cacheDir, remoteDir, c.Ref, "")
if !a.NoError(err) || !a.NotEmpty(sha) { //nolint:testifylint // pre-existing issue from nektos/act
return
}
atar, err := cache.GetTarArchive(ctx, cacheDir, sha, "js")
// NotNil, not NotEmpty: atar is a live io.PipeReader whose producer goroutine is
// writing concurrently; NotEmpty deep-reflects over its internals and races.
if !a.NoError(err) || !a.NotNil(atar) { //nolint:testifylint // pre-existing issue from nektos/act
return
}
// GetTarArchive streams from a background goroutine walking the shared repo.
// Drain and close so it finishes before the next subtest fetches into the same
// repo; otherwise the lingering walk races with that fetch.
defer func() {
_, _ = io.Copy(io.Discard, atar)
_ = atar.Close()
}()
mytar := tar.NewReader(atar)
th, err := mytar.Next()
if !a.NoError(err) || !a.NotEqual(0, th.Size) { //nolint:testifylint // pre-existing issue from nektos/act
return
}
buf := &bytes.Buffer{}
// G110: Potential DoS vulnerability via decompression bomb (gosec)
_, err = io.Copy(buf, mytar)
a.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
str := buf.String()
a.NotEmpty(str)
})
}
}
+59 -53
View File
@@ -7,6 +7,7 @@ package runner
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"regexp" "regexp"
"slices" "slices"
"strconv" "strconv"
@@ -17,7 +18,7 @@ import (
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
) )
func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step actionStep) map[string]string { func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step actionStep) (map[string]string, error) {
env := make(map[string]string) env := make(map[string]string)
stepEnv := *step.getEnv() stepEnv := *step.getEnv()
for k, v := range stepEnv { for k, v := range stepEnv {
@@ -28,7 +29,7 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
} }
} }
ee := parent.NewStepExpressionEvaluator(ctx, step) ee := parent.NewActionInputsExpressionEvaluator(ctx, step)
for inputID, input := range step.getActionModel().Inputs { for inputID, input := range step.getActionModel().Inputs {
envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_") envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_")
@@ -36,55 +37,78 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
// lookup if key is defined in the step but the already // lookup if key is defined in the step but the already
// evaluated value from the environment // evaluated value from the environment
_, defined := step.getStepModel().With[inputID] defined := false
for key := range step.getStepModel().With {
if strings.EqualFold(key, inputID) {
defined = true
break
}
}
if value, ok := stepEnv[envKey]; defined && ok { if value, ok := stepEnv[envKey]; defined && ok {
env[envKey] = value env[envKey] = value
} else { } else {
// defaults could contain expressions // defaults could contain expressions
env[envKey] = ee.Interpolate(ctx, input.Default) var err error
if env[envKey], err = ee.Interpolate(ctx, input.Default); err != nil {
return nil, fmt.Errorf("unable to interpolate the default of input %s: %w", inputID, err)
}
} }
} }
gh := step.getGithubContext(ctx) gh := step.getGithubContext(ctx)
env["GITHUB_ACTION_REPOSITORY"] = gh.ActionRepository env["GITHUB_ACTION_REPOSITORY"] = gh.ActionRepository
env["GITHUB_ACTION_REF"] = gh.ActionRef env["GITHUB_ACTION_REF"] = gh.ActionRef
return env return env, nil
} }
func newCompositeRunContext(ctx context.Context, parent *RunContext, step actionStep, actionPath string) *RunContext { func (rc *RunContext) setCompositeActionEnv(env map[string]string) {
env := evaluateCompositeInputAndEnv(ctx, parent, step) rc.setActionEnv(env)
for key := range rc.Env {
if strings.HasPrefix(key, "INPUT_") {
delete(rc.Env, key)
}
}
}
func newCompositeRunContext(ctx context.Context, parent *RunContext, step actionStep, actionPath string) (*RunContext, error) {
env, err := evaluateCompositeInputAndEnv(ctx, parent, step)
if err != nil {
return nil, err
}
// run with the global config but without secrets // run with the global config but without secrets
configCopy := *(parent.Config) configCopy := *parent.Config
configCopy.Secrets = nil configCopy.Secrets = nil
// create a run context for the composite action to run in // create a run context for the composite action to run in
compositerc := &RunContext{ compositerc := &RunContext{
Name: parent.Name, Name: parent.Name,
JobName: parent.JobName, JobName: parent.JobName,
Matrix: parent.Matrix,
Run: &model.Run{ Run: &model.Run{
JobID: parent.Run.JobID, JobID: parent.Run.JobID,
Workflow: &model.Workflow{ Workflow: &model.Workflow{
Name: parent.Run.Workflow.Name, Name: parent.Run.Workflow.Name,
Jobs: map[string]*model.Job{ Jobs: map[string]*model.Job{
parent.Run.JobID: {}, parent.Run.JobID: {Strategy: parent.Run.Job().Strategy},
}, },
}, },
}, },
Config: &configCopy, Config: &configCopy,
StepResults: map[string]*model.StepResult{}, StepResults: map[string]*model.StepResult{},
JobContainer: parent.JobContainer, JobContainer: parent.JobContainer,
ActionPath: actionPath, ActionPath: actionPath,
Env: env, GlobalEnv: parent.GlobalEnv,
GlobalEnv: parent.GlobalEnv, Masks: parent.Masks,
Masks: parent.Masks, ExtraPath: parent.ExtraPath,
ExtraPath: parent.ExtraPath, Parent: parent,
Parent: parent, EventJSON: parent.EventJSON,
EventJSON: parent.EventJSON, platformImage: parent.platformImage,
} }
compositerc.setCompositeActionEnv(env)
compositerc.ExprEval = compositerc.NewExpressionEvaluator(ctx) compositerc.ExprEval = compositerc.NewExpressionEvaluator(ctx)
return compositerc return compositerc, nil
} }
// appendUniqueMasks appends the masks from src to dst, skipping any mask that // appendUniqueMasks appends the masks from src to dst, skipping any mask that
@@ -105,7 +129,10 @@ func execAsComposite(step actionStep) common.Executor {
action := step.getActionModel() action := step.getActionModel()
return func(ctx context.Context) error { return func(ctx context.Context) error {
compositeRC := step.getCompositeRunContext(ctx) compositeRC, err := step.getCompositeRunContext(ctx)
if err != nil {
return err
}
steps := step.getCompositeSteps() steps := step.getCompositeSteps()
@@ -115,14 +142,17 @@ func execAsComposite(step actionStep) common.Executor {
ctx = WithCompositeLogger(ctx, &compositeRC.Masks) ctx = WithCompositeLogger(ctx, &compositeRC.Masks)
err := steps.main(ctx) err = steps.main(ctx)
// Map outputs from composite RunContext to job RunContext // Map outputs from composite RunContext to job RunContext
eval := compositeRC.NewExpressionEvaluator(ctx) eval := compositeRC.NewExpressionEvaluator(ctx)
for outputName, output := range action.Outputs { for outputName, output := range action.Outputs {
rc.setOutput(ctx, map[string]string{ value, outputErr := eval.Interpolate(ctx, output.Value)
"name": outputName, if outputErr != nil {
}, eval.Interpolate(ctx, output.Value)) err = errors.Join(err, fmt.Errorf("unable to interpolate output %s: %w", outputName, outputErr))
continue
}
rc.setOutput(ctx, map[string]string{"name": outputName}, value)
} }
// compositeRC.Masks is seeded with rc.Masks (see newCompositeRunContext) // compositeRC.Masks is seeded with rc.Masks (see newCompositeRunContext)
@@ -131,7 +161,7 @@ func execAsComposite(step actionStep) common.Executor {
// repeated composite actions grow rc.Masks exponentially. // repeated composite actions grow rc.Masks exponentially.
rc.Masks = appendUniqueMasks(rc.Masks, compositeRC.Masks) rc.Masks = appendUniqueMasks(rc.Masks, compositeRC.Masks)
rc.ExtraPath = compositeRC.ExtraPath rc.ExtraPath = compositeRC.ExtraPath
// compositeRC.Env is dirty, contains INPUT_ and merged step env, only rely on compositeRC.GlobalEnv // Propagate GlobalEnv only, so composite inputs and step-local values do not escape.
mergeIntoMap := mergeIntoMapCaseSensitive mergeIntoMap := mergeIntoMapCaseSensitive
if rc.JobContainer.IsEnvironmentCaseInsensitive() { if rc.JobContainer.IsEnvironmentCaseInsensitive() {
mergeIntoMap = mergeIntoMapCaseInsensitive mergeIntoMap = mergeIntoMapCaseInsensitive
@@ -181,20 +211,7 @@ func (rc *RunContext) compositeExecutor(action *model.Action) *compositeSteps {
stepPre := rc.newCompositeCommandExecutor(step.pre()) stepPre := rc.newCompositeCommandExecutor(step.pre())
preSteps = append(preSteps, newCompositeStepLogExecutor(stepPre, stepID)) preSteps = append(preSteps, newCompositeStepLogExecutor(stepPre, stepID))
steps = append(steps, func(ctx context.Context) error { steps = append(steps, newCompositeStepLogExecutor(rc.newCompositeCommandExecutor(step.main()), stepID))
ctx = WithCompositeStepLogger(ctx, stepID)
logger := common.Logger(ctx)
err := rc.newCompositeCommandExecutor(step.main())(ctx)
if err != nil {
logger.Errorf("##[error]%s", EscapeCommandData(err.Error()))
common.SetJobError(ctx, err)
} else if ctx.Err() != nil {
logger.Errorf("##[error]%s", EscapeCommandData(ctx.Err().Error()))
common.SetJobError(ctx, ctx.Err())
}
return nil
})
// run the post executor in reverse order // run the post executor in reverse order
if postExecutor != nil { if postExecutor != nil {
@@ -207,6 +224,7 @@ func (rc *RunContext) compositeExecutor(action *model.Action) *compositeSteps {
} }
steps = append(steps, common.JobError) steps = append(steps, common.JobError)
preSteps = append(preSteps, common.JobError)
return &compositeSteps{ return &compositeSteps{
pre: func(ctx context.Context) error { pre: func(ctx context.Context) error {
return common.NewPipelineExecutor(preSteps...)(common.WithJobErrorContainer(ctx)) return common.NewPipelineExecutor(preSteps...)(common.WithJobErrorContainer(ctx))
@@ -222,19 +240,7 @@ func (rc *RunContext) newCompositeCommandExecutor(executor common.Executor) comm
return func(ctx context.Context) error { return func(ctx context.Context) error {
ctx = WithCompositeLogger(ctx, &rc.Masks) ctx = WithCompositeLogger(ctx, &rc.Masks)
// We need to inject a composite RunContext related command logWriter := rc.commandLogWriter(ctx)
// handler into the current running job container
// We need this, to support scoping commands to the composite action
// executing.
rawLogger := common.Logger(ctx).WithField("raw_output", true)
logWriter := common.NewLineWriter(rc.commandHandler(ctx), func(s string) bool {
if rc.Config.LogOutput {
rawLogger.Infof("%s", s)
} else {
rawLogger.Debugf("%s", s)
}
return true
})
oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter) oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter)
defer rc.JobContainer.ReplaceLogWriter(oldout, olderr) defer rc.JobContainer.ReplaceLogWriter(oldout, olderr)
+49
View File
@@ -6,9 +6,58 @@ package runner
import ( import (
"testing" "testing"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
) )
func TestCompositeActionParity(t *testing.T) {
t.Run("inherits contexts without leaking inputs", func(t *testing.T) {
ctx := t.Context()
strategy := &model.Strategy{MaxParallel: 3}
parent := &RunContext{
Config: &Config{},
Matrix: map[string]any{"os": "linux"},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Name: "workflow", Jobs: map[string]*model.Job{"job": {Strategy: strategy}}}},
JobContainer: &jobContainerMock{},
platformImage: "-self-hosted",
}
composite, err := newCompositeRunContext(ctx, parent, &stepActionRemote{
Step: &model.Step{With: map[string]string{"SHARED": "outer"}},
RunContext: parent,
action: &model.Action{Inputs: map[string]model.Input{"shared": {Default: "outer-default"}}},
env: map[string]string{"INPUT_SHARED": "outer"},
}, "/action")
require.NoError(t, err)
assert.Same(t, strategy, composite.Run.Job().Strategy)
assert.True(t, composite.IsHostEnv())
interpolated, err := composite.NewExpressionEvaluator(ctx).Interpolate(ctx,
"${{ matrix.os }}|${{ strategy.max-parallel }}|${{ inputs.shared }}")
require.NoError(t, err)
assert.Equal(t, "linux|3|outer", interpolated)
assert.NotContains(t, composite.Env, "INPUT_SHARED")
nestedEnv := composite.GetEnv()
require.NoError(t, populateEnvsFromInput(ctx, &nestedEnv, &model.Action{Inputs: map[string]model.Input{"shared": {Default: "inner-default"}}}, composite))
assert.Equal(t, "inner-default", nestedEnv["INPUT_SHARED"])
})
t.Run("propagates pre failures", func(t *testing.T) {
setCloneExecutor(t, func(git.NewGitCloneExecutorInput) common.Executor { return common.NewErrorExecutor(assert.AnError) })
rc := &RunContext{
Config: &Config{GitHubInstance: "github.com", ActionCacheDir: t.TempDir()},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{},
}
require.ErrorIs(t, rc.compositeExecutor(&model.Action{Runs: model.ActionRuns{Using: "composite", Steps: []model.Step{{ID: "nested", Uses: "org/action@v1"}}}}).pre(t.Context()), assert.AnError)
})
}
func TestAppendUniqueMasks(t *testing.T) { func TestAppendUniqueMasks(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
+102 -93
View File
@@ -8,6 +8,10 @@ import (
"context" "context"
"io" "io"
"io/fs" "io/fs"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -23,12 +27,10 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
type closerMock struct { type closerFunc func()
mock.Mock
}
func (m *closerMock) Close() error { func (close closerFunc) Close() error {
m.Called() close()
return nil return nil
} }
@@ -39,6 +41,15 @@ runs:
using: 'node16' using: 'node16'
main: 'main.js' main: 'main.js'
`, "\t", " ") `, "\t", " ")
yamlAction := &model.Action{
Name: "name",
Runs: model.ActionRuns{
Using: "node16",
Main: "main.js",
PreIf: "always()",
PostIf: "always()",
},
}
table := []struct { table := []struct {
name string name string
@@ -52,30 +63,14 @@ runs:
step: &model.Step{}, step: &model.Step{},
filename: "action.yml", filename: "action.yml",
fileContent: yaml, fileContent: yaml,
expected: &model.Action{ expected: yamlAction,
Name: "name",
Runs: model.ActionRuns{
Using: "node16",
Main: "main.js",
PreIf: "always()",
PostIf: "always()",
},
},
}, },
{ {
name: "readActionYaml", name: "readActionYaml",
step: &model.Step{}, step: &model.Step{},
filename: "action.yaml", filename: "action.yaml",
fileContent: yaml, fileContent: yaml,
expected: &model.Action{ expected: yamlAction,
Name: "name",
Runs: model.ActionRuns{
Using: "node16",
Main: "main.js",
PreIf: "always()",
PostIf: "always()",
},
},
}, },
{ {
name: "readDockerfile", name: "readDockerfile",
@@ -121,14 +116,14 @@ runs:
for _, tt := range table { for _, tt := range table {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
closerMock := &closerMock{} closed := false
readFile := func(filename string) (io.Reader, io.Closer, error) { readFile := func(filename string) (io.Reader, io.Closer, error) {
if tt.filename != filename { if tt.filename != filename {
return nil, nil, fs.ErrNotExist return nil, nil, fs.ErrNotExist
} }
return strings.NewReader(tt.fileContent), closerMock, nil return strings.NewReader(tt.fileContent), closerFunc(func() { closed = true }), nil
} }
writeFile := func(filename string, data []byte, perm fs.FileMode) error { writeFile := func(filename string, data []byte, perm fs.FileMode) error {
@@ -137,58 +132,16 @@ runs:
return nil return nil
} }
if tt.filename != "" {
closerMock.On("Close")
}
action, err := readActionImpl(context.Background(), tt.step, "actionDir", "actionPath", readFile, writeFile) action, err := readActionImpl(context.Background(), tt.step, "actionDir", "actionPath", readFile, writeFile)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, tt.expected, action) assert.Equal(t, tt.expected, action)
closerMock.AssertExpectations(t) assert.Equal(t, tt.filename != "", closed)
}) })
} }
} }
// With AutoRemove the daemon reaps the container on exit, so act must not remove it afterwards.
func TestExecAsDockerAutoRemove(t *testing.T) {
orig := ContainerNewContainer
defer func() { ContainerNewContainer = orig }()
for _, tc := range []struct {
autoRemove bool
removes int
}{
{false, 2}, // stale + post-run
{true, 1}, // post-run skipped
} {
cm := &containerMock{}
ContainerNewContainer = func(*container.NewContainerInput) container.ExecutionsEnvironment { return cm }
step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "org/action@v1"},
RunContext: &RunContext{
Config: &Config{AutoRemove: tc.autoRemove},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: cm,
},
action: &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "docker://node:14"}},
}
removes := 0
cm.On("Pull", false).Return(func(context.Context) error { return nil })
cm.On("Remove").Return(func(context.Context) error { removes++; return nil })
cm.On("Create", []string(nil), []string(nil)).Return(func(context.Context) error { return nil })
cm.On("Start", true).Return(func(context.Context) error { return nil })
cm.On("Close").Return(func(context.Context) error { return nil })
require.NoError(t, execAsDocker(context.Background(), step, "action", t.TempDir(), t.TempDir(), false, stepStageMain))
cm.AssertExpectations(t)
assert.Equal(t, tc.removes, removes)
}
}
func TestActionRunner(t *testing.T) { func TestActionRunner(t *testing.T) {
table := []struct { table := []struct {
name string name string
@@ -274,7 +227,7 @@ func TestActionRunner(t *testing.T) {
ctx := context.Background() ctx := context.Background()
cm := &containerMock{} cm := &containerMock{}
cm.On("CopyDir", "/var/run/act/actions/dir/", "dir/", false).Return(func(ctx context.Context) error { return nil }) cm.On("CopyDir", "/var/run/act/actions/dir/", "dir/", false, true).Return(func(ctx context.Context) error { return nil })
envMatcher := mock.MatchedBy(func(env map[string]string) bool { envMatcher := mock.MatchedBy(func(env map[string]string) bool {
for k, v := range tt.expectedEnv { for k, v := range tt.expectedEnv {
@@ -285,7 +238,7 @@ func TestActionRunner(t *testing.T) {
return true return true
}) })
cm.On("Exec", []string{"node", "/var/run/act/actions/dir/path"}, envMatcher, "", "").Return(func(ctx context.Context) error { return nil }) cm.On("Exec", []string{"node", "--preserve-symlinks-main", "/var/run/act/actions/dir/path"}, envMatcher, "", "").Return(func(ctx context.Context) error { return nil })
tt.step.getRunContext().JobContainer = cm tt.step.getRunContext().JobContainer = cm
@@ -297,6 +250,32 @@ func TestActionRunner(t *testing.T) {
} }
} }
func TestNodeActionCommandPreservesSymlinkedEntrypoint(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("symlink creation requires privileges on Windows")
}
requireHostTools(t, "node")
actionDir := t.TempDir()
entrypoint := filepath.Join(actionDir, "index.mjs")
require.NoError(t, os.WriteFile(entrypoint, []byte(`
import {fileURLToPath} from "node:url";
const self = fileURLToPath(import.meta.url);
if (process.argv[1] !== self) {
console.log("argv[1]:", process.argv[1], "import.meta.url:", self);
process.exitCode = 1;
}
`), 0o600))
symlinkedActionDir := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.Symlink(actionDir, symlinkedActionDir))
symlinkedEntrypoint := filepath.Join(symlinkedActionDir, "index.mjs")
args := nodeActionCommand(symlinkedEntrypoint)
output, err := exec.CommandContext(t.Context(), args[0], args[1:]...).CombinedOutput()
require.NoError(t, err, "%s", output)
}
func TestNewStepContainerDoesNotUseDockerSecrets(t *testing.T) { func TestNewStepContainerDoesNotUseDockerSecrets(t *testing.T) {
cm := &containerMock{} cm := &containerMock{}
@@ -337,11 +316,12 @@ func TestNewStepContainerDoesNotUseDockerSecrets(t *testing.T) {
step.On("getStepModel").Return(&model.Step{ID: "action"}) step.On("getStepModel").Return(&model.Step{ID: "action"})
step.On("getEnv").Return(&env) step.On("getEnv").Return(&env)
_ = newStepContainer(ctx, step, "registry.example.com/action:tag", nil, nil) _, _ = newStepContainer(ctx, step, "registry.example.com/action:tag", nil, nil, "")
// DOCKER_USERNAME/DOCKER_PASSWORD should not be injected as pull credentials for docker action containers. // DOCKER_USERNAME/DOCKER_PASSWORD should not be injected as pull credentials for docker action containers.
assert.Empty(t, captured.Username) assert.Empty(t, captured.Username)
assert.Empty(t, captured.Password) assert.Empty(t, captured.Password)
assert.True(t, captured.AutoRemove)
step.AssertExpectations(t) step.AssertExpectations(t)
} }
@@ -357,7 +337,7 @@ func TestMaybeCopyToActionDirHoldsCloneLock(t *testing.T) {
copyEntered := make(chan struct{}) copyEntered := make(chan struct{})
cm := &containerMock{} cm := &containerMock{}
cm.On("CopyDir", "/var/run/act/actions/", actionDir+"/", false).Return(func(ctx context.Context) error { cm.On("CopyDir", "/var/run/act/actions/", actionDir+"/", false, true).Return(func(ctx context.Context) error {
close(copyEntered) close(copyEntered)
<-releaseCopy <-releaseCopy
return nil return nil
@@ -542,8 +522,6 @@ func TestDockerActionImageTag(t *testing.T) {
) )
} }
// Only the entrypoint is stage specific: every stage of a docker action receives runs.args
// and runs.env, and the `entrypoint` input applies to the main stage alone.
func TestExecAsDockerStageEntrypoint(t *testing.T) { func TestExecAsDockerStageEntrypoint(t *testing.T) {
orig := ContainerNewContainer orig := ContainerNewContainer
defer func() { ContainerNewContainer = orig }() defer func() { ContainerNewContainer = orig }()
@@ -551,25 +529,62 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
stage stepStage stage stepStage
with map[string]string
runs model.ActionRuns
env map[string]string
wantCmd []string
wantEntrypoint []string wantEntrypoint []string
}{ }{
{ {
name: "main stage prefers the entrypoint input", name: "main stage prefers manifest values",
stage: stepStageMain, stage: stepStageMain,
wantEntrypoint: []string{"input.sh"}, with: map[string]string{"args": "caller", "entrypoint": "input.sh"},
runs: model.ActionRuns{
Entrypoint: "main.sh",
Args: []string{"manifest"},
Env: map[string]string{"ACTION_ONLY": "manifest"},
},
wantCmd: []string{"manifest"},
wantEntrypoint: []string{"main.sh"},
}, },
{ {
name: "pre stage uses runs.pre-entrypoint", name: "main stage uses caller fallbacks",
stage: stepStagePre, stage: stepStageMain,
with: map[string]string{"args": "caller --flag", "entrypoint": "input.sh --verbose"},
runs: model.ActionRuns{Env: map[string]string{"ACTION_ONLY": "manifest"}},
wantCmd: []string{"caller", "--flag"},
wantEntrypoint: []string{"input.sh", "--verbose"},
},
{
name: "explicit empty manifest args suppress caller args",
stage: stepStageMain,
with: map[string]string{"args": "caller"},
runs: model.ActionRuns{Args: []string{}},
wantCmd: []string{},
},
{
name: "pre stage keeps step environment",
stage: stepStagePre,
with: map[string]string{"entrypoint": "input.sh"},
runs: model.ActionRuns{
PreEntrypoint: "pre.sh --verbose",
Args: []string{"hello"},
Env: map[string]string{"SHARED": "manifest"},
},
env: map[string]string{"SHARED": "step"},
wantCmd: []string{"hello"},
wantEntrypoint: []string{"pre.sh", "--verbose"}, wantEntrypoint: []string{"pre.sh", "--verbose"},
}, },
{ {
name: "post stage uses runs.post-entrypoint", name: "post stage uses manifest entrypoint",
stage: stepStagePost, stage: stepStagePost,
runs: model.ActionRuns{PostEntrypoint: "post.sh", Args: []string{"hello"}},
wantCmd: []string{"hello"},
wantEntrypoint: []string{"post.sh"}, wantEntrypoint: []string{"post.sh"},
}, },
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
tc.runs.Using, tc.runs.Image = "docker", "docker://node:14"
cm := &containerMock{} cm := &containerMock{}
var input *container.NewContainerInput var input *container.NewContainerInput
ContainerNewContainer = func(in *container.NewContainerInput) container.ExecutionsEnvironment { ContainerNewContainer = func(in *container.NewContainerInput) container.ExecutionsEnvironment {
@@ -578,22 +593,14 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
} }
step := &stepActionRemote{ step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "org/action@v1", With: map[string]string{"entrypoint": "input.sh"}}, Step: &model.Step{ID: "1", Uses: "org/action@v1", With: tc.with},
RunContext: &RunContext{ RunContext: &RunContext{
Config: &Config{}, Config: &Config{},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}}, Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: cm, JobContainer: cm,
}, },
action: &model.Action{Runs: model.ActionRuns{ action: &model.Action{Runs: tc.runs},
Using: "docker", env: mergeMaps(tc.env),
Image: "docker://node:14",
PreEntrypoint: "pre.sh --verbose",
Entrypoint: "main.sh",
PostEntrypoint: "post.sh",
Args: []string{"hello"},
Env: map[string]string{"MY_VAR": "world"},
}},
env: map[string]string{},
} }
cm.On("Pull", false).Return(func(context.Context) error { return nil }) cm.On("Pull", false).Return(func(context.Context) error { return nil })
@@ -604,9 +611,11 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
require.NoError(t, execAsDocker(context.Background(), step, "action", t.TempDir(), t.TempDir(), false, tc.stage)) require.NoError(t, execAsDocker(context.Background(), step, "action", t.TempDir(), t.TempDir(), false, tc.stage))
require.NotNil(t, input) require.NotNil(t, input)
assert.Equal(t, tc.wantCmd, input.Cmd)
assert.Equal(t, tc.wantEntrypoint, input.Entrypoint) assert.Equal(t, tc.wantEntrypoint, input.Entrypoint)
assert.Equal(t, []string{"hello"}, input.Cmd) for key, value := range mergeMaps(tc.runs.Env, tc.env) {
assert.Contains(t, input.Env, "MY_VAR=world") assert.Contains(t, input.Env, key+"="+value)
}
}) })
} }
} }
+5 -6
View File
@@ -61,6 +61,8 @@ func TestCancelledJobStatusEnablesAlwaysAndCancelledSteps(t *testing.T) {
enabled, err := interp.Evaluate("always()", exprparser.DefaultStatusCheckSuccess) enabled, err := interp.Evaluate("always()", exprparser.DefaultStatusCheckSuccess)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, true, enabled, "`if: always()` step must run on a cancelled job") assert.Equal(t, true, enabled, "`if: always()` step must run on a cancelled job")
setJobResult(context.Background(), rc, rc, true)
assert.Equal(t, "cancelled", rc.Run.Job().Result)
} }
// TestMainStepsExecutorRunsAlwaysStepsAfterCancel verifies that newMainStepsExecutor does // TestMainStepsExecutorRunsAlwaysStepsAfterCancel verifies that newMainStepsExecutor does
@@ -107,17 +109,14 @@ func TestMainStepsExecutorMarksFailedOnTimeoutBetweenSteps(t *testing.T) {
"job1": createJob(t, `runs-on: ubuntu-latest`, ""), "job1": createJob(t, `runs-on: ubuntu-latest`, ""),
}) })
// A short deadline that we let elapse between steps, so no step records the error itself. ctx := newControllableDeadlineContext(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
defer cancel()
var ran []string var ran []string
var laterStepCtxErr error var laterStepCtxErr error
steps := []common.Executor{ steps := []common.Executor{
func(c context.Context) error { func(context.Context) error {
ran = append(ran, "step1") ran = append(ran, "step1")
// Block until the job deadline elapses, then return cleanly: the interrupt lands in the loop's between-steps check, not inside a step. ctx.expire()
<-c.Done()
return nil return nil
}, },
func(c context.Context) error { func(c context.Context) error {
+43 -16
View File
@@ -18,11 +18,17 @@ var commandPatternGA *regexp.Regexp
var commandPatternADO *regexp.Regexp var commandPatternADO *regexp.Regexp
func init() { func init() {
commandPatternGA = regexp.MustCompile("^::([^ ]+)( (.+))?::([^\r\n]*)[\r\n]+$") commandPatternGA = regexp.MustCompile("^::([^ ]+?)( (.+?))?::([^\r\n]*)[\r\n]*$")
commandPatternADO = regexp.MustCompile("^##\\[([^ ]+)( (.+))?]([^\r\n]*)[\r\n]+$") // excluding ']' ends the command info at the first bracket, as GitHub does
commandPatternADO = regexp.MustCompile("^##\\[([^ \\]]+)( ([^\\]]*))?]([^\r\n]*)[\r\n]*$")
} }
func tryParseRawActionCommand(line string) (command string, kvPairs map[string]string, arg string, ok bool) { func tryParseRawActionCommand(line string) (command string, kvPairs map[string]string, arg string, ok bool) {
command, kvPairs, arg, _, ok = tryParseActionCommand(line)
return command, kvPairs, arg, ok
}
func tryParseActionCommand(line string) (command string, kvPairs map[string]string, arg string, legacy, ok bool) {
if m := commandPatternGA.FindStringSubmatch(line); m != nil { if m := commandPatternGA.FindStringSubmatch(line); m != nil {
command = m[1] command = m[1]
kvPairs = parseKeyValuePairs(m[3], ",") kvPairs = parseKeyValuePairs(m[3], ",")
@@ -32,19 +38,21 @@ func tryParseRawActionCommand(line string) (command string, kvPairs map[string]s
command = m[1] command = m[1]
kvPairs = parseKeyValuePairs(m[3], ";") kvPairs = parseKeyValuePairs(m[3], ";")
arg = m[4] arg = m[4]
legacy = true
ok = true ok = true
} }
return command, kvPairs, arg, ok return command, kvPairs, arg, legacy, ok
} }
func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler { func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
logger := common.Logger(ctx) logger := common.Logger(ctx)
resumeCommand := "" resumeCommand := ""
return func(line string) bool { return func(line string) bool {
command, kvPairs, arg, ok := tryParseRawActionCommand(line) command, kvPairs, arg, legacy, ok := tryParseActionCommand(line)
if !ok { if !ok {
return true return true
} }
command = strings.ToLower(command)
if resumeCommand != "" { if resumeCommand != "" {
// There should not be any emojis in the log output for Gitea. // There should not be any emojis in the log output for Gitea.
@@ -54,19 +62,24 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
logger.Infof("%s", line) logger.Infof("%s", line)
// Resumed here rather than from the switch, because the end token is arbitrary // Resumed here rather than from the switch, because the end token is arbitrary
// and a token naming a real command would otherwise never resume. // and a token naming a real command would otherwise never resume.
if command == resumeCommand { if strings.EqualFold(command, resumeCommand) {
resumeCommand = "" resumeCommand = ""
} }
return true return true
} }
arg = UnescapeCommandData(arg) if legacy {
kvPairs = unescapeKvPairs(kvPairs) arg = UnescapeLegacyCommand(arg)
kvPairs = unescapeKvPairs(kvPairs, UnescapeLegacyCommand)
} else {
arg = UnescapeCommandData(arg)
kvPairs = unescapeKvPairs(kvPairs, unescapeCommandProperty)
}
if (command == "set-env" || command == "add-path") && rc.refuseUnsecureCommand(ctx, command) { if (command == "set-env" || command == "add-path") && rc.refuseUnsecureCommand(ctx, command) {
return true return true
} }
switch command { switch command {
case "set-env": case "set-env":
rc.setEnv(ctx, kvPairs, arg) rc.setEnv(ctx, kvPairs, arg, true)
case "set-output": case "set-output":
rc.setOutput(ctx, kvPairs, arg) rc.setOutput(ctx, kvPairs, arg)
case "add-path": case "add-path":
@@ -139,8 +152,16 @@ func (rc *RunContext) takeUnsecureCommandError() error {
return err return err
} }
func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string) { func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string, fromCommand bool) {
name := kvPairs["name"] name := kvPairs["name"]
if strings.EqualFold(name, "NODE_OPTIONS") {
message := "Can't store NODE_OPTIONS output parameter using '$GITHUB_ENV' command."
if fromCommand {
message = "Can't update NODE_OPTIONS environment variable using ::set-env:: command."
}
common.Logger(ctx).WithField(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(message))
return
}
common.Logger(ctx).Infof("::set-env:: %s=%s", name, arg) common.Logger(ctx).Infof("::set-env:: %s=%s", name, arg)
if rc.Env == nil { if rc.Env == nil {
rc.Env = make(map[string]string) rc.Env = make(map[string]string)
@@ -159,14 +180,14 @@ func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg
mergeIntoMap(rc.GlobalEnv, newenv) mergeIntoMap(rc.GlobalEnv, newenv)
} }
func (rc *RunContext) setEnvFile(ctx context.Context, kvPairs map[string]string, arg string) {
rc.setEnv(ctx, kvPairs, arg, false)
}
func (rc *RunContext) setOutput(ctx context.Context, kvPairs map[string]string, arg string) { func (rc *RunContext) setOutput(ctx context.Context, kvPairs map[string]string, arg string) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
stepID := rc.CurrentStep stepID := rc.CurrentStep
outputName := kvPairs["name"] outputName := kvPairs["name"]
if outputMapping, ok := rc.OutputMappings[MappableOutput{StepID: stepID, OutputName: outputName}]; ok {
stepID = outputMapping.StepID
outputName = outputMapping.OutputName
}
result, ok := rc.StepResults[stepID] result, ok := rc.StepResults[stepID]
if !ok { if !ok {
@@ -193,7 +214,7 @@ func parseKeyValuePairs(kvPairs, separator string) map[string]string {
rtn := make(map[string]string) rtn := make(map[string]string)
kvPairList := strings.SplitSeq(kvPairs, separator) kvPairList := strings.SplitSeq(kvPairs, separator)
for kvPair := range kvPairList { for kvPair := range kvPairList {
kv := strings.Split(kvPair, "=") kv := strings.SplitN(kvPair, "=", 2)
if len(kv) == 2 { if len(kv) == 2 {
rtn[kv[0]] = kv[1] rtn[kv[0]] = kv[1]
} }
@@ -206,6 +227,7 @@ var (
commandDataEscaper = strings.NewReplacer("%", "%25", "\r", "%0D", "\n", "%0A") commandDataEscaper = strings.NewReplacer("%", "%25", "\r", "%0D", "\n", "%0A")
commandDataUnescaper = strings.NewReplacer("%25", "%", "%0D", "\r", "%0A", "\n") commandDataUnescaper = strings.NewReplacer("%25", "%", "%0D", "\r", "%0A", "\n")
commandPropertyUnescaper = strings.NewReplacer("%25", "%", "%0D", "\r", "%0A", "\n", "%3A", ":", "%2C", ",") commandPropertyUnescaper = strings.NewReplacer("%25", "%", "%0D", "\r", "%0A", "\n", "%3A", ":", "%2C", ",")
legacyCommandUnescaper = strings.NewReplacer("%3B", ";", "%0D", "\r", "%0A", "\n", "%5D", "]", "%25", "%")
) )
// EscapeCommandData encodes the data part of a "::cmd::" or "##[cmd]" line the runner writes itself, // EscapeCommandData encodes the data part of a "::cmd::" or "##[cmd]" line the runner writes itself,
@@ -222,9 +244,14 @@ func unescapeCommandProperty(arg string) string {
return commandPropertyUnescaper.Replace(arg) return commandPropertyUnescaper.Replace(arg)
} }
func unescapeKvPairs(kvPairs map[string]string) map[string]string { // UnescapeLegacyCommand decodes a "##[cmd]" line, which also spells ";" and "]" escaped.
func UnescapeLegacyCommand(arg string) string {
return legacyCommandUnescaper.Replace(arg)
}
func unescapeKvPairs(kvPairs map[string]string, unescape func(string) string) map[string]string {
for k, v := range kvPairs { for k, v := range kvPairs {
kvPairs[k] = unescapeCommandProperty(v) kvPairs[k] = unescape(v)
} }
return kvPairs return kvPairs
} }
+23 -4
View File
@@ -26,12 +26,21 @@ func unsecureRC() *RunContext {
func TestSetEnv(t *testing.T) { func TestSetEnv(t *testing.T) {
a := assert.New(t) a := assert.New(t)
ctx := context.Background() logger, hook := test.NewNullLogger()
ctx := common.WithLogger(context.Background(), logger)
rc := unsecureRC() rc := unsecureRC()
handler := rc.commandHandler(ctx) handler := rc.commandHandler(ctx)
handler("::set-env name=x::valz\n") handler("::set-env name=x::valz\n")
a.Equal("valz", rc.Env["x"]) a.Equal("valz", rc.Env["x"])
handler("::SET-ENV name=NODE_OPTIONS::--require command.js\n")
rc.setEnvFile(ctx, map[string]string{"name": "node_options"}, "--require env.js")
a.NotContains(rc.Env, "NODE_OPTIONS")
a.NotContains(rc.Env, "node_options")
entries := hook.AllEntries()
require.Len(t, entries, 3)
a.Equal("##[error]Can't update NODE_OPTIONS environment variable using ::set-env:: command.", entries[1].Message)
a.Equal("##[error]Can't store NODE_OPTIONS output parameter using '$GITHUB_ENV' command.", entries[2].Message)
} }
func TestStopCommandsKeepsSuppressedLinesInLog(t *testing.T) { func TestStopCommandsKeepsSuppressedLinesInLog(t *testing.T) {
@@ -85,6 +94,16 @@ func TestSetOutput(t *testing.T) {
handler("::set-output name=x%3A%2C%0A%25%0D%3A::percent2%25%0Atest\n") handler("::set-output name=x%3A%2C%0A%25%0D%3A::percent2%25%0Atest\n")
a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x:,\n%\r:"]) a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x:,\n%\r:"])
handler("::set-output name=symbol::std::vector")
a.Equal("std::vector", rc.StepResults["my-step"].Outputs["symbol"])
handler("::set-output name=a=b::value\n")
a.Equal("value", rc.StepResults["my-step"].Outputs["a=b"])
handler("##[set-output name=legacy%3B%5D]value%3B%5D")
a.Equal("value;]", rc.StepResults["my-step"].Outputs["legacy;]"])
handler("##[set-output name=bracket]value]tail")
a.Equal("value]tail", rc.StepResults["my-step"].Outputs["bracket"])
handler("::set-output name=modern%3B%5D::value%3B%5D\n")
a.Equal("value%3B%5D", rc.StepResults["my-step"].Outputs["modern%3B%5D"])
} }
func TestAddpath(t *testing.T) { func TestAddpath(t *testing.T) {
@@ -110,7 +129,7 @@ func TestStopCommands(t *testing.T) {
handler("::set-env name=x::valz\n") handler("::set-env name=x::valz\n")
a.Equal("valz", rc.Env["x"]) a.Equal("valz", rc.Env["x"])
handler("::stop-commands::my-end-token\n") handler("::stop-commands::MY-END-TOKEN\n")
handler("::set-env name=x::abcd\n") handler("::set-env name=x::abcd\n")
a.Equal("valz", rc.Env["x"]) a.Equal("valz", rc.Env["x"])
handler("::my-end-token::\n") handler("::my-end-token::\n")
@@ -163,10 +182,10 @@ func TestAddmask(t *testing.T) {
rc := new(RunContext) rc := new(RunContext)
handler := rc.commandHandler(loggerCtx) handler := rc.commandHandler(loggerCtx)
handler("::add-mask::my-secret-value\n") handler("::ADD-MASK::my::secret")
a.Equal("***", hook.LastEntry().Message) a.Equal("***", hook.LastEntry().Message)
a.NotEqual("*my-secret-value", hook.LastEntry().Message) a.Equal([]string{"my::secret"}, rc.Masks)
} }
// based on https://stackoverflow.com/a/10476304 // based on https://stackoverflow.com/a/10476304
+4 -7
View File
@@ -14,6 +14,8 @@ import (
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
) )
var noopExecutor = func(context.Context) error { return nil }
type containerMock struct { type containerMock struct {
mock.Mock mock.Mock
container.Container container.Container
@@ -50,18 +52,13 @@ func (cm *containerMock) UpdateFromEnv(srcPath string, env *map[string]string) c
return args.Get(0).(func(context.Context) error) return args.Get(0).(func(context.Context) error)
} }
func (cm *containerMock) UpdateFromImageEnv(env *map[string]string) common.Executor {
args := cm.Called(env)
return args.Get(0).(func(context.Context) error)
}
func (cm *containerMock) Copy(destPath string, files ...*container.FileEntry) common.Executor { func (cm *containerMock) Copy(destPath string, files ...*container.FileEntry) common.Executor {
args := cm.Called(destPath, files) args := cm.Called(destPath, files)
return args.Get(0).(func(context.Context) error) return args.Get(0).(func(context.Context) error)
} }
func (cm *containerMock) CopyDir(destPath, srcPath string, useGitIgnore bool) common.Executor { func (cm *containerMock) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
args := cm.Called(destPath, srcPath, useGitIgnore) args := cm.Called(destPath, srcPath, useGitIgnore, skipGitDir)
return args.Get(0).(func(context.Context) error) return args.Get(0).(func(context.Context) error)
} }
+102
View File
@@ -0,0 +1,102 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
package runner
import (
"context"
"crypto/rand"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model"
cerrdefs "github.com/containerd/errdefs"
"github.com/moby/moby/client"
"github.com/sirupsen/logrus/hooks/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestDockerProxyMountedJob(t *testing.T) {
mode := os.Getenv("ACT_TEST_DOCKER_PROXY")
if mode == "" {
t.Skip("set ACT_TEST_DOCKER_PROXY=proxy or direct to verify mounted Docker access")
}
require.Contains(t, []string{"proxy", "direct"}, mode)
ctx, cancel := context.WithTimeout(t.Context(), 2*time.Minute)
defer cancel()
dockerClient, err := container.GetDockerClient(ctx)
require.NoError(t, err)
t.Cleanup(func() { assert.NoError(t, dockerClient.Close()) })
require.True(t, strings.HasPrefix(dockerClient.DaemonHost(), "unix://"), "mounted Docker coverage requires a Unix daemon socket")
fixtureDir, err := filepath.Abs("testdata/docker-proxy")
require.NoError(t, err)
resourceName := "gitea-proxy-test-" + strings.ToLower(rand.Text())
runner, err := New(&Config{
Workdir: fixtureDir,
ActionCacheDir: t.TempDir(),
EventName: "push",
PlatformPicker: mapPlatformPicker(platforms),
ContainerNamePrefix: resourceName,
ContainerDaemonSocket: dockerClient.DaemonHost(),
ContainerMaxLifetime: 2 * time.Minute,
Env: map[string]string{"PROXY_TEST_RESOURCE": resourceName, "PROXY_TEST_MODE": mode},
})
require.NoError(t, err)
planner, err := model.NewWorkflowPlanner(filepath.Join(fixtureDir, "push.yml"), true)
require.NoError(t, err)
plan, err := planner.PlanEvent("push")
require.NoError(t, err)
runContext, err := runner.newRunContext(ctx, plan.Stages[0].Runs[0], nil)
require.NoError(t, err)
jobName := runContext.jobContainerName()
jobNetwork, _ := runContext.networkNameForGitea()
t.Cleanup(func() {
cleanCtx, cleanCancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cleanCancel()
if _, err := dockerClient.ContainerRemove(cleanCtx, jobName, client.ContainerRemoveOptions{Force: true, RemoveVolumes: true}); !cerrdefs.IsNotFound(err) {
assert.NoError(t, err)
}
for _, name := range []string{resourceName, jobNetwork} {
if _, err := dockerClient.NetworkRemove(cleanCtx, name, client.NetworkRemoveOptions{}); !cerrdefs.IsNotFound(err) {
assert.NoError(t, err)
}
}
for _, name := range []string{resourceName, jobName, jobName + "-env"} {
if _, err := dockerClient.VolumeRemove(cleanCtx, name, client.VolumeRemoveOptions{}); !cerrdefs.IsNotFound(err) {
assert.NoError(t, err)
}
}
})
hook := &test.Hook{}
require.NoError(t, runner.NewPlanExecutor(plan)(common.WithLoggerHook(ctx, hook)))
var messages []string
for _, entry := range hook.AllEntries() {
messages = append(messages, strings.TrimSpace(entry.Message))
}
require.Contains(t, messages, "docker proxy post verified")
_, err = dockerClient.ContainerInspect(ctx, jobName, client.ContainerInspectOptions{})
assert.True(t, cerrdefs.IsNotFound(err), "job container survived cleanup: %v", err)
_, err = dockerClient.NetworkInspect(ctx, resourceName, client.NetworkInspectOptions{})
if mode == "proxy" {
assert.True(t, cerrdefs.IsNotFound(err), "labelled network survived cleanup: %v", err)
} else {
require.NoError(t, err)
}
_, err = dockerClient.VolumeInspect(ctx, resourceName, client.VolumeInspectOptions{})
if mode == "proxy" {
assert.True(t, cerrdefs.IsNotFound(err), "labelled volume survived cleanup: %v", err)
} else {
require.NoError(t, err)
}
}
+77 -82
View File
@@ -13,6 +13,7 @@ import (
"reflect" "reflect"
"regexp" "regexp"
"strings" "strings"
"sync"
"time" "time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -26,20 +27,12 @@ import (
"go.yaml.in/yaml/v4" "go.yaml.in/yaml/v4"
) )
// ExpressionEvaluator is the interface for evaluating expressions
type ExpressionEvaluator interface {
evaluate(context.Context, string, exprparser.DefaultStatusCheck) (any, error)
interpolate(context.Context, string) (string, error)
EvaluateYamlNode(context.Context, *yaml.Node) error
Interpolate(context.Context, string) string
}
// NewExpressionEvaluator creates a new evaluator // NewExpressionEvaluator creates a new evaluator
func (rc *RunContext) NewExpressionEvaluator(ctx context.Context) ExpressionEvaluator { func (rc *RunContext) NewExpressionEvaluator(ctx context.Context) *ExpressionEvaluator {
return rc.NewExpressionEvaluatorWithEnv(ctx, rc.GetEnv()) return rc.NewExpressionEvaluatorWithEnv(ctx, rc.GetEnv())
} }
func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map[string]string) ExpressionEvaluator { func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map[string]string) *ExpressionEvaluator {
var workflowCallResult map[string]*model.WorkflowCallResult var workflowCallResult map[string]*model.WorkflowCallResult
// todo: cleanup EvaluationEnvironment creation // todo: cleanup EvaluationEnvironment creation
@@ -79,7 +72,7 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
} }
ghc := rc.getGithubContext(ctx) ghc := rc.getGithubContext(ctx)
inputs := getEvaluatorInputs(ctx, rc, nil, ghc) inputs := getEvaluatorInputs(rc, rc.actionInputs, ghc)
ee := &exprparser.EvaluationEnvironment{ ee := &exprparser.EvaluationEnvironment{
Github: ghc, Github: ghc,
@@ -89,7 +82,7 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
// todo: should be unavailable // todo: should be unavailable
// but required to interpolate/evaluate the step outputs on the job // but required to interpolate/evaluate the step outputs on the job
Steps: rc.getStepsContext(), Steps: rc.getStepsContext(),
Secrets: getWorkflowSecrets(ctx, rc), Secrets: getWorkflowSecrets(rc),
Vars: getWorkflowVars(ctx, rc), Vars: getWorkflowVars(ctx, rc),
Strategy: strategy, Strategy: strategy,
Matrix: rc.Matrix, Matrix: rc.Matrix,
@@ -98,7 +91,7 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
HashFiles: getHashFilesFunction(ctx, rc), HashFiles: getHashFilesFunction(ctx, rc),
} }
ee.Runner = rc.getRunnerContext(ctx) ee.Runner = rc.getRunnerContext(ctx)
return expressionEvaluator{ return &expressionEvaluator{
interpreter: exprparser.NewInterpeter(ee, exprparser.Config{ interpreter: exprparser.NewInterpeter(ee, exprparser.Config{
Run: rc.Run, Run: rc.Run,
WorkingDir: rc.Config.Workdir, WorkingDir: rc.Config.Workdir,
@@ -110,8 +103,17 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
//go:embed hashfiles/index.js //go:embed hashfiles/index.js
var hashfiles string var hashfiles string
// NewStepExpressionEvaluator creates a new evaluator // NewStepExpressionEvaluator creates a new evaluator with the `inputs` of the enclosing workflow or composite action
func (rc *RunContext) NewStepExpressionEvaluator(ctx context.Context, step step) ExpressionEvaluator { func (rc *RunContext) NewStepExpressionEvaluator(ctx context.Context, step step) *ExpressionEvaluator {
return rc.newStepExpressionEvaluator(ctx, step, rc.actionInputs)
}
// NewActionInputsExpressionEvaluator creates a new evaluator with the step's own with: values as `inputs`
func (rc *RunContext) NewActionInputsExpressionEvaluator(ctx context.Context, step step) *ExpressionEvaluator {
return rc.newStepExpressionEvaluator(ctx, step, inputsFromEnv(*step.getEnv()))
}
func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step, stepInputs map[string]any) *ExpressionEvaluator {
// todo: cleanup EvaluationEnvironment creation // todo: cleanup EvaluationEnvironment creation
job := rc.Run.Job() job := rc.Run.Job()
strategy := make(map[string]any) strategy := make(map[string]any)
@@ -131,26 +133,23 @@ func (rc *RunContext) NewStepExpressionEvaluator(ctx context.Context, step step)
} }
} }
ghc := rc.getGithubContext(ctx)
inputs := getEvaluatorInputs(ctx, rc, step, ghc)
ee := &exprparser.EvaluationEnvironment{ ee := &exprparser.EvaluationEnvironment{
Github: step.getGithubContext(ctx), Github: step.getGithubContext(ctx),
Env: *step.getEnv(), Env: *step.getEnv(),
Job: rc.getJobContext(), Job: rc.getJobContext(),
Steps: rc.getStepsContext(), Steps: rc.getStepsContext(),
Secrets: getWorkflowSecrets(ctx, rc), Secrets: getWorkflowSecrets(rc),
Vars: getWorkflowVars(ctx, rc), Vars: getWorkflowVars(ctx, rc),
Strategy: strategy, Strategy: strategy,
Matrix: rc.Matrix, Matrix: rc.Matrix,
Needs: using, Needs: using,
// todo: should be unavailable // todo: should be unavailable
// but required to interpolate/evaluate the inputs in actions/composite // but required to interpolate/evaluate the inputs in actions/composite
Inputs: inputs, Inputs: getEvaluatorInputs(rc, stepInputs, rc.getGithubContext(ctx)),
HashFiles: getHashFilesFunction(ctx, rc), HashFiles: getHashFilesFunction(ctx, rc),
} }
ee.Runner = rc.getRunnerContext(ctx) ee.Runner = rc.getRunnerContext(ctx)
return expressionEvaluator{ return &expressionEvaluator{
interpreter: exprparser.NewInterpeter(ee, exprparser.Config{ interpreter: exprparser.NewInterpeter(ee, exprparser.Config{
Run: rc.Run, Run: rc.Run,
WorkingDir: rc.Config.Workdir, WorkingDir: rc.Config.Workdir,
@@ -178,7 +177,7 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
followSymlink = true followSymlink = true
continue continue
} }
return "", fmt.Errorf("Invalid glob option %s, available option: '--follow-symbolic-links'", s) return "", fmt.Errorf("invalid glob option %s, available option: '--follow-symbolic-links'", s)
} }
} }
patterns = append(patterns, s) patterns = append(patterns, s)
@@ -196,7 +195,7 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
Mode: 0o644, Mode: 0o644,
Body: hashfiles, Body: hashfiles,
}). }).
Then(rc.execJobContainer([]string{"node", path.Join(rc.JobContainer.GetActPath(), name)}, Then(rc.JobContainer.Exec([]string{"node", path.Join(rc.JobContainer.GetActPath(), name)},
env, "", "")). env, "", "")).
Finally(func(context.Context) error { Finally(func(context.Context) error {
rc.JobContainer.ReplaceLogWriter(stdout, stderr) rc.JobContainer.ReplaceLogWriter(stdout, stderr)
@@ -222,6 +221,8 @@ type expressionEvaluator struct {
interpreter exprparser.Interpreter interpreter exprparser.Interpreter
} }
type ExpressionEvaluator = expressionEvaluator
func (ee expressionEvaluator) evaluate(ctx context.Context, in string, defaultStatusCheck exprparser.DefaultStatusCheck) (any, error) { func (ee expressionEvaluator) evaluate(ctx context.Context, in string, defaultStatusCheck exprparser.DefaultStatusCheck) (any, error) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
logger.Debugf("evaluating expression '%s'", in) logger.Debugf("evaluating expression '%s'", in)
@@ -246,44 +247,43 @@ func (ee expressionEvaluator) EvaluateYamlNode(ctx context.Context, node *yaml.N
return ee.shared(ctx).EvaluateYamlNode(node) return ee.shared(ctx).EvaluateYamlNode(node)
} }
func (ee expressionEvaluator) Interpolate(ctx context.Context, in string) string { func (ee expressionEvaluator) Interpolate(ctx context.Context, in string) (string, error) {
out, err := ee.interpolate(ctx, in) return ee.shared(ctx).Interpolate(in)
}
// InterpolateName keeps the source text of a job or step name that cannot be evaluated, as GitHub does.
func (ee expressionEvaluator) InterpolateName(ctx context.Context, in string) string {
out, err := ee.Interpolate(ctx, in)
if err != nil { if err != nil {
common.Logger(ctx).Errorf("Unable to interpolate expression '%s': %s", in, err) common.Logger(ctx).Warnf("Unable to evaluate the display name '%s': %s", in, err)
return "" return in
} }
return out return out
} }
func (ee expressionEvaluator) interpolate(ctx context.Context, in string) (string, error) {
return ee.shared(ctx).Interpolate(in)
}
// EvalBool evaluates an expression against given evaluator. An `if:` is an expression even without // EvalBool evaluates an expression against given evaluator. An `if:` is an expression even without
// `${{ }}`, while literal text around one makes the whole value a string. // `${{ }}`, while literal text around one makes the whole value a string.
func EvalBool(ctx context.Context, evaluator ExpressionEvaluator, expr string, defaultStatusCheck exprparser.DefaultStatusCheck) (bool, error) { func EvalBool(ctx context.Context, evaluator *expressionEvaluator, expr string, defaultStatusCheck exprparser.DefaultStatusCheck) (bool, error) {
return expreval.New(func(in string, dsc exprparser.DefaultStatusCheck) (any, error) { return expreval.New(func(in string, dsc exprparser.DefaultStatusCheck) (any, error) {
return evaluator.evaluate(ctx, in, dsc) return evaluator.evaluate(ctx, in, dsc)
}).EvalBool(expr, defaultStatusCheck) }).EvalBool(expr, defaultStatusCheck)
} }
func getEvaluatorInputs(ctx context.Context, rc *RunContext, step step, ghc *model.GithubContext) map[string]any { func inputsFromEnv(env map[string]string) map[string]any {
inputs := map[string]any{} inputs := map[string]any{}
setupWorkflowInputs(ctx, &inputs, rc)
var env map[string]string
if step != nil {
env = *step.getEnv()
} else {
env = rc.GetEnv()
}
for k, v := range env { for k, v := range env {
if after, ok := strings.CutPrefix(k, "INPUT_"); ok { if after, ok := strings.CutPrefix(k, "INPUT_"); ok {
inputs[strings.ToLower(after)] = v inputs[strings.ToLower(after)] = v
} }
} }
return inputs
}
func getEvaluatorInputs(rc *RunContext, stepInputs map[string]any, ghc *model.GithubContext) map[string]any {
inputs := map[string]any{}
maps.Copy(inputs, rc.workflowCallInputs)
maps.Copy(inputs, stepInputs)
if ghc.EventName == "workflow_dispatch" { if ghc.EventName == "workflow_dispatch" {
config := rc.Run.Workflow.WorkflowDispatchConfig() config := rc.Run.Workflow.WorkflowDispatchConfig()
@@ -326,54 +326,49 @@ func coerceInputValue(value any, inputType string) any {
return value == "true" return value == "true"
} }
func setupWorkflowInputs(ctx context.Context, inputs *map[string]any, rc *RunContext) { // resolveWorkflowCall evaluates the caller's with: and secrets: once, as the server does before dispatching a called workflow.
if rc.caller != nil { func (rc *RunContext) resolveWorkflowCall(ctx context.Context) error {
config := rc.Run.Workflow.WorkflowCallConfig() if rc.caller == nil {
return nil
}
callerJob := rc.caller.runContext.Run.Job()
callerEval := rc.caller.runContext.ExprEval
calleeEval := sync.OnceValue(func() *expressionEvaluator { return rc.NewExpressionEvaluator(ctx) })
config := rc.Run.Workflow.WorkflowCallConfig()
for name, input := range config.Inputs { rc.workflowCallInputs = make(map[string]any, len(config.Inputs))
value := rc.caller.runContext.Run.Job().With[name] for name, input := range config.Inputs {
if value != nil { value, eval, label := callerJob.With[name], callerEval, "input"
if str, ok := value.(string); ok { if value == nil {
// evaluate using the calling RunContext (outside) value, eval, label = input.Default, calleeEval(), "the default of input"
value = rc.caller.runContext.ExprEval.Interpolate(ctx, str) }
} if str, ok := value.(string); ok {
var err error
if value, err = eval.Interpolate(ctx, str); err != nil {
return fmt.Errorf("unable to interpolate %s %s: %w", label, name, err)
} }
}
rc.workflowCallInputs[name] = coerceInputValue(value, input.Type)
}
if value == nil && config != nil && config.Inputs != nil { secrets := callerJob.Secrets()
value = input.Default if secrets == nil && callerJob.InheritSecrets() {
if rc.ExprEval != nil { secrets = rc.caller.runContext.Config.Secrets
if str, ok := value.(string); ok { }
// evaluate using the called RunContext (inside) rc.workflowCallSecrets = make(map[string]string, len(secrets))
value = rc.ExprEval.Interpolate(ctx, str) for k, v := range secrets {
} var err error
} if rc.workflowCallSecrets[k], err = callerEval.Interpolate(ctx, v); err != nil {
} return fmt.Errorf("unable to interpolate secret %s: %w", k, err)
(*inputs)[name] = coerceInputValue(value, input.Type)
} }
} }
return nil
} }
func getWorkflowSecrets(ctx context.Context, rc *RunContext) map[string]string { func getWorkflowSecrets(rc *RunContext) map[string]string {
if rc.caller != nil { if rc.caller != nil {
job := rc.caller.runContext.Run.Job() return rc.workflowCallSecrets
secrets := job.Secrets()
if secrets == nil && job.InheritSecrets() {
secrets = rc.caller.runContext.Config.Secrets
}
// Interpolate into a new map. secrets may be the shared Config.Secrets (or the job's
// map), which other parallel jobs read concurrently (e.g. log masking), so mutating it
// in place is a data race.
interpolated := make(map[string]string, len(secrets))
for k, v := range secrets {
interpolated[k] = rc.caller.runContext.ExprEval.Interpolate(ctx, v)
}
return interpolated
} }
return rc.Config.Secrets return rc.Config.Secrets
} }
+35 -7
View File
@@ -156,8 +156,10 @@ func TestEvaluateRunContext(t *testing.T) {
func TestEvaluateStep(t *testing.T) { func TestEvaluateStep(t *testing.T) {
rc := createRunContext(t) rc := createRunContext(t)
rc.Env["INPUT_FORGED"] = "leaked"
step := &stepRun{ step := &stepRun{
RunContext: rc, RunContext: rc,
env: map[string]string{"INPUT_FORGED": "leaked"},
} }
ee := rc.NewStepExpressionEvaluator(context.Background(), step) ee := rc.NewStepExpressionEvaluator(context.Background(), step)
@@ -176,6 +178,7 @@ func TestEvaluateStep(t *testing.T) {
{"steps.id_with_underscores.conclusion", model.StepStatusSuccess.String(), ""}, {"steps.id_with_underscores.conclusion", model.StepStatusSuccess.String(), ""},
{"steps.id_with_underscores.outcome", model.StepStatusFailure.String(), ""}, {"steps.id_with_underscores.outcome", model.StepStatusFailure.String(), ""},
{"steps.id_with_underscores.outputs.foo_with_underscores", "bar_with_underscores", ""}, {"steps.id_with_underscores.outputs.foo_with_underscores", "bar_with_underscores", ""},
{"inputs.forged", nil, ""}, // INPUT_* env is not an input
} }
for _, table := range tables { for _, table := range tables {
@@ -263,19 +266,21 @@ func TestInterpolate(t *testing.T) {
{"${{ null }}", ""}, {"${{ null }}", ""},
{"${{ fromJSON('[1,2]') }}", "Array"}, {"${{ fromJSON('[1,2]') }}", "Array"},
{"${{ fromJSON('{\"a\":1}') }}", "Object"}, {"${{ fromJSON('{\"a\":1}') }}", "Object"},
// a malformed part must not restructure its neighbours, and it interpolates to nothing
{"${{ 1) && (2 }}", ""},
{"run ${{ 1) && (2 }} now", ""},
{"${{ 1", "${{ 1"}, {"${{ 1", "${{ 1"},
} }
for _, table := range tables { for _, table := range tables {
t.Run("interpolate", func(t *testing.T) { t.Run("interpolate", func(t *testing.T) {
assertObject := assert.New(t) out, err := ee.Interpolate(context.Background(), table.in)
out := ee.Interpolate(context.Background(), table.in) require.NoError(t, err, table.in)
assertObject.Equal(table.out, out, table.in) assert.Equal(t, table.out, out, table.in)
}) })
} }
for _, in := range []string{"${{ 1) && (2 }}", "run ${{ 1) && (2 }} now"} {
_, err := ee.Interpolate(context.Background(), in)
assert.Error(t, err, in)
}
} }
func TestGetEvaluatorInputsBoolean(t *testing.T) { func TestGetEvaluatorInputsBoolean(t *testing.T) {
@@ -349,10 +354,33 @@ on:
} }
ghc := &model.GithubContext{EventName: eventName, Event: table.event} ghc := &model.GithubContext{EventName: eventName, Event: table.event}
inputs := getEvaluatorInputs(context.Background(), rc, nil, ghc) inputs := getEvaluatorInputs(rc, nil, ghc)
assert.Equal(t, table.flag, inputs["flag"]) assert.Equal(t, table.flag, inputs["flag"])
assert.Equal(t, "gitea", inputs["name"]) assert.Equal(t, "gitea", inputs["name"])
}) })
} }
} }
} }
func TestJobNameMasksSecrets(t *testing.T) {
workflow, err := model.ReadWorkflow(strings.NewReader(`
jobs:
a:
name: deploy ${{ secrets.A }}
b:
name: deploy ${{ secrets.B }}
`))
require.NoError(t, err)
runner := &runnerImpl{config: &Config{Secrets: map[string]string{"A": "s3cr3t-a", "B": "s3cr3t-b"}}}
containerName := func(jobID string) string {
rc, err := runner.newRunContext(t.Context(), &model.Run{JobID: jobID, Workflow: workflow}, nil)
require.NoError(t, err)
assert.NotContains(t, rc.Name, "s3cr3t")
return rc.jobContainerName()
}
a, b := containerName("a"), containerName("b")
assert.NotContains(t, a, "s3cr3t") // it reaches the container name, which no log masker covers
assert.NotEqual(t, a, b) // masking the name must not collapse two jobs onto one container
}
+91 -84
View File
@@ -9,7 +9,7 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/base64" "encoding/base64"
"encoding/json" "encoding/json/v2"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -18,6 +18,7 @@ import (
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"unicode" "unicode"
@@ -154,11 +155,9 @@ func newJobExecutor(info jobInfo, sf stepFactory, rc *RunContext) common.Executo
if rc.Run == nil { if rc.Run == nil {
return nil return nil
} }
rc.ExprEval = rc.NewExpressionEvaluator(ctx) if err := evaluateJobEnvAndDefaults(ctx, rc); err != nil {
// evaluate environment variables since they can contain reportStepError(ctx, rc, err)
// GitHub's special environment variables. return err
for k, v := range rc.GetEnv() {
rc.Env[k] = rc.ExprEval.Interpolate(ctx, v)
} }
return nil return nil
}) })
@@ -236,67 +235,47 @@ func newJobExecutor(info jobInfo, sf stepFactory, rc *RunContext) common.Executo
// Ahead of the teardown below, while the job environment is still up. // Ahead of the teardown below, while the job environment is still up.
postExecutor = postExecutor.Finally(rc.runJobCompletedHook) postExecutor = postExecutor.Finally(rc.runJobCompletedHook)
postExecutor = postExecutor.Finally(func(ctx context.Context) error {
// swallowed: a bad output fails this job, it must not abandon the rest of the plan
if err := info.interpolateOutputs()(ctx); err != nil {
reportStepError(ctx, rc, err)
} else if err := setJobOutputs(ctx, rc); err != nil {
reportStepError(ctx, rc, err)
}
return nil
})
postExecutor = postExecutor.Finally(func(ctx context.Context) error { postExecutor = postExecutor.Finally(func(ctx context.Context) error {
jobError := common.JobError(ctx) jobError := common.JobError(ctx)
var err error var err error
// jobError == nil keeps a failed job's container alive for post-mortem debugging when // always allow 1 min for stopping and removing the runner, even if we were cancelled
// AutoRemove is off (the act-CLI --rm behavior; the shipped runner always sets ctx, cancel := context.WithTimeout(common.WithLogger(context.Background(), common.Logger(ctx)), time.Minute)
// AutoRemove). A cancelled run is not a failure to inspect, and the cancel-path post defer cancel()
// context now carries its own error container so a failing post step makes jobError
// non-nil — OR in rc.jobCancelled so cancellation still always tears the container down.
if rc.Config.AutoRemove || jobError == nil || rc.jobCancelled {
// always allow 1 min for stopping and removing the runner, even if we were cancelled
ctx, cancel := context.WithTimeout(common.WithLogger(context.Background(), common.Logger(ctx)), time.Minute)
defer cancel()
logger := common.Logger(ctx) logger := common.Logger(ctx)
tryUploadJobSummary(ctx, rc) tryUploadJobSummary(ctx, rc)
// For Gitea logger.Infof("Cleaning up container for job %s", rc.JobName)
// We don't need to call `stopServiceContainers` here since it will be called by following `info.stopContainer` if err = info.stopContainer()(ctx); err != nil {
// logger.Infof("Cleaning up services for job %s", rc.JobName) logger.Errorf("##[error]%s", EscapeCommandData("Error while stop job container: "+err.Error()))
// if err := rc.stopServiceContainers()(ctx); err != nil {
// logger.Errorf("Error while cleaning services: %v", err)
// }
logger.Infof("Cleaning up container for job %s", rc.JobName)
if err = info.stopContainer()(ctx); err != nil {
logger.Errorf("##[error]%s", EscapeCommandData("Error while stop job container: "+err.Error()))
}
// For Gitea
// We don't need to call `NewDockerNetworkRemoveExecutor` here since it is called by above `info.stopContainer`
// if !rc.IsHostEnv(ctx) && rc.Config.ContainerNetworkMode == "" {
// // clean network in docker mode only
// // if the value of `ContainerNetworkMode` is empty string,
// // it means that the network to which containers are connecting is created by `runner`,
// // so, we should remove the network at last.
// networkName, _ := rc.networkName()
// logger.Infof("Cleaning up network for job %s, and network name is: %s", rc.JobName, networkName)
// if err := container.NewDockerNetworkRemoveExecutor(networkName)(ctx); err != nil {
// logger.Errorf("Error while cleaning network: %v", err)
// }
// }
} }
setJobResult(ctx, info, rc, jobError == nil) setJobResult(ctx, info, rc, jobError == nil)
setJobOutputs(ctx, rc)
return err return err
}) })
stepsExecutor := newStepsExecutor(rc, preSteps, steps) return common.Executor(func(ctx context.Context) error {
if err := info.startContainer()(ctx); err != nil {
return common.NewPipelineExecutor(info.startContainer(), stepsExecutor. return err
Finally(func(ctx context.Context) error { }
return newStepsExecutor(rc, preSteps, steps).Finally(func(ctx context.Context) error {
// Record an interrupt (backstop for interrupts that land outside the main // Record an interrupt (backstop for interrupts that land outside the main
// step loop) so the post steps observe the cancelled/failed job status. // step loop) so the post steps observe the cancelled/failed job status.
rc.markInterrupted(ctx.Err()) rc.markInterrupted(ctx.Err())
postCtx, cancel := postStepsContext(ctx) postCtx, cancel := postStepsContext(ctx)
defer cancel() defer cancel()
return postExecutor(postCtx) return postExecutor(postCtx)
}). })(ctx)
Finally(info.interpolateOutputs()). }).Finally(info.closeContainer())
Finally(info.closeContainer()))
} }
// postStepsContext derives the context used to run the job's post/cleanup steps from the // postStepsContext derives the context used to run the job's post/cleanup steps from the
@@ -393,7 +372,7 @@ func setJobResult(ctx context.Context, info jobInfo, rc *RunContext, success boo
// concurrent succeeding one. // concurrent succeeding one.
job := rc.Run.Job() job := rc.Run.Job()
var continueOnError bool var continueOnError bool
if !success { if !success && !rc.jobCancelled {
// Use a fresh context so an expired job timeout cannot block expression evaluation. // Use a fresh context so an expired job timeout cannot block expression evaluation.
evalCtx := common.WithLogger(context.Background(), common.Logger(ctx)) evalCtx := common.WithLogger(context.Background(), common.Logger(ctx))
continueOnError = evaluateJobContinueOnError(evalCtx, rc, job) continueOnError = evaluateJobContinueOnError(evalCtx, rc, job)
@@ -406,7 +385,11 @@ func setJobResult(ctx context.Context, info jobInfo, rc *RunContext, success boo
if len(info.matrix()) > 0 && job.Result != "" { if len(info.matrix()) > 0 && job.Result != "" {
result = job.Result result = job.Result
} }
if !success { // cancelled is sticky, so a sibling combination finishing last cannot mask it
switch {
case rc.jobCancelled:
result = "cancelled"
case !success && result != "cancelled":
result = "failure" result = "failure"
job.SetContinueOnError(continueOnError) job.SetContinueOnError(continueOnError)
} }
@@ -416,42 +399,73 @@ func setJobResult(ctx context.Context, info jobInfo, rc *RunContext, success boo
if rc.caller != nil { if rc.caller != nil {
// set reusable workflow job result // set reusable workflow job result
rc.caller.setReusedWorkflowJobResult(rc.JobName, jobResult) // For Gitea rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, jobResult) // For Gitea
return return
} }
jobResultMessage := "succeeded" jobResultMessage := "failed"
if jobResult != "success" { switch jobResult {
jobResultMessage = "failed" case "success":
jobResultMessage = "succeeded"
case "cancelled":
jobResultMessage = "cancelled"
} }
logger.WithField("jobResult", jobResult).Infof("Job %s", jobResultMessage) logger.WithField("jobResult", jobResult).Infof("Job %s", jobResultMessage)
} }
func setJobOutputs(ctx context.Context, rc *RunContext) { // evaluateJobEnvAndDefaults resolves the job's env and defaults.run once, as GitHub does at job setup.
if rc.caller != nil { func evaluateJobEnvAndDefaults(ctx context.Context, rc *RunContext) error {
// map outputs for reusable workflows rc.ExprEval = rc.NewExpressionEvaluator(ctx)
callerOutputs := make(map[string]string) var err error
for k, v := range rc.GetEnv() {
ee := rc.NewExpressionEvaluator(ctx) if rc.Env[k], err = rc.ExprEval.Interpolate(ctx, v); err != nil {
return fmt.Errorf("unable to interpolate env %s: %w", k, err)
for k, v := range rc.Run.Workflow.WorkflowCallConfig().Outputs {
callerOutputs[k] = ee.Interpolate(ctx, ee.Interpolate(ctx, v.Value))
} }
// Matrix combinations of a reusable-workflow caller share the caller's *model.Job;
// serialize the write so parallel combos don't race on its Outputs field.
callerJob := rc.caller.runContext.Run.Job()
defer lockJob(callerJob)()
callerJob.Outputs = callerOutputs
} }
defaults := rc.Run.Job().Defaults.Run
if rc.jobRunDefaults.Shell, err = rc.ExprEval.Interpolate(ctx, defaults.Shell); err != nil {
return fmt.Errorf("unable to interpolate defaults.run.shell: %w", err)
}
if rc.jobRunDefaults.WorkingDirectory, err = rc.ExprEval.Interpolate(ctx, defaults.WorkingDirectory); err != nil {
return fmt.Errorf("unable to interpolate defaults.run.working-directory: %w", err)
}
return nil
}
func setJobOutputs(ctx context.Context, rc *RunContext) error {
if rc.caller == nil {
return nil
}
outputs := rc.Run.Workflow.WorkflowCallConfig().Outputs
callerOutputs := make(map[string]string, len(outputs))
ee := sync.OnceValue(func() *expressionEvaluator { return rc.NewExpressionEvaluator(ctx) })
for k, v := range outputs {
value := v.Value
for range 2 { // two passes, the value resolves through a job output
var err error
if value, err = ee().Interpolate(ctx, value); err != nil {
return fmt.Errorf("unable to interpolate workflow output %s: %w", k, err)
}
}
callerOutputs[k] = value
}
// Matrix combinations of a reusable-workflow caller share the caller's *model.Job;
// serialize the write so parallel combos don't race on its Outputs field.
callerJob := rc.caller.runContext.Run.Job()
defer lockJob(callerJob)()
callerJob.Outputs = callerOutputs
return nil
} }
// applyJobTimeout applies the job-level timeout-minutes to ctx, mirroring the // applyJobTimeout applies the job-level timeout-minutes to ctx, mirroring the
// step-level evaluateStepTimeout in step.go. // step-level evaluateStepTimeout in step.go.
func applyJobTimeout(ctx context.Context, rc *RunContext, job *model.Job) (context.Context, context.CancelFunc) { func applyJobTimeout(ctx context.Context, rc *RunContext, job *model.Job) (context.Context, context.CancelFunc) {
timeout := rc.ExprEval.Interpolate(ctx, job.TimeoutMinutes) timeout, err := rc.ExprEval.Interpolate(ctx, job.TimeoutMinutes)
if timeout != "" { if err != nil {
common.Logger(ctx).Errorf("An error occurred when attempting to determine the job timeout: %s", err)
} else if timeout != "" {
if timeoutMinutes, err := strconv.ParseInt(timeout, 10, 64); err == nil { if timeoutMinutes, err := strconv.ParseInt(timeout, 10, 64); err == nil {
return context.WithTimeout(ctx, time.Duration(timeoutMinutes)*time.Minute) return context.WithTimeout(ctx, time.Duration(timeoutMinutes)*time.Minute)
} }
@@ -515,7 +529,8 @@ func tryUploadJobSummary(ctx context.Context, rc *RunContext) {
if !ok || len(body) == 0 { if !ok || len(body) == 0 {
continue continue
} }
uploadJobSummary(ctx, client, base+strconv.Itoa(i)+"/summary", runtimeToken, body) // Gitea renders summaries on the run page, so mask before the upload.
uploadJobSummary(ctx, client, base+strconv.Itoa(i)+"/summary", runtimeToken, []byte(rc.maskSecrets(string(body))))
} }
} }
@@ -649,17 +664,9 @@ func archiveEntryMatchesPath(entryName, requestedPath string) bool {
func useStepLogger(rc *RunContext, stepModel *model.Step, stage stepStage, executor common.Executor) common.Executor { func useStepLogger(rc *RunContext, stepModel *model.Step, stage stepStage, executor common.Executor) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
ctx = withStepLogger(ctx, stepModel.Number, stepModel.ID, rc.ExprEval.Interpolate(ctx, stepModel.String()), stage.String()) ctx = withStepLogger(ctx, stepModel.Number, stepModel.ID, rc.ExprEval.InterpolateName(ctx, stepModel.String()), stage.String())
rawLogger := common.Logger(ctx).WithField("raw_output", true) logWriter := rc.commandLogWriter(ctx)
logWriter := common.NewLineWriter(rc.commandHandler(ctx), func(s string) bool {
if rc.Config.LogOutput {
rawLogger.Infof("%s", s)
} else {
rawLogger.Debugf("%s", s)
}
return true
})
oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter) oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter)
defer rc.JobContainer.ReplaceLogWriter(oldout, olderr) defer rc.JobContainer.ReplaceLogWriter(oldout, olderr)
+129 -30
View File
@@ -299,6 +299,9 @@ func TestNewJobExecutor(t *testing.T) {
executedSteps []string executedSteps []string
result string result string
hasError bool hasError bool
output string
startError error
cancelOnStart bool
}{ }{
{ {
name: "zeroSteps", name: "zeroSteps",
@@ -319,8 +322,8 @@ func TestNewJobExecutor(t *testing.T) {
executedSteps: []string{ executedSteps: []string{
"startContainer", "startContainer",
"step1", "step1",
"stopContainer",
"interpolateOutputs", "interpolateOutputs",
"stopContainer",
"closeContainer", "closeContainer",
}, },
result: "success", result: "success",
@@ -337,6 +340,7 @@ func TestNewJobExecutor(t *testing.T) {
"startContainer", "startContainer",
"step1", "step1",
"interpolateOutputs", "interpolateOutputs",
"stopContainer",
"closeContainer", "closeContainer",
}, },
result: "failure", result: "failure",
@@ -353,8 +357,8 @@ func TestNewJobExecutor(t *testing.T) {
"startContainer", "startContainer",
"pre1", "pre1",
"step1", "step1",
"stopContainer",
"interpolateOutputs", "interpolateOutputs",
"stopContainer",
"closeContainer", "closeContainer",
}, },
result: "success", result: "success",
@@ -371,8 +375,8 @@ func TestNewJobExecutor(t *testing.T) {
"startContainer", "startContainer",
"step1", "step1",
"post1", "post1",
"stopContainer",
"interpolateOutputs", "interpolateOutputs",
"stopContainer",
"closeContainer", "closeContainer",
}, },
result: "success", result: "success",
@@ -390,8 +394,8 @@ func TestNewJobExecutor(t *testing.T) {
"pre1", "pre1",
"step1", "step1",
"post1", "post1",
"stopContainer",
"interpolateOutputs", "interpolateOutputs",
"stopContainer",
"closeContainer", "closeContainer",
}, },
result: "success", result: "success",
@@ -417,13 +421,37 @@ func TestNewJobExecutor(t *testing.T) {
"step3", "step3",
"post3", "post3",
"post2", "post2",
"stopContainer",
"interpolateOutputs", "interpolateOutputs",
"stopContainer",
"closeContainer", "closeContainer",
}, },
result: "success", result: "success",
hasError: false, hasError: false,
}, },
{
name: "jobOutputExpressionFailure",
steps: []*model.Step{{ID: "1"}},
preSteps: []bool{false},
postSteps: []bool{false},
executedSteps: []string{"startContainer", "step1", "interpolateOutputs", "stopContainer", "closeContainer"},
result: "failure",
output: "${{ 'test' != test }}",
},
{
name: "start failure",
steps: []*model.Step{{ID: "1"}},
executedSteps: []string{"startContainer", "closeContainer"},
startError: errors.New("start failed"),
},
{
name: "cancelled at startup boundary",
steps: []*model.Step{{ID: "1"}},
preSteps: []bool{false},
postSteps: []bool{true},
executedSteps: []string{"startContainer", "step1", "post1", "interpolateOutputs", "stopContainer", "closeContainer"},
result: "cancelled",
cancelOnStart: true,
},
} }
contains := func(needle string, haystack []string) bool { contains := func(needle string, haystack []string) bool {
@@ -434,7 +462,8 @@ func TestNewJobExecutor(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
fmt.Printf("::group::%s\n", tt.name) //nolint:forbidigo // pre-existing issue from nektos/act fmt.Printf("::group::%s\n", tt.name) //nolint:forbidigo // pre-existing issue from nektos/act
ctx := common.WithJobErrorContainer(context.Background()) ctx, cancel := context.WithCancel(common.WithJobErrorContainer(context.Background()))
defer cancel()
jim := &jobInfoMock{} jim := &jobInfoMock{}
sfm := &stepFactoryMock{} sfm := &stepFactoryMock{}
rc := &RunContext{ rc := &RunContext{
@@ -449,15 +478,22 @@ func TestNewJobExecutor(t *testing.T) {
}, },
Config: &Config{}, Config: &Config{},
} }
if tt.output != "" {
rc.Run.Job().Outputs = map[string]string{"bad": tt.output}
rc.outputTemplate = map[string]string{"bad": tt.output}
}
rc.ExprEval = rc.NewExpressionEvaluator(ctx) rc.ExprEval = rc.NewExpressionEvaluator(ctx)
executorOrder := make([]string, 0) executorOrder := make([]string, 0)
jim.On("steps").Return(tt.steps) jim.On("steps").Return(tt.steps)
if len(tt.steps) > 0 { if len(tt.steps) > 0 {
jim.On("startContainer").Return(func(ctx context.Context) error { jim.On("startContainer").Return(func(_ context.Context) error {
executorOrder = append(executorOrder, "startContainer") executorOrder = append(executorOrder, "startContainer")
return nil if tt.cancelOnStart {
cancel()
}
return tt.startError
}) })
} }
@@ -491,23 +527,31 @@ func TestNewJobExecutor(t *testing.T) {
defer sm.AssertExpectations(t) defer sm.AssertExpectations(t)
} }
if len(tt.steps) > 0 { if len(tt.steps) > 0 && tt.startError == nil {
jim.On("matrix").Return(map[string]any{}) jim.On("matrix").Return(map[string]any{})
jim.On("interpolateOutputs").Return(func(ctx context.Context) error { jim.On("interpolateOutputs").Return(func(ctx context.Context) error {
executorOrder = append(executorOrder, "interpolateOutputs") executorOrder = append(executorOrder, "interpolateOutputs")
if tt.output != "" {
return rc.interpolateOutputs()(ctx)
}
return nil return nil
}) })
if contains("stopContainer", tt.executedSteps) { if contains("stopContainer", tt.executedSteps) {
jim.On("stopContainer").Return(func(ctx context.Context) error { jim.On("stopContainer").Return(func(ctx context.Context) error {
executorOrder = append(executorOrder, "stopContainer") executorOrder = append(executorOrder, "stopContainer")
require.NoError(t, ctx.Err())
_, bounded := ctx.Deadline()
require.True(t, bounded)
return nil return nil
}) })
} }
jim.On("result", tt.result) jim.On("result", tt.result)
}
if len(tt.steps) > 0 {
jim.On("closeContainer").Return(func(ctx context.Context) error { jim.On("closeContainer").Return(func(ctx context.Context) error {
executorOrder = append(executorOrder, "closeContainer") executorOrder = append(executorOrder, "closeContainer")
return nil return nil
@@ -516,7 +560,15 @@ func TestNewJobExecutor(t *testing.T) {
executor := newJobExecutor(jim, sfm, rc) executor := newJobExecutor(jim, sfm, rc)
err := executor(ctx) err := executor(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act switch {
case tt.startError != nil:
require.ErrorIs(t, err, tt.startError)
case tt.cancelOnStart:
require.ErrorIs(t, err, context.Canceled)
default:
require.NoError(t, err)
}
assert.Empty(t, rc.Run.Job().Outputs["bad"])
assert.Equal(t, tt.executedSteps, executorOrder) assert.Equal(t, tt.executedSteps, executorOrder)
jim.AssertExpectations(t) jim.AssertExpectations(t)
@@ -527,18 +579,39 @@ func TestNewJobExecutor(t *testing.T) {
} }
} }
type controllableDeadlineContext struct {
context.Context
done chan struct{}
}
func newControllableDeadlineContext(parent context.Context) *controllableDeadlineContext {
return &controllableDeadlineContext{Context: parent, done: make(chan struct{})}
}
func (ctx *controllableDeadlineContext) Done() <-chan struct{} {
return ctx.done
}
func (ctx *controllableDeadlineContext) Err() error {
select {
case <-ctx.done:
return context.DeadlineExceeded
default:
return nil
}
}
func (ctx *controllableDeadlineContext) expire() {
close(ctx.done)
}
// TestNewJobExecutorRunsPostStepsAfterTimeout guards the timeout-minutes cleanup // TestNewJobExecutorRunsPostStepsAfterTimeout guards the timeout-minutes cleanup
// path: when a job exceeds its timeout the job context is DeadlineExceeded, but // path: when a job exceeds its timeout the job context is DeadlineExceeded, but
// the post steps (cleanup hooks like actions/checkout post and cache save) must // the post steps (cleanup hooks like actions/checkout post and cache save) must
// still run against a fresh, non-expired context, and the job must still be // still run against a fresh, non-expired context, and the job must still be
// reported as failed. // reported as failed.
func TestNewJobExecutorRunsPostStepsAfterTimeout(t *testing.T) { func TestNewJobExecutorRunsPostStepsAfterTimeout(t *testing.T) {
ctx := common.WithJobErrorContainer(context.Background()) ctx := newControllableDeadlineContext(common.WithJobErrorContainer(context.Background()))
// The timeout is generous so the main step (which blocks on ctx.Done below) is
// always reached before the deadline fires; otherwise the pipeline would
// short-circuit before the step runs and the job error would never be set.
ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
defer cancel()
jim := &jobInfoMock{} jim := &jobInfoMock{}
sfm := &stepFactoryMock{} sfm := &stepFactoryMock{}
@@ -562,19 +635,16 @@ func TestNewJobExecutorRunsPostStepsAfterTimeout(t *testing.T) {
jim.On("startContainer").Return(func(ctx context.Context) error { return nil }) jim.On("startContainer").Return(func(ctx context.Context) error { return nil })
jim.On("interpolateOutputs").Return(func(ctx context.Context) error { return nil }) jim.On("interpolateOutputs").Return(func(ctx context.Context) error { return nil })
jim.On("closeContainer").Return(func(ctx context.Context) error { return nil }) jim.On("closeContainer").Return(func(ctx context.Context) error { return nil })
// The job timed out, so it must be reported as failed. stopContainer is left // The job timed out, so it must be reported as failed and still cleaned up.
// unexpected on purpose: a timed-out (failed) job preserves its error state, so jim.On("stopContainer").Return(func(context.Context) error { return nil })
// the graceful stop is skipped exactly like any other failure without AutoRemove.
jim.On("result", "failure") jim.On("result", "failure")
sm := &stepMock{} sm := &stepMock{}
sfm.On("newStep", stepModel, rc).Return(sm, nil) sfm.On("newStep", stepModel, rc).Return(sm, nil)
sm.On("pre").Return(func(ctx context.Context) error { return nil }) sm.On("pre").Return(func(ctx context.Context) error { return nil })
// The main step runs past the job timeout: it blocks until the job context is sm.On("main").Return(func(stepCtx context.Context) error {
// done, mirroring a step that overruns timeout-minutes. ctx.expire()
sm.On("main").Return(func(ctx context.Context) error { return stepCtx.Err()
<-ctx.Done()
return ctx.Err()
}) })
var postRan bool var postRan bool
@@ -984,12 +1054,7 @@ func tarArchive(t *testing.T, entries ...tarEntry) []byte {
func newTestRC(wf *model.Workflow, matrix map[string]any) *RunContext { func newTestRC(wf *model.Workflow, matrix map[string]any) *RunContext {
return &RunContext{ return &RunContext{
Config: &Config{ Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }},
Workdir: ".",
Platforms: map[string]string{
"ubuntu-latest": "ubuntu-latest",
},
},
StepResults: map[string]*model.StepResult{}, StepResults: map[string]*model.StepResult{},
Env: map[string]string{}, Env: map[string]string{},
Matrix: matrix, Matrix: matrix,
@@ -1082,3 +1147,37 @@ func TestJobSetContinueOnError(t *testing.T) {
assert.True(t, j.ContinueOnError) assert.True(t, j.ContinueOnError)
}) })
} }
func TestTryUploadJobSummaryMasksSecrets(t *testing.T) {
var got string
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
got = string(body)
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
cm := &containerMock{}
cm.On("GetContainerArchive", mock.Anything, "/var/run/act/workflow/step-summary-0.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{
name: "step-summary-0.md", body: "deployed true with s3cr3t and runtime-added via pr0xypw",
}))),
nil,
).Once()
rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": "job-summary",
"ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": fakeRuntimeToken(34),
"GITEA_RUN_ID": "12",
}, cm, 1)
rc.Config.Secrets = map[string]string{"TOK": "s3cr3t", "ACTIONS_STEP_DEBUG": "true"}
rc.Config.ExtraMasks = []string{"pr0xypw"}
rc.Masks = []string{"runtime-added"}
tryUploadJobSummary(context.Background(), rc)
assert.Equal(t, "deployed true with *** and *** via ***", got)
cm.AssertExpectations(t)
}
+9 -5
View File
@@ -51,10 +51,11 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
rawLogger.Infof("shell: %s", shell) rawLogger.Infof("shell: %s", shell)
} }
env := maps.Clone(rc.GetEnv()) env := map[string]string{}
if jobContainer := rc.Run.Job().Container(); jobContainer != nil { if jobContainer := rc.Run.Job().Container(); jobContainer != nil {
maps.Copy(env, jobContainer.Env) maps.Copy(env, jobContainer.Env)
} }
maps.Copy(env, rc.GetEnv())
rc.withGithubEnv(ctx, rc.getGithubContext(ctx), env) rc.withGithubEnv(ctx, rc.getGithubContext(ctx), env)
rc.ApplyExtraPath(ctx, &env) rc.ApplyExtraPath(ctx, &env)
@@ -64,7 +65,9 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
} }
// Processed even on failure, so a hook that exports what it managed to set up before // Processed even on failure, so a hook that exports what it managed to set up before
// failing still hands it to the job. // failing still hands it to the job.
err = cmp.Or(err, rc.processHookFileCommands(ctx)) if processErr := rc.processHookFileCommands(ctx); err == nil {
err = processErr
}
if err == nil { if err == nil {
return nil return nil
} }
@@ -93,10 +96,11 @@ func (rc *RunContext) setupHookFileCommands(ctx context.Context, env map[string]
} }
func (rc *RunContext) processHookFileCommands(ctx context.Context) error { func (rc *RunContext) processHookFileCommands(ctx context.Context) error {
if err := processRunnerEnvFileCommand(ctx, hookEnvFileCommand, rc, rc.setEnv); err != nil { err := processRunnerEnvFileCommand(ctx, hookEnvFileCommand, rc, rc.setEnvFile)
return err if pathErr := rc.UpdateExtraPath(ctx, path.Join(rc.JobContainer.GetActPath(), hookPathFileCommand)); pathErr != nil && err == nil {
err = pathErr
} }
return rc.UpdateExtraPath(ctx, path.Join(rc.JobContainer.GetActPath(), hookPathFileCommand)) return err
} }
// hookCommand mirrors actions/runner, which deliberately does not apply the shell flags it // hookCommand mirrors actions/runner, which deliberately does not apply the shell flags it
-95
View File
@@ -1,95 +0,0 @@
// Copyright 2024 The Gitea Authors. All rights reserved.
// Copyright 2024 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package runner
import (
"archive/tar"
"bytes"
"context"
"fmt"
"io"
"io/fs"
goURL "net/url"
"os"
"path/filepath"
"strings"
"gitea.com/gitea/runner/act/filecollector"
)
type LocalRepositoryCache struct {
Parent ActionCache
LocalRepositories map[string]string
CacheDirCache map[string]string
}
func (l *LocalRepositoryCache) Fetch(ctx context.Context, cacheDir, url, ref, token string) (string, error) {
if dest, ok := l.LocalRepositories[fmt.Sprintf("%s@%s", url, ref)]; ok {
l.CacheDirCache[fmt.Sprintf("%s@%s", cacheDir, ref)] = dest
return ref, nil
}
if purl, err := goURL.Parse(url); err == nil {
if dest, ok := l.LocalRepositories[fmt.Sprintf("%s@%s", strings.TrimPrefix(purl.Path, "/"), ref)]; ok {
l.CacheDirCache[fmt.Sprintf("%s@%s", cacheDir, ref)] = dest
return ref, nil
}
}
return l.Parent.Fetch(ctx, cacheDir, url, ref, token)
}
func (l *LocalRepositoryCache) GetTarArchive(ctx context.Context, cacheDir, sha, includePrefix string) (io.ReadCloser, error) {
// sha is mapped to ref in fetch if there is a local override
if dest, ok := l.CacheDirCache[fmt.Sprintf("%s@%s", cacheDir, sha)]; ok {
srcPath := filepath.Join(dest, includePrefix)
buf := &bytes.Buffer{}
tw := tar.NewWriter(buf)
defer tw.Close()
srcPath = filepath.Clean(srcPath)
fi, err := os.Lstat(srcPath)
if err != nil {
return nil, err
}
tc := &filecollector.TarCollector{
TarWriter: tw,
}
if fi.IsDir() {
srcPrefix := srcPath
if !strings.HasSuffix(srcPrefix, string(filepath.Separator)) {
srcPrefix += string(filepath.Separator)
}
fc := &filecollector.FileCollector{
Fs: &filecollector.DefaultFs{},
SrcPath: srcPath,
SrcPrefix: srcPrefix,
Handler: tc,
}
err = filepath.Walk(srcPath, fc.CollectFiles(ctx, []string{}))
if err != nil {
return nil, err
}
} else {
var f io.ReadCloser
var linkname string
if fi.Mode()&fs.ModeSymlink != 0 {
linkname, err = os.Readlink(srcPath)
if err != nil {
return nil, err
}
} else {
f, err = os.Open(srcPath)
if err != nil {
return nil, err
}
defer f.Close()
}
err := tc.WriteFile(fi.Name(), fi, linkname, f)
if err != nil {
return nil, err
}
}
return io.NopCloser(buf), nil
}
return l.Parent.GetTarArchive(ctx, cacheDir, sha, includePrefix)
}
+138 -84
View File
@@ -8,7 +8,8 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/base64" "encoding/base64"
"encoding/json" "encoding/json/jsontext"
"encoding/json/v2"
"fmt" "fmt"
"io" "io"
"net/url" "net/url"
@@ -78,7 +79,7 @@ type JobLoggerFactory interface {
type jobLoggerFactoryContextKey string type jobLoggerFactoryContextKey string
var jobLoggerFactoryContextKeyVal = (jobLoggerFactoryContextKey)("jobloggerkey") var jobLoggerFactoryContextKeyVal = jobLoggerFactoryContextKey("jobloggerkey")
func WithJobLoggerFactory(ctx context.Context, factory JobLoggerFactory) context.Context { func WithJobLoggerFactory(ctx context.Context, factory JobLoggerFactory) context.Context {
return context.WithValue(ctx, jobLoggerFactoryContextKeyVal, factory) return context.WithValue(ctx, jobLoggerFactoryContextKeyVal, factory)
@@ -99,10 +100,7 @@ func WithJobLogger(ctx context.Context, jobID, jobName string, config *Config, m
mux.Lock() mux.Lock()
defer mux.Unlock() defer mux.Unlock()
nextColor++ nextColor++
formatter = &jobLogFormatter{ formatter = &jobLogFormatter{color: colors[nextColor%len(colors)]}
color: colors[nextColor%len(colors)],
logPrefixJobID: config.LogPrefixJobID,
}
} }
logger = logrus.New() logger = logrus.New()
@@ -124,7 +122,7 @@ func WithJobLogger(ctx context.Context, jobID, jobName string, config *Config, m
logger.SetFormatter(&maskedFormatter{ logger.SetFormatter(&maskedFormatter{
Formatter: logger.Formatter, Formatter: logger.Formatter,
masker: valueMasker(config.InsecureSecrets, config.Secrets), masker: valueMasker(config.InsecureSecrets, config.maskers()),
}) })
rtn := logger.WithFields(logrus.Fields{ rtn := logger.WithFields(logrus.Fields{
"job": jobName, "job": jobName,
@@ -170,55 +168,83 @@ func withStepLogger(ctx context.Context, stepNumber int, stepID, stepName, stage
type entryProcessor func(entry *logrus.Entry) *logrus.Entry type entryProcessor func(entry *logrus.Entry) *logrus.Entry
// secretValueEncoders are the shapes a secret takes on its way into a log: a base64
// payload, a JSON string, or a URL component. An action that serializes a secret leaks
// it in one of these forms, which a mask of the verbatim value alone does not catch, so
// every form is masked as well. This mirrors the value encoders of GitHub's runner.
var secretValueEncoders = []func(string) string{ var secretValueEncoders = []func(string) string{
func(v string) string { return base64.StdEncoding.EncodeToString([]byte(v)) }, func(v string) string { return base64.StdEncoding.EncodeToString([]byte(v)) },
base64ShiftEncoder(1), base64ShiftEncoder(1),
base64ShiftEncoder(2), base64ShiftEncoder(2),
base64InteriorEncoder(0),
base64InteriorEncoder(1),
base64InteriorEncoder(2),
expressionStringEscape,
jsonStringEscape, jsonStringEscape,
jsonStringEscapeNoHTML, jsonStringEscapeNoHTML,
url.QueryEscape, uriDataEscape,
url.QueryEscape, // the form-encoded twin of uriDataEscape, which spells a space "+"
url.PathEscape, url.PathEscape,
xmlDataEscape,
trimDoubleQuotes,
} }
// minShiftedBase64Len is the shortest shifted base64 fragment worth masking. A shorter // base64ShiftEncoder reproduces the 3-byte alignments of `Basic base64("user:token")`, and
// one carries too few bytes of the secret to identify it and would mask unrelated output. // its padded tail only matches a secret that ends the payload.
const minShiftedBase64Len = 8
// base64ShiftEncoder returns the part of a secret's base64 form that survives when the
// secret does not start on a 3-byte boundary of the payload it is embedded in. base64
// encodes three bytes at a time, so `Authorization: Basic base64("user:token")` contains
// the base64 of the token alone only when the prefix length happens to be a multiple of
// three; at the other two alignments the encoding of the whole value differs. Encoding
// the secret behind shift filler bytes reproduces those alignments, which is what the
// Base64StringEscapeShift1/2 encoders of GitHub's runner do.
//
// The leading group (filler mixed with the secret's first bytes) and the trailing group
// (padded here, but continuing into whatever follows the secret) are dropped, leaving the
// group-aligned middle that does appear verbatim in the log.
func base64ShiftEncoder(shift int) func(string) string { func base64ShiftEncoder(shift int) func(string) string {
return func(v string) string {
value := []byte(v)
if len(value) > shift {
value = value[shift:]
}
return base64.StdEncoding.EncodeToString(value)
}
}
const minInteriorBase64Len = 8 // below this a fragment matches unrelated output
// base64InteriorEncoder keeps the aligned middle, so a secret with data after it still matches.
func base64InteriorEncoder(shift int) func(string) string {
return func(v string) string { return func(v string) string {
buf := make([]byte, shift+len(v)) buf := make([]byte, shift+len(v))
copy(buf[shift:], v) copy(buf[shift:], v)
encoded := base64.StdEncoding.EncodeToString(buf) encoded := base64.StdEncoding.EncodeToString(buf)
// Keep only the aligned middle, and only when enough of it is left to be a if len(encoded) < 8+minInteriorBase64Len {
// distinctive pattern rather than a fragment that matches unrelated output.
if len(encoded) < 8+minShiftedBase64Len {
return "" return ""
} }
return encoded[4 : len(encoded)-4] return encoded[4 : len(encoded)-4]
} }
} }
func expressionStringEscape(v string) string {
return strings.ReplaceAll(v, "'", "''")
}
func uriDataEscape(v string) string {
return strings.ReplaceAll(url.QueryEscape(v), "+", "%20")
}
var xmlDataEscaper = strings.NewReplacer(
"&", "&amp;",
"<", "&lt;",
">", "&gt;",
`"`, "&quot;",
"'", "&apos;",
)
func xmlDataEscape(v string) string {
return xmlDataEscaper.Replace(v)
}
func trimDoubleQuotes(v string) string {
if len(v) > 8 && strings.HasPrefix(v, `"`) && strings.HasSuffix(v, `"`) {
return v[1 : len(v)-1]
}
return ""
}
// jsonStringEscape returns v as it appears inside a JSON string, without the quotes, // jsonStringEscape returns v as it appears inside a JSON string, without the quotes,
// which is what `toJSON(secrets)` or any action logging a JSON body produces. Go's encoder // which is what `toJSON(secrets)` or any action logging a JSON body produces. Go's encoder
// escapes <, >, & (as act's own toJSON does); the non-HTML variant below covers the runtimes // escapes <, >, & (as act's own toJSON does); the non-HTML variant below covers the runtimes
// that do not. When v has none of those characters both forms are equal and deduplicated. // that do not. When v has none of those characters both forms are equal and deduplicated.
func jsonStringEscape(v string) string { func jsonStringEscape(v string) string {
encoded, err := json.Marshal(v) encoded, err := json.Marshal(v, jsontext.EscapeForHTML(true))
if err != nil { if err != nil {
return v return v
} }
@@ -229,59 +255,60 @@ func jsonStringEscape(v string) string {
// JavaScript (JSON.stringify) or .NET action emits, so a secret containing < > or & is // JavaScript (JSON.stringify) or .NET action emits, so a secret containing < > or & is
// masked in that form too. // masked in that form too.
func jsonStringEscapeNoHTML(v string) string { func jsonStringEscapeNoHTML(v string) string {
var buf bytes.Buffer encoded, err := json.Marshal(v)
enc := json.NewEncoder(&buf) if err != nil {
enc.SetEscapeHTML(false)
if err := enc.Encode(v); err != nil {
return v return v
} }
// Encode appends a newline; drop it along with the surrounding quotes. return string(encoded[1 : len(encoded)-1])
encoded := strings.TrimRight(buf.String(), "\n")
return encoded[1 : len(encoded)-1]
} }
func AppendSecretMasker(oldnew []string, v string) []string { // AppendSecretMaskers skips the debug settings, as GitHub does: they arrive as secrets, but
ret := oldnew // masking "true" would corrupt unrelated log lines and drop job outputs that say it.
func AppendSecretMaskers(oldnew []string, secrets map[string]string) []string {
for l := range strings.SplitSeq(v, "\n") { for k, v := range secrets {
tm := strings.TrimSpace(l) if k != "ACTIONS_STEP_DEBUG" && k != "ACTIONS_RUNNER_DEBUG" {
// formatted JSON secrets could otherwise mask {,[,],} everywhere oldnew = AppendSecretMasker(oldnew, v)
if len(tm) > 1 {
ret = append(ret, tm, "***")
// command data reaches the log escaped, so "pass%word" also arrives as "pass%25word"
if strings.ContainsAny(tm, "%\r\n") {
ret = append(ret, EscapeCommandData(tm), "***")
}
} }
} }
return oldnew
}
// The encoded forms are derived from the whole value: a multi-line secret is // AppendSecretMasker registers v and each of its lines, as GitHub does.
// encoded as one string, not line by line. func AppendSecretMasker(oldnew []string, v string) []string {
trimmed := strings.TrimSpace(v) ret := appendMaskedValue(oldnew, v)
if len(trimmed) <= 1 { for l := range strings.FieldsFuncSeq(v, func(r rune) bool { return r == '\r' || r == '\n' }) {
ret = appendMaskedValue(ret, strings.TrimSpace(l))
}
return ret
}
// appendMaskedValue registers one value and every shape it takes on its way into a log.
func appendMaskedValue(ret []string, v string) []string {
// formatted JSON secrets could otherwise mask {,[,],} everywhere
if len(strings.TrimSpace(v)) <= 1 || slices.Contains(ret, v) {
return ret return ret
} }
ret = append(ret, v, "***")
// command data reaches the log escaped, so "pass%word" also arrives as "pass%25word"
if strings.ContainsAny(v, "%\r\n") {
ret = append(ret, EscapeCommandData(v), "***")
}
for _, encode := range secretValueEncoders { for _, encode := range secretValueEncoders {
encoded := encode(trimmed) encoded := encode(v)
// An encoding that leaves the value unchanged is already masked above. // An encoding that leaves the value unchanged is already masked above.
if encoded == trimmed || len(encoded) <= 1 || slices.Contains(ret, encoded) { if encoded == v || len(encoded) <= 1 || slices.Contains(ret, encoded) {
continue continue
} }
ret = append(ret, encoded, "***") ret = append(ret, encoded, "***")
} }
return ret return ret
} }
// valueMasker applies secrets and ::add-mask:: patterns to every log entry, including // valueMasker applies secrets and ::add-mask:: patterns to every log entry, including
// raw_output (command/stream) lines; there is no bypass by field. // raw_output (command/stream) lines; there is no bypass by field.
func valueMasker(insecureSecrets bool, secrets map[string]string) entryProcessor { func valueMasker(insecureSecrets bool, oldnew []string) entryProcessor {
var oldnew []string
for _, v := range secrets {
oldnew = AppendSecretMasker(oldnew, v)
}
oldnew = slices.Clip(oldnew) oldnew = slices.Clip(oldnew)
defReplacer := strings.NewReplacer(oldnew...) defReplacer := NewSecretReplacer(oldnew)
// A ::add-mask:: only ever appends to the job's mask slice, so the replacer built for // A ::add-mask:: only ever appends to the job's mask slice, so the replacer built for
// it stays valid until the slice grows. Cache it, keyed by the slice itself and its // it stays valid until the slice grows. Cache it, keyed by the slice itself and its
@@ -317,7 +344,7 @@ func valueMasker(insecureSecrets bool, secrets map[string]string) entryProcessor
pairs = AppendSecretMasker(pairs, v) pairs = AppendSecretMasker(pairs, v)
} }
masked = len(*masks) masked = len(*masks)
replacer = strings.NewReplacer(pairs...) replacer = NewSecretReplacer(pairs)
} }
cmasker := replacer cmasker := replacer
mu.Unlock() mu.Unlock()
@@ -338,8 +365,7 @@ func (f *maskedFormatter) Format(entry *logrus.Entry) ([]byte, error) {
} }
type jobLogFormatter struct { type jobLogFormatter struct {
color int color int
logPrefixJobID bool
} }
func (f *jobLogFormatter) Format(entry *logrus.Entry) ([]byte, error) { func (f *jobLogFormatter) Format(entry *logrus.Entry) ([]byte, error) {
@@ -363,27 +389,23 @@ func (f *jobLogFormatter) Format(entry *logrus.Entry) ([]byte, error) {
func (f *jobLogFormatter) printColored(b *bytes.Buffer, entry *logrus.Entry) { func (f *jobLogFormatter) printColored(b *bytes.Buffer, entry *logrus.Entry) {
entry.Message = strings.TrimSuffix(entry.Message, "\n") entry.Message = strings.TrimSuffix(entry.Message, "\n")
var job any job := entry.Data["job"]
if f.logPrefixJobID {
job = entry.Data["jobID"]
} else {
job = entry.Data["job"]
}
debugFlag := "" debugFlag := ""
if entry.Level == logrus.DebugLevel { if entry.Level == logrus.DebugLevel {
debugFlag = "[DEBUG] " debugFlag = "[DEBUG] "
} }
if entry.Data[rawOutputField] == true { switch {
case entry.Data[rawOutputField] == true:
if entry.Data[scriptLineCyanField] == true { if entry.Data[scriptLineCyanField] == true {
fmt.Fprintf(b, "\x1b[%dm|\x1b[0m \x1b[36;1m%s\x1b[0m", f.color, entry.Message) fmt.Fprintf(b, "\x1b[%dm|\x1b[0m \x1b[36;1m%s\x1b[0m", f.color, entry.Message)
} else { } else {
fmt.Fprintf(b, "\x1b[%dm|\x1b[0m %s", f.color, entry.Message) fmt.Fprintf(b, "\x1b[%dm|\x1b[0m %s", f.color, entry.Message)
} }
} else if entry.Data["dryrun"] == true { case entry.Data["dryrun"] == true:
fmt.Fprintf(b, "\x1b[1m\x1b[%dm\x1b[7m*DRYRUN*\x1b[0m \x1b[%dm[%s] \x1b[0m%s%s", gray, f.color, job, debugFlag, entry.Message) fmt.Fprintf(b, "\x1b[1m\x1b[%dm\x1b[7m*DRYRUN*\x1b[0m \x1b[%dm[%s] \x1b[0m%s%s", gray, f.color, job, debugFlag, entry.Message)
} else { default:
fmt.Fprintf(b, "\x1b[%dm[%s] \x1b[0m%s%s", f.color, job, debugFlag, entry.Message) fmt.Fprintf(b, "\x1b[%dm[%s] \x1b[0m%s%s", f.color, job, debugFlag, entry.Message)
} }
} }
@@ -391,23 +413,19 @@ func (f *jobLogFormatter) printColored(b *bytes.Buffer, entry *logrus.Entry) {
func (f *jobLogFormatter) print(b *bytes.Buffer, entry *logrus.Entry) { func (f *jobLogFormatter) print(b *bytes.Buffer, entry *logrus.Entry) {
entry.Message = strings.TrimSuffix(entry.Message, "\n") entry.Message = strings.TrimSuffix(entry.Message, "\n")
var job any job := entry.Data["job"]
if f.logPrefixJobID {
job = entry.Data["jobID"]
} else {
job = entry.Data["job"]
}
debugFlag := "" debugFlag := ""
if entry.Level == logrus.DebugLevel { if entry.Level == logrus.DebugLevel {
debugFlag = "[DEBUG] " debugFlag = "[DEBUG] "
} }
if entry.Data[rawOutputField] == true { switch {
case entry.Data[rawOutputField] == true:
fmt.Fprintf(b, "[%s] | %s", job, entry.Message) fmt.Fprintf(b, "[%s] | %s", job, entry.Message)
} else if entry.Data["dryrun"] == true { case entry.Data["dryrun"] == true:
fmt.Fprintf(b, "*DRYRUN* [%s] %s%s", job, debugFlag, entry.Message) fmt.Fprintf(b, "*DRYRUN* [%s] %s%s", job, debugFlag, entry.Message)
} else { default:
fmt.Fprintf(b, "[%s] %s%s", job, debugFlag, entry.Message) fmt.Fprintf(b, "[%s] %s%s", job, debugFlag, entry.Message)
} }
} }
@@ -434,3 +452,39 @@ func checkIfTerminal(w io.Writer) bool {
return false return false
} }
} }
// maskSecrets hides this job's secrets in a value that reaches somewhere the log maskers cannot,
// such as a container name or a job summary. Masks added at runtime count, so a summary written
// after ::add-mask:: is covered too.
func (rc *RunContext) maskSecrets(value string) string {
oldnew := rc.Config.maskers()
for _, mask := range rc.Masks {
oldnew = AppendSecretMasker(oldnew, mask)
}
return NewSecretReplacer(oldnew).Replace(value)
}
// maskers is every value this job's configuration says to hide, whatever the sink.
func (c *Config) maskers() []string {
oldnew := AppendSecretMaskers(nil, c.Secrets)
for _, mask := range c.ExtraMasks {
oldnew = AppendSecretMasker(oldnew, mask)
}
return oldnew
}
// NewSecretReplacer masks the longest secret first. Replacer matches in argument order, so a
// secret that prefixes another would otherwise mask only that prefix and print the rest.
func NewSecretReplacer(oldnew []string) *strings.Replacer {
pairs := make([][2]string, 0, len(oldnew)/2)
for i := 0; i+1 < len(oldnew); i += 2 {
pairs = append(pairs, [2]string{oldnew[i], oldnew[i+1]})
}
slices.SortFunc(pairs, func(a, b [2]string) int { return len(b[0]) - len(a[0]) })
sorted := make([]string, 0, len(pairs)*2)
for _, pair := range pairs {
sorted = append(sorted, pair[0], pair[1])
}
return strings.NewReplacer(sorted...)
}
+48 -34
View File
@@ -47,7 +47,7 @@ func TestValueMasker(t *testing.T) {
for _, entry := range table { for _, entry := range table {
t.Run(entry.name, func(t *testing.T) { t.Run(entry.name, func(t *testing.T) {
ctx := WithMasks(t.Context(), &entry.masks) ctx := WithMasks(t.Context(), &entry.masks)
masker := valueMasker(false, entry.secrets) masker := valueMasker(false, AppendSecretMaskers(nil, entry.secrets))
for line := range strings.SplitSeq(entry.lines, "\n") { for line := range strings.SplitSeq(entry.lines, "\n") {
lentry := masker(&logrus.Entry{ lentry := masker(&logrus.Entry{
Context: ctx, Context: ctx,
@@ -64,26 +64,27 @@ func TestValueMasker(t *testing.T) {
// A secret that reaches the log through an encoding — a base64 payload, a JSON body, a // A secret that reaches the log through an encoding — a base64 payload, a JSON body, a
// URL — must be masked as well: masking only the verbatim value leaks it. // URL — must be masked as well: masking only the verbatim value leaks it.
func TestValueMaskerEncodedSecrets(t *testing.T) { func TestValueMaskerEncodedSecrets(t *testing.T) {
secret := `p@ss w"rd/1`
masker := valueMasker(false, map[string]string{"TOKEN": secret})
for _, tc := range []struct { for _, tc := range []struct {
name string name, secret string
line string encoded []string
}{ }{
{"verbatim", "the token is " + secret}, {"common encodings", `p@ss w"rd/1`, []string{
{"base64", "Authorization: Basic " + base64.StdEncoding.EncodeToString([]byte(secret))}, `p@ss w"rd/1`, base64.StdEncoding.EncodeToString([]byte(`p@ss w"rd/1`)),
{"json", `{"token":"` + jsonStringEscape(secret) + `"}`}, jsonStringEscape(`p@ss w"rd/1`), url.PathEscape(`p@ss w"rd/1`),
{"query escaped", "https://example.com/?token=" + url.QueryEscape(secret)}, }},
{"path escaped", "https://example.com/" + url.PathEscape(secret) + "/x"}, {"XML expression and quotes", `"a'b&c<d>"`, []string{
`&quot;a&apos;b&amp;c&lt;d&gt;&quot;`, `"a''b&c<d>"`, `a'b&c<d>`,
}},
{"URI spaces", "a b", []string{"a%20b", "a+b"}},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
entry := masker(&logrus.Entry{Context: t.Context(), Message: tc.line}) masker := valueMasker(false, AppendSecretMaskers(nil, map[string]string{"TOKEN": tc.secret}))
entry := masker(&logrus.Entry{Context: t.Context(), Message: strings.Join(tc.encoded, " ")})
assert.Contains(t, entry.Message, "***") assert.Contains(t, entry.Message, "***")
assert.NotContains(t, entry.Message, secret) for _, disallowed := range tc.encoded {
assert.NotContains(t, entry.Message, base64.StdEncoding.EncodeToString([]byte(secret))) assert.NotContains(t, entry.Message, disallowed)
assert.NotContains(t, entry.Message, url.QueryEscape(secret)) }
}) })
} }
} }
@@ -94,7 +95,7 @@ func TestValueMaskerEncodedSecrets(t *testing.T) {
// form, so a JS-serialized JSON body does not leak it. // form, so a JS-serialized JSON body does not leak it.
func TestValueMaskerJSONEscapesBothWays(t *testing.T) { func TestValueMaskerJSONEscapesBothWays(t *testing.T) {
secret := `a"<b>&c` secret := `a"<b>&c`
masker := valueMasker(false, map[string]string{"TOKEN": secret}) masker := valueMasker(false, AppendSecretMaskers(nil, map[string]string{"TOKEN": secret}))
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -112,17 +113,33 @@ func TestValueMaskerJSONEscapesBothWays(t *testing.T) {
} }
} }
// With debug logging on, the job logger writes to stdout, which no reporter masks, so it has
// to hide the values that are not job secrets too.
func TestValueMaskerHidesExtraMasks(t *testing.T) {
masker := valueMasker(false, (&Config{ExtraMasks: []string{"pr0xypw"}}).maskers())
entry := masker(&logrus.Entry{Context: t.Context(), Message: "proxy is http://user:pr0xypw@proxy:3128"})
assert.Equal(t, "proxy is http://user:***@proxy:3128", entry.Message)
}
// ::add-mask:: values go through the same masker, so they get the same treatment. // ::add-mask:: values go through the same masker, so they get the same treatment.
func TestValueMaskerEncodedMasks(t *testing.T) { func TestValueMaskerEncodedMasks(t *testing.T) {
masks := []string{"s3cr3t value"} masks := []string{"s3cr3t value", "first\rsecond", " s3cr3t "}
masker := valueMasker(false, nil) masker := valueMasker(false, AppendSecretMaskers(nil, nil))
entry := masker(&logrus.Entry{ for _, tc := range []struct {
Context: WithMasks(t.Context(), &masks), line, want string
Message: "encoded: " + base64.StdEncoding.EncodeToString([]byte("s3cr3t value")), }{
}) {"encoded: " + base64.StdEncoding.EncodeToString([]byte("s3cr3t value")), "encoded: ***"},
{"first and second", "*** and ***"},
assert.Equal(t, "encoded: ***", entry.Message) {"encoded: " + base64.StdEncoding.EncodeToString([]byte(" s3cr3t ")), "encoded: ***"},
{"encoded: " + base64.StdEncoding.EncodeToString([]byte("s3cr3t")), "encoded: ***"},
{"encoded: " + base64.StdEncoding.EncodeToString([]byte("first")), "encoded: ***"},
} {
entry := masker(&logrus.Entry{Context: WithMasks(t.Context(), &masks), Message: tc.line})
assert.Equal(t, tc.want, entry.Message, tc.line)
}
} }
// A token in a Basic auth header is base64'd together with the user name, so the token's // A token in a Basic auth header is base64'd together with the user name, so the token's
@@ -130,8 +147,8 @@ func TestValueMaskerEncodedMasks(t *testing.T) {
// alignments must be masked as well, or `Authorization: Basic base64("user:token")` leaks // alignments must be masked as well, or `Authorization: Basic base64("user:token")` leaks
// the token to anyone who can decode the log. // the token to anyone who can decode the log.
func TestValueMaskerBase64Alignments(t *testing.T) { func TestValueMaskerBase64Alignments(t *testing.T) {
secret := "s3cr3t-token-value" secret := "s3cr3t"
masker := valueMasker(false, map[string]string{"TOKEN": secret}) masker := valueMasker(false, AppendSecretMaskers(nil, map[string]string{"TOKEN": secret}))
// One prefix per alignment: len%3 of 0, 1 and 2. // One prefix per alignment: len%3 of 0, 1 and 2.
for _, prefix := range []string{"x-access-token:", "user:", "ab:"} { for _, prefix := range []string{"x-access-token:", "user:", "ab:"} {
@@ -140,8 +157,6 @@ func TestValueMaskerBase64Alignments(t *testing.T) {
entry := masker(&logrus.Entry{Context: t.Context(), Message: "Authorization: Basic " + encoded}) entry := masker(&logrus.Entry{Context: t.Context(), Message: "Authorization: Basic " + encoded})
assert.Contains(t, entry.Message, "***") assert.Contains(t, entry.Message, "***")
// The aligned middle of the secret must be gone, so the payload can no longer be
// decoded back into the token.
assert.NotEqual(t, "Authorization: Basic "+encoded, entry.Message) assert.NotEqual(t, "Authorization: Basic "+encoded, entry.Message)
decodable := strings.TrimPrefix(entry.Message, "Authorization: Basic ") decodable := strings.TrimPrefix(entry.Message, "Authorization: Basic ")
decoded, err := base64.StdEncoding.DecodeString(decodable) decoded, err := base64.StdEncoding.DecodeString(decodable)
@@ -155,7 +170,7 @@ func TestValueMaskerBase64Alignments(t *testing.T) {
// The masker caches its replacer, so it has to notice both a mask appended to the same // The masker caches its replacer, so it has to notice both a mask appended to the same
// slice and a composite action logging with a slice of its own. // slice and a composite action logging with a slice of its own.
func TestValueMaskerCachedReplacerSeesNewMasks(t *testing.T) { func TestValueMaskerCachedReplacerSeesNewMasks(t *testing.T) {
masker := valueMasker(false, map[string]string{"TOKEN": "secret-token"}) masker := valueMasker(false, AppendSecretMaskers(nil, map[string]string{"TOKEN": "secret-token"}))
mask := func(masks *[]string, message string) string { mask := func(masks *[]string, message string) string {
return masker(&logrus.Entry{Context: WithMasks(t.Context(), masks), Message: message}).Message return masker(&logrus.Entry{Context: WithMasks(t.Context(), masks), Message: message}).Message
} }
@@ -180,15 +195,14 @@ func TestAppendSecretMaskerSkipsUselessEncodings(t *testing.T) {
// JSON, query and path escaping all leave it unchanged. // JSON, query and path escaping all leave it unchanged.
pairs := AppendSecretMasker(nil, "plaintoken") pairs := AppendSecretMasker(nil, "plaintoken")
assert.Equal(t, []string{ assert.Equal(t, []string{
"plaintoken", "***", "plaintoken", "***", "cGxhaW50b2tlbg==", "***", "bGFpbnRva2Vu", "***", "YWludG9rZW4=", "***",
base64.StdEncoding.EncodeToString([]byte("plaintoken")), "***", "aW50b2tl", "***", "YWludG9r", "***", "bGFpbnRv", "***",
// The two shifted alignments, each without its leading and trailing group.
"YWludG9r", "***",
"bGFpbnRv", "***",
}, pairs) }, pairs)
// Too short to mask. // Too short to mask.
assert.Empty(t, AppendSecretMasker(nil, "x")) assert.Empty(t, AppendSecretMasker(nil, "x"))
assert.Empty(t, AppendSecretMasker(nil, " \t"))
assert.NotContains(t, AppendSecretMasker(nil, `"123456"`), "123456")
} }
func TestJobLogFormatterDecodesCommandData(t *testing.T) { func TestJobLogFormatterDecodesCommandData(t *testing.T) {
+18
View File
@@ -8,6 +8,7 @@ import (
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.yaml.in/yaml/v4" "go.yaml.in/yaml/v4"
) )
@@ -64,3 +65,20 @@ func TestMaxParallelStrategy(t *testing.T) {
}) })
} }
} }
func TestNewPlanExecutorInvalidMatrix(t *testing.T) {
var rawMatrix yaml.Node
require.NoError(t, rawMatrix.Encode(map[string]any{
"config": map[string]any{"nested": "value"},
}))
plan := &model.Plan{Stages: []*model.Stage{{Runs: []*model.Run{{
Workflow: &model.Workflow{Jobs: map[string]*model.Job{
"test": {Strategy: &model.Strategy{RawMatrix: rawMatrix}},
}},
JobID: "test",
}}}}}
runner := &runnerImpl{config: &Config{}}
require.ErrorContains(t, runner.NewPlanExecutor(plan)(t.Context()), "could not get job matrix:")
}
+1 -1
View File
@@ -244,7 +244,7 @@ func TestPatchActionsAtTheContainerCopy(t *testing.T) {
require.NoError(t, os.WriteFile(script, []byte(gateTSC), 0o600)) require.NoError(t, os.WriteFile(script, []byte(gateTSC), 0o600))
var copied string var copied string
cm.On("CopyDir", mock.Anything, mock.Anything, mock.Anything).Return(func(context.Context) error { cm.On("CopyDir", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(func(context.Context) error {
body, err := os.ReadFile(script) body, err := os.ReadFile(script)
require.NoError(t, err) require.NoError(t, err)
copied = string(body) copied = string(body)
+13 -61
View File
@@ -5,7 +5,6 @@
package runner package runner
import ( import (
"archive/tar"
"context" "context"
"fmt" "fmt"
"net/url" "net/url"
@@ -78,10 +77,6 @@ func newRemoteReusableWorkflowExecutor(rc *RunContext) common.Executor {
filename := fmt.Sprintf("%s/%s@%s", remoteReusableWorkflow.Org, remoteReusableWorkflow.Repo, remoteReusableWorkflow.Ref) filename := fmt.Sprintf("%s/%s@%s", remoteReusableWorkflow.Org, remoteReusableWorkflow.Repo, remoteReusableWorkflow.Ref)
workflowDir := fmt.Sprintf("%s/%s", rc.ActionCacheDir(), safeFilename(filename)) workflowDir := fmt.Sprintf("%s/%s", rc.ActionCacheDir(), safeFilename(filename))
if rc.Config.ActionCache != nil {
return newActionCacheReusableWorkflowExecutor(rc, filename, remoteReusableWorkflow)
}
token := getGitCloneToken(rc.Config, remoteReusableWorkflow.CloneURL()) token := getGitCloneToken(rc.Config, remoteReusableWorkflow.CloneURL())
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
@@ -90,41 +85,6 @@ func newRemoteReusableWorkflowExecutor(rc *RunContext) common.Executor {
) )
} }
func newActionCacheReusableWorkflowExecutor(rc *RunContext, filename string, remoteReusableWorkflow *remoteReusableWorkflow) common.Executor {
return func(ctx context.Context) error {
ghctx := rc.getGithubContext(ctx)
remoteReusableWorkflow.URL = ghctx.ServerURL
sha, err := rc.Config.ActionCache.Fetch(ctx, filename, remoteReusableWorkflow.CloneURL(), remoteReusableWorkflow.Ref, ghctx.Token)
if err != nil {
return err
}
archive, err := rc.Config.ActionCache.GetTarArchive(ctx, filename, sha, ".github/workflows/"+remoteReusableWorkflow.Filename)
if err != nil {
return err
}
defer archive.Close()
treader := tar.NewReader(archive)
if _, err = treader.Next(); err != nil {
return err
}
planner, err := model.NewSingleWorkflowPlanner(remoteReusableWorkflow.Filename, treader)
if err != nil {
return err
}
plan, err := planner.PlanEvent("workflow_call")
if err != nil {
return err
}
runner, err := NewReusableWorkflowRunner(rc)
if err != nil {
return err
}
return runner.NewPlanExecutor(plan)(ctx)
}
}
// cloneRemoteReusableWorkflow always invokes the clone executor — moving refs // cloneRemoteReusableWorkflow always invokes the clone executor — moving refs
// (branches, tags) must be re-resolved each run, matching GitHub Actions. // (branches, tags) must be re-resolved each run, matching GitHub Actions.
// //
@@ -135,9 +95,12 @@ func newActionCacheReusableWorkflowExecutor(rc *RunContext, filename string, rem
// remoteReusableWorkflow.URL = rc.getGithubContext(ctx).ServerURL // remoteReusableWorkflow.URL = rc.getGithubContext(ctx).ServerURL
func cloneRemoteReusableWorkflow(rc *RunContext, cloneURL, ref, targetDirectory, token string) common.Executor { func cloneRemoteReusableWorkflow(rc *RunContext, cloneURL, ref, targetDirectory, token string) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
cloneURL = rc.NewExpressionEvaluator(ctx).Interpolate(ctx, cloneURL) interpolatedURL, err := rc.NewExpressionEvaluator(ctx).Interpolate(ctx, cloneURL)
if err != nil {
return fmt.Errorf("unable to interpolate the workflow clone URL: %w", err)
}
return git.NewGitCloneExecutor(git.NewGitCloneExecutorInput{ return git.NewGitCloneExecutor(git.NewGitCloneExecutorInput{
URL: cloneURL, URL: interpolatedURL,
Ref: ref, Ref: ref,
Dir: targetDirectory, Dir: targetDirectory,
Token: token, Token: token,
@@ -147,15 +110,12 @@ func cloneRemoteReusableWorkflow(rc *RunContext, cloneURL, ref, targetDirectory,
} }
} }
var modelNewWorkflowPlanner = model.NewWorkflowPlanner
func newReusableWorkflowExecutor(rc *RunContext, directory, workflow string) common.Executor { func newReusableWorkflowExecutor(rc *RunContext, directory, workflow string) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
// Scoped to the yaml read so concurrent invocations don't serialize // Serialize workflow reads with cache updates.
// on the whole job run.
planner, err := func() (model.WorkflowPlanner, error) { planner, err := func() (model.WorkflowPlanner, error) {
defer git.AcquireCloneLock(directory)() defer git.AcquireCloneLock(directory)()
return modelNewWorkflowPlanner(path.Join(directory, workflow), true) return model.NewWorkflowPlanner(path.Join(directory, workflow), true)
}() }()
if err != nil { if err != nil {
return err return err
@@ -166,12 +126,11 @@ func newReusableWorkflowExecutor(rc *RunContext, directory, workflow string) com
return err return err
} }
runner, err := NewReusableWorkflowRunner(rc) runner, err := newReusableWorkflowRunner(rc)
if err != nil { if err != nil {
return err return err
} }
// return runner.NewPlanExecutor(plan)(ctx)
return common.NewPipelineExecutor( // For Gitea return common.NewPipelineExecutor( // For Gitea
runner.NewPlanExecutor(plan), runner.NewPlanExecutor(plan),
setReusedWorkflowCallerResult(rc, runner), setReusedWorkflowCallerResult(rc, runner),
@@ -179,7 +138,7 @@ func newReusableWorkflowExecutor(rc *RunContext, directory, workflow string) com
} }
} }
func NewReusableWorkflowRunner(rc *RunContext) (Runner, error) { func newReusableWorkflowRunner(rc *RunContext) (*runnerImpl, error) {
runner := &runnerImpl{ runner := &runnerImpl{
config: rc.Config, config: rc.Config,
eventJSON: rc.EventJSON, eventJSON: rc.EventJSON,
@@ -255,16 +214,9 @@ func newRemoteReusableWorkflowFromAbsoluteURL(uses string) *remoteReusableWorkfl
} }
// For Gitea // For Gitea
func setReusedWorkflowCallerResult(rc *RunContext, runner Runner) common.Executor { func setReusedWorkflowCallerResult(rc *RunContext, runner *runnerImpl) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := common.Logger(ctx) caller := runner.caller
runnerImpl, ok := runner.(*runnerImpl)
if !ok {
logger.Warn("Failed to get caller from runner")
return nil
}
caller := runnerImpl.caller
allJobDone := true allJobDone := true
hasFailure := false hasFailure := false
@@ -287,14 +239,14 @@ func setReusedWorkflowCallerResult(rc *RunContext, runner Runner) common.Executo
} }
if rc.caller != nil { if rc.caller != nil {
rc.caller.setReusedWorkflowJobResult(rc.JobName, reusedWorkflowJobResult) rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, reusedWorkflowJobResult)
} else { } else {
// Serialize this shared Job.Result write against the other matrix combos // Serialize this shared Job.Result write against the other matrix combos
// and setJobResult (same lockJob key). // and setJobResult (same lockJob key).
unlock := lockJob(rc.Run.Job()) unlock := lockJob(rc.Run.Job())
rc.result(reusedWorkflowJobResult) rc.result(reusedWorkflowJobResult)
unlock() unlock()
logger.WithField("jobResult", reusedWorkflowJobResult).Infof("Job %s", reusedWorkflowJobResultMessage) common.Logger(ctx).WithField("jobResult", reusedWorkflowJobResult).Infof("Job %s", reusedWorkflowJobResultMessage)
} }
} }
+3 -20
View File
@@ -5,7 +5,6 @@ package runner
import ( import (
"context" "context"
"errors"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
@@ -77,19 +76,11 @@ func TestReusableWorkflowCachedBranchRefRefreshes(t *testing.T) {
func TestNewReusableWorkflowExecutorHoldsCloneLock(t *testing.T) { func TestNewReusableWorkflowExecutorHoldsCloneLock(t *testing.T) {
workflowDir := t.TempDir() workflowDir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(workflowDir, "reusable.yml"), []byte(":"), 0o644))
unlockOnce := sync.OnceFunc(git.AcquireCloneLock(workflowDir)) unlockOnce := sync.OnceFunc(git.AcquireCloneLock(workflowDir))
defer unlockOnce() defer unlockOnce()
plannerCalled := make(chan struct{})
origPlanner := modelNewWorkflowPlanner
modelNewWorkflowPlanner = func(string, bool) (model.WorkflowPlanner, error) {
close(plannerCalled)
return nil, errors.New("stop")
}
defer func() { modelNewWorkflowPlanner = origPlanner }()
rc := &RunContext{ rc := &RunContext{
Config: &Config{}, Config: &Config{},
Run: &model.Run{Workflow: &model.Workflow{Jobs: map[string]*model.Job{}}}, Run: &model.Run{Workflow: &model.Workflow{Jobs: map[string]*model.Job{}}},
@@ -100,26 +91,18 @@ func TestNewReusableWorkflowExecutorHoldsCloneLock(t *testing.T) {
go func() { done <- exec(context.Background()) }() go func() { done <- exec(context.Background()) }()
select { select {
case <-plannerCalled:
t.Fatal("planner ran while clone lock was held")
case err := <-done: case err := <-done:
t.Fatalf("executor returned before planner was reached: %v", err) t.Fatalf("executor returned while clone lock was held: %v", err)
case <-time.After(50 * time.Millisecond): case <-time.After(50 * time.Millisecond):
} }
unlockOnce() unlockOnce()
select {
case <-plannerCalled:
case <-time.After(time.Second):
t.Fatal("planner not called after lock was released")
}
select { select {
case err := <-done: case err := <-done:
require.Error(t, err) require.Error(t, err)
case <-time.After(time.Second): case <-time.After(time.Second):
t.Fatal("executor did not return after planner ran") t.Fatal("executor did not return after lock was released")
} }
} }
+298 -243
View File
@@ -11,7 +11,7 @@ import (
"crypto/rand" "crypto/rand"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json/v2"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -37,6 +37,8 @@ import (
"github.com/moby/moby/api/types/mount" "github.com/moby/moby/api/types/mount"
"github.com/opencontainers/selinux/go-selinux" "github.com/opencontainers/selinux/go-selinux"
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
"golang.org/x/text/encoding/unicode"
"golang.org/x/text/transform"
) )
// RunContext contains info about current job // RunContext contains info about current job
@@ -56,16 +58,20 @@ type RunContext struct {
CurrentStepIndex int CurrentStepIndex int
StepResults map[string]*model.StepResult StepResults map[string]*model.StepResult
IntraActionState map[string]map[string]string IntraActionState map[string]map[string]string
ExprEval ExpressionEvaluator ExprEval *expressionEvaluator
JobContainer container.ExecutionsEnvironment JobContainer container.ExecutionsEnvironment
serviceContainers []*serviceContainer serviceContainers []*serviceContainer
OutputMappings map[MappableOutput]MappableOutput
JobName string JobName string
ActionPath string ActionPath string
Parent *RunContext Parent *RunContext
actionInputs map[string]any // inputs of the composite action this runs, nil for a job
Masks []string Masks []string
cleanUpJobContainer common.Executor cleanUpJobContainer common.Executor
caller *caller // job calling this RunContext (reusable workflows) caller *caller // job calling this RunContext (reusable workflows)
workflowCallInputs map[string]any // the caller's with:, resolved once by resolveWorkflowCall
workflowCallSecrets map[string]string // the caller's secrets:, resolved once by resolveWorkflowCall
jobRunDefaults model.RunDefaults // defaults.run, resolved once at job setup as GitHub does
platformImage string // container.image or the runs-on pick, resolved once by isEnabled
// summaryFileInitialized tracks which per-step summary files (workflow/step-summary-N.md) // summaryFileInitialized tracks which per-step summary files (workflow/step-summary-N.md)
// have already been created on the JobContainer. The runner sets up file-command files // have already been created on the JobContainer. The runner sets up file-command files
// via JobContainer.Copy at the start of every phase, which truncates them — fine for // via JobContainer.Copy at the start of every phase, which truncates them — fine for
@@ -88,7 +94,9 @@ type RunContext struct {
jobFailed bool jobFailed bool
// empty for a host-mode job, which starts no container // empty for a host-mode job, which starts no container
jobContainerID string jobContainerID string
hasBash *bool // memoized implicit-shell probe, only set on the top-level RunContext
jobNetworkName string jobNetworkName string
dockerProxy *container.DockerProxy
// stepEnv is a copy of the running step's environment, so that workflow commands parsed out // stepEnv is a copy of the running step's environment, so that workflow commands parsed out
// of the container's output can be judged against it. Written by runStepExecutor and read on // of the container's output can be judged against it. Written by runStepExecutor and read on
// the log-writer goroutine, hence unsecureCommandMu, which also guards unsecureCommandErr. // the log-writer goroutine, hence unsecureCommandMu, which also guards unsecureCommandErr.
@@ -148,11 +156,6 @@ func (rc *RunContext) AddMask(mask string) {
rc.Masks = append(rc.Masks, mask) rc.Masks = append(rc.Masks, mask)
} }
type MappableOutput struct {
StepID string
OutputName string
}
func (rc *RunContext) String() string { func (rc *RunContext) String() string {
name := fmt.Sprintf("%s/%s", rc.Run.Workflow.Name, rc.Name) name := fmt.Sprintf("%s/%s", rc.Run.Workflow.Name, rc.Name)
if rc.caller != nil { if rc.caller != nil {
@@ -185,8 +188,16 @@ func (rc *RunContext) GetEnv() map[string]string {
return rc.Env return rc.Env
} }
// setActionEnv sets a composite action's env, keeping the `inputs` context it derives from
// in sync. Remote actions re-evaluate it per stage, so inputs may change between them.
func (rc *RunContext) setActionEnv(env map[string]string) {
rc.Env = env
rc.actionInputs = inputsFromEnv(env)
}
func (rc *RunContext) jobContainerName() string { func (rc *RunContext) jobContainerName() string {
nameParts := []string{rc.Config.ContainerNamePrefix, "WORKFLOW-" + rc.Run.Workflow.Name, "JOB-" + rc.Name} // The job id, never evaluated, keeps two jobs apart when masking collapses their names.
nameParts := []string{rc.Config.ContainerNamePrefix, "WORKFLOW-" + rc.Run.Workflow.Name, "JOB-" + rc.Run.JobID, rc.Name}
if rc.caller != nil { if rc.caller != nil {
nameParts = append(nameParts, "CALLED-BY-"+rc.caller.runContext.JobName) nameParts = append(nameParts, "CALLED-BY-"+rc.caller.runContext.JobName)
} }
@@ -204,14 +215,15 @@ func (rc *RunContext) networkNameForGitea() (string, bool) {
func getDockerDaemonSocketMountPath(daemonPath string) string { func getDockerDaemonSocketMountPath(daemonPath string) string {
if before, after, ok := strings.Cut(daemonPath, "://"); ok { if before, after, ok := strings.Cut(daemonPath, "://"); ok {
scheme := before scheme := before
if strings.EqualFold(scheme, "npipe") { switch {
case strings.EqualFold(scheme, "npipe"):
// linux container mount on windows, use the default socket path of the VM / wsl2 // linux container mount on windows, use the default socket path of the VM / wsl2
return "/var/run/docker.sock" return "/var/run/docker.sock"
} else if strings.EqualFold(scheme, "unix") { case strings.EqualFold(scheme, "unix"):
return after return after
} else if strings.IndexFunc(scheme, func(r rune) bool { case strings.IndexFunc(scheme, func(r rune) bool {
return (r < 'a' || r > 'z') && (r < 'A' || r > 'Z') return (r < 'a' || r > 'z') && (r < 'A' || r > 'Z')
}) == -1 { }) == -1:
// unknown protocol use default // unknown protocol use default
return "/var/run/docker.sock" return "/var/run/docker.sock"
} }
@@ -231,20 +243,39 @@ func (rc *RunContext) containerDaemonSocket() string {
const sharedToolCacheVolume = "act-toolcache" // mounted only when the tool cache is shared const sharedToolCacheVolume = "act-toolcache" // mounted only when the tool cache is shared
// validVolumes returns the volumes allowed on this job's containers: the configured base // validVolumes returns what the job and action containers may mount, the configured base plus
// plus the volumes the runner mounts automatically. It derives a fresh slice every call and // the runner's own volumes. Fresh slice per call, the shared Config is never mutated.
// never mutates the shared Config (see containerDaemonSocket).
func (rc *RunContext) validVolumes() []string { func (rc *RunContext) validVolumes() []string {
name := rc.jobContainerName() name := rc.jobContainerName()
volumes := slices.Clone(rc.Config.ValidVolumes) volumes := slices.Clone(rc.Config.ValidVolumes)
if rc.Config.SharedToolCache { if rc.Config.SharedToolCache {
volumes = append(volumes, sharedToolCacheVolume) volumes = append(volumes, sharedToolCacheVolume)
} }
if rc.Config.BindWorkdir {
volumes = append(volumes, rc.Config.Workdir)
}
if rc.dockerProxy != nil {
volumes = append(volumes, rc.dockerProxy.Socket)
}
// TODO: add a new configuration to control whether the docker daemon can be mounted // TODO: add a new configuration to control whether the docker daemon can be mounted
return append(volumes, name, name+"-env", return append(volumes, name, name+"-env",
getDockerDaemonSocketMountPath(rc.containerDaemonSocket())) getDockerDaemonSocketMountPath(rc.containerDaemonSocket()))
} }
func (rc *RunContext) jobDockerSocket() string {
if rc.dockerProxy != nil {
return rc.dockerProxy.Socket
}
return getDockerDaemonSocketMountPath(rc.containerDaemonSocket())
}
func (rc *RunContext) startDockerProxy(ctx context.Context) {
if !filepath.IsAbs(strings.TrimPrefix(rc.containerDaemonSocket(), "unix://")) || common.Dryrun(ctx) {
return
}
rc.dockerProxy = container.NewDockerProxy(ctx, rc.jobContainerName())
}
// toolCache returns the tool cache path the job sees, relocatable through RUNNER_TOOL_CACHE. // toolCache returns the tool cache path the job sees, relocatable through RUNNER_TOOL_CACHE.
func (rc *RunContext) toolCache(fallback string) string { func (rc *RunContext) toolCache(fallback string) string {
if path := rc.GetEnv()["RUNNER_TOOL_CACHE"]; path != "" { if path := rc.GetEnv()["RUNNER_TOOL_CACHE"]; path != "" {
@@ -279,7 +310,7 @@ func splitVolumes(specs []string) ([]string, map[string]string, map[string]bool)
for _, spec := range specs { for _, spec := range specs {
parsed, err := loader.ParseVolume(spec) parsed, err := loader.ParseVolume(spec)
if err != nil { if err != nil {
binds = append(binds, spec) // let Docker report the malformed spec binds = append(binds, spec) // unclassifiable, sanitizeConfig warns and drops it
continue continue
} }
targets[parsed.Target] = true targets[parsed.Target] = true
@@ -293,7 +324,7 @@ func splitVolumes(specs []string) ([]string, map[string]string, map[string]bool)
} }
// Returns the binds and mounts for the container, resolving paths as appopriate // Returns the binds and mounts for the container, resolving paths as appopriate
func (rc *RunContext) GetBindsAndMounts() ([]string, map[string]string) { func (rc *RunContext) GetBindsAndMounts() ([]string, map[string]string, error) {
name := rc.jobContainerName() name := rc.jobContainerName()
ext := container.LinuxContainerEnvironmentExtensions{} ext := container.LinuxContainerEnvironmentExtensions{}
@@ -302,7 +333,10 @@ func (rc *RunContext) GetBindsAndMounts() ([]string, map[string]string) {
if container := job.Container(); container != nil { if container := job.Container(); container != nil {
for _, v := range container.Volumes { for _, v := range container.Volumes {
if rc.ExprEval != nil { if rc.ExprEval != nil {
v = rc.ExprEval.Interpolate(context.Background(), v) var err error
if v, err = rc.ExprEval.Interpolate(context.Background(), v); err != nil {
return nil, nil, fmt.Errorf("unable to interpolate container.volumes: %w", err)
}
} }
volumes = append(volumes, v) volumes = append(volumes, v)
} }
@@ -311,8 +345,8 @@ func (rc *RunContext) GetBindsAndMounts() ([]string, map[string]string) {
// the runner's own mounts below yield to the targets the job claims // the runner's own mounts below yield to the targets the job claims
binds, mounts, claimed := splitVolumes(volumes) binds, mounts, claimed := splitVolumes(volumes)
if daemonSocket := rc.containerDaemonSocket(); daemonSocket != "-" && !claimed["/var/run/docker.sock"] { if rc.containerDaemonSocket() != "-" && !claimed["/var/run/docker.sock"] {
binds = append(binds, getDockerDaemonSocketMountPath(daemonSocket)+":/var/run/docker.sock") binds = append(binds, rc.jobDockerSocket()+":/var/run/docker.sock")
} }
if rc.Config.SharedToolCache { if rc.Config.SharedToolCache {
if toolCache := rc.toolCache(container.DefaultToolCache); !claimed[toolCache] { if toolCache := rc.toolCache(container.DefaultToolCache); !claimed[toolCache] {
@@ -336,21 +370,12 @@ func (rc *RunContext) GetBindsAndMounts() ([]string, map[string]string) {
} }
} }
return binds, mounts return binds, mounts, nil
} }
func (rc *RunContext) startHostEnvironment() common.Executor { func (rc *RunContext) startHostEnvironment() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := common.Logger(ctx) logWriter := rc.commandLogWriter(ctx)
rawLogger := logger.WithField(rawOutputField, true)
logWriter := common.NewLineWriter(rc.commandHandler(ctx), func(s string) bool {
if rc.Config.LogOutput {
rawLogger.Infof("%s", s)
} else {
rawLogger.Debugf("%s", s)
}
return true
})
cacheDir := rc.ActionCacheDir() cacheDir := rc.ActionCacheDir()
randBytes := make([]byte, 8) randBytes := make([]byte, 8)
_, _ = rand.Read(randBytes) _, _ = rand.Read(randBytes)
@@ -436,16 +461,8 @@ var newContainer = container.NewContainer
func (rc *RunContext) startJobContainer() common.Executor { func (rc *RunContext) startJobContainer() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
image := rc.platformImage(ctx) image := rc.platformImage
rawLogger := logger.WithField(rawOutputField, true) logWriter := rc.commandLogWriter(ctx)
logWriter := common.NewLineWriter(rc.commandHandler(ctx), func(s string) bool {
if rc.Config.LogOutput {
rawLogger.Infof("%s", s)
} else {
rawLogger.Debugf("%s", s)
}
return true
})
username, password, err := rc.handleCredentials(ctx) username, password, err := rc.handleCredentials(ctx)
if err != nil { if err != nil {
@@ -462,19 +479,20 @@ func (rc *RunContext) startJobContainer() common.Executor {
envList = append(envList, rc.runnerEnv(ctx)...) envList = append(envList, rc.runnerEnv(ctx)...)
envList = append(envList, fmt.Sprintf("%s=%s", "LANG", "C.UTF-8")) // Use same locale as GitHub Actions envList = append(envList, fmt.Sprintf("%s=%s", "LANG", "C.UTF-8")) // Use same locale as GitHub Actions
ext := container.LinuxContainerEnvironmentExtensions{}
binds, mounts := rc.GetBindsAndMounts()
// specify the network to which the container will connect when `docker create` stage. (like execute command line: docker create --network <networkName> <image>) // specify the network to which the container will connect when `docker create` stage. (like execute command line: docker create --network <networkName> <image>)
// if using service containers, will create a new network for the containers. // if using service containers, will create a new network for the containers.
// and it will be removed after at last. // and it will be removed after at last.
networkName, createAndDeleteNetwork := rc.networkNameForGitea() networkName, createAndDeleteNetwork := rc.networkNameForGitea()
rc.cleanUpJobContainer = rc.cleanupJobResources(networkName, createAndDeleteNetwork)
// add service containers // add service containers
for serviceID, spec := range rc.Run.Job().Services { for serviceID, spec := range rc.Run.Job().Services {
// GitHub compatibility: skip services whose image evaluates to an // GitHub compatibility: skip services whose image evaluates to an
// empty string, enabling conditional services via expressions // empty string, enabling conditional services via expressions
serviceImage := rc.ExprEval.Interpolate(ctx, spec.Image) serviceImage, err := rc.ExprEval.Interpolate(ctx, spec.Image)
if err != nil {
return fmt.Errorf("unable to interpolate service %s image: %w", serviceID, err)
}
if serviceImage == "" { if serviceImage == "" {
logger.Infof("The service '%s' will not be started because the container definition has an empty image.", serviceID) logger.Infof("The service '%s' will not be started because the container definition has an empty image.", serviceID)
continue continue
@@ -484,105 +502,128 @@ func (rc *RunContext) startJobContainer() common.Executor {
// a service reaches the internet the way the job does; its own env still wins // a service reaches the internet the way the job does; its own env still wins
maps0.Copy(interpolatedEnvs, rc.Config.ProxyEnv) maps0.Copy(interpolatedEnvs, rc.Config.ProxyEnv)
for k, v := range spec.Env { for k, v := range spec.Env {
interpolatedEnvs[k] = rc.ExprEval.Interpolate(ctx, v) if interpolatedEnvs[k], err = rc.ExprEval.Interpolate(ctx, v); err != nil {
return fmt.Errorf("unable to interpolate service %s env %s: %w", serviceID, k, err)
}
} }
envs := make([]string, 0, len(interpolatedEnvs)) envs := make([]string, 0, len(interpolatedEnvs))
for k, v := range interpolatedEnvs { for k, v := range interpolatedEnvs {
envs = append(envs, fmt.Sprintf("%s=%s", k, v)) envs = append(envs, fmt.Sprintf("%s=%s", k, v))
} }
// interpolate cmd // interpolate cmd
interpolatedCmd := make([]string, 0, len(spec.Cmd)) interpolatedCmd := make([]string, len(spec.Cmd))
for _, v := range spec.Cmd { for i, v := range spec.Cmd {
interpolatedCmd = append(interpolatedCmd, rc.ExprEval.Interpolate(ctx, v)) if interpolatedCmd[i], err = rc.ExprEval.Interpolate(ctx, v); err != nil {
return fmt.Errorf("unable to interpolate service %s command: %w", serviceID, err)
}
} }
// keep these local: reusing username/password would overwrite the // keep these local: reusing username/password would overwrite the
// credentials the job container is pulled with further down // credentials the job container is pulled with further down
serviceUsername, servicePassword, err := rc.handleServiceCredentials(ctx, spec.Credentials) serviceUsername, servicePassword, err := rc.interpolateCredentials(ctx, spec.Credentials, "")
if err != nil { if err != nil {
return fmt.Errorf("failed to handle service %s credentials: %w", serviceID, err) return fmt.Errorf("failed to handle service %s credentials: %w", serviceID, err)
} }
interpolatedVolumes := make([]string, 0, len(spec.Volumes)) interpolatedVolumes := make([]string, len(spec.Volumes))
for _, volume := range spec.Volumes { for i, volume := range spec.Volumes {
interpolatedVolumes = append(interpolatedVolumes, rc.ExprEval.Interpolate(ctx, volume)) if interpolatedVolumes[i], err = rc.ExprEval.Interpolate(ctx, volume); err != nil {
return fmt.Errorf("unable to interpolate service %s volumes: %w", serviceID, err)
}
} }
serviceBinds, serviceMounts := rc.GetServiceBindsAndMounts(interpolatedVolumes) serviceBinds, serviceMounts, _ := splitVolumes(interpolatedVolumes)
interpolatedPorts := make([]string, 0, len(spec.Ports)) interpolatedPorts := make([]string, len(spec.Ports))
for _, port := range spec.Ports { for i, port := range spec.Ports {
interpolatedPorts = append(interpolatedPorts, rc.ExprEval.Interpolate(ctx, port)) if interpolatedPorts[i], err = rc.ExprEval.Interpolate(ctx, port); err != nil {
return fmt.Errorf("unable to interpolate service %s ports: %w", serviceID, err)
}
} }
exposedPorts, portBindings, err := nat.ParsePortSpecs(interpolatedPorts) exposedPorts, portBindings, err := nat.ParsePortSpecs(interpolatedPorts)
if err != nil { if err != nil {
return fmt.Errorf("failed to parse service %s ports: %w", serviceID, err) return fmt.Errorf("failed to parse service %s ports: %w", serviceID, err)
} }
serviceOptions, err := rc.ExprEval.Interpolate(ctx, spec.Options)
if err != nil {
return fmt.Errorf("unable to interpolate service %s options: %w", serviceID, err)
}
serviceContainerName := createContainerName(rc.jobContainerName(), serviceID) serviceContainerName := createContainerName(rc.jobContainerName(), serviceID)
c := newContainer(&container.NewContainerInput{ c := newContainer(&container.NewContainerInput{
Name: serviceContainerName, Name: serviceContainerName,
WorkingDir: ext.ToContainerPath(rc.Config.Workdir), Image: serviceImage,
Image: serviceImage, Username: serviceUsername,
Username: serviceUsername, Password: servicePassword,
Password: servicePassword, Cmd: interpolatedCmd,
Cmd: interpolatedCmd, Env: envs,
Env: envs, Mounts: serviceMounts,
Mounts: serviceMounts, Binds: serviceBinds,
Binds: serviceBinds, Stdout: logWriter,
Stdout: logWriter, Stderr: logWriter,
Stderr: logWriter, Privileged: rc.Config.Privileged,
Privileged: rc.Config.Privileged, UsernsMode: rc.Config.UsernsMode,
UsernsMode: rc.Config.UsernsMode, Platform: rc.Config.ContainerArchitecture,
Platform: rc.Config.ContainerArchitecture, AutoRemove: false, // so a dead service's log survives, cleanupJobResources removes it
AutoRemove: false, // so a dead service's log survives, cleanupJobResources removes it WorkflowOptions: serviceOptions,
Options: rc.ExprEval.Interpolate(ctx, spec.Options), NetworkMode: networkName,
NetworkMode: networkName, NetworkAliases: []string{serviceID},
NetworkAliases: []string{serviceID}, ExposedPorts: exposedPorts,
ExposedPorts: exposedPorts, PortBindings: portBindings,
PortBindings: portBindings, ValidVolumes: rc.Config.ValidVolumes, // not validVolumes(), a service gets no docker socket
AllocatePTY: rc.Config.AllocatePTY, AllocatePTY: rc.Config.AllocatePTY,
}) })
rc.serviceContainers = append(rc.serviceContainers, &serviceContainer{name: serviceID, image: serviceImage, container: c}) rc.serviceContainers = append(rc.serviceContainers, &serviceContainer{name: serviceID, image: serviceImage, container: c})
} }
rc.cleanUpJobContainer = rc.cleanupJobResources(networkName, createAndDeleteNetwork)
// For Gitea, `jobContainerNetwork` should be the same as `networkName` // For Gitea, `jobContainerNetwork` should be the same as `networkName`
jobContainerNetwork := networkName jobContainerNetwork := networkName
rc.JobContainer = newContainer(&container.NewContainerInput{ workflowOptions, err := rc.workflowOptions(ctx)
Cmd: nil, if err != nil {
Entrypoint: []string{"/bin/sleep", fmt.Sprint(rc.Config.ContainerMaxLifetime.Round(time.Second).Seconds())}, return err
WorkingDir: ext.ToContainerPath(rc.Config.Workdir),
Image: image,
Username: username,
Password: password,
Name: name,
Env: envList,
Mounts: mounts,
NetworkMode: jobContainerNetwork,
NetworkAliases: []string{rc.Name},
Binds: binds,
Stdout: logWriter,
Stderr: logWriter,
Privileged: rc.Config.Privileged,
UsernsMode: rc.Config.UsernsMode,
Platform: rc.Config.ContainerArchitecture,
Options: rc.options(ctx),
AutoRemove: rc.Config.AutoRemove,
ValidVolumes: rc.validVolumes(),
AllocatePTY: rc.Config.AllocatePTY,
})
if rc.JobContainer == nil {
return errors.New("Failed to create job container")
} }
ext := container.LinuxContainerEnvironmentExtensions{}
containerInput := &container.NewContainerInput{
Cmd: nil,
Entrypoint: []string{"/bin/sleep", fmt.Sprint(rc.Config.ContainerMaxLifetime.Round(time.Second).Seconds())},
WorkingDir: ext.ToContainerPath(rc.Config.Workdir),
Image: image,
Username: username,
Password: password,
Name: name,
Env: envList,
NetworkMode: jobContainerNetwork,
NetworkAliases: []string{rc.Name},
Stdout: logWriter,
Stderr: logWriter,
Privileged: rc.Config.Privileged,
UsernsMode: rc.Config.UsernsMode,
Platform: rc.Config.ContainerArchitecture,
RunnerOptions: rc.Config.ContainerOptions,
WorkflowOptions: workflowOptions,
AutoRemove: true,
AllocatePTY: rc.Config.AllocatePTY,
}
rc.JobContainer = newContainer(containerInput)
if rc.JobContainer == nil {
return errors.New("failed to create job container")
}
defer printStartJobContainerGroup(ctx, image, name, networkName)()
if err := common.NewPipelineExecutor(
rc.stopJobContainer(),
rc.pullServicesImages(rc.Config.ForcePull),
rc.JobContainer.Pull(rc.Config.ForcePull),
).Finally(rc.closeContainer())(ctx); err != nil {
return err
}
rc.startDockerProxy(ctx)
if containerInput.Binds, containerInput.Mounts, err = rc.GetBindsAndMounts(); err != nil {
return err
}
containerInput.ValidVolumes = rc.validVolumes()
rc.jobNetworkName = networkName rc.jobNetworkName = networkName
defer printStartJobContainerGroup(ctx, image, name, networkName)()
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
rc.pullServicesImages(rc.Config.ForcePull),
rc.JobContainer.Pull(rc.Config.ForcePull),
rc.stopJobContainer(),
container.NewDockerNetworkCreateExecutor(networkName, rc.Config.ContainerNetworkCreateOptions). container.NewDockerNetworkCreateExecutor(networkName, rc.Config.ContainerNetworkCreateOptions).
IfBool(createAndDeleteNetwork), IfBool(createAndDeleteNetwork),
rc.startServiceContainers(), rc.startServiceContainers(),
@@ -604,45 +645,52 @@ func (rc *RunContext) startJobContainer() common.Executor {
} }
} }
// cleanupJobResources removes everything the job created, continuing past failures. func (rc *RunContext) commandLogWriter(ctx context.Context) io.Writer {
// Only job container and volume errors are returned, the rest are logged. rawLogger := common.Logger(ctx).WithField(rawOutputField, true)
return common.NewLineWriter(rc.commandHandler(ctx), func(line string) bool {
rawLogger.Infof("%s", line)
return true
})
}
func (rc *RunContext) cleanupJobResources(networkName string, createAndDeleteNetwork bool) common.Executor { func (rc *RunContext) cleanupJobResources(networkName string, createAndDeleteNetwork bool) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
logger := common.Logger(ctx) logger := common.Logger(ctx)
removeJobContainer := rc.JobContainer != nil && !rc.Config.ReuseContainers errs := []error{rc.closeDockerProxy(ctx)}
if rc.JobContainer != nil {
var errs []error
if removeJobContainer {
errs = append(errs, rc.JobContainer.Remove()(ctx)) errs = append(errs, rc.JobContainer.Remove()(ctx))
} }
if len(rc.serviceContainers) > 0 { if len(rc.serviceContainers) > 0 {
logger.Infof("Cleaning up services for job %s", rc.JobName) logger.Infof("Cleaning up services for job %s", rc.JobName)
if err := rc.stopServiceContainers()(ctx); err != nil { errs = append(errs, rc.stopServiceContainers()(ctx))
logger.Errorf("Error while cleaning services: %v", err)
}
} }
if removeJobContainer { if !common.Dryrun(ctx) {
// after the containers using them, services can hold these via `--volumes-from` errs = append(errs, container.RemoveDockerJobResources(ctx, rc.jobContainerName()))
}
if rc.JobContainer != nil {
name := rc.jobContainerName() name := rc.jobContainerName()
errs = append(errs, errs = append(errs,
container.NewDockerVolumeRemoveExecutor(name, false)(ctx), container.NewDockerVolumeRemoveExecutor(name, false)(ctx),
container.NewDockerVolumeRemoveExecutor(name+"-env", false)(ctx)) container.NewDockerVolumeRemoveExecutor(name+"-env", false)(ctx))
} }
if createAndDeleteNetwork { if createAndDeleteNetwork {
// last, once every container has detached
logger.Infof("Cleaning up network for job %s, and network name is: %s", rc.JobName, networkName) logger.Infof("Cleaning up network for job %s, and network name is: %s", rc.JobName, networkName)
if err := container.NewDockerNetworkRemoveExecutor(networkName)(ctx); err != nil { errs = append(errs, container.NewDockerNetworkRemoveExecutor(networkName)(ctx))
logger.Errorf("Error while cleaning network: %v", err)
}
} }
return errors.Join(errs...) return errors.Join(errs...)
} }
} }
func (rc *RunContext) execJobContainer(cmd []string, env map[string]string, user, workdir string) common.Executor { //nolint:unparam // pre-existing issue from nektos/act func (rc *RunContext) closeDockerProxy(ctx context.Context) error {
return func(ctx context.Context) error { if rc.dockerProxy == nil {
return rc.JobContainer.Exec(cmd, env, user, workdir)(ctx) return nil
} }
err := rc.dockerProxy.Close(ctx)
rc.dockerProxy = nil
if err != nil {
return fmt.Errorf("close docker proxy: %w", err)
}
return nil
} }
func (rc *RunContext) ApplyExtraPath(ctx context.Context, env *map[string]string) { func (rc *RunContext) ApplyExtraPath(ctx context.Context, env *map[string]string) {
@@ -689,17 +737,21 @@ func (rc *RunContext) UpdateExtraPath(ctx context.Context, githubEnvPath string)
if err != nil && err != io.EOF { if err != nil && err != io.EOF {
return err return err
} }
s := bufio.NewScanner(reader) decoded := transform.NewReader(reader, unicode.BOMOverride(unicode.UTF8.NewDecoder()))
s := bufio.NewScanner(decoded)
s.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
for s.Scan() { for s.Scan() {
line := s.Text() line := s.Text()
if len(line) > 0 { if len(line) > 0 {
rc.addPath(ctx, line) rc.addPath(ctx, line)
} }
} }
if err := s.Err(); err != nil {
return fmt.Errorf("reading path file: %w", err)
}
return nil return nil
} }
// stopJobContainer removes the job container (if it exists) and its volume (if it exists)
func (rc *RunContext) stopJobContainer() common.Executor { func (rc *RunContext) stopJobContainer() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
if rc.cleanUpJobContainer != nil { if rc.cleanUpJobContainer != nil {
@@ -736,10 +788,17 @@ func (rc *RunContext) startServiceContainers() common.Executor {
func (rc *RunContext) stopServiceContainers() common.Executor { func (rc *RunContext) stopServiceContainers() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
execs := []common.Executor{} execs := []common.Executor{}
for _, svc := range rc.serviceContainers { errs := make([]error, len(rc.serviceContainers))
execs = append(execs, svc.container.Remove().Finally(svc.container.Close())) for index, svc := range rc.serviceContainers {
execs = append(execs, func(ctx context.Context) error {
if err := errors.Join(svc.container.Remove()(ctx), svc.container.Close()(ctx)); err != nil {
errs[index] = fmt.Errorf("clean service %s: %w", svc.name, err)
}
return nil
})
} }
return common.NewParallelExecutor(len(execs), execs...)(ctx) errs = append(errs, common.NewParallelExecutor(len(execs), execs...)(ctx))
return errors.Join(errs...)
} }
} }
@@ -885,6 +944,9 @@ func (rc *RunContext) captureJobContainerInfo() common.Executor {
return nil return nil
} }
rc.jobContainerID = info.ID rc.jobContainerID = info.ID
if source := info.Mounts[rc.githubWorkspace()]; source != "" {
rc.Env["GITEA_DOCKER_WORKSPACE"] = source
}
return nil return nil
} }
} }
@@ -927,18 +989,31 @@ func (rc *RunContext) interpolateOutputs() common.Executor {
// pristine snapshot (outputTemplate) and write under the lock, so each combo overwrites // pristine snapshot (outputTemplate) and write under the lock, so each combo overwrites
// with its own resolved values (last wins, as on GitHub) instead of the first combo's // with its own resolved values (last wins, as on GitHub) instead of the first combo's
// resolved values freezing the shared template against later combos. // resolved values freezing the shared template against later combos.
defer lockJob(job)() // Resolved up front so one failure publishes none of them, as GitHub does.
outputs := make(map[string]string, len(rc.outputTemplate))
var err error
for k, v := range rc.outputTemplate { for k, v := range rc.outputTemplate {
job.Outputs[k] = ee.Interpolate(ctx, v) if outputs[k], err = ee.Interpolate(ctx, v); err != nil {
err = fmt.Errorf("failed to evaluate job output %q: %w", k, err)
break
}
} }
return nil defer lockJob(job)()
for k := range rc.outputTemplate {
if err != nil {
job.Outputs[k] = ""
continue
}
job.Outputs[k] = outputs[k]
}
return err
} }
} }
func (rc *RunContext) startContainer() common.Executor { func (rc *RunContext) startContainer() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
var err error var err error
if rc.IsHostEnv(ctx) { if rc.IsHostEnv() {
err = rc.startHostEnvironment()(ctx) err = rc.startHostEnvironment()(ctx)
} else { } else {
err = rc.startJobContainer()(ctx) err = rc.startJobContainer()(ctx)
@@ -953,25 +1028,19 @@ func (rc *RunContext) startContainer() common.Executor {
} }
func (rc *RunContext) cleanupFailedStart(ctx context.Context) { func (rc *RunContext) cleanupFailedStart(ctx context.Context) {
if rc.cleanUpJobContainer == nil { cleanCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
return defer cancel()
cleanup := rc.cleanUpJobContainer
if cleanup == nil {
cleanup = rc.closeDockerProxy
} }
cleanCtx := ctx if err := cleanup(cleanCtx); err != nil {
if ctx.Err() != nil {
// the start likely failed because ctx was cancelled, detach so teardown still runs
var cancel context.CancelFunc
cleanCtx, cancel = context.WithTimeout(common.WithLogger(context.Background(), common.Logger(ctx)), time.Minute)
defer cancel()
}
if err := rc.cleanUpJobContainer(cleanCtx); err != nil {
common.Logger(ctx).Errorf("Error while cleaning up after failed container start for job %s: %v", rc.JobName, err) common.Logger(ctx).Errorf("Error while cleaning up after failed container start for job %s: %v", rc.JobName, err)
} }
} }
func (rc *RunContext) IsHostEnv(ctx context.Context) bool { func (rc *RunContext) IsHostEnv() bool {
platform := rc.runsOnImage(ctx) return strings.EqualFold(rc.platformImage, "-self-hosted")
image := rc.containerImage(ctx)
return image == "" && strings.EqualFold(platform, "-self-hosted")
} }
func (rc *RunContext) stopContainer() common.Executor { func (rc *RunContext) stopContainer() common.Executor {
@@ -1045,7 +1114,7 @@ func (rc *RunContext) Executor() (common.Executor, error) {
// unfinished. rc.caller is only set for reusable workflows. // unfinished. rc.caller is only set for reusable workflows.
rc.result("failure") rc.result("failure")
if rc.caller != nil { // For Gitea if rc.caller != nil { // For Gitea
rc.caller.setReusedWorkflowJobResult(rc.JobName, "failure") rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, "failure")
} }
return err return err
} }
@@ -1056,42 +1125,35 @@ func (rc *RunContext) Executor() (common.Executor, error) {
}, nil }, nil
} }
func (rc *RunContext) containerImage(ctx context.Context) string { func (rc *RunContext) containerImage(ctx context.Context) (string, error) {
job := rc.Run.Job() c := rc.Run.Job().Container()
if c == nil {
c := job.Container() return "", nil
if c != nil {
return rc.ExprEval.Interpolate(ctx, c.Image)
} }
image, err := rc.ExprEval.Interpolate(ctx, c.Image)
return "" if err != nil {
return "", fmt.Errorf("unable to interpolate container.image: %w", err)
}
return image, nil
} }
func (rc *RunContext) runsOnImage(ctx context.Context) string { func (rc *RunContext) runsOnImage(ctx context.Context) (string, error) {
if rc.Run.Job().RunsOn() == nil { if rc.Run.Job().RunsOn() == nil {
common.Logger(ctx).Errorf("'runs-on' key not defined in %s", rc.String()) common.Logger(ctx).Errorf("'runs-on' key not defined in %s", rc.String())
} }
job := rc.Run.Job() runsOn := rc.Run.Job().RunsOn()
runsOn := job.RunsOn()
for i, v := range runsOn { for i, v := range runsOn {
runsOn[i] = rc.ExprEval.Interpolate(ctx, v) var err error
} if runsOn[i], err = rc.ExprEval.Interpolate(ctx, v); err != nil {
return "", fmt.Errorf("unable to interpolate runs-on: %w", err)
if pick := rc.Config.PlatformPicker; pick != nil {
if image := pick(runsOn); image != "" {
return image
} }
} }
for _, platformName := range rc.runsOnPlatformNames(ctx) { if rc.Config.PlatformPicker != nil {
image := rc.Config.Platforms[strings.ToLower(platformName)] return rc.Config.PlatformPicker(runsOn), nil
if image != "" {
return image
}
} }
return "", nil
return ""
} }
func (rc *RunContext) runsOnPlatformNames(ctx context.Context) []string { func (rc *RunContext) runsOnPlatformNames(ctx context.Context) []string {
@@ -1112,22 +1174,26 @@ func (rc *RunContext) runsOnPlatformNames(ctx context.Context) []string {
return model.RunsOnFromNode(rawRunsOn) return model.RunsOnFromNode(rawRunsOn)
} }
func (rc *RunContext) platformImage(ctx context.Context) string { // resolvePlatformImage evaluates the job's image once, so every consumer sees the image the job started with.
if containerImage := rc.containerImage(ctx); containerImage != "" { func (rc *RunContext) resolvePlatformImage(ctx context.Context) error {
return containerImage image, err := rc.containerImage(ctx)
if err == nil && image == "" {
image, err = rc.runsOnImage(ctx)
} }
rc.platformImage = image
return rc.runsOnImage(ctx) return err
} }
func (rc *RunContext) options(ctx context.Context) string { func (rc *RunContext) workflowOptions(ctx context.Context) (string, error) {
job := rc.Run.Job() c := rc.Run.Job().Container()
c := job.Container() if c == nil {
if c != nil { return "", nil
return rc.Config.ContainerOptions + " " + rc.ExprEval.Interpolate(ctx, c.Options)
} }
options, err := rc.ExprEval.Interpolate(ctx, c.Options)
return rc.Config.ContainerOptions if err != nil {
return "", fmt.Errorf("unable to interpolate container.options: %w", err)
}
return options, nil
} }
func (rc *RunContext) isEnabled(ctx context.Context) (bool, error) { func (rc *RunContext) isEnabled(ctx context.Context) (bool, error) {
@@ -1146,7 +1212,7 @@ func (rc *RunContext) isEnabled(ctx context.Context) (bool, error) {
if !runJob { if !runJob {
if rc.caller != nil { // For Gitea if rc.caller != nil { // For Gitea
rc.caller.setReusedWorkflowJobResult(rc.JobName, "skipped") rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, "skipped")
return false, nil return false, nil
} }
l.WithField("jobResult", "skipped").Debugf("Skipping job '%s' due to '%s'", job.Name, job.If.Value) l.WithField("jobResult", "skipped").Debugf("Skipping job '%s' due to '%s'", job.Name, job.If.Value)
@@ -1157,8 +1223,10 @@ func (rc *RunContext) isEnabled(ctx context.Context) (bool, error) {
return true, nil return true, nil
} }
img := rc.platformImage(ctx) if err := rc.resolvePlatformImage(ctx); err != nil {
if img == "" { return false, err
}
if rc.platformImage == "" {
for _, platformName := range rc.runsOnPlatformNames(ctx) { for _, platformName := range rc.runsOnPlatformNames(ctx) {
l.Infof("Skipping unsupported platform -- Try running with `-P %+v=...`", platformName) l.Infof("Skipping unsupported platform -- Try running with `-P %+v=...`", platformName)
} }
@@ -1266,12 +1334,23 @@ func (rc *RunContext) getRunnerContext(ctx context.Context) map[string]any {
} }
runnerContext["name"] = rc.Config.RunnerName runnerContext["name"] = rc.Config.RunnerName
runnerContext["environment"] = "self-hosted" runnerContext["environment"] = "self-hosted"
runnerContext["workspace"] = parentDir(rc.githubWorkspace())
if rc.Config.RunnerDebug() { if rc.Config.RunnerDebug() {
runnerContext["debug"] = "1" runnerContext["debug"] = "1"
} }
return runnerContext return runnerContext
} }
func (rc *RunContext) githubWorkspace() string {
if rc.JobContainer != nil {
return rc.JobContainer.ToContainerPath(rc.Config.Workdir)
}
if workspace := rc.Config.Env["GITHUB_WORKSPACE"]; workspace != "" {
return workspace
}
return rc.Config.Workdir
}
func (rc *RunContext) getGithubContext(ctx context.Context) *model.GithubContext { func (rc *RunContext) getGithubContext(ctx context.Context) *model.GithubContext {
logger := common.Logger(ctx) logger := common.Logger(ctx)
ghc := &model.GithubContext{ ghc := &model.GithubContext{
@@ -1298,11 +1377,10 @@ func (rc *RunContext) getGithubContext(ctx context.Context) *model.GithubContext
RefType: rc.Config.Env["GITHUB_REF_TYPE"], RefType: rc.Config.Env["GITHUB_REF_TYPE"],
BaseRef: rc.Config.Env["GITHUB_BASE_REF"], BaseRef: rc.Config.Env["GITHUB_BASE_REF"],
HeadRef: rc.Config.Env["GITHUB_HEAD_REF"], HeadRef: rc.Config.Env["GITHUB_HEAD_REF"],
Workspace: rc.Config.Env["GITHUB_WORKSPACE"], Workspace: rc.githubWorkspace(),
} }
if rc.JobContainer != nil { if rc.JobContainer != nil {
ghc.EventPath = rc.JobContainer.GetActPath() + "/workflow/event.json" ghc.EventPath = rc.JobContainer.GetActPath() + "/workflow/event.json"
ghc.Workspace = rc.JobContainer.ToContainerPath(rc.Config.Workdir)
} }
if ghc.RunID == "" { if ghc.RunID == "" {
@@ -1367,9 +1445,9 @@ func (rc *RunContext) getGithubContext(ctx context.Context) *model.GithubContext
ghc.SetBaseAndHeadRef() ghc.SetBaseAndHeadRef()
repoPath := rc.Config.Workdir repoPath := rc.Config.Workdir
ghcontext.SetRepositoryAndOwner(ctx, ghc, rc.Config.GitHubInstance, rc.Config.RemoteName, repoPath) ghcontext.SetRepositoryAndOwner(ctx, ghc, rc.Config.GitHubInstance, repoPath)
if ghc.Ref == "" { if ghc.Ref == "" {
ghcontext.SetRef(ctx, ghc, rc.Config.DefaultBranch, repoPath) ghcontext.SetRef(ctx, ghc, repoPath)
} }
if ghc.Sha == "" { if ghc.Sha == "" {
ghcontext.SetSha(ctx, ghc, repoPath) ghcontext.SetSha(ctx, ghc, repoPath)
@@ -1535,7 +1613,7 @@ func (rc *RunContext) imageOS(ctx context.Context) string {
// log that runs-on is missing. // log that runs-on is missing.
return "" return ""
} }
if imageOS := imageOSFromImage(rc.platformImage(ctx)); imageOS != "" { if imageOS := imageOSFromImage(rc.platformImage); imageOS != "" {
return imageOS return imageOS
} }
@@ -1585,58 +1663,35 @@ func (rc *RunContext) handleCredentials(ctx context.Context) (string, string, er
return "", "", nil return "", "", nil
} }
if len(container.Credentials) != 2 { return rc.interpolateCredentials(ctx, container.Credentials, "container.")
err := errors.New("invalid property count for key 'credentials:'") }
return "", "", err
func (rc *RunContext) interpolateCredentials(ctx context.Context, credentials map[string]string, prefix string) (string, string, error) {
if credentials == nil {
return "", "", nil
}
if len(credentials) != 2 {
return "", "", errors.New("invalid property count for key 'credentials:'")
} }
ee := rc.NewExpressionEvaluator(ctx) ee := rc.NewExpressionEvaluator(ctx)
var username, password string interpolate := func(key string) (string, error) {
if username = ee.Interpolate(ctx, container.Credentials["username"]); username == "" { value, err := ee.Interpolate(ctx, credentials[key])
err := errors.New("failed to interpolate container.credentials.username") if err != nil {
return "", fmt.Errorf("failed to interpolate %scredentials.%s: %w", prefix, key, err)
}
if value == "" {
return "", fmt.Errorf("failed to interpolate %scredentials.%s", prefix, key)
}
return value, nil
}
username, err := interpolate("username")
if err != nil {
return "", "", err return "", "", err
} }
if password = ee.Interpolate(ctx, container.Credentials["password"]); password == "" { password, err := interpolate("password")
err := errors.New("failed to interpolate container.credentials.password") if err != nil {
return "", "", err return "", "", err
} }
if container.Credentials["username"] == "" || container.Credentials["password"] == "" {
err := errors.New("container.credentials cannot be empty")
return "", "", err
}
return username, password, nil return username, password, nil
} }
func (rc *RunContext) handleServiceCredentials(ctx context.Context, creds map[string]string) (username, password string, err error) {
if creds == nil {
return username, password, err
}
if len(creds) != 2 {
err = errors.New("invalid property count for key 'credentials:'")
return username, password, err
}
ee := rc.NewExpressionEvaluator(ctx)
if username = ee.Interpolate(ctx, creds["username"]); username == "" {
err = errors.New("failed to interpolate credentials.username")
return username, password, err
}
if password = ee.Interpolate(ctx, creds["password"]); password == "" {
err = errors.New("failed to interpolate credentials.password")
return username, password, err
}
return username, password, err
}
// GetServiceBindsAndMounts returns the binds and mounts for the service container, resolving paths as appopriate
func (rc *RunContext) GetServiceBindsAndMounts(svcVolumes []string) ([]string, map[string]string) {
binds, mounts, claimed := splitVolumes(svcVolumes)
if daemonSocket := rc.containerDaemonSocket(); daemonSocket != "-" && !claimed["/var/run/docker.sock"] {
binds = append(binds, getDockerDaemonSocketMountPath(daemonSocket)+":/var/run/docker.sock")
}
return binds, mounts
}
+263 -128
View File
@@ -9,6 +9,9 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"io"
"net/http"
"net/http/httptest"
"os" "os"
"runtime" "runtime"
"strings" "strings"
@@ -214,11 +217,15 @@ type fakeContainer struct {
container.ExecutionsEnvironment container.ExecutionsEnvironment
} }
func (fakeContainer) Pull(bool) common.Executor { return func(context.Context) error { return nil } } func (fakeContainer) Pull(bool) common.Executor { return func(context.Context) error { return nil } }
func (fakeContainer) Start(bool) common.Executor { return func(context.Context) error { return nil } } func (fakeContainer) Start(bool) common.Executor { return func(context.Context) error { return nil } }
func (fakeContainer) Remove() common.Executor { return func(context.Context) error { return nil } }
func (fakeContainer) Close() common.Executor { return func(context.Context) error { return nil } } func (fakeContainer) Remove() common.Executor { return func(context.Context) error { return nil } }
func (fakeContainer) GetActPath() string { return "/var/run/act" }
func (fakeContainer) Close() common.Executor { return func(context.Context) error { return nil } }
func (fakeContainer) GetActPath() string { return "/var/run/act" }
func (fakeContainer) ToContainerPath(path string) string { return path }
func (fakeContainer) Create([]string, []string) common.Executor { func (fakeContainer) Create([]string, []string) common.Executor {
return func(context.Context) error { return nil } return func(context.Context) error { return nil }
} }
@@ -233,10 +240,66 @@ func (fakeContainer) Inspect(context.Context) (*container.Info, error) {
func (fakeContainer) DumpLogs(context.Context) error { return nil } func (fakeContainer) DumpLogs(context.Context) error { return nil }
// startJobContainerInputs runs startJobContainer against fakeContainer and returns the
// inputs it built, one per container.
func startJobContainerInputs(t *testing.T, workflowYAML string, cfg *Config) []*container.NewContainerInput {
t.Helper()
workflow, err := model.ReadWorkflow(strings.NewReader(workflowYAML))
require.NoError(t, err)
var inputs []*container.NewContainerInput
origNewContainer := newContainer
newContainer = func(input *container.NewContainerInput) container.ExecutionsEnvironment {
inputs = append(inputs, input)
return fakeContainer{}
}
t.Cleanup(func() { newContainer = origNewContainer })
cfg.Workdir = "/tmp"
cfg.ContainerNetworkMode = "host" // an explicit network mode creates no network
cfg.Env = map[string]string{}
cfg.Secrets = map[string]string{}
rc := &RunContext{
Name: "test",
Config: cfg,
Env: map[string]string{},
Run: &model.Run{
JobID: "job",
Workflow: workflow,
},
}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
require.NoError(t, rc.resolvePlatformImage(t.Context()))
daemon := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch {
case strings.HasSuffix(r.URL.Path, "/_ping"):
w.Header().Set("API-Version", "1.47")
_, _ = io.WriteString(w, "OK")
case strings.HasSuffix(r.URL.Path, "/containers/json"), strings.HasSuffix(r.URL.Path, "/networks"):
_, _ = io.WriteString(w, "[]")
case strings.HasSuffix(r.URL.Path, "/volumes"):
_, _ = io.WriteString(w, `{"Volumes":[]}`)
case strings.HasSuffix(r.URL.Path, "/info"):
_, _ = io.WriteString(w, `{"Architecture":"amd64","OSType":"linux"}`)
default:
t.Errorf("unexpected Docker request: %s %s", r.Method, r.URL)
http.NotFound(w, r)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
require.NoError(t, rc.startJobContainer()(t.Context()))
return inputs
}
// Regression test: a service without a `credentials:` block resolves to empty // Regression test: a service without a `credentials:` block resolves to empty
// credentials, which used to overwrite the job container's own credentials. // credentials, which used to overwrite the job container's own credentials.
func TestStartJobContainerKeepsJobCredentialsWithServices(t *testing.T) { func TestStartJobContainerKeepsJobCredentialsWithServices(t *testing.T) {
workflow, err := model.ReadWorkflow(strings.NewReader(` inputs := startJobContainerInputs(t, `
name: test name: test
on: push on: push
jobs: jobs:
@@ -256,37 +319,7 @@ jobs:
username: db-user username: db-user
password: db-password password: db-password
steps: [] steps: []
`)) `, &Config{})
require.NoError(t, err)
var inputs []*container.NewContainerInput
origNewContainer := newContainer
newContainer = func(input *container.NewContainerInput) container.ExecutionsEnvironment {
inputs = append(inputs, input)
return fakeContainer{}
}
t.Cleanup(func() { newContainer = origNewContainer })
rc := &RunContext{
Name: "test",
Config: &Config{
Workdir: "/tmp",
// no daemon: an explicit network mode creates no network, and
// reusing containers short-circuits the volume cleanup executors
ContainerNetworkMode: "host",
ReuseContainers: true,
Env: map[string]string{},
Secrets: map[string]string{},
},
Env: map[string]string{},
Run: &model.Run{
JobID: "job",
Workflow: workflow,
},
}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
require.NoError(t, rc.startJobContainer()(t.Context()))
credentials := map[string][2]string{} credentials := map[string][2]string{}
for _, in := range inputs { for _, in := range inputs {
@@ -300,10 +333,57 @@ jobs:
require.Equal(t, [2]string{"", ""}, credentials["redis:latest"]) require.Equal(t, [2]string{"", ""}, credentials["redis:latest"])
} }
func TestStartJobContainerGivesServicesTheirVolumes(t *testing.T) {
redis := startJobContainerInputs(t, `
jobs:
job:
services:
redis:
image: redis:latest
volumes:
- data:/data
`, &Config{ValidVolumes: []string{"data"}})[0]
require.Equal(t, "redis:latest", redis.Image) // services are built before the job container
require.Equal(t, []string{"data"}, redis.ValidVolumes)
require.Equal(t, map[string]string{"data": "/data"}, redis.Mounts)
require.Empty(t, redis.Binds) // the docker socket is the job container's alone
require.Empty(t, redis.WorkingDir)
}
// Only the workflow's options may be stripped later, so the two sources have to reach the
// container apart from each other.
func TestStartJobContainerKeepsRunnerOptionsApartFromWorkflowOptions(t *testing.T) {
inputs := startJobContainerInputs(t, `
name: test
on: push
jobs:
job:
runs-on: ubuntu-latest
container:
image: registry.example/job:latest
options: --cap-add SYS_PTRACE
services:
redis:
image: redis:latest
options: --shm-size 1g
steps: []
`, &Config{ContainerOptions: "--device /dev/fuse"})
options := map[string][2]string{}
for _, in := range inputs {
options[in.Image] = [2]string{in.RunnerOptions, in.WorkflowOptions}
}
require.Equal(t, [2]string{"--device /dev/fuse", "--cap-add SYS_PTRACE"}, options["registry.example/job:latest"])
// a service container gets no options from the runner's config today
require.Equal(t, [2]string{"", "--shm-size 1g"}, options["redis:latest"])
}
// A service container reaches the internet the same way the job does, so it inherits the // A service container reaches the internet the same way the job does, so it inherits the
// job's proxy; a service that sets the variable itself keeps its own value. // job's proxy; a service that sets the variable itself keeps its own value.
func TestStartJobContainerGivesServicesTheJobProxy(t *testing.T) { func TestStartJobContainerGivesServicesTheJobProxy(t *testing.T) {
workflow, err := model.ReadWorkflow(strings.NewReader(` inputs := startJobContainerInputs(t, `
name: test name: test
on: push on: push
jobs: jobs:
@@ -319,36 +399,7 @@ jobs:
env: env:
no_proxy: db-only.example no_proxy: db-only.example
steps: [] steps: []
`)) `, &Config{ProxyEnv: map[string]string{"http_proxy": "http://proxy:3128", "no_proxy": "internal.example"}})
require.NoError(t, err)
var inputs []*container.NewContainerInput
origNewContainer := newContainer
newContainer = func(input *container.NewContainerInput) container.ExecutionsEnvironment {
inputs = append(inputs, input)
return fakeContainer{}
}
t.Cleanup(func() { newContainer = origNewContainer })
rc := &RunContext{
Name: "test",
Config: &Config{
Workdir: "/tmp",
ContainerNetworkMode: "host",
ReuseContainers: true,
Env: map[string]string{},
ProxyEnv: map[string]string{"http_proxy": "http://proxy:3128", "no_proxy": "internal.example"},
Secrets: map[string]string{},
},
Env: map[string]string{},
Run: &model.Run{
JobID: "job",
Workflow: workflow,
},
}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
require.NoError(t, rc.startJobContainer()(t.Context()))
env := map[string][]string{} env := map[string][]string{}
for _, in := range inputs { for _, in := range inputs {
@@ -421,7 +472,8 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
config := testcase.rc.Config config := testcase.rc.Config
config.Workdir = testcase.name config.Workdir = testcase.name
config.BindWorkdir = bindWorkDir config.BindWorkdir = bindWorkDir
gotbind, gotmount := rctemplate.GetBindsAndMounts() gotbind, gotmount, err := rctemplate.GetBindsAndMounts()
require.NoError(t, err)
// Name binds/mounts are either/or // Name binds/mounts are either/or
if config.BindWorkdir { if config.BindWorkdir {
@@ -480,7 +532,8 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
rc.Run.Workflow.Jobs = map[string]*model.Job{"job1": job} rc.Run.Workflow.Jobs = map[string]*model.Job{"job1": job}
rc.ExprEval = rc.NewExpressionEvaluator(context.Background()) rc.ExprEval = rc.NewExpressionEvaluator(context.Background())
gotbind, gotmount := rc.GetBindsAndMounts() gotbind, gotmount, err := rc.GetBindsAndMounts()
require.NoError(t, err)
assert.Contains(t, gotbind, "/host/mame/roms:/root/.mame/roms:ro") assert.Contains(t, gotbind, "/host/mame/roms:/root/.mame/roms:ro")
assert.NotContains(t, gotbind, "${{ secrets.MAME }}") assert.NotContains(t, gotbind, "${{ secrets.MAME }}")
assert.NotContains(t, gotmount, "${{ secrets.MAME }}") assert.NotContains(t, gotmount, "${{ secrets.MAME }}")
@@ -509,37 +562,30 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
rc.Run.JobID = "job1" rc.Run.JobID = "job1"
rc.Run.Workflow.Jobs = map[string]*model.Job{"job1": job} rc.Run.Workflow.Jobs = map[string]*model.Job{"job1": job}
jobBinds, jobMounts := rc.GetBindsAndMounts() gotbind, gotmount, err := rc.GetBindsAndMounts()
svcBinds, svcMounts := rc.GetServiceBindsAndMounts(testcase.volumes) require.NoError(t, err)
// job and service containers classify volumes alike, only their own mounts differ
for _, got := range []struct {
binds []string
mounts map[string]string
}{{jobBinds, jobMounts}, {svcBinds, svcMounts}} {
gotbind, gotmount := got.binds, got.mounts
if len(testcase.wantbind) > 0 { if len(testcase.wantbind) > 0 {
assert.Contains(t, gotbind, testcase.wantbind) assert.Contains(t, gotbind, testcase.wantbind)
} }
for k, v := range testcase.wantmount { for k, v := range testcase.wantmount {
assert.Contains(t, gotmount, k) assert.Contains(t, gotmount, k)
assert.Equal(t, gotmount[k], v) assert.Equal(t, gotmount[k], v)
} }
// Docker rejects a container with two mounts on one target, so the job's own // Docker rejects a container with two mounts on one target, so the job's own
// volumes must displace the runner's rather than pile up next to them. // volumes must displace the runner's rather than pile up next to them.
targets := map[string]bool{} targets := map[string]bool{}
for _, bind := range gotbind { for _, bind := range gotbind {
parsed, err := loader.ParseVolume(bind) parsed, err := loader.ParseVolume(bind)
require.NoError(t, err) require.NoError(t, err)
assert.NotContains(t, targets, parsed.Target, "%s mounts an already mounted target", bind) assert.NotContains(t, targets, parsed.Target, "%s mounts an already mounted target", bind)
targets[parsed.Target] = true targets[parsed.Target] = true
} }
for source, target := range gotmount { for source, target := range gotmount {
assert.NotContains(t, targets, target, "%s mounts an already mounted target", source) assert.NotContains(t, targets, target, "%s mounts an already mounted target", source)
targets[target] = true targets[target] = true
}
} }
}) })
} }
@@ -552,11 +598,13 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
Config: &Config{}, Config: &Config{},
} }
_, gotmount := rc.GetBindsAndMounts() _, gotmount, err := rc.GetBindsAndMounts()
require.NoError(t, err)
assert.NotContains(t, gotmount, sharedToolCacheVolume) assert.NotContains(t, gotmount, sharedToolCacheVolume)
rc.Config.SharedToolCache = true rc.Config.SharedToolCache = true
_, gotmount = rc.GetBindsAndMounts() _, gotmount, err = rc.GetBindsAndMounts()
require.NoError(t, err)
assert.Equal(t, container.DefaultToolCache, gotmount[sharedToolCacheVolume]) assert.Equal(t, container.DefaultToolCache, gotmount[sharedToolCacheVolume])
}) })
} }
@@ -582,6 +630,11 @@ func TestRunContextValidVolumes(t *testing.T) {
// a job may mount it only while the runner does // a job may mount it only while the runner does
rc.Config.SharedToolCache = false rc.Config.SharedToolCache = false
assert.NotContains(t, rc.validVolumes(), sharedToolCacheVolume) assert.NotContains(t, rc.validVolumes(), sharedToolCacheVolume)
rc.Config.Workdir = "/workspace/1/owner/repo"
assert.NotContains(t, rc.validVolumes(), rc.Config.Workdir)
rc.Config.BindWorkdir = true
assert.Contains(t, rc.validVolumes(), rc.Config.Workdir)
} }
func TestCleanupJobResourcesCleansServicesWithoutJobContainer(t *testing.T) { func TestCleanupJobResourcesCleansServicesWithoutJobContainer(t *testing.T) {
@@ -594,7 +647,7 @@ func TestCleanupJobResourcesCleansServicesWithoutJobContainer(t *testing.T) {
serviceContainers: []*serviceContainer{{name: "svc", container: service}}, serviceContainers: []*serviceContainer{{name: "svc", container: service}},
} }
err := rc.cleanupJobResources("external-network", false)(context.Background()) err := rc.cleanupJobResources("external-network", false)(common.WithDryrun(t.Context(), true))
require.NoError(t, err) require.NoError(t, err)
service.AssertExpectations(t) service.AssertExpectations(t)
} }
@@ -603,11 +656,12 @@ func TestCleanupJobResourcesCleansServicesWithoutJobContainer(t *testing.T) {
func TestCleanupJobResourcesContinuesAfterFailure(t *testing.T) { func TestCleanupJobResourcesContinuesAfterFailure(t *testing.T) {
t.Setenv("DOCKER_HOST", "unix:///nonexistent.sock") t.Setenv("DOCKER_HOST", "unix:///nonexistent.sock")
removeError, closeError := errors.New("remove service"), errors.New("close service")
jobContainer := &containerMock{} jobContainer := &containerMock{}
jobContainer.On("Remove").Return(func(context.Context) error { return errors.New("removal failed") }).Once() jobContainer.On("Remove").Return(func(context.Context) error { return errors.New("removal failed") }).Once()
service := &containerMock{} service := &containerMock{}
service.On("Remove").Return(func(context.Context) error { return nil }).Once() service.On("Remove").Return(func(context.Context) error { return removeError }).Once()
service.On("Close").Return(func(context.Context) error { return nil }).Once() service.On("Close").Return(func(context.Context) error { return closeError }).Once()
rc := &RunContext{ rc := &RunContext{
Name: "job", Name: "job",
@@ -619,7 +673,11 @@ func TestCleanupJobResourcesContinuesAfterFailure(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() cancel()
require.Error(t, rc.cleanupJobResources("job-network", true)(ctx)) err := rc.cleanupJobResources("job-network", true)(ctx)
require.ErrorContains(t, err, "removal failed")
require.ErrorIs(t, err, removeError)
require.ErrorIs(t, err, closeError)
require.ErrorIs(t, err, context.Canceled)
jobContainer.AssertExpectations(t) jobContainer.AssertExpectations(t)
service.AssertExpectations(t) service.AssertExpectations(t)
} }
@@ -633,8 +691,10 @@ func TestInterpolateOutputsIsPerMatrixCombo(t *testing.T) {
r := &runnerImpl{config: &Config{}} r := &runnerImpl{config: &Config{}}
ctx := context.Background() ctx := context.Background()
rcA := r.newRunContext(ctx, run, map[string]any{"v": "a"}) rcA, err := r.newRunContext(ctx, run, map[string]any{"v": "a"})
rcB := r.newRunContext(ctx, run, map[string]any{"v": "b"}) require.NoError(t, err)
rcB, err := r.newRunContext(ctx, run, map[string]any{"v": "b"})
require.NoError(t, err)
require.NoError(t, rcA.interpolateOutputs()(ctx)) require.NoError(t, rcA.interpolateOutputs()(ctx))
require.NoError(t, rcB.interpolateOutputs()(ctx)) require.NoError(t, rcB.interpolateOutputs()(ctx))
@@ -662,13 +722,12 @@ func TestGetGitHubContext(t *testing.T) {
Name: "GitHubContextTest", Name: "GitHubContextTest",
}, },
}, },
Name: "GitHubContextTest", Name: "GitHubContextTest",
CurrentStep: "step", CurrentStep: "step",
Matrix: map[string]any{}, Matrix: map[string]any{},
Env: map[string]string{}, Env: map[string]string{},
ExtraPath: []string{}, ExtraPath: []string{},
StepResults: map[string]*model.StepResult{}, StepResults: map[string]*model.StepResult{},
OutputMappings: map[MappableOutput]MappableOutput{},
} }
rc.Run.JobID = "job1" rc.Run.JobID = "job1"
@@ -745,13 +804,8 @@ func TestGetGithubContextRef(t *testing.T) {
func createIfTestRunContext(jobs map[string]*model.Job) *RunContext { func createIfTestRunContext(jobs map[string]*model.Job) *RunContext {
rc := &RunContext{ rc := &RunContext{
Config: &Config{ Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }},
Workdir: ".", Env: map[string]string{},
Platforms: map[string]string{
"ubuntu-latest": "ubuntu-latest",
},
},
Env: map[string]string{},
Run: &model.Run{ Run: &model.Run{
JobID: "job1", JobID: "job1",
Workflow: &model.Workflow{ Workflow: &model.Workflow{
@@ -987,6 +1041,14 @@ func TestRunContextGetEnv(t *testing.T) {
} }
} }
// a remote composite action re-evaluates its env per stage, so inputs must follow it
func TestSetActionEnvRefreshesInputs(t *testing.T) {
rc := &RunContext{}
rc.setActionEnv(map[string]string{"INPUT_MSG": "pre"})
rc.setActionEnv(map[string]string{"INPUT_MSG": "main"})
assert.Equal(t, map[string]any{"msg": "main"}, rc.actionInputs)
}
func TestCreateContainerNameBoundedForLongMatrixInput(t *testing.T) { func TestCreateContainerNameBoundedForLongMatrixInput(t *testing.T) {
longMatrixValue := strings.Repeat("os=ubuntu-latest-go=1.24-node=22-", 20) longMatrixValue := strings.Repeat("os=ubuntu-latest-go=1.24-node=22-", 20)
name := createContainerName( name := createContainerName(
@@ -1032,12 +1094,19 @@ func TestRunContext_cleanupFailedStart(t *testing.T) {
calls int calls int
err error err error
sentinel any sentinel any
cancel context.CancelFunc
} }
newRC := func(c *capture) *RunContext { newRC := func(c *capture) *RunContext {
return &RunContext{ return &RunContext{
JobName: "job", JobName: "job",
cleanUpJobContainer: func(ctx context.Context) error { cleanUpJobContainer: func(ctx context.Context) error {
c.calls++ c.calls++
if c.cancel != nil {
c.cancel()
}
deadline, ok := ctx.Deadline()
require.True(t, ok)
assert.WithinDuration(t, time.Now().Add(time.Minute), deadline, time.Second)
c.err = ctx.Err() c.err = ctx.Err()
c.sentinel = ctx.Value(sentinel) c.sentinel = ctx.Value(sentinel)
return nil return nil
@@ -1045,9 +1114,10 @@ func TestRunContext_cleanupFailedStart(t *testing.T) {
} }
} }
t.Run("runs teardown on the live context", func(t *testing.T) { t.Run("detaches teardown from cancellation during cleanup", func(t *testing.T) {
var c capture ctx, cancel := context.WithCancel(context.WithValue(context.Background(), sentinel, "v"))
ctx := context.WithValue(context.Background(), sentinel, "v") defer cancel()
c := capture{cancel: cancel}
newRC(&c).cleanupFailedStart(ctx) newRC(&c).cleanupFailedStart(ctx)
@@ -1065,7 +1135,7 @@ func TestRunContext_cleanupFailedStart(t *testing.T) {
assert.Equal(t, 1, c.calls) assert.Equal(t, 1, c.calls)
require.NoError(t, c.err) require.NoError(t, c.err)
assert.Nil(t, c.sentinel) assert.Equal(t, "v", c.sentinel)
}) })
t.Run("no-op when there is nothing to clean up", func(t *testing.T) { t.Run("no-op when there is nothing to clean up", func(t *testing.T) {
@@ -1257,6 +1327,29 @@ func TestGetJobContextReportsContainers(t *testing.T) {
}, jobContext.Services) }, jobContext.Services)
} }
func TestCaptureJobContainerInfoExportsDockerWorkspace(t *testing.T) {
job := &containerMock{}
job.On("Inspect", mock.Anything).Return(&container.Info{
ID: "job-container-id",
Mounts: map[string]string{"/workspace/owner/repo": "/var/lib/docker/volumes/job/_data"},
}, nil)
rc := &RunContext{
Config: &Config{Workdir: "/workspace/owner/repo/"},
Env: map[string]string{},
JobContainer: job,
}
require.NoError(t, rc.captureJobContainerInfo()(context.Background()))
assert.Equal(t, "job-container-id", rc.jobContainerID)
assert.Equal(t, "/var/lib/docker/volumes/job/_data", rc.Env["GITEA_DOCKER_WORKSPACE"])
rc.Config.Workdir = "/elsewhere"
rc.Env = map[string]string{}
require.NoError(t, rc.captureJobContainerInfo()(context.Background()))
assert.NotContains(t, rc.Env, "GITEA_DOCKER_WORKSPACE")
}
// A job that never started a container reports an empty context, not a placeholder. // A job that never started a container reports an empty context, not a placeholder.
func TestGetJobContextWithoutContainer(t *testing.T) { func TestGetJobContextWithoutContainer(t *testing.T) {
jobContext := (&RunContext{}).getJobContext() jobContext := (&RunContext{}).getJobContext()
@@ -1304,15 +1397,15 @@ func TestRunContextImageOS(t *testing.T) {
t.Run("prefers the release in the resolved image tag", func(t *testing.T) { t.Run("prefers the release in the resolved image tag", func(t *testing.T) {
rc := createRunsOnRunContext(t, "ubuntu-latest") rc := createRunsOnRunContext(t, "ubuntu-latest")
rc.Config.Platforms = map[string]string{ rc.Config.PlatformPicker = func([]string) string { return "docker.gitea.com/runner-images:ubuntu-24.04" }
"ubuntu-latest": "docker.gitea.com/runner-images:ubuntu-24.04", require.NoError(t, rc.resolvePlatformImage(ctx))
}
assert.Equal(t, "ubuntu24", rc.imageOS(ctx)) assert.Equal(t, "ubuntu24", rc.imageOS(ctx))
}) })
t.Run("falls back to the runs-on label", func(t *testing.T) { t.Run("falls back to the runs-on label", func(t *testing.T) {
rc := createRunsOnRunContext(t, "ubuntu-22.04") rc := createRunsOnRunContext(t, "ubuntu-22.04")
rc.Config.Platforms = map[string]string{"ubuntu-22.04": "some-image"} rc.Config.PlatformPicker = func([]string) string { return "some-image" }
require.NoError(t, rc.resolvePlatformImage(ctx))
assert.Equal(t, "ubuntu22", rc.imageOS(ctx)) assert.Equal(t, "ubuntu22", rc.imageOS(ctx))
}) })
@@ -1332,10 +1425,15 @@ func TestRunContextGetRunnerContext(t *testing.T) {
t.Run("adds the runner values the container cannot know", func(t *testing.T) { t.Run("adds the runner values the container cannot know", func(t *testing.T) {
rc := createRunsOnRunContext(t, "ubuntu-latest") rc := createRunsOnRunContext(t, "ubuntu-latest")
rc.Config.RunnerName = "runner-1" rc.Config.RunnerName = "runner-1"
rc.Config.Workdir = "/workspace/owner/repo"
runnerContext := rc.getRunnerContext(ctx) runnerContext := rc.getRunnerContext(ctx)
assert.Equal(t, "runner-1", runnerContext["name"]) assert.Equal(t, "runner-1", runnerContext["name"])
assert.Equal(t, "self-hosted", runnerContext["environment"]) assert.Equal(t, "self-hosted", runnerContext["environment"])
assert.Equal(t, "/workspace/owner", runnerContext["workspace"])
assert.Equal(t, "/workspace/owner/repo", rc.getGithubContext(ctx).Workspace)
rc.Config.Env = map[string]string{"GITHUB_WORKSPACE": "/configured/work"}
assert.Equal(t, "/configured/work", rc.getGithubContext(ctx).Workspace)
assert.NotContains(t, runnerContext, "debug") assert.NotContains(t, runnerContext, "debug")
}) })
@@ -1348,15 +1446,52 @@ func TestRunContextGetRunnerContext(t *testing.T) {
t.Run("keeps the execution environment values", func(t *testing.T) { t.Run("keeps the execution environment values", func(t *testing.T) {
rc := createRunsOnRunContext(t, "ubuntu-latest") rc := createRunsOnRunContext(t, "ubuntu-latest")
rc.JobContainer = &container.HostEnvironment{TmpDir: "/tmp/act", ToolCache: "/tmp/tool_cache"} rc.Config.Workdir = "/host/work/owner/repo"
rc.JobContainer = &container.HostEnvironment{Workdir: rc.Config.Workdir, Path: "/container/work/owner/repo", TmpDir: "/tmp/act", ToolCache: "/tmp/tool_cache"}
runnerContext := rc.getRunnerContext(ctx) runnerContext := rc.getRunnerContext(ctx)
assert.Equal(t, "/tmp/act", runnerContext["temp"]) assert.Equal(t, "/tmp/act", runnerContext["temp"])
assert.Equal(t, "/tmp/tool_cache", runnerContext["tool_cache"]) assert.Equal(t, "/tmp/tool_cache", runnerContext["tool_cache"])
assert.Equal(t, "/container/work/owner", runnerContext["workspace"])
assert.Equal(t, "/container/work/owner/repo", rc.getGithubContext(ctx).Workspace)
assert.NotEmpty(t, runnerContext["os"]) assert.NotEmpty(t, runnerContext["os"])
}) })
} }
func TestRunContextUpdateExtraPath(t *testing.T) {
longPath := strings.Repeat("x", 64*1024+1)
for _, testcase := range []struct {
name string
content []byte
want string
wantErr bool
}{
{name: "UTF-8 BOM", content: []byte("\xef\xbb\xbf/utf8/tools\n"), want: "/utf8/tools"},
{name: "UTF-16 BOM", content: []byte{0xff, 0xfe, 'C', 0, ':', 0, '\\', 0, 't', 0, 'o', 0, 'o', 0, 'l', 0, 's', 0, '\n', 0}, want: `C:\tools`},
{name: "over 64 KiB", content: []byte(longPath + "\n"), want: longPath},
{name: "over 16 MiB", content: []byte(strings.Repeat("x", 16*1024*1024+1)), wantErr: true},
} {
t.Run(testcase.name, func(t *testing.T) {
logger := log.New()
logger.SetOutput(io.Discard)
ctx := common.WithLogger(t.Context(), logger)
jobContainer := &containerMock{}
jobContainer.On("GetContainerArchive", mock.Anything, "/github/path").
Return(io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "path", body: string(testcase.content)}))), nil).Once()
defer jobContainer.AssertExpectations(t)
rc := &RunContext{JobContainer: jobContainer}
err := rc.UpdateExtraPath(ctx, "/github/path")
if testcase.wantErr {
require.ErrorContains(t, err, "reading path file")
return
}
require.NoError(t, err)
assert.Equal(t, []string{testcase.want}, rc.ExtraPath)
})
}
}
func TestParentDir(t *testing.T) { func TestParentDir(t *testing.T) {
assert.Empty(t, parentDir("")) assert.Empty(t, parentDir(""))
assert.Empty(t, parentDir("repo")) assert.Empty(t, parentDir("repo"))
+74 -112
View File
@@ -6,7 +6,6 @@ package runner
import ( import (
"context" "context"
"encoding/json"
"fmt" "fmt"
"maps" "maps"
"os" "os"
@@ -22,61 +21,45 @@ import (
log "github.com/sirupsen/logrus" log "github.com/sirupsen/logrus"
) )
// Runner provides capabilities to run GitHub actions
type Runner interface {
NewPlanExecutor(plan *model.Plan) common.Executor
}
// Config contains the config for a new runner // Config contains the config for a new runner
type Config struct { type Config struct {
Actor string // the user that triggered the event Actor string // the user that triggered the event
Workdir string // path to working directory Workdir string // path to working directory
ActionCacheDir string // path used for caching action contents ActionCacheDir string // path used for caching action contents
ActionOfflineMode bool // when offline, use cached action contents ActionOfflineMode bool // when offline, use cached action contents
ActionCloneDepth int // limit history when cloning an action repo; 0 clones every branch in full ActionCloneDepth int // limit history when cloning an action repo, 0 clones every branch in full
BindWorkdir bool // bind the workdir to the job container BindWorkdir bool // bind the workdir to the job container
EventName string // name of event to run EventName string // name of event to run
EventPath string // path to JSON file to use for event.json in containers EventPath string // path to JSON file to use for event.json in containers
DefaultBranch string // name of the main branch for this repository ForcePull bool // force pulling of the image, even if already present
ReuseContainers bool // reuse containers to maintain state ForceRebuild bool // force rebuilding local docker image action
ForcePull bool // force pulling of the image, even if already present JSONLogger bool // use json or text logger
ForceRebuild bool // force rebuilding local docker image action Env map[string]string // env for containers
LogOutput bool // log the output from docker run Secrets map[string]string // list of secrets
JSONLogger bool // use json or text logger ExtraMasks []string // values to hide that are not in Secrets, such as the proxy password
LogPrefixJobID bool // switches from the full job name to the job id Vars map[string]string // list of vars
Env map[string]string // env for containers Token string // GitHub token
Inputs map[string]string // manually passed action inputs InsecureSecrets bool // switch hiding output when printing to terminal
Secrets map[string]string // list of secrets Privileged bool // use privileged mode
Vars map[string]string // list of vars UsernsMode string // user namespace to use
Token string // GitHub token ContainerArchitecture string // Desired OS/architecture platform for running containers
InsecureSecrets bool // switch hiding output when printing to terminal ContainerDaemonSocket string // Path to Docker daemon socket
Platforms map[string]string // list of platforms ContainerOptions string // Options for the job container
Privileged bool // use privileged mode UseGitIgnore bool // controls if paths in .gitignore should not be copied into container, default true
UsernsMode string // user namespace to use GitHubInstance string // GitHub instance to use, default "github.com"
ContainerArchitecture string // Desired OS/architecture platform for running containers ContainerCapAdd []string // list of kernel capabilities to add to the containers
ContainerDaemonSocket string // Path to Docker daemon socket ContainerCapDrop []string // list of kernel capabilities to remove from the containers
ContainerOptions string // Options for the job container ArtifactServerPath string // the path where the artifact server stores uploads
UseGitIgnore bool // controls if paths in .gitignore should not be copied into container, default true ArtifactServerAddr string // the address the artifact server binds to
GitHubInstance string // GitHub instance to use, default "github.com" ArtifactServerPort string // the port the artifact server binds to
ContainerCapAdd []string // list of kernel capabilities to add to the containers NoSkipCheckout bool // do not skip actions/checkout
ContainerCapDrop []string // list of kernel capabilities to remove from the containers DisableActEnv bool // do not inject the ACT=true environment variable into jobs
AutoRemove bool // controls if the container is automatically removed upon workflow completion ContainerNetworkMode docker_container.NetworkMode // the network mode of job containers (the value of --network)
ArtifactServerPath string // the path where the artifact server stores uploads ContainerNetworkCreateOptions container.NewDockerNetworkCreateExecutorInput // the default network create options
ArtifactServerAddr string // the address the artifact server binds to ProxyEnv map[string]string // the proxy variables the job runs with, also given to service containers and image builds
ArtifactServerPort string // the port the artifact server binds to NoActionPatch bool // run actions exactly as published, applying no compatibility patches, see patch_actions.go
NoSkipCheckout bool // do not skip actions/checkout
DisableActEnv bool // do not inject the ACT=true environment variable into jobs
RemoteName string // remote name in local git repo config
ReplaceGheActionWithGithubCom []string // Use actions from GitHub Enterprise instance to GitHub
ReplaceGheActionTokenWithGithubCom string // Token of private action repo on GitHub.
Matrix map[string]map[string]bool // Matrix config to run
ContainerNetworkMode docker_container.NetworkMode // the network mode of job containers (the value of --network)
ContainerNetworkCreateOptions container.NewDockerNetworkCreateExecutorInput // the default network create options
ActionCache ActionCache // Use a custom ActionCache Implementation
ProxyEnv map[string]string // the proxy variables the job runs with, also given to service containers and image builds
NoActionPatch bool // run actions exactly as published, applying no compatibility patches, see patch_actions.go
PresetGitHubContext *model.GithubContext // the preset github context, overrides some fields like DefaultBranch, Env, Secrets etc. PresetGitHubContext *model.GithubContext // overrides actor, ref, repository, token and related context fields
EventJSON string // the content of JSON file to use for event.json in containers, overrides EventPath EventJSON string // the content of JSON file to use for event.json in containers, overrides EventPath
ContainerNamePrefix string // the prefix of container name ContainerNamePrefix string // the prefix of container name
ContainerMaxLifetime time.Duration // the max lifetime of job containers ContainerMaxLifetime time.Duration // the max lifetime of job containers
@@ -88,17 +71,17 @@ type Config struct {
// differ from GitHubInstance when the runner registered with a different hostname than // differ from GitHubInstance when the runner registered with a different hostname than
// AppURL. It is never set for github.com or a GithubMirror, so the token stays on-instance. // AppURL. It is never set for github.com or a GithubMirror, so the token stays on-instance.
DefaultActionInstanceIsSelfHosted bool DefaultActionInstanceIsSelfHosted bool
PlatformPicker func(labels []string) string // platform picker, it will take precedence over Platforms if isn't nil PlatformPicker func(labels []string) string
JobLoggerLevel *log.Level // the level of job logger JobLoggerLevel *log.Level // the level of job logger
ValidVolumes []string // only volumes (and bind mounts) in this slice can be mounted on the job container or service containers ValidVolumes []string // only volumes (and bind mounts) in this slice can be mounted on the job container or service containers
SharedToolCache bool // one tool cache for all jobs instead of one per job SharedToolCache bool // one tool cache for all jobs instead of one per job
InsecureSkipTLS bool // whether to skip verifying TLS certificate of the Gitea instance InsecureSkipTLS bool // whether to skip verifying TLS certificate of the Gitea instance
MaxParallel int // max parallel jobs to run across all workflows (0 = no limit, uses CPU count) MaxParallel int // max parallel jobs to run across all workflows (0 = no limit, uses CPU count)
AllocatePTY bool // allocate a pseudo-TTY for each step's process AllocatePTY bool // allocate a pseudo-TTY for each step's process
ServiceReadyTimeout time.Duration // how long a job waits for its service containers to report healthy (0 uses the default) ServiceReadyTimeout time.Duration // how long a job waits for its service containers to report healthy (0 uses the default)
RunnerName string // name this runner registered with, reported as `runner.name`, defaults to the hostname RunnerName string // name this runner registered with, reported as `runner.name`, defaults to the hostname
JobStartedHook string // script run inside the job environment before the job's first step; ACTIONS_RUNNER_HOOK_JOB_STARTED is read from Env when empty JobStartedHook string // script run inside the job environment before the job's first step; ACTIONS_RUNNER_HOOK_JOB_STARTED is read from Env when empty
JobCompletedHook string // script run inside the job environment after the job's last step; ACTIONS_RUNNER_HOOK_JOB_COMPLETED is read from Env when empty JobCompletedHook string // script run inside the job environment after the job's last step; ACTIONS_RUNNER_HOOK_JOB_COMPLETED is read from Env when empty
} }
// RunnerDebug reports whether debug logging is on, exposed as `runner.debug` and // RunnerDebug reports whether debug logging is on, exposed as `runner.debug` and
@@ -141,8 +124,10 @@ type runnerImpl struct {
caller *caller // the job calling this runner (caller of a reusable workflow) caller *caller // the job calling this runner (caller of a reusable workflow)
} }
type Runner = runnerImpl
// New Creates a new Runner // New Creates a new Runner
func New(runnerConfig *Config) (Runner, error) { func New(runnerConfig *Config) (*Runner, error) {
runner := &runnerImpl{ runner := &runnerImpl{
config: runnerConfig, config: runnerConfig,
} }
@@ -150,7 +135,7 @@ func New(runnerConfig *Config) (Runner, error) {
return runner.configure() return runner.configure()
} }
func (runner *runnerImpl) configure() (Runner, error) { func (runner *runnerImpl) configure() (*runnerImpl, error) {
if runner.config.RunnerName == "" { if runner.config.RunnerName == "" {
// Callers that do not register, such as `exec`, still get a `runner.name`. // Callers that do not register, such as `exec`, still get a `runner.name`.
runner.config.RunnerName, _ = os.Hostname() runner.config.RunnerName, _ = os.Hostname()
@@ -166,15 +151,6 @@ func (runner *runnerImpl) configure() (Runner, error) {
return nil, err return nil, err
} }
runner.eventJSON = string(eventJSONBytes) runner.eventJSON = string(eventJSONBytes)
} else if len(runner.config.Inputs) != 0 {
eventMap := map[string]map[string]string{
"inputs": runner.config.Inputs,
}
eventJSON, err := json.Marshal(eventMap)
if err != nil {
return nil, err
}
runner.eventJSON = string(eventJSON)
} }
return runner, nil return runner, nil
} }
@@ -213,7 +189,6 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
log.Debugf("Job.Outputs: %v", job.Outputs) log.Debugf("Job.Outputs: %v", job.Outputs)
log.Debugf("Job.Uses: %v", job.Uses) log.Debugf("Job.Uses: %v", job.Uses)
log.Debugf("Job.With: %v", job.With) log.Debugf("Job.With: %v", job.With)
// log.Debugf("Job.RawSecrets: %v", job.RawSecrets)
log.Debugf("Job.Result: %v", job.Result) log.Debugf("Job.Result: %v", job.Result)
if job.Strategy != nil { if job.Strategy != nil {
@@ -223,7 +198,10 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
log.Debugf("Job.Strategy.MaxParallelString: %v", job.Strategy.MaxParallelString) log.Debugf("Job.Strategy.MaxParallelString: %v", job.Strategy.MaxParallelString)
log.Debugf("Job.Strategy.RawMatrix: %v", job.Strategy.RawMatrix) log.Debugf("Job.Strategy.RawMatrix: %v", job.Strategy.RawMatrix)
strategyRc := runner.newRunContext(ctx, run, nil) strategyRc, err := runner.newRunContext(ctx, run, nil)
if err != nil {
return err
}
// Resolve template expressions in the matrix node before Matrix() is called. // Resolve template expressions in the matrix node before Matrix() is called.
// On failure the literal string is kept and normalizeMatrixValue wraps it as a fallback. // On failure the literal string is kept and normalizeMatrixValue wraps it as a fallback.
if err := strategyRc.NewExpressionEvaluator(ctx).EvaluateYamlNode(ctx, &job.Strategy.RawMatrix); err != nil { if err := strategyRc.NewExpressionEvaluator(ctx).EvaluateYamlNode(ctx, &job.Strategy.RawMatrix); err != nil {
@@ -231,15 +209,11 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
} }
} }
var matrixes []map[string]any matrixes, err := job.GetMatrixes()
if m, err := job.GetMatrixes(); err != nil { if err != nil {
log.Errorf("Error while get job's matrix: %v", err) return fmt.Errorf("could not get job matrix: %w", err)
} else {
log.Debugf("Job Matrices: %v", m)
log.Debugf("Runner Matrices: %v", runner.config.Matrix)
matrixes = selectMatrixes(m, runner.config.Matrix)
} }
log.Debugf("Final matrix after applying user inclusions '%v'", matrixes) log.Debugf("Job Matrices: %v", matrixes)
maxParallel := 4 maxParallel := 4
if job.Strategy != nil { if job.Strategy != nil {
@@ -259,7 +233,10 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
log.Infof("Running job with maxParallel=%d for %d matrix combinations", maxParallel, len(matrixes)) log.Infof("Running job with maxParallel=%d for %d matrix combinations", maxParallel, len(matrixes))
for i, matrix := range matrixes { for i, matrix := range matrixes {
rc := runner.newRunContext(ctx, run, matrix) rc, err := runner.newRunContext(ctx, run, matrix)
if err != nil {
return err
}
rc.JobName = rc.Name rc.JobName = rc.Name
if len(matrixes) > 1 { if len(matrixes) > 1 {
rc.Name = fmt.Sprintf("%s-%d", rc.Name, i+1) rc.Name = fmt.Sprintf("%s-%d", rc.Name, i+1)
@@ -268,7 +245,7 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
maxJobNameLen = len(rc.String()) maxJobNameLen = len(rc.String())
} }
if rc.caller != nil { // For Gitea if rc.caller != nil { // For Gitea
rc.caller.setReusedWorkflowJobResult(rc.JobName, "pending") rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, "pending")
} }
stageExecutor = append(stageExecutor, func(ctx context.Context) error { stageExecutor = append(stageExecutor, func(ctx context.Context) error {
jobName := fmt.Sprintf("%-*s", maxJobNameLen, rc.String()) jobName := fmt.Sprintf("%-*s", maxJobNameLen, rc.String())
@@ -336,7 +313,7 @@ func handleFailure(plan *model.Plan) common.Executor {
for _, stage := range plan.Stages { for _, stage := range plan.Stages {
for _, run := range stage.Runs { for _, run := range stage.Runs {
if run.Job().Result == "failure" && !run.Job().ContinueOnError { if run.Job().Result == "failure" && !run.Job().ContinueOnError {
return fmt.Errorf("Job '%s' failed", run.String()) return fmt.Errorf("job '%s' failed", run.String())
} }
} }
} }
@@ -344,26 +321,7 @@ func handleFailure(plan *model.Plan) common.Executor {
} }
} }
func selectMatrixes(originalMatrixes []map[string]any, targetMatrixValues map[string]map[string]bool) []map[string]any { func (runner *runnerImpl) newRunContext(ctx context.Context, run *model.Run, matrix map[string]any) (*RunContext, error) {
matrixes := make([]map[string]any, 0)
for _, original := range originalMatrixes {
flag := true
for key, val := range original {
if allowedVals, ok := targetMatrixValues[key]; ok {
valToString := fmt.Sprintf("%v", val)
if _, ok := allowedVals[valToString]; !ok {
flag = false
}
}
}
if flag {
matrixes = append(matrixes, original)
}
}
return matrixes
}
func (runner *runnerImpl) newRunContext(ctx context.Context, run *model.Run, matrix map[string]any) *RunContext {
rc := &RunContext{ rc := &RunContext{
Config: runner.config, Config: runner.config,
Run: run, Run: run,
@@ -372,20 +330,24 @@ func (runner *runnerImpl) newRunContext(ctx context.Context, run *model.Run, mat
Matrix: matrix, Matrix: matrix,
caller: runner.caller, caller: runner.caller,
} }
if err := rc.resolveWorkflowCall(ctx); err != nil {
return nil, err
}
rc.ExprEval = rc.NewExpressionEvaluator(ctx) rc.ExprEval = rc.NewExpressionEvaluator(ctx)
rc.Name = rc.ExprEval.Interpolate(ctx, run.String()) rc.Name = rc.maskSecrets(rc.ExprEval.InterpolateName(ctx, run.String()))
// Snapshot the job's pristine output expressions now, before any matrix combo runs and // Snapshot the job's pristine output expressions now, before any matrix combo runs and
// rewrites the shared Job.Outputs (see interpolateOutputs). // rewrites the shared Job.Outputs (see interpolateOutputs).
if job := run.Job(); job != nil { if job := run.Job(); job != nil {
rc.outputTemplate = maps.Clone(job.Outputs) rc.outputTemplate = maps.Clone(job.Outputs)
} }
return rc return rc, nil
} }
// For Gitea // For Gitea
func (c *caller) setReusedWorkflowJobResult(jobName, result string) { // Keyed by job id, not name: only the values are read, and masking can collapse two names into one.
func (c *caller) setReusedWorkflowJobResult(jobID, result string) {
c.updateResultLock.Lock() c.updateResultLock.Lock()
defer c.updateResultLock.Unlock() defer c.updateResultLock.Unlock()
c.reusedWorkflowJobResults[jobName] = result c.reusedWorkflowJobResults[jobID] = result
} }
+52 -184
View File
@@ -8,12 +8,9 @@ import (
"bytes" "bytes"
"context" "context"
"fmt" "fmt"
"io"
"os" "os"
"path"
"path/filepath" "path/filepath"
"runtime" "runtime"
"slices"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -24,7 +21,6 @@ import (
"github.com/joho/godotenv" "github.com/joho/godotenv"
log "github.com/sirupsen/logrus" log "github.com/sirupsen/logrus"
assert "github.com/stretchr/testify/assert" assert "github.com/stretchr/testify/assert"
"go.yaml.in/yaml/v4"
) )
var ( var (
@@ -35,6 +31,17 @@ var (
secrets map[string]string secrets map[string]string
) )
func mapPlatformPicker(platforms map[string]string) func([]string) string {
return func(labels []string) string {
for _, label := range labels {
if image := platforms[strings.ToLower(label)]; image != "" {
return image
}
}
return ""
}
}
func init() { func init() {
if p := os.Getenv("ACT_TEST_IMAGE"); p != "" { if p := os.Getenv("ACT_TEST_IMAGE"); p != "" {
baseImage = p baseImage = p
@@ -162,10 +169,24 @@ func TestGraphEvent(t *testing.T) {
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.NotNil(t, plan) assert.NotNil(t, plan)
assert.Empty(t, plan.Stages) assert.Empty(t, plan.Stages)
}
// these two build the same action Dockerfiles into one image tag, so they cannot overlap for _, workflowPath := range []string{
var sharedImageWorkflows = []string{"local-action-dockerfile", "local-action-via-composite-dockerfile"} "testdata/workflow_dispatch_no_inputs_mapping/workflow_dispatch.yml",
"testdata/workflow_dispatch-scalar/workflow_dispatch.yml",
} {
planner, err := model.NewWorkflowPlanner(workflowPath, true)
if !assert.NoError(t, err, workflowPath) { //nolint:testifylint // pre-existing issue from nektos/act
continue
}
plan, err := planner.PlanEvent("workflow_dispatch")
if !assert.NoError(t, err, workflowPath) || !assert.NotNil(t, plan, workflowPath) { //nolint:testifylint // pre-existing issue from nektos/act
continue
}
if assert.Len(t, plan.Stages, 1, workflowPath) {
assert.Len(t, plan.Stages[0].Runs, 1, workflowPath)
}
}
}
// bounds concurrent plans: each job holds a network, and the daemon's address pool is finite // bounds concurrent plans: each job holds a network, and the daemon's address pool is finite
var planSlots = make(chan struct{}, 4) var planSlots = make(chan struct{}, 4)
@@ -189,29 +210,22 @@ func (j *TestJobFileInfo) runTest(ctx context.Context, t *testing.T, cfg *Config
fullWorkflowPath := filepath.Join(workdir, j.workflowPath) fullWorkflowPath := filepath.Join(workdir, j.workflowPath)
runnerConfig := &Config{ runnerConfig := &Config{
Workdir: workdir, Workdir: workdir,
BindWorkdir: false, BindWorkdir: false,
EventName: j.eventName, EventName: j.eventName,
EventPath: cfg.EventPath, EventPath: cfg.EventPath,
Platforms: j.platforms, PlatformPicker: mapPlatformPicker(j.platforms),
// fixtures reuse workflow and job names, so parallel tests would collide without this // fixtures reuse workflow and job names, so parallel tests would collide without this
ContainerNamePrefix: strings.ReplaceAll(t.Name(), "/", "-"), ContainerNamePrefix: strings.ReplaceAll(t.Name(), "/", "-"),
ReuseContainers: false,
// as the shipped runner does, else a fixture asserting a job failure keeps its
// container, and its network, on the daemon forever
AutoRemove: true,
// 0 would run jobs runtime.NumCPU()-wide, making the network peak machine-dependent // 0 would run jobs runtime.NumCPU()-wide, making the network peak machine-dependent
MaxParallel: 2, MaxParallel: 2,
ForceRebuild: true, ForceRebuild: true,
Env: cfg.Env, Env: cfg.Env,
Secrets: cfg.Secrets, Secrets: cfg.Secrets,
Inputs: cfg.Inputs,
GitHubInstance: "github.com", GitHubInstance: "github.com",
DefaultActionInstance: cfg.DefaultActionInstance, DefaultActionInstance: cfg.DefaultActionInstance,
ContainerArchitecture: cfg.ContainerArchitecture, ContainerArchitecture: cfg.ContainerArchitecture,
ContainerMaxLifetime: time.Hour, ContainerMaxLifetime: time.Hour,
Matrix: cfg.Matrix,
ActionCache: cfg.ActionCache,
ValidVolumes: []string{"**"}, // allow workflow-declared volumes (e.g. container-volumes) ValidVolumes: []string{"**"}, // allow workflow-declared volumes (e.g. container-volumes)
} }
@@ -224,11 +238,18 @@ func (j *TestJobFileInfo) runTest(ctx context.Context, t *testing.T, cfg *Config
plan, err := planner.PlanEvent(j.eventName) plan, err := planner.PlanEvent(j.eventName)
assert.True(t, (err == nil) != (plan == nil), "PlanEvent should return either a plan or an error") //nolint:testifylint // pre-existing issue from nektos/act assert.True(t, (err == nil) != (plan == nil), "PlanEvent should return either a plan or an error") //nolint:testifylint // pre-existing issue from nektos/act
if err == nil && plan != nil { if err == nil && plan != nil {
err = func() error { usesDocker := false
for _, platform := range j.platforms {
if platform != "-self-hosted" {
usesDocker = true
break
}
}
if usesDocker && !common.Dryrun(ctx) {
planSlots <- struct{}{} planSlots <- struct{}{}
defer func() { <-planSlots }() defer func() { <-planSlots }()
return runner.NewPlanExecutor(plan)(ctx) }
}() err = runner.NewPlanExecutor(plan)(ctx)
if j.errorMessage == "" { if j.errorMessage == "" {
assert.NoError(t, err, fullWorkflowPath) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err, fullWorkflowPath) //nolint:testifylint // pre-existing issue from nektos/act
} else { } else {
@@ -239,10 +260,6 @@ func (j *TestJobFileInfo) runTest(ctx context.Context, t *testing.T, cfg *Config
fmt.Println("::endgroup::") //nolint:forbidigo // pre-existing issue from nektos/act fmt.Println("::endgroup::") //nolint:forbidigo // pre-existing issue from nektos/act
} }
type TestConfig struct {
LocalRepositories map[string]string `yaml:"local-repositories"`
}
func TestRunEvent(t *testing.T) { func TestRunEvent(t *testing.T) {
requireDocker(t) requireDocker(t)
t.Parallel() t.Parallel()
@@ -265,6 +282,7 @@ func TestRunEvent(t *testing.T) {
{workdir, "uses-composite", "push", "", platforms, secrets}, {workdir, "uses-composite", "push", "", platforms, secrets},
{workdir, "uses-composite-with-error", "push", "Job 'failing-composite-action' failed", platforms, secrets}, {workdir, "uses-composite-with-error", "push", "Job 'failing-composite-action' failed", platforms, secrets},
{workdir, "uses-docker-url", "push", "", platforms, secrets}, {workdir, "uses-docker-url", "push", "", platforms, secrets},
{workdir, "uses-step-if-inputs-not-leaked", "push", "", platforms, secrets},
{workdir, "act-composite-env-test", "push", "", platforms, secrets}, {workdir, "act-composite-env-test", "push", "", platforms, secrets},
// Eval // Eval
@@ -276,9 +294,7 @@ func TestRunEvent(t *testing.T) {
{workdir, "basic", "push", "", platforms, secrets}, {workdir, "basic", "push", "", platforms, secrets},
{workdir, "fail", "push", "exit with `FAILURE`: 1", platforms, secrets}, {workdir, "fail", "push", "exit with `FAILURE`: 1", platforms, secrets},
{workdir, "checkout", "push", "", platforms, secrets},
{workdir, "job-container", "push", "", platforms, secrets}, {workdir, "job-container", "push", "", platforms, secrets},
{workdir, "job-container-non-root", "push", "", platforms, secrets},
{workdir, "job-container-invalid-credentials", "push", "failed to handle credentials: failed to interpolate container.credentials.password", platforms, secrets}, {workdir, "job-container-invalid-credentials", "push", "failed to handle credentials: failed to interpolate container.credentials.password", platforms, secrets},
{workdir, "container-hostname", "push", "", platforms, secrets}, {workdir, "container-hostname", "push", "", platforms, secrets},
{workdir, "matrix", "push", "", platforms, secrets}, {workdir, "matrix", "push", "", platforms, secrets},
@@ -288,17 +304,14 @@ func TestRunEvent(t *testing.T) {
{workdir, "defaults-run", "push", "", platforms, secrets}, {workdir, "defaults-run", "push", "", platforms, secrets},
{workdir, "composite-fail-with-output", "push", "", platforms, secrets}, {workdir, "composite-fail-with-output", "push", "", platforms, secrets},
{workdir, "issue-597", "push", "", platforms, secrets}, {workdir, "issue-597", "push", "", platforms, secrets},
{workdir, "issue-598", "push", "", platforms, secrets},
{workdir, "if-env-act", "push", "", platforms, secrets}, {workdir, "if-env-act", "push", "", platforms, secrets},
{workdir, "env-and-path", "push", "", platforms, secrets},
{workdir, "environment-files", "push", "", platforms, secrets}, {workdir, "environment-files", "push", "", platforms, secrets},
{workdir, "GITHUB_STATE", "push", "", platforms, secrets}, {workdir, "GITHUB_STATE", "push", "", platforms, secrets},
{workdir, "environment-files-parser-bug", "push", "", platforms, secrets}, {workdir, "environment-files-parser-bug", "push", "", platforms, secrets},
{workdir, "non-existent-action", "push", "Job 'nopanic' failed", platforms, secrets}, {workdir, "non-existent-action", "push", "Job 'nopanic' failed", platforms, secrets},
{workdir, "outputs", "push", "", platforms, secrets}, {workdir, "outputs", "push", "", platforms, secrets},
{workdir, "networking", "push", "", platforms, secrets}, {workdir, "networking", "push", "", platforms, secrets},
{workdir, "steps-context/conclusion", "push", "", platforms, secrets}, {workdir, "steps-context", "push", "", platforms, secrets},
{workdir, "steps-context/outcome", "push", "", platforms, secrets},
{workdir, "job-status-check", "push", "job 'fail' failed", platforms, secrets}, {workdir, "job-status-check", "push", "job 'fail' failed", platforms, secrets},
{workdir, "if-expressions", "push", "Job 'mytest' failed", platforms, secrets}, {workdir, "if-expressions", "push", "Job 'mytest' failed", platforms, secrets},
{workdir, "actions-environment-and-context-tests", "push", "", platforms, secrets}, {workdir, "actions-environment-and-context-tests", "push", "", platforms, secrets},
@@ -308,8 +321,6 @@ func TestRunEvent(t *testing.T) {
{workdir, "ensure-post-steps", "push", "Job 'second-post-step-should-fail' failed", platforms, secrets}, {workdir, "ensure-post-steps", "push", "Job 'second-post-step-should-fail' failed", platforms, secrets},
{workdir, "workflow_call_inputs", "workflow_call", "", platforms, secrets}, {workdir, "workflow_call_inputs", "workflow_call", "", platforms, secrets},
{workdir, "workflow_dispatch", "workflow_dispatch", "", platforms, secrets}, {workdir, "workflow_dispatch", "workflow_dispatch", "", platforms, secrets},
{workdir, "workflow_dispatch_no_inputs_mapping", "workflow_dispatch", "", platforms, secrets},
{workdir, "workflow_dispatch-scalar", "workflow_dispatch", "", platforms, secrets},
{workdir, "workflow_dispatch-scalar-composite-action", "workflow_dispatch", "", platforms, secrets}, {workdir, "workflow_dispatch-scalar-composite-action", "workflow_dispatch", "", platforms, secrets},
{workdir, "job-needs-context-contains-result", "push", "", platforms, secrets}, {workdir, "job-needs-context-contains-result", "push", "", platforms, secrets},
{workdir, "container-volumes", "push", "", platforms, secrets}, {workdir, "container-volumes", "push", "", platforms, secrets},
@@ -323,9 +334,6 @@ func TestRunEvent(t *testing.T) {
{workdir, "services", "push", "", platforms, secrets}, {workdir, "services", "push", "", platforms, secrets},
{workdir, "services-with-container", "push", "", platforms, secrets}, {workdir, "services-with-container", "push", "", platforms, secrets},
{workdir, "services-empty-image", "push", "", platforms, secrets}, {workdir, "services-empty-image", "push", "", platforms, secrets},
// local remote action overrides
{workdir, "local-remote-action-overrides", "push", "", platforms, secrets},
} }
for _, table := range tables { for _, table := range tables {
@@ -334,12 +342,11 @@ func TestRunEvent(t *testing.T) {
// host /proc bind mounts are Linux-Docker-only // host /proc bind mounts are Linux-Docker-only
requireLinuxDocker(t) requireLinuxDocker(t)
} }
if !slices.Contains(sharedImageWorkflows, table.workflowPath) { t.Parallel()
t.Parallel()
}
config := &Config{ config := &Config{
Secrets: table.secrets, Secrets: table.secrets,
Env: map[string]string{"GITHUB_REPOSITORY": t.Name()},
} }
eventFile := filepath.Join(workdir, table.workflowPath, "event.json") eventFile := filepath.Join(workdir, table.workflowPath, "event.json")
@@ -347,22 +354,6 @@ func TestRunEvent(t *testing.T) {
config.EventPath = eventFile config.EventPath = eventFile
} }
testConfigFile := filepath.Join(workdir, table.workflowPath, "config.yml")
if file, err := os.ReadFile(testConfigFile); err == nil {
testConfig := &TestConfig{}
if yaml.Unmarshal(file, testConfig) == nil {
if testConfig.LocalRepositories != nil {
config.ActionCache = &LocalRepositoryCache{
Parent: GoGitActionCache{
path.Clean(path.Join(workdir, "cache")),
},
LocalRepositories: testConfig.LocalRepositories,
CacheDirCache: map[string]string{},
}
}
}
}
table.runTest(ctx, t, config) table.runTest(ctx, t, config)
}) })
} }
@@ -407,20 +398,16 @@ func TestRunEventHostEnvironment(t *testing.T) {
{workdir, "evalmatrix-merge-map", "push", "", platforms, secrets}, {workdir, "evalmatrix-merge-map", "push", "", platforms, secrets},
{workdir, "evalmatrix-merge-array", "push", "", platforms, secrets}, {workdir, "evalmatrix-merge-array", "push", "", platforms, secrets},
{workdir, "fail", "push", "exit with `FAILURE`: 1", platforms, secrets},
{workdir, "checkout", "push", "", platforms, secrets},
{workdir, "matrix", "push", "", platforms, secrets}, {workdir, "matrix", "push", "", platforms, secrets},
{workdir, "commands", "push", "", platforms, secrets}, {workdir, "commands", "push", "", platforms, secrets},
{workdir, "defaults-run", "push", "", platforms, secrets}, {workdir, "defaults-run", "push", "", platforms, secrets},
{workdir, "composite-fail-with-output", "push", "", platforms, secrets}, {workdir, "composite-fail-with-output", "push", "", platforms, secrets},
{workdir, "issue-597", "push", "", platforms, secrets}, {workdir, "issue-597", "push", "", platforms, secrets},
{workdir, "issue-598", "push", "", platforms, secrets},
{workdir, "if-env-act", "push", "", platforms, secrets}, {workdir, "if-env-act", "push", "", platforms, secrets},
{workdir, "env-and-path", "push", "", platforms, secrets}, {workdir, "env-and-path", "push", "", platforms, secrets},
{workdir, "non-existent-action", "push", "Job 'nopanic' failed", platforms, secrets}, {workdir, "non-existent-action", "push", "Job 'nopanic' failed", platforms, secrets},
{workdir, "outputs", "push", "", platforms, secrets}, {workdir, "outputs", "push", "", platforms, secrets},
{workdir, "steps-context/conclusion", "push", "", platforms, secrets}, {workdir, "steps-context", "push", "", platforms, secrets},
{workdir, "steps-context/outcome", "push", "", platforms, secrets},
{workdir, "job-status-check", "push", "job 'fail' failed", platforms, secrets}, {workdir, "job-status-check", "push", "job 'fail' failed", platforms, secrets},
{workdir, "if-expressions", "push", "Job 'mytest' failed", platforms, secrets}, {workdir, "if-expressions", "push", "Job 'mytest' failed", platforms, secrets},
{workdir, "evalenv", "push", "", platforms, secrets}, {workdir, "evalenv", "push", "", platforms, secrets},
@@ -453,6 +440,7 @@ func TestRunEventHostEnvironment(t *testing.T) {
}...) }...)
} }
hostPlanSlots := make(chan struct{}, 2)
for _, table := range tables { for _, table := range tables {
t.Run(table.workflowPath, func(t *testing.T) { t.Run(table.workflowPath, func(t *testing.T) {
switch table.workflowPath { switch table.workflowPath {
@@ -461,6 +449,9 @@ func TestRunEventHostEnvironment(t *testing.T) {
case "nix-prepend-path": case "nix-prepend-path":
requireHostTools(t, "nix") requireHostTools(t, "nix")
} }
t.Parallel()
hostPlanSlots <- struct{}{}
defer func() { <-hostPlanSlots }()
table.runTest(ctx, t, &Config{}) table.runTest(ctx, t, &Config{})
}) })
} }
@@ -503,47 +494,6 @@ func TestReusableWorkflowCaller(t *testing.T) {
table.runTest(context.Background(), t, &Config{Secrets: table.secrets}) table.runTest(context.Background(), t, &Config{Secrets: table.secrets})
} }
type maskJobLoggerFactory struct {
Output bytes.Buffer
}
func (f *maskJobLoggerFactory) WithJobLogger() *log.Logger {
logger := log.New()
logger.SetOutput(io.MultiWriter(&f.Output, os.Stdout))
logger.SetLevel(log.DebugLevel)
return logger
}
func TestMaskValues(t *testing.T) {
t.Parallel()
assertNoSecret := func(text, secret string) { //nolint:unparam // pre-existing issue from nektos/act
found := strings.Contains(text, "composite secret")
if found {
fmt.Printf("\nFound Secret in the given text:\n%s\n", text) //nolint:forbidigo // pre-existing issue from nektos/act
}
assert.False(t, strings.Contains(text, "composite secret")) //nolint:testifylint // pre-existing issue from nektos/act
}
requireDocker(t)
log.SetLevel(log.DebugLevel)
tjfi := TestJobFileInfo{
workdir: workdir,
workflowPath: "mask-values",
eventName: "push",
errorMessage: "",
platforms: platforms,
}
logger := &maskJobLoggerFactory{}
tjfi.runTest(WithJobLoggerFactory(common.WithLogger(context.Background(), logger.WithJobLogger()), logger), t, &Config{})
output := logger.Output.String()
assertNoSecret(output, "secret value")
assertNoSecret(output, "YWJjCg==")
}
func TestRunEventSecrets(t *testing.T) { func TestRunEventSecrets(t *testing.T) {
requireDocker(t) requireDocker(t)
t.Parallel() t.Parallel()
@@ -565,62 +515,6 @@ func TestRunEventSecrets(t *testing.T) {
tjfi.runTest(context.Background(), t, &Config{Secrets: secrets, Env: env}) tjfi.runTest(context.Background(), t, &Config{Secrets: secrets, Env: env})
} }
func TestRunWithService(t *testing.T) {
requireDocker(t)
log.SetLevel(log.DebugLevel)
ctx := context.Background()
platforms := map[string]string{
"ubuntu-latest": "node:24-bookworm-slim",
}
workflowPath := "services"
eventName := "push"
workdir, err := filepath.Abs("testdata")
assert.NoError(t, err, workflowPath) //nolint:testifylint // pre-existing issue from nektos/act
runnerConfig := &Config{
Workdir: workdir,
EventName: eventName,
Platforms: platforms,
ReuseContainers: false,
ContainerMaxLifetime: time.Hour, // otherwise the job container is `sleep 0` and exits at once
}
runner, err := New(runnerConfig)
assert.NoError(t, err, workflowPath) //nolint:testifylint // pre-existing issue from nektos/act
planner, err := model.NewWorkflowPlanner("testdata/"+workflowPath, true)
assert.NoError(t, err, workflowPath) //nolint:testifylint // pre-existing issue from nektos/act
plan, err := planner.PlanEvent(eventName)
assert.NoError(t, err, workflowPath) //nolint:testifylint // pre-existing issue from nektos/act
err = runner.NewPlanExecutor(plan)(ctx)
assert.NoError(t, err, workflowPath)
}
func TestRunActionInputs(t *testing.T) {
t.Parallel()
requireDocker(t)
workflowPath := "input-from-cli"
tjfi := TestJobFileInfo{
workdir: workdir,
workflowPath: workflowPath,
eventName: "workflow_dispatch",
errorMessage: "",
platforms: platforms,
}
inputs := map[string]string{
"SOME_INPUT": "input",
}
tjfi.runTest(context.Background(), t, &Config{Inputs: inputs})
}
func TestRunEventPullRequest(t *testing.T) { func TestRunEventPullRequest(t *testing.T) {
t.Parallel() t.Parallel()
requireDocker(t) requireDocker(t)
@@ -637,29 +531,3 @@ func TestRunEventPullRequest(t *testing.T) {
tjfi.runTest(context.Background(), t, &Config{EventPath: filepath.Join(workdir, workflowPath, "event.json")}) tjfi.runTest(context.Background(), t, &Config{EventPath: filepath.Join(workdir, workflowPath, "event.json")})
} }
func TestRunMatrixWithUserDefinedInclusions(t *testing.T) {
t.Parallel()
requireDocker(t)
workflowPath := "matrix-with-user-inclusions"
tjfi := TestJobFileInfo{
workdir: workdir,
workflowPath: workflowPath,
eventName: "push",
errorMessage: "",
platforms: platforms,
}
matrix := map[string]map[string]bool{
"node": {
"8": true,
"8.x": true,
},
"os": {
"ubuntu-18.04": true,
},
}
tjfi.runTest(context.Background(), t, &Config{Matrix: matrix})
}
+43 -48
View File
@@ -11,6 +11,7 @@ import (
"path" "path"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
@@ -58,13 +59,10 @@ func (s stepStage) String() string {
func processRunnerEnvFileCommand(ctx context.Context, fileName string, rc *RunContext, setter func(context.Context, map[string]string, string)) error { func processRunnerEnvFileCommand(ctx context.Context, fileName string, rc *RunContext, setter func(context.Context, map[string]string, string)) error {
env := map[string]string{} env := map[string]string{}
err := rc.JobContainer.UpdateFromEnv(path.Join(rc.JobContainer.GetActPath(), fileName), &env)(ctx) err := rc.JobContainer.UpdateFromEnv(path.Join(rc.JobContainer.GetActPath(), fileName), &env)(ctx)
if err != nil {
return err
}
for k, v := range env { for k, v := range env {
setter(ctx, map[string]string{"name": k}, v) setter(ctx, map[string]string{"name": k}, v)
} }
return nil return err
} }
func runStepExecutor(step step, stage stepStage, executor common.Executor) common.Executor { func runStepExecutor(step step, stage stepStage, executor common.Executor) common.Executor {
@@ -86,25 +84,25 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
} }
err := setupEnv(ctx, step) err := setupEnv(ctx, step)
if err != nil { var runStep bool
return err if err == nil {
runStep, err = isStepEnabled(ctx, ifExpression, step, stage)
} }
runStep, err := isStepEnabled(ctx, ifExpression, step, stage)
if err != nil { if err != nil {
stepResult.Conclusion = model.StepStatusFailure stepResult.Conclusion = model.StepStatusFailure
stepResult.Outcome = model.StepStatusFailure stepResult.Outcome = model.StepStatusFailure
logger.WithField("stepResult", stepResult.Conclusion).Infof("Failure - %s %s", stage, stepModel)
return err return err
} }
if !runStep { if !runStep {
stepResult.Conclusion = model.StepStatusSkipped stepResult.Conclusion = model.StepStatusSkipped
stepResult.Outcome = model.StepStatusSkipped stepResult.Outcome = model.StepStatusSkipped
logger.WithField("stepResult", stepResult.Outcome).Debugf("Skipping step '%s' due to '%s'", stepModel, ifExpression) logger.WithField("stepResult", stepResult.Conclusion).Debugf("Skipping step '%s' due to '%s'", stepModel, ifExpression)
return nil return nil
} }
stepString := rc.ExprEval.Interpolate(ctx, stepModel.String()) stepString := rc.ExprEval.InterpolateName(ctx, stepModel.String())
if strings.Contains(stepString, "::add-mask::") { if strings.Contains(stepString, "::add-mask::") {
stepString = "add-mask command" stepString = "add-mask command"
} }
@@ -160,11 +158,11 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
if topRC.summaryFileInitialized == nil { if topRC.summaryFileInitialized == nil {
topRC.summaryFileInitialized = map[int]bool{} topRC.summaryFileInitialized = map[int]bool{}
} }
_ = rc.JobContainer.Copy(actPath, files...)(ctx)
if !topRC.summaryFileInitialized[stepSummaryIndex] { if !topRC.summaryFileInitialized[stepSummaryIndex] {
files = append(files, &container.FileEntry{Name: summaryFileCommand, Mode: 0o666}) _ = rc.JobContainer.Copy(actPath, &container.FileEntry{Name: summaryFileCommand, Mode: 0o666})(ctx)
topRC.summaryFileInitialized[stepSummaryIndex] = true topRC.summaryFileInitialized[stepSummaryIndex] = true
} }
_ = rc.JobContainer.Copy(actPath, files...)(ctx)
// The command handler needs the step's env to judge ACTIONS_ALLOW_UNSECURE_COMMANDS. // The command handler needs the step's env to judge ACTIONS_ALLOW_UNSECURE_COMMANDS.
// Cloned: the step executor keeps writing to its own env map after this point, on a // Cloned: the step executor keeps writing to its own env map after this point, on a
@@ -182,15 +180,29 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
if err == nil { if err == nil {
err = insecureErr err = insecureErr
} }
if fileErr := processRunnerEnvFileCommand(ctx, envFileCommand, rc, rc.setEnvFile); fileErr != nil && err == nil {
err = fileErr
}
if fileErr := processRunnerEnvFileCommand(ctx, stateFileCommand, rc, rc.saveState); fileErr != nil && err == nil {
err = fileErr
}
if fileErr := processRunnerEnvFileCommand(ctx, outputFileCommand, rc, rc.setOutput); fileErr != nil && err == nil {
err = fileErr
}
if fileErr := rc.UpdateExtraPath(ctx, path.Join(actPath, pathFileCommand)); fileErr != nil && err == nil {
err = fileErr
}
_ = rc.JobContainer.Copy(actPath, files...)(ctx)
if err == nil { if err == nil {
logger.WithField("stepResult", stepResult.Outcome).Infof("Success - %s %s", stage, stepString) logger.WithField("stepResult", stepResult.Conclusion).Infof("Success - %s %s", stage, stepString)
} else { } else {
stepResult.Outcome = model.StepStatusFailure stepResult.Outcome = model.StepStatusFailure
continueOnError, parseErr := isContinueOnError(ctx, stepModel.RawContinueOnError, step, stage) continueOnError, parseErr := isContinueOnError(ctx, stepModel.RawContinueOnError, step, stage)
if parseErr != nil { if parseErr != nil {
stepResult.Conclusion = model.StepStatusFailure stepResult.Conclusion = model.StepStatusFailure
logger.WithField("stepResult", stepResult.Conclusion).Infof("Failure - %s %s", stage, stepString)
return parseErr return parseErr
} }
@@ -205,68 +217,50 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
// Infof: Errorf entries are promoted to the user log by the reporter, // Infof: Errorf entries are promoted to the user log by the reporter,
// which would duplicate the ##[error] annotation emitted elsewhere. // which would duplicate the ##[error] annotation emitted elsewhere.
logger.WithField("stepResult", stepResult.Outcome).Infof("Failure - %s %s", stage, stepString) logger.WithField("stepResult", stepResult.Conclusion).Infof("Failure - %s %s", stage, stepString)
}
// Process Runner File Commands
orgerr := err
err = processRunnerEnvFileCommand(ctx, envFileCommand, rc, rc.setEnv)
if err != nil {
return err
}
err = processRunnerEnvFileCommand(ctx, stateFileCommand, rc, rc.saveState)
if err != nil {
return err
}
err = processRunnerEnvFileCommand(ctx, outputFileCommand, rc, rc.setOutput)
if err != nil {
return err
}
err = rc.UpdateExtraPath(ctx, path.Join(actPath, pathFileCommand))
if err != nil {
return err
}
if orgerr != nil {
return orgerr
} }
return err return err
} }
} }
func evaluateStepTimeout(ctx context.Context, exprEval ExpressionEvaluator, stepModel *model.Step) (context.Context, context.CancelFunc) { func evaluateStepTimeout(ctx context.Context, exprEval *expressionEvaluator, stepModel *model.Step) (context.Context, context.CancelFunc) {
timeout := exprEval.Interpolate(ctx, stepModel.TimeoutMinutes) timeout, err := exprEval.Interpolate(ctx, stepModel.TimeoutMinutes)
if timeout != "" { if err != nil {
if timeOutMinutes, err := strconv.ParseInt(timeout, 10, 64); err == nil { common.Logger(ctx).Errorf("An error occurred when attempting to determine the step timeout: %s", err)
} else if timeout != "" {
if timeOutMinutes, err := strconv.ParseInt(timeout, 10, 64); err == nil && timeOutMinutes > 0 {
return context.WithTimeout(ctx, time.Duration(timeOutMinutes)*time.Minute) return context.WithTimeout(ctx, time.Duration(timeOutMinutes)*time.Minute)
} }
} }
return ctx, func() {} return ctx, func() {}
} }
func setupEnv(ctx context.Context, step step) error { //nolint:unparam // pre-existing issue from nektos/act func setupEnv(ctx context.Context, step step) error {
rc := step.getRunContext() rc := step.getRunContext()
mergeEnv(ctx, step) mergeEnv(ctx, step)
// merge step env last, since it should not be overwritten // merge step env last, since it should not be overwritten
mergeIntoMap(step, step.getEnv(), step.getStepModel().GetEnv()) mergeIntoMap(step, step.getEnv(), step.getStepModel().GetEnv())
var err error
exprEval := rc.NewExpressionEvaluator(ctx) exprEval := rc.NewExpressionEvaluator(ctx)
for k, v := range *step.getEnv() { for k, v := range *step.getEnv() {
if !strings.HasPrefix(k, "INPUT_") { if !strings.HasPrefix(k, "INPUT_") {
(*step.getEnv())[k] = exprEval.Interpolate(ctx, v) if (*step.getEnv())[k], err = exprEval.Interpolate(ctx, v); err != nil {
return fmt.Errorf("unable to interpolate env %s: %w", k, err)
}
} }
} }
// after we have an evaluated step context, update the expressions evaluator with a new env context // after we have an evaluated step context, update the expressions evaluator with a new env context
// you can use step level env in the with property of a uses construct // you can use step level env in the with property of a uses construct
exprEval = rc.NewExpressionEvaluatorWithEnv(ctx, *step.getEnv()) inputEval := sync.OnceValue(func() *expressionEvaluator { return rc.NewExpressionEvaluatorWithEnv(ctx, *step.getEnv()) })
for k, v := range *step.getEnv() { for k, v := range *step.getEnv() {
if strings.HasPrefix(k, "INPUT_") { if strings.HasPrefix(k, "INPUT_") {
(*step.getEnv())[k] = exprEval.Interpolate(ctx, v) if (*step.getEnv())[k], err = inputEval().Interpolate(ctx, v); err != nil {
return fmt.Errorf("unable to interpolate env %s: %w", k, err)
}
} }
} }
// For Gitea, reduce log noise
// common.Logger(ctx).Debugf("setupEnv => %v", *step.getEnv())
return nil return nil
} }
@@ -277,7 +271,8 @@ func mergeEnv(ctx context.Context, step step) {
c := job.Container() c := job.Container()
if c != nil { if c != nil {
mergeIntoMap(step, env, rc.GetEnv(), c.Env) // container env is the image's baseline, which job env and $GITHUB_ENV override
mergeIntoMap(step, env, c.Env, rc.GetEnv())
} else { } else {
mergeIntoMap(step, env, rc.GetEnv()) mergeIntoMap(step, env, rc.GetEnv())
} }
+33 -34
View File
@@ -33,10 +33,7 @@ type stepActionLocal struct {
func (sal *stepActionLocal) pre() common.Executor { func (sal *stepActionLocal) pre() common.Executor {
sal.env = map[string]string{} sal.env = map[string]string{}
return common.NewPipelineExecutor()
return func(ctx context.Context) error {
return nil
}
} }
func (sal *stepActionLocal) main() common.Executor { func (sal *stepActionLocal) main() common.Executor {
@@ -50,39 +47,37 @@ func (sal *stepActionLocal) main() common.Executor {
defer rawLogger.Infof("::endgroup::") defer rawLogger.Infof("::endgroup::")
actionDir := filepath.Join(sal.getRunContext().Config.Workdir, sal.Step.Uses) actionDir := filepath.Join(sal.getRunContext().Config.Workdir, sal.Step.Uses)
_, containerActionPath := getContainerActionPaths(sal.Step, path.Join(actionDir, ""), sal.RunContext)
localReader := func(ctx context.Context) actionYamlReader { localReader := func(filename string) (io.Reader, io.Closer, error) {
_, cpath := getContainerActionPaths(sal.Step, path.Join(actionDir, ""), sal.RunContext) spath := path.Join(containerActionPath, filename)
return func(filename string) (io.Reader, io.Closer, error) { for range maxSymlinkDepth {
spath := path.Join(cpath, filename) tars, err := sal.RunContext.JobContainer.GetContainerArchive(ctx, spath)
for range maxSymlinkDepth { if errors.Is(err, fs.ErrNotExist) {
tars, err := sal.RunContext.JobContainer.GetContainerArchive(ctx, spath) return nil, nil, err
if errors.Is(err, fs.ErrNotExist) { } else if err != nil {
return nil, nil, fs.ErrNotExist
}
treader := tar.NewReader(tars)
header, err := treader.Next()
if errors.Is(err, io.EOF) {
return nil, nil, os.ErrNotExist
} else if err != nil {
return nil, nil, err
}
if header.FileInfo().Mode()&os.ModeSymlink == os.ModeSymlink {
spath, err = symlinkJoin(spath, header.Linkname, containerActionPath)
if err != nil {
return nil, nil, err return nil, nil, err
} else if err != nil {
return nil, nil, fs.ErrNotExist
} }
treader := tar.NewReader(tars) } else {
header, err := treader.Next() return treader, tars, nil
if errors.Is(err, io.EOF) {
return nil, nil, os.ErrNotExist
} else if err != nil {
return nil, nil, err
}
if header.FileInfo().Mode()&os.ModeSymlink == os.ModeSymlink {
spath, err = symlinkJoin(spath, header.Linkname, cpath)
if err != nil {
return nil, nil, err
}
} else {
return treader, tars, nil
}
} }
return nil, nil, fmt.Errorf("max depth %d of symlinks exceeded while reading %s", maxSymlinkDepth, spath)
} }
return nil, nil, fmt.Errorf("max depth %d of symlinks exceeded while reading %s", maxSymlinkDepth, spath)
} }
actionModel, err := sal.readAction(ctx, sal.Step, actionDir, "", localReader(ctx), os.WriteFile) actionModel, err := sal.readAction(ctx, sal.Step, actionDir, "", localReader, os.WriteFile)
if err != nil { if err != nil {
return err return err
} }
@@ -127,15 +122,19 @@ func (sal *stepActionLocal) getActionModel() *model.Action {
return sal.action return sal.action
} }
func (sal *stepActionLocal) getCompositeRunContext(ctx context.Context) *RunContext { func (sal *stepActionLocal) getCompositeRunContext(ctx context.Context) (*RunContext, error) {
if sal.compositeRunContext == nil { if sal.compositeRunContext == nil {
actionDir := filepath.Join(sal.RunContext.Config.Workdir, sal.Step.Uses) actionDir := filepath.Join(sal.RunContext.Config.Workdir, sal.Step.Uses)
_, containerActionDir := getContainerActionPaths(sal.getStepModel(), actionDir, sal.RunContext) _, containerActionDir := getContainerActionPaths(sal.getStepModel(), actionDir, sal.RunContext)
sal.compositeRunContext = newCompositeRunContext(ctx, sal.RunContext, sal, containerActionDir) compositeRunContext, err := newCompositeRunContext(ctx, sal.RunContext, sal, containerActionDir)
sal.compositeSteps = sal.compositeRunContext.compositeExecutor(sal.action) if err != nil {
return nil, err
}
sal.compositeRunContext = compositeRunContext
sal.compositeSteps = compositeRunContext.compositeExecutor(sal.action)
} }
return sal.compositeRunContext return sal.compositeRunContext, nil
} }
func (sal *stepActionLocal) getCompositeSteps() *compositeSteps { func (sal *stepActionLocal) getCompositeSteps() *compositeSteps {
+11 -28
View File
@@ -74,27 +74,17 @@ func TestStepActionLocalTest(t *testing.T) {
salm.On("readAction", sal.Step, filepath.Clean("/tmp/path/to/action"), "", mock.Anything, mock.Anything). salm.On("readAction", sal.Step, filepath.Clean("/tmp/path/to/action"), "", mock.Anything, mock.Anything).
Return(&model.Action{}, nil) Return(&model.Action{}, nil)
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(func(ctx context.Context) error { cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil) cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
salm.On("runAction", sal, filepath.Clean("/tmp/path/to/action"), (*remoteAction)(nil)).Return(func(ctx context.Context) error { salm.On("runAction", sal, filepath.Clean("/tmp/path/to/action"), (*remoteAction)(nil)).Return(noopExecutor)
return nil
})
err := sal.pre()(ctx) err := sal.pre()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
@@ -264,26 +254,19 @@ func TestStepActionLocalPost(t *testing.T) {
if tt.mocks.exec { if tt.mocks.exec {
suffixMatcher := func(suffix string) any { suffixMatcher := func(suffix string) any {
return mock.MatchedBy(func(array []string) bool { return mock.MatchedBy(func(array []string) bool {
return strings.HasSuffix(array[1], suffix) return len(array) == 3 && array[0] == "node" && array[1] == "--preserve-symlinks-main" &&
strings.HasSuffix(array[2], suffix)
}) })
} }
cm.On("Exec", suffixMatcher("runner/local/action/post.js"), sal.env, "", "").Return(func(ctx context.Context) error { return tt.err }) cm.On("Exec", suffixMatcher("runner/local/action/post.js"), sal.env, "", "").Return(func(ctx context.Context) error { return tt.err })
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(func(ctx context.Context) error { cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil) cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
} }
+34 -75
View File
@@ -5,7 +5,6 @@
package runner package runner
import ( import (
"archive/tar"
"context" "context"
"errors" "errors"
"fmt" "fmt"
@@ -33,7 +32,6 @@ type stepActionRemote struct {
action *model.Action action *model.Action
env map[string]string env map[string]string
remoteAction *remoteAction remoteAction *remoteAction
cacheDir string
resolvedSha string resolvedSha string
} }
@@ -53,7 +51,11 @@ func (sar *stepActionRemote) prepareActionExecutor() common.Executor {
// Since actions can specify the download source via a url prefix. // Since actions can specify the download source via a url prefix.
// The prefix may contain some sensitive information that needs to be stored in secrets, // The prefix may contain some sensitive information that needs to be stored in secrets,
// so we need to interpolate the expression value for uses first. // so we need to interpolate the expression value for uses first.
sar.Step.Uses = sar.RunContext.NewExpressionEvaluator(ctx).Interpolate(ctx, sar.Step.Uses) uses, err := sar.RunContext.NewExpressionEvaluator(ctx).Interpolate(ctx, sar.Step.Uses)
if err != nil {
return fmt.Errorf("unable to interpolate uses: %w", err)
}
sar.Step.Uses = uses
github := sar.getGithubContext(ctx) // read before remoteAction is set, so `$/` resolves against the enclosing action github := sar.getGithubContext(ctx) // read before remoteAction is set, so `$/` resolves against the enclosing action
if strings.HasPrefix(sar.Step.Uses, selfRepoPrefix) { if strings.HasPrefix(sar.Step.Uses, selfRepoPrefix) {
@@ -62,65 +64,13 @@ func (sar *stepActionRemote) prepareActionExecutor() common.Executor {
sar.remoteAction = newRemoteAction(sar.Step.Uses) sar.remoteAction = newRemoteAction(sar.Step.Uses)
} }
if sar.remoteAction == nil { if sar.remoteAction == nil {
return fmt.Errorf("Expected format {org}/{repo}[/path]@ref or %s{path}. Actual '%s' Input string was not in a correct format", selfRepoPrefix, sar.Step.Uses) return fmt.Errorf("expected format {org}/{repo}[/path]@ref or %s{path}. Actual '%s' Input string was not in a correct format", selfRepoPrefix, sar.Step.Uses)
} }
if sar.remoteAction.IsCheckout() && isLocalCheckout(github, sar.Step) && !sar.RunContext.Config.NoSkipCheckout { if sar.remoteAction.IsCheckout() && isLocalCheckout(github, sar.Step) && !sar.RunContext.Config.NoSkipCheckout {
common.Logger(ctx).Debugf("Skipping local actions/checkout because workdir was already copied") common.Logger(ctx).Debugf("Skipping local actions/checkout because workdir was already copied")
return nil return nil
} }
for _, action := range sar.RunContext.Config.ReplaceGheActionWithGithubCom {
if strings.EqualFold(fmt.Sprintf("%s/%s", sar.remoteAction.Org, sar.remoteAction.Repo), action) {
sar.remoteAction.URL = "https://github.com"
github.Token = sar.RunContext.Config.ReplaceGheActionTokenWithGithubCom
}
}
// Actions served from the action cache are read out of a git object store rather than a
// directory, so they never reach the bundle patch below and keep to the v1 cache API.
if sar.RunContext.Config.ActionCache != nil {
cache := sar.RunContext.Config.ActionCache
var err error
sar.cacheDir = fmt.Sprintf("%s/%s", sar.remoteAction.Org, sar.remoteAction.Repo)
repoURL := sar.remoteAction.URL + "/" + sar.cacheDir
repoRef := sar.remoteAction.Ref
sar.resolvedSha, err = cache.Fetch(ctx, sar.cacheDir, repoURL, repoRef, github.Token)
if err != nil {
return fmt.Errorf("failed to fetch \"%s\" version \"%s\": %w", repoURL, repoRef, err)
}
remoteReader := func(ctx context.Context) actionYamlReader {
return func(filename string) (io.Reader, io.Closer, error) {
spath := path.Join(sar.remoteAction.Path, filename)
for range maxSymlinkDepth {
tars, err := cache.GetTarArchive(ctx, sar.cacheDir, sar.resolvedSha, spath)
if err != nil {
return nil, nil, os.ErrNotExist
}
treader := tar.NewReader(tars)
header, err := treader.Next()
if err != nil {
return nil, nil, os.ErrNotExist
}
if header.FileInfo().Mode()&os.ModeSymlink == os.ModeSymlink {
spath, err = symlinkJoin(spath, header.Linkname, ".")
if err != nil {
return nil, nil, err
}
} else {
return treader, tars, nil
}
}
return nil, nil, fmt.Errorf("max depth %d of symlinks exceeded while reading %s", maxSymlinkDepth, spath)
}
}
actionModel, err := sar.readAction(ctx, sar.Step, sar.resolvedSha, sar.remoteAction.Path, remoteReader(ctx), os.WriteFile)
sar.action = actionModel
return err
}
actionDir := sar.actionDir() actionDir := sar.actionDir()
defaultActionURL := sar.RunContext.Config.DefaultActionURL() defaultActionURL := sar.RunContext.Config.DefaultActionURL()
// For Gitea // For Gitea
@@ -148,12 +98,13 @@ func (sar *stepActionRemote) prepareActionExecutor() common.Executor {
var ntErr common.Executor var ntErr common.Executor
if err := gitClone(ctx); err != nil { if err := gitClone(ctx); err != nil {
var refErr *git.Error var refErr *git.Error
if errors.As(err, &refErr) && errors.Is(err, git.ErrShortRef) { switch {
return fmt.Errorf("Unable to resolve action `%s`, the provided ref `%s` is the shortened version of a commit SHA, which is not supported. Please use the full commit SHA `%s` instead", case errors.As(err, &refErr) && errors.Is(err, git.ErrShortRef):
return fmt.Errorf("unable to resolve action `%s`, the provided ref `%s` is the shortened version of a commit SHA, which is not supported. Please use the full commit SHA `%s` instead",
sar.Step.Uses, sar.remoteAction.Ref, refErr.Commit()) sar.Step.Uses, sar.remoteAction.Ref, refErr.Commit())
} else if errors.Is(err, gogit.ErrForceNeeded) { // TODO: figure out if it will be easy to shadow/alias go-git err's case errors.Is(err, gogit.ErrForceNeeded): // TODO: figure out if it will be easy to shadow/alias go-git err's
ntErr = common.NewInfoExecutor("Non-terminating error while running 'git clone': %v", err) ntErr = common.NewInfoExecutor("Non-terminating error while running 'git clone': %v", err)
} else { default:
return err return err
} }
} }
@@ -165,18 +116,16 @@ func (sar *stepActionRemote) prepareActionExecutor() common.Executor {
sar.resolvedSha = sha sar.resolvedSha = sha
} }
remoteReader := func(ctx context.Context) actionYamlReader { //nolint:unparam // pre-existing issue from nektos/act remoteReader := func(filename string) (io.Reader, io.Closer, error) {
return func(filename string) (io.Reader, io.Closer, error) { f, err := os.Open(filepath.Join(actionDir, sar.remoteAction.Path, filename))
f, err := os.Open(filepath.Join(actionDir, sar.remoteAction.Path, filename)) return f, f, err
return f, f, err
}
} }
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
ntErr, ntErr,
func(ctx context.Context) error { func(ctx context.Context) error {
defer git.AcquireCloneLock(actionDir)() defer git.AcquireCloneLock(actionDir)()
actionModel, err := sar.readAction(ctx, sar.Step, actionDir, sar.remoteAction.Path, remoteReader(ctx), os.WriteFile) actionModel, err := sar.readAction(ctx, sar.Step, actionDir, sar.remoteAction.Path, remoteReader, os.WriteFile)
sar.action = actionModel sar.action = actionModel
return err return err
}, },
@@ -215,9 +164,12 @@ func (sar *stepActionRemote) main() common.Executor {
common.Logger(ctx).Debugf("Skipping local actions/checkout because you bound your workspace") common.Logger(ctx).Debugf("Skipping local actions/checkout because you bound your workspace")
return nil return nil
} }
eval := sar.RunContext.NewExpressionEvaluator(ctx) checkoutPath, err := sar.RunContext.NewExpressionEvaluator(ctx).Interpolate(ctx, sar.Step.With["path"])
copyToPath := path.Join(sar.RunContext.JobContainer.ToContainerPath(sar.RunContext.Config.Workdir), eval.Interpolate(ctx, sar.Step.With["path"])) if err != nil {
return sar.RunContext.JobContainer.CopyDir(copyToPath, sar.RunContext.Config.Workdir+string(filepath.Separator)+".", sar.RunContext.Config.UseGitIgnore)(ctx) return fmt.Errorf("unable to interpolate with.path: %w", err)
}
copyToPath := path.Join(sar.RunContext.JobContainer.ToContainerPath(sar.RunContext.Config.Workdir), checkoutPath)
return sar.RunContext.JobContainer.CopyDir(copyToPath, sar.RunContext.Config.Workdir+string(filepath.Separator)+".", sar.RunContext.Config.UseGitIgnore, false)(ctx)
} }
return sar.runAction(sar, sar.actionDir(), sar.remoteAction)(ctx) return sar.runAction(sar, sar.actionDir(), sar.remoteAction)(ctx)
@@ -284,14 +236,18 @@ func (sar *stepActionRemote) getActionModel() *model.Action {
return sar.action return sar.action
} }
func (sar *stepActionRemote) getCompositeRunContext(ctx context.Context) *RunContext { func (sar *stepActionRemote) getCompositeRunContext(ctx context.Context) (*RunContext, error) {
if sar.compositeRunContext == nil { if sar.compositeRunContext == nil {
actionDir := sar.actionDir() actionDir := sar.actionDir()
actionLocation := path.Join(actionDir, sar.remoteAction.Path) actionLocation := path.Join(actionDir, sar.remoteAction.Path)
_, containerActionDir := getContainerActionPaths(sar.getStepModel(), actionLocation, sar.RunContext) _, containerActionDir := getContainerActionPaths(sar.getStepModel(), actionLocation, sar.RunContext)
sar.compositeRunContext = newCompositeRunContext(ctx, sar.RunContext, sar, containerActionDir) compositeRunContext, err := newCompositeRunContext(ctx, sar.RunContext, sar, containerActionDir)
sar.compositeSteps = sar.compositeRunContext.compositeExecutor(sar.action) if err != nil {
return nil, err
}
sar.compositeRunContext = compositeRunContext
sar.compositeSteps = compositeRunContext.compositeExecutor(sar.action)
} else { } else {
// Re-evaluate environment here. For remote actions the environment // Re-evaluate environment here. For remote actions the environment
// need to be re-created for every stage (pre, main, post) as there // need to be re-created for every stage (pre, main, post) as there
@@ -299,11 +255,14 @@ func (sar *stepActionRemote) getCompositeRunContext(ctx context.Context) *RunCon
// stages are executed. (e.g. the output of another action is the // stages are executed. (e.g. the output of another action is the
// input for this action during the main stage, but the env // input for this action during the main stage, but the env
// was already created during the pre stage) // was already created during the pre stage)
env := evaluateCompositeInputAndEnv(ctx, sar.RunContext, sar) env, err := evaluateCompositeInputAndEnv(ctx, sar.RunContext, sar)
sar.compositeRunContext.Env = env if err != nil {
return nil, err
}
sar.compositeRunContext.setCompositeActionEnv(env)
sar.compositeRunContext.ExtraPath = sar.RunContext.ExtraPath sar.compositeRunContext.ExtraPath = sar.RunContext.ExtraPath
} }
return sar.compositeRunContext return sar.compositeRunContext, nil
} }
func (sar *stepActionRemote) getCompositeSteps() *compositeSteps { func (sar *stepActionRemote) getCompositeSteps() *compositeSteps {
+111 -318
View File
@@ -31,6 +31,52 @@ type stepActionRemoteMocks struct {
mock.Mock mock.Mock
} }
func actionDirSuffix(suffix string) any {
return mock.MatchedBy(func(actionDir string) bool { return strings.HasSuffix(actionDir, suffix) })
}
func setCloneExecutor(t *testing.T, executor func(git.NewGitCloneExecutorInput) common.Executor) {
original := stepActionRemoteNewCloneExecutor
stepActionRemoteNewCloneExecutor = executor
t.Cleanup(func() { stepActionRemoteNewCloneExecutor = original })
}
func TestShortSHAActionRejected(t *testing.T) {
actionRoot := t.TempDir()
repo := filepath.Join(actionRoot, "actions", "hello-world-docker-action")
require.NoError(t, os.MkdirAll(repo, 0o755))
gitMust(t, "", "init", "--initial-branch=main", repo)
gitMust(t, repo, "config", "user.email", "test@test")
gitMust(t, repo, "config", "user.name", "test")
require.NoError(t, os.WriteFile(filepath.Join(repo, "action.yml"),
[]byte("name: hello\nruns:\n using: node24\n main: index.js\n"), 0o644))
gitMust(t, repo, "add", ".")
gitMust(t, repo, "commit", "-m", "initial")
output, err := exec.Command("git", "-C", repo, "rev-parse", "--short=7", "HEAD").Output()
require.NoError(t, err)
workflowDir := t.TempDir()
workflow := fmt.Sprintf("on: push\njobs:\n test:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/hello-world-docker-action@%s\n", strings.TrimSpace(string(output)))
require.NoError(t, os.WriteFile(filepath.Join(workflowDir, "push.yml"), []byte(workflow), 0o644))
runner, err := New(&Config{
Workdir: workflowDir,
EventName: "push",
GitHubInstance: "github.com",
DefaultActionInstance: actionRoot,
ContainerMaxLifetime: time.Hour,
PlatformPicker: func([]string) string { return baseImage },
})
require.NoError(t, err)
planner, err := model.NewWorkflowPlanner(workflowDir, true)
require.NoError(t, err)
plan, err := planner.PlanEvent("push")
require.NoError(t, err)
err = runner.NewPlanExecutor(plan)(common.WithDryrun(t.Context(), true))
require.ErrorContains(t, err, "shortened version of a commit SHA")
}
func (sarm *stepActionRemoteMocks) readAction(_ context.Context, step *model.Step, actionDir, actionPath string, readFile actionYamlReader, writeFile fileWriter) (*model.Action, error) { func (sarm *stepActionRemoteMocks) readAction(_ context.Context, step *model.Step, actionDir, actionPath string, readFile actionYamlReader, writeFile fileWriter) (*model.Action, error) {
args := sarm.Called(step, actionDir, actionPath, readFile, writeFile) args := sarm.Called(step, actionDir, actionPath, readFile, writeFile)
return args.Get(0).(*model.Action), args.Error(1) return args.Get(0).(*model.Action), args.Error(1)
@@ -136,16 +182,12 @@ func TestStepActionRemote(t *testing.T) {
clonedAction := false clonedAction := false
origStepAtionRemoteNewCloneExecutor := stepActionRemoteNewCloneExecutor setCloneExecutor(t, func(input git.NewGitCloneExecutorInput) common.Executor {
stepActionRemoteNewCloneExecutor = func(input git.NewGitCloneExecutorInput) common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
clonedAction = true clonedAction = true
return nil return nil
} }
} })
defer (func() {
stepActionRemoteNewCloneExecutor = origStepAtionRemoteNewCloneExecutor
})()
sar := &stepActionRemote{ sar := &stepActionRemote{
RunContext: &RunContext{ RunContext: &RunContext{
@@ -170,33 +212,19 @@ func TestStepActionRemote(t *testing.T) {
} }
sar.RunContext.ExprEval = sar.RunContext.NewExpressionEvaluator(ctx) sar.RunContext.ExprEval = sar.RunContext.NewExpressionEvaluator(ctx)
suffixMatcher := func(suffix string) any {
return mock.MatchedBy(func(actionDir string) bool {
return strings.HasSuffix(actionDir, suffix)
})
}
if tt.mocks.read { if tt.mocks.read {
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), "", mock.Anything, mock.Anything).Return(&model.Action{}, nil) sarm.On("readAction", sar.Step, actionDirSuffix(sar.Step.UsesHash()), "", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
} }
if tt.mocks.run { if tt.mocks.run {
sarm.On("runAction", sar, suffixMatcher(sar.Step.UsesHash()), newRemoteAction(sar.Step.Uses)).Return(func(ctx context.Context) error { return tt.runError }) sarm.On("runAction", sar, actionDirSuffix(sar.Step.UsesHash()), newRemoteAction(sar.Step.Uses)).Return(func(ctx context.Context) error { return tt.runError })
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(func(ctx context.Context) error { cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil) cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
} }
@@ -214,279 +242,61 @@ func TestStepActionRemote(t *testing.T) {
cm.AssertExpectations(t) cm.AssertExpectations(t)
}) })
} }
}
func TestStepActionRemotePre(t *testing.T) { t.Run("refreshes composite inputs without leaking environment", func(t *testing.T) {
table := []struct { step := &stepActionRemote{
name string Step: &model.Step{Uses: "org/composite@v1", With: map[string]string{"SHARED": "first"}},
stepModel *model.Step RunContext: &RunContext{Config: &Config{ActionCacheDir: t.TempDir()}, Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}}, JobContainer: &jobContainerMock{}},
}{ action: &model.Action{Inputs: map[string]model.Input{"shared": {}}},
{ env: map[string]string{},
name: "run-pre", remoteAction: newRemoteAction("org/composite@v1"),
stepModel: &model.Step{
Uses: "org/repo/path@ref",
},
},
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
ctx := context.Background()
clonedAction := false
sarm := &stepActionRemoteMocks{}
origStepAtionRemoteNewCloneExecutor := stepActionRemoteNewCloneExecutor
stepActionRemoteNewCloneExecutor = func(input git.NewGitCloneExecutorInput) common.Executor {
return func(ctx context.Context) error {
clonedAction = true
return nil
}
}
defer (func() {
stepActionRemoteNewCloneExecutor = origStepAtionRemoteNewCloneExecutor
})()
sar := &stepActionRemote{
Step: tt.stepModel,
RunContext: &RunContext{
Config: &Config{
GitHubInstance: "https://github.com",
ActionCacheDir: "/tmp/test-cache",
},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"1": {},
},
},
},
},
readAction: sarm.readAction,
}
suffixMatcher := func(suffix string) any {
return mock.MatchedBy(func(actionDir string) bool {
return strings.HasSuffix(actionDir, suffix)
})
}
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), "path", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
err := sar.pre()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, clonedAction)
sarm.AssertExpectations(t)
})
}
}
func TestStepActionRemotePreThroughAction(t *testing.T) {
table := []struct {
name string
stepModel *model.Step
}{
{
name: "run-pre",
stepModel: &model.Step{
Uses: "org/repo/path@ref",
},
},
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
ctx := context.Background()
clonedAction := false
sarm := &stepActionRemoteMocks{}
origStepAtionRemoteNewCloneExecutor := stepActionRemoteNewCloneExecutor
stepActionRemoteNewCloneExecutor = func(input git.NewGitCloneExecutorInput) common.Executor {
return func(ctx context.Context) error {
if input.URL == "https://github.com/org/repo" {
clonedAction = true
}
return nil
}
}
defer (func() {
stepActionRemoteNewCloneExecutor = origStepAtionRemoteNewCloneExecutor
})()
sar := &stepActionRemote{
Step: tt.stepModel,
RunContext: &RunContext{
Config: &Config{
GitHubInstance: "https://enterprise.github.com",
ReplaceGheActionWithGithubCom: []string{"org/repo"},
ActionCacheDir: "/tmp/test-cache",
},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"1": {},
},
},
},
},
readAction: sarm.readAction,
}
suffixMatcher := func(suffix string) any {
return mock.MatchedBy(func(actionDir string) bool {
return strings.HasSuffix(actionDir, suffix)
})
}
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), "path", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
err := sar.pre()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, clonedAction)
sarm.AssertExpectations(t)
})
}
}
func TestStepActionRemotePreThroughActionToken(t *testing.T) {
table := []struct {
name string
stepModel *model.Step
}{
{
name: "run-pre",
stepModel: &model.Step{
Uses: "org/repo/path@ref",
},
},
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
ctx := context.Background()
var actualURL string
var actualToken string
sarm := &stepActionRemoteMocks{}
origStepAtionRemoteNewCloneExecutor := stepActionRemoteNewCloneExecutor
stepActionRemoteNewCloneExecutor = func(input git.NewGitCloneExecutorInput) common.Executor {
return func(ctx context.Context) error {
actualURL = input.URL
actualToken = input.Token
return nil
}
}
defer (func() {
stepActionRemoteNewCloneExecutor = origStepAtionRemoteNewCloneExecutor
})()
// Use unique cache directory to ensure action gets cloned, not served from cache
uniqueCacheDir := fmt.Sprintf("/tmp/test-cache-token-%d", time.Now().UnixNano())
sar := &stepActionRemote{
Step: tt.stepModel,
RunContext: &RunContext{
Config: &Config{
GitHubInstance: "https://enterprise.github.com",
ReplaceGheActionWithGithubCom: []string{"org/repo"},
ReplaceGheActionTokenWithGithubCom: "PRIVATE_ACTIONS_TOKEN_ON_GITHUB",
ActionCacheDir: uniqueCacheDir,
Token: "PRIVATE_ACTIONS_TOKEN_ON_GITHUB",
},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"1": {},
},
},
},
},
readAction: sarm.readAction,
}
suffixMatcher := func(suffix string) any {
return mock.MatchedBy(func(actionDir string) bool {
return strings.HasSuffix(actionDir, suffix)
})
}
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), "path", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
err := sar.pre()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
// Verify that the clone was called (URL should be redirected to github.com)
assert.True(t, actualURL != "", "Expected clone to be called") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, "https://github.com/org/repo", actualURL, "URL should be redirected to github.com")
// Note: Token might be empty because getGitCloneToken doesn't check ReplaceGheActionTokenWithGithubCom
// The important part is that the URL replacement works
if actualToken != "" {
assert.Equal(t, "PRIVATE_ACTIONS_TOKEN_ON_GITHUB", actualToken, "If token is set, it should be the replacement token")
}
sarm.AssertExpectations(t)
})
}
}
func TestStepActionRemoteUsesGitHubInstanceWhenDefaultActionInstanceEmpty(t *testing.T) {
ctx := context.Background()
var actualURL string
sarm := &stepActionRemoteMocks{}
origStepAtionRemoteNewCloneExecutor := stepActionRemoteNewCloneExecutor
stepActionRemoteNewCloneExecutor = func(input git.NewGitCloneExecutorInput) common.Executor {
return func(ctx context.Context) error {
actualURL = input.URL
return nil
} }
}
defer func() {
stepActionRemoteNewCloneExecutor = origStepAtionRemoteNewCloneExecutor
}()
sar := &stepActionRemote{ for _, value := range []string{"first", "second"} {
Step: &model.Step{ step.env["INPUT_SHARED"] = value
Uses: "actions/setup-go@v4", composite, err := step.getCompositeRunContext(t.Context())
}, require.NoError(t, err)
RunContext: &RunContext{ assert.Equal(t, map[string]any{"shared": value}, composite.actionInputs)
Config: &Config{ assert.NotContains(t, composite.Env, "INPUT_SHARED")
GitHubInstance: "gitea.example", }
DefaultActionInstance: "", })
ActionCacheDir: t.TempDir(), }
},
Run: &model.Run{ func TestStepActionRemotePrepare(t *testing.T) {
JobID: "1", for _, test := range []struct {
Workflow: &model.Workflow{ name, uses, instance, actionPath, wantURL string
Jobs: map[string]*model.Job{ }{
"1": {}, {name: "nested action", uses: "org/repo/path@ref", instance: "https://github.com", actionPath: "path", wantURL: "https://github.com/org/repo"},
}, {name: "instance fallback", uses: "actions/setup-go@v4", instance: "gitea.example", wantURL: "https://gitea.example/actions/setup-go"},
} {
t.Run(test.name, func(t *testing.T) {
var actualURL string
setCloneExecutor(t, func(input git.NewGitCloneExecutorInput) common.Executor {
return func(context.Context) error {
actualURL = input.URL
return nil
}
})
actionMocks := &stepActionRemoteMocks{}
action := &stepActionRemote{
Step: &model.Step{Uses: test.uses},
RunContext: &RunContext{
Config: &Config{GitHubInstance: test.instance, ActionCacheDir: t.TempDir()},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{
Jobs: map[string]*model.Job{"1": {}},
}},
}, },
}, readAction: actionMocks.readAction,
}, }
readAction: sarm.readAction, actionMocks.On("readAction", action.Step, actionDirSuffix(action.Step.UsesHash()), test.actionPath,
} mock.Anything, mock.Anything).Return(&model.Action{}, nil)
suffixMatcher := func(suffix string) any { require.NoError(t, action.prepareActionExecutor()(t.Context()))
return mock.MatchedBy(func(actionDir string) bool { assert.Equal(t, test.wantURL, actualURL)
return strings.HasSuffix(actionDir, suffix) actionMocks.AssertExpectations(t)
}) })
} }
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), "", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
require.NoError(t, sar.prepareActionExecutor()(ctx))
assert.Equal(t, "https://gitea.example/actions/setup-go", actualURL)
sarm.AssertExpectations(t)
} }
func TestStepActionRemotePost(t *testing.T) { func TestStepActionRemotePost(t *testing.T) {
@@ -661,29 +471,21 @@ func TestStepActionRemotePost(t *testing.T) {
if tt.mocks.exec { if tt.mocks.exec {
// Use mock.MatchedBy to match the exec command with hash-based path // Use mock.MatchedBy to match the exec command with hash-based path
execMatcher := mock.MatchedBy(func(args []string) bool { execMatcher := mock.MatchedBy(func(args []string) bool {
if len(args) != 2 { if len(args) != 3 {
return false return false
} }
return args[0] == "node" && strings.Contains(args[1], "post.js") return args[0] == "node" && args[1] == "--preserve-symlinks-main" && strings.Contains(args[2], "post.js")
}) })
cm.On("Exec", execMatcher, sar.env, "", "").Return(func(ctx context.Context) error { return tt.err }) cm.On("Exec", execMatcher, sar.env, "", "").Return(func(ctx context.Context) error { return tt.err })
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(func(ctx context.Context) error { cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil) cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
} }
@@ -1032,14 +834,10 @@ func TestStepActionRemoteCloneTokenSurvivesNilSecrets(t *testing.T) {
ctx := context.Background() ctx := context.Background()
var capturedToken string var capturedToken string
origStepAtionRemoteNewCloneExecutor := stepActionRemoteNewCloneExecutor setCloneExecutor(t, func(input git.NewGitCloneExecutorInput) common.Executor {
stepActionRemoteNewCloneExecutor = func(input git.NewGitCloneExecutorInput) common.Executor {
capturedToken = input.Token capturedToken = input.Token
return func(ctx context.Context) error { return nil } return func(ctx context.Context) error { return nil }
} })
defer (func() {
stepActionRemoteNewCloneExecutor = origStepAtionRemoteNewCloneExecutor
})()
sarm := &stepActionRemoteMocks{} sarm := &stepActionRemoteMocks{}
sar := &stepActionRemote{ sar := &stepActionRemote{
@@ -1066,12 +864,7 @@ func TestStepActionRemoteCloneTokenSurvivesNilSecrets(t *testing.T) {
} }
sar.RunContext.ExprEval = sar.RunContext.NewExpressionEvaluator(ctx) sar.RunContext.ExprEval = sar.RunContext.NewExpressionEvaluator(ctx)
suffixMatcher := func(suffix string) any { sarm.On("readAction", sar.Step, actionDirSuffix(sar.Step.UsesHash()), "", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
return mock.MatchedBy(func(actionDir string) bool {
return strings.HasSuffix(actionDir, suffix)
})
}
sarm.On("readAction", sar.Step, suffixMatcher(sar.Step.UsesHash()), "", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
err := sar.prepareActionExecutor()(ctx) err := sar.prepareActionExecutor()(ctx)
require.NoError(t, err) require.NoError(t, err)
+17 -63
View File
@@ -22,11 +22,7 @@ type stepDocker struct {
env map[string]string env map[string]string
} }
func (sd *stepDocker) pre() common.Executor { func (sd *stepDocker) pre() common.Executor { return common.NewPipelineExecutor() }
return func(ctx context.Context) error {
return nil
}
}
func (sd *stepDocker) main() common.Executor { func (sd *stepDocker) main() common.Executor {
sd.env = map[string]string{} sd.env = map[string]string{}
@@ -34,11 +30,7 @@ func (sd *stepDocker) main() common.Executor {
return runStepExecutor(sd, stepStageMain, sd.runUsesContainer()) return runStepExecutor(sd, stepStageMain, sd.runUsesContainer())
} }
func (sd *stepDocker) post() common.Executor { func (sd *stepDocker) post() common.Executor { return common.NewPipelineExecutor() }
return func(ctx context.Context) error {
return nil
}
}
func (sd *stepDocker) getRunContext() *RunContext { func (sd *stepDocker) getRunContext() *RunContext {
return sd.RunContext return sd.RunContext
@@ -67,74 +59,36 @@ func (sd *stepDocker) runUsesContainer() common.Executor {
return func(ctx context.Context) error { return func(ctx context.Context) error {
image := strings.TrimPrefix(step.Uses, "docker://") image := strings.TrimPrefix(step.Uses, "docker://")
eval := rc.NewExpressionEvaluator(ctx) eval := rc.NewExpressionEvaluator(ctx)
cmd, err := shellquote.Split(eval.Interpolate(ctx, step.With["args"])) args, err := eval.Interpolate(ctx, step.With["args"])
if err != nil {
return fmt.Errorf("unable to interpolate with.args: %w", err)
}
cmd, err := shellquote.Split(args)
if err != nil { if err != nil {
return err return err
} }
var entrypoint []string var entrypoint []string
if entry := eval.Interpolate(ctx, step.With["entrypoint"]); entry != "" { entry, err := eval.Interpolate(ctx, step.With["entrypoint"])
if err != nil {
return fmt.Errorf("unable to interpolate with.entrypoint: %w", err)
}
if entry != "" {
entrypoint = []string{entry} entrypoint = []string{entry}
} }
stepContainer := sd.newStepContainer(ctx, image, cmd, entrypoint) stepContainer, err := newStepContainer(ctx, sd, image, cmd, entrypoint, "")
if err != nil {
return err
}
return common.NewPipelineExecutor( return common.NewPipelineExecutor(
stepContainer.Pull(rc.Config.ForcePull), stepContainer.Pull(rc.Config.ForcePull),
stepContainer.Remove().IfBool(!rc.Config.ReuseContainers), stepContainer.Remove(),
stepContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop), stepContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop),
stepContainer.Start(true), stepContainer.Start(true),
).Finally(
stepContainer.Remove().IfBool(!rc.Config.ReuseContainers && !rc.Config.AutoRemove),
).Finally(stepContainer.Close())(ctx) ).Finally(stepContainer.Close())(ctx)
} }
} }
var ContainerNewContainer = container.NewContainer var ContainerNewContainer = container.NewContainer
func (sd *stepDocker) newStepContainer(ctx context.Context, image string, cmd, entrypoint []string) container.Container {
rc := sd.RunContext
step := sd.Step
rawLogger := common.Logger(ctx).WithField("raw_output", true)
logWriter := common.NewLineWriter(rc.commandHandler(ctx), func(s string) bool {
if rc.Config.LogOutput {
rawLogger.Infof("%s", s)
} else {
rawLogger.Debugf("%s", s)
}
return true
})
envList := make([]string, 0)
for k, v := range sd.env {
envList = append(envList, fmt.Sprintf("%s=%s", k, v))
}
envList = append(envList, rc.runnerEnv(ctx)...)
binds, mounts := rc.GetBindsAndMounts()
networkMode := "container:" + rc.jobContainerName()
if rc.IsHostEnv(ctx) {
networkMode = "default"
}
stepContainer := ContainerNewContainer(&container.NewContainerInput{
Cmd: cmd,
Entrypoint: entrypoint,
WorkingDir: rc.JobContainer.ToContainerPath(rc.Config.Workdir),
Image: image,
Name: createContainerName(rc.jobContainerName(), "STEP-"+step.ID),
Env: envList,
Mounts: mounts,
NetworkMode: networkMode,
Binds: binds,
Stdout: logWriter,
Stderr: logWriter,
Privileged: rc.Config.Privileged,
UsernsMode: rc.Config.UsernsMode,
Platform: rc.Config.ContainerArchitecture,
AutoRemove: rc.Config.AutoRemove,
ValidVolumes: rc.validVolumes(),
AllocatePTY: rc.Config.AllocatePTY,
})
return stepContainer
}
+16 -81
View File
@@ -30,9 +30,9 @@ func TestStepDockerMain(t *testing.T) {
input = containerInput input = containerInput
return cm return cm
} }
defer (func() { defer func() {
ContainerNewContainer = origContainerNewContainer ContainerNewContainer = origContainerNewContainer
})() }()
ctx := context.Background() ctx := context.Background()
@@ -69,41 +69,23 @@ func TestStepDockerMain(t *testing.T) {
} }
sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx) sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx)
cm.On("Pull", false).Return(func(ctx context.Context) error { cm.On("Pull", false).Return(noopExecutor)
return nil
})
cm.On("Remove").Return(func(ctx context.Context) error { cm.On("Remove").Return(noopExecutor)
return nil
})
cm.On("Create", []string(nil), []string(nil)).Return(func(ctx context.Context) error { cm.On("Create", []string(nil), []string(nil)).Return(noopExecutor)
return nil
})
cm.On("Start", true).Return(func(ctx context.Context) error { cm.On("Start", true).Return(noopExecutor)
return nil
})
cm.On("Close").Return(func(ctx context.Context) error { cm.On("Close").Return(noopExecutor)
return nil
})
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(func(ctx context.Context) error { cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil) cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
@@ -115,47 +97,11 @@ func TestStepDockerMain(t *testing.T) {
// DOCKER_USERNAME/DOCKER_PASSWORD secrets should not be used as implicit pull credentials for docker:// action containers. // DOCKER_USERNAME/DOCKER_PASSWORD secrets should not be used as implicit pull credentials for docker:// action containers.
assert.Empty(t, input.Username) assert.Empty(t, input.Username)
assert.Empty(t, input.Password) assert.Empty(t, input.Password)
assert.True(t, input.AutoRemove)
cm.AssertExpectations(t) cm.AssertExpectations(t)
} }
// With AutoRemove the daemon reaps the container on exit, so act must not remove it afterwards.
func TestStepDockerAutoRemove(t *testing.T) {
orig := ContainerNewContainer
defer func() { ContainerNewContainer = orig }()
for _, tc := range []struct {
autoRemove bool
removes int
}{
{false, 2}, // stale + post-run
{true, 1}, // post-run skipped
} {
cm := &containerMock{}
ContainerNewContainer = func(*container.NewContainerInput) container.ExecutionsEnvironment { return cm }
sd := &stepDocker{
RunContext: &RunContext{
Config: &Config{AutoRemove: tc.autoRemove},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: cm,
},
Step: &model.Step{ID: "1", Uses: "docker://node:14"},
}
removes := 0
cm.On("Pull", false).Return(func(context.Context) error { return nil })
cm.On("Remove").Return(func(context.Context) error { removes++; return nil })
cm.On("Create", []string(nil), []string(nil)).Return(func(context.Context) error { return nil })
cm.On("Start", true).Return(func(context.Context) error { return nil })
cm.On("Close").Return(func(context.Context) error { return nil })
require.NoError(t, sd.runUsesContainer()(context.Background()))
cm.AssertExpectations(t)
assert.Equal(t, tc.removes, removes)
}
}
func TestStepDockerNewStepContainerAllocatePTY(t *testing.T) { func TestStepDockerNewStepContainerAllocatePTY(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -199,23 +145,12 @@ func TestStepDockerNewStepContainerAllocatePTY(t *testing.T) {
} }
sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx) sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx)
_ = sd.newStepContainer(ctx, "node:14", []string{"echo", "hi"}, nil) _, _ = newStepContainer(ctx, sd, "node:14", []string{"echo", "hi"}, nil, "")
assert.Equal(t, tc.allocPTY, captured.AllocatePTY) assert.Equal(t, tc.allocPTY, captured.AllocatePTY)
}) })
} }
} }
func TestStepDockerPrePost(t *testing.T) {
ctx := context.Background()
sd := &stepDocker{}
err := sd.pre()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
err = sd.post()(ctx)
assert.NoError(t, err)
}
func TestStepDockerNewStepContainerNetworkMode(t *testing.T) { func TestStepDockerNewStepContainerNetworkMode(t *testing.T) {
cases := []struct { cases := []struct {
name string name string
@@ -276,10 +211,10 @@ func TestStepDockerNewStepContainerNetworkMode(t *testing.T) {
} }
sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx) sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx)
assert.Equal(t, tc.expectDefault, sd.RunContext.IsHostEnv(ctx), require.NoError(t, sd.RunContext.resolvePlatformImage(ctx))
"IsHostEnv mismatch for platform %q", tc.platform) assert.Equal(t, tc.expectDefault, sd.RunContext.IsHostEnv(), "IsHostEnv mismatch for platform %q", tc.platform)
_ = sd.newStepContainer(ctx, "alpine:3.20", []string{"echo", "hello"}, nil) _, _ = newStepContainer(ctx, sd, "alpine:3.20", []string{"echo", "hello"}, nil, "")
if tc.expectDefault { if tc.expectDefault {
assert.Equal(t, "default", captured.NetworkMode, assert.Equal(t, "default", captured.NetworkMode,
+2 -2
View File
@@ -19,7 +19,7 @@ type stepFactoryImpl struct{}
func (sf *stepFactoryImpl) newStep(stepModel *model.Step, rc *RunContext) (step, error) { func (sf *stepFactoryImpl) newStep(stepModel *model.Step, rc *RunContext) (step, error) {
switch stepModel.Type() { switch stepModel.Type() {
case model.StepTypeInvalid: case model.StepTypeInvalid:
return nil, fmt.Errorf("Invalid run/uses syntax for job:%s step:%+v", rc.Run, stepModel) return nil, fmt.Errorf("invalid run/uses syntax for job:%s step:%+v", rc.Run, stepModel)
case model.StepTypeRun: case model.StepTypeRun:
return &stepRun{ return &stepRun{
Step: stepModel, Step: stepModel,
@@ -46,5 +46,5 @@ func (sf *stepFactoryImpl) newStep(stepModel *model.Step, rc *RunContext) (step,
}, nil }, nil
} }
return nil, fmt.Errorf("Unable to determine how to run job:%s step:%+v", rc.Run, stepModel) return nil, fmt.Errorf("unable to determine how to run job:%s step:%+v", rc.Run, stepModel)
} }
+1 -1
View File
@@ -65,7 +65,7 @@ func TestStepFactoryNewStep(t *testing.T) {
step, err := sf.newStep(tt.model, &RunContext{}) step, err := sf.newStep(tt.model, &RunContext{})
assert.True(t, tt.check((step))) assert.True(t, tt.check(step))
assert.NoError(t, err) assert.NoError(t, err)
}) })
} }
+74 -61
View File
@@ -7,6 +7,7 @@ package runner
import ( import (
"context" "context"
"fmt" "fmt"
"io"
"maps" "maps"
"runtime" "runtime"
"slices" "slices"
@@ -22,6 +23,8 @@ import (
yaml "go.yaml.in/yaml/v4" yaml "go.yaml.in/yaml/v4"
) )
var builtinShells = []string{"bash", "sh", "pwsh", "powershell", "cmd", "python"}
type stepRun struct { type stepRun struct {
Step *model.Step Step *model.Step
RunContext *RunContext RunContext *RunContext
@@ -33,11 +36,7 @@ type stepRun struct {
shellCommand string shellCommand string
} }
func (sr *stepRun) pre() common.Executor { func (sr *stepRun) pre() common.Executor { return common.NewPipelineExecutor() }
return func(ctx context.Context) error {
return nil
}
}
func (sr *stepRun) main() common.Executor { func (sr *stepRun) main() common.Executor {
sr.env = map[string]string{} sr.env = map[string]string{}
@@ -202,11 +201,7 @@ func stepDeclaredEnvKeysInOrder(step *model.Step) []string {
return keys return keys
} }
func (sr *stepRun) post() common.Executor { func (sr *stepRun) post() common.Executor { return common.NewPipelineExecutor() }
return func(ctx context.Context) error {
return nil
}
}
func (sr *stepRun) getRunContext() *RunContext { func (sr *stepRun) getRunContext() *RunContext {
return sr.RunContext return sr.RunContext
@@ -259,15 +254,35 @@ func getScriptName(rc *RunContext, step *model.Step) string {
// OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "${{": executable file not found in $PATH: unknown // OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "${{": executable file not found in $PATH: unknown
func (sr *stepRun) setupShellCommand(ctx context.Context) (name, script string, err error) { func (sr *stepRun) setupShellCommand(ctx context.Context) (name, script string, err error) {
logger := common.Logger(ctx) logger := common.Logger(ctx)
sr.setupShell(ctx) eval := sr.RunContext.NewStepExpressionEvaluator(ctx, sr)
sr.setupWorkingDirectory(ctx) implicitShell, err := sr.setupShell(ctx, eval)
if err != nil {
return "", "", err
}
if err := sr.setupWorkingDirectory(ctx, eval); err != nil {
return "", "", err
}
step := sr.Step step := sr.Step
script = sr.RunContext.NewStepExpressionEvaluator(ctx, sr).Interpolate(ctx, step.Run) script, err = eval.Interpolate(ctx, step.Run)
if err != nil {
return "", "", fmt.Errorf("unable to interpolate the run script: %w", err)
}
sr.interpolatedScript = script sr.interpolatedScript = script
// GitHub matches the built-in names case-insensitively, so `shell: PWSH` is valid
if slices.Contains(builtinShells, strings.ToLower(step.Shell)) {
step.Shell = strings.ToLower(step.Shell)
}
scCmd := step.ShellCommand() scCmd := step.ShellCommand()
if implicitShell && (step.Shell == "bash" || step.Shell == "sh") {
scCmd = step.Shell + " -e {0}"
}
if !strings.Contains(scCmd, "{0}") {
return "", "", fmt.Errorf("invalid shell option %q: format must contain {0}", step.Shell)
}
sr.shellCommand = scCmd sr.shellCommand = scCmd
name = getScriptName(sr.RunContext, step) name = getScriptName(sr.RunContext, step)
@@ -276,7 +291,8 @@ func (sr *stepRun) setupShellCommand(ctx context.Context) (name, script string,
// Reference: https://github.com/actions/runner/blob/8109c962f09d9acc473d92c595ff43afceddb347/src/Runner.Worker/Handlers/ScriptHandlerHelpers.cs#L19-L27 // Reference: https://github.com/actions/runner/blob/8109c962f09d9acc473d92c595ff43afceddb347/src/Runner.Worker/Handlers/ScriptHandlerHelpers.cs#L19-L27
runPrepend := "" runPrepend := ""
runAppend := "" runAppend := ""
switch step.Shell { shellCommand, _, _ := strings.Cut(step.Shell, " ")
switch shellCommand {
case "bash", "sh": case "bash", "sh":
name += ".sh" name += ".sh"
case "pwsh", "powershell": case "pwsh", "powershell":
@@ -300,81 +316,78 @@ func (sr *stepRun) setupShellCommand(ctx context.Context) (name, script string,
rc := sr.getRunContext() rc := sr.getRunContext()
scriptPath := fmt.Sprintf("%s/%s", rc.JobContainer.GetActPath(), name) scriptPath := fmt.Sprintf("%s/%s", rc.JobContainer.GetActPath(), name)
sr.cmdline = strings.Replace(scCmd, `{0}`, scriptPath, 1) sr.cmdline = strings.ReplaceAll(scCmd, `{0}`, scriptPath)
sr.cmd, err = shellquote.Split(sr.cmdline) sr.cmd, err = shellquote.Split(sr.cmdline)
return name, script, err return name, script, err
} }
type localEnv struct { func (sr *stepRun) setupShell(ctx context.Context, eval *expressionEvaluator) (bool, error) {
env map[string]string
}
func (l *localEnv) Getenv(name string) string {
if runtime.GOOS == "windows" {
for k, v := range l.env {
if strings.EqualFold(name, k) {
return v
}
}
return ""
}
return l.env[name]
}
func (sr *stepRun) setupShell(ctx context.Context) {
rc := sr.RunContext rc := sr.RunContext
step := sr.Step step := sr.Step
if step.Shell == "" { shell, err := eval.Interpolate(ctx, step.Shell)
step.Shell = rc.Run.Job().Defaults.Run.Shell if err != nil {
return false, fmt.Errorf("unable to interpolate the shell: %w", err)
} }
if shell == "" {
step.Shell = rc.NewExpressionEvaluator(ctx).Interpolate(ctx, step.Shell) shell = rc.jobRunDefaults.Shell
if step.Shell == "" {
step.Shell = rc.Run.Workflow.Defaults.Run.Shell
} }
if shell == "" {
shell = rc.Run.Workflow.Defaults.Run.Shell
}
step.Shell = shell
if step.Shell == "" { implicitShell := step.Shell == ""
if implicitShell {
shellWithFallback := []string{"bash", "sh"}
env := maps.Clone(sr.env)
rc.ApplyExtraPath(ctx, &env)
if _, ok := rc.JobContainer.(*container.HostEnvironment); ok { if _, ok := rc.JobContainer.(*container.HostEnvironment); ok {
shellWithFallback := []string{"bash", "sh"}
// Don't use bash on windows by default, if not using a docker container // Don't use bash on windows by default, if not using a docker container
if runtime.GOOS == "windows" { if runtime.GOOS == "windows" {
shellWithFallback = []string{"pwsh", "powershell"} shellWithFallback = []string{"pwsh", "powershell"}
} }
step.Shell = shellWithFallback[0] step.Shell = shellWithFallback[0]
lenv := &localEnv{env: map[string]string{}} _, err := lookpath.LookPath2(shellWithFallback[0], env)
maps.Copy(lenv.env, sr.env)
sr.getRunContext().ApplyExtraPath(ctx, &lenv.env)
_, err := lookpath.LookPath2(shellWithFallback[0], lenv)
if err != nil { if err != nil {
step.Shell = shellWithFallback[1] step.Shell = shellWithFallback[1]
} }
} else if containerImage := rc.containerImage(ctx); containerImage != "" { } else {
// Currently only linux containers are supported, use sh by default like actions/runner step.Shell = shellWithFallback[0]
step.Shell = "sh" if !rc.containerHasBash(ctx, env) {
step.Shell = shellWithFallback[1]
}
} }
} }
return implicitShell, nil
} }
func (sr *stepRun) setupWorkingDirectory(ctx context.Context) { // containerHasBash probes once per job, else every implicit-shell step pays for an exec.
rc := sr.RunContext func (rc *RunContext) containerHasBash(ctx context.Context, env map[string]string) bool {
step := sr.Step top := rc.topLevelRunContext()
var workingdirectory string if top.hasBash == nil {
stdout, stderr := rc.JobContainer.ReplaceLogWriter(io.Discard, io.Discard)
if step.WorkingDirectory == "" { found := rc.JobContainer.Exec([]string{"sh", "-c", "command -v bash >/dev/null 2>&1"}, env, "", "")(ctx) == nil
workingdirectory = rc.Run.Job().Defaults.Run.WorkingDirectory rc.JobContainer.ReplaceLogWriter(stdout, stderr)
} else { top.hasBash = &found
workingdirectory = step.WorkingDirectory
} }
return *top.hasBash
}
// jobs can receive context values, so we interpolate func (sr *stepRun) setupWorkingDirectory(ctx context.Context, eval *expressionEvaluator) error {
workingdirectory = rc.NewExpressionEvaluator(ctx).Interpolate(ctx, workingdirectory) rc := sr.RunContext
workingdirectory, err := eval.Interpolate(ctx, sr.Step.WorkingDirectory)
// but top level keys in workflow file like `defaults` or `env` can't if err != nil {
return fmt.Errorf("unable to interpolate the working directory: %w", err)
}
if workingdirectory == "" {
workingdirectory = rc.jobRunDefaults.WorkingDirectory
}
// top level keys in workflow file like `defaults` or `env` can't hold expressions
if workingdirectory == "" { if workingdirectory == "" {
workingdirectory = rc.Run.Workflow.Defaults.Run.WorkingDirectory workingdirectory = rc.Run.Workflow.Defaults.Run.WorkingDirectory
} }
sr.WorkingDirectory = workingdirectory sr.WorkingDirectory = workingdirectory
return nil
} }
+129 -33
View File
@@ -6,8 +6,12 @@ package runner
import ( import (
"bytes" "bytes"
"cmp"
"context" "context"
"io" "io"
"os"
"path/filepath"
"runtime"
"testing" "testing"
"gitea.com/gitea/runner/act/container" "gitea.com/gitea/runner/act/container"
@@ -15,8 +19,13 @@ import (
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
) )
type shellContainerMock struct{ *containerMock }
func (*shellContainerMock) ReplaceLogWriter(_, _ io.Writer) (io.Writer, io.Writer) { return nil, nil }
func TestStepRun(t *testing.T) { func TestStepRun(t *testing.T) {
cm := &containerMock{} cm := &containerMock{}
fileEntry := &container.FileEntry{ fileEntry := &container.FileEntry{
@@ -34,17 +43,12 @@ func TestStepRun(t *testing.T) {
JobID: "1", JobID: "1",
Workflow: &model.Workflow{ Workflow: &model.Workflow{
Jobs: map[string]*model.Job{ Jobs: map[string]*model.Job{
"1": { "1": {},
Defaults: model.Defaults{
Run: model.RunDefaults{
Shell: "bash",
},
},
},
}, },
}, },
}, },
JobContainer: cm, JobContainer: cm,
jobRunDefaults: model.RunDefaults{Shell: "bash"},
}, },
Step: &model.Step{ Step: &model.Step{
ID: "1", ID: "1",
@@ -53,28 +57,16 @@ func TestStepRun(t *testing.T) {
}, },
} }
cm.On("Copy", "/var/run/act", []*container.FileEntry{fileEntry}).Return(func(ctx context.Context) error { cm.On("Copy", "/var/run/act", []*container.FileEntry{fileEntry}).Return(noopExecutor)
return nil cm.On("Exec", []string{"bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "/var/run/act/workflow/1.sh"}, mock.AnythingOfType("map[string]string"), "", "workdir").Return(noopExecutor)
})
cm.On("Exec", []string{"bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "/var/run/act/workflow/1.sh"}, mock.AnythingOfType("map[string]string"), "", "workdir").Return(func(ctx context.Context) error {
return nil
})
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(func(ctx context.Context) error { cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(func(ctx context.Context) error { cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
return nil
})
ctx := context.Background() ctx := context.Background()
@@ -86,13 +78,117 @@ func TestStepRun(t *testing.T) {
cm.AssertExpectations(t) cm.AssertExpectations(t)
} }
func TestStepRunPrePost(t *testing.T) { func TestStepRunShellParity(t *testing.T) {
ctx := context.Background() tests := []struct {
sr := &stepRun{} name, run, shell, workingDir string
env map[string]string
host bool
probeErr error
wantExt string
wantCmd []string
wantErr string
}{
{
name: "implicit host bash",
host: true,
wantCmd: []string{"bash", "-e", "/var/run/act/workflow/1.sh"},
},
{
name: "implicit container bash",
wantCmd: []string{"bash", "-e", "/var/run/act/workflow/1.sh"},
},
{
name: "implicit container sh fallback",
probeErr: assert.AnError,
wantCmd: []string{"sh", "-e", "/var/run/act/workflow/1.sh"},
},
{
name: "custom pwsh template",
shell: "pwsh -NoProfile -File {0}",
wantExt: ".ps1",
wantCmd: []string{"pwsh", "-NoProfile", "-File", "/var/run/act/workflow/1.ps1"},
},
{
name: "missing placeholder",
shell: "bash -e",
wantErr: `invalid shell option "bash -e": format must contain {0}`,
},
{
name: "all placeholders",
shell: "bash -c '. {0}; . {0}'",
wantCmd: []string{"bash", "-c", ". /var/run/act/workflow/1.sh; . /var/run/act/workflow/1.sh"},
},
{
name: "step env expressions",
shell: "${{ env.SHELL }}",
workingDir: "${{ env.DIR }}",
env: map[string]string{"SHELL": "python {0}", "DIR": "subdir"},
wantExt: ".py",
wantCmd: []string{"python", "/var/run/act/workflow/1.py"},
},
{
name: "run expression without a context",
run: "echo ${{ COMMIT_SHA }}",
wantErr: "unable to interpolate the run script:",
},
{
name: "shell expression without a context",
shell: "${{ SHELL }}",
wantErr: "unable to interpolate the shell:",
},
{
name: "working directory expression without a context",
workingDir: "${{ DIR }}",
wantErr: "unable to interpolate the working directory:",
},
}
err := sr.pre()(ctx) for _, test := range tests {
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act t.Run(test.name, func(t *testing.T) {
cm := &containerMock{}
var jobContainer container.ExecutionsEnvironment = &shellContainerMock{cm}
if test.host {
if runtime.GOOS == "windows" {
t.Skip("Linux host shell selection")
}
test.env = map[string]string{"PATH": t.TempDir()}
require.NoError(t, os.WriteFile(filepath.Join(test.env["PATH"], "bash"), nil, 0o755))
jobContainer = &container.HostEnvironment{ActPath: "/var/run/act"}
} else if test.shell == "" {
cm.On("Exec", []string{"sh", "-c", "command -v bash >/dev/null 2>&1"},
mock.AnythingOfType("map[string]string"), "", "").Return(func(context.Context) error {
return test.probeErr
})
}
err = sr.post()(ctx) sr := &stepRun{
assert.NoError(t, err) RunContext: &RunContext{
Config: &Config{},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: jobContainer,
},
Step: &model.Step{ID: "1", Run: cmp.Or(test.run, "echo hi"), Shell: test.shell, WorkingDirectory: test.workingDir},
env: test.env,
}
name, script, err := sr.setupShellCommand(t.Context())
if test.wantErr != "" {
require.ErrorContains(t, err, test.wantErr)
return
}
require.NoError(t, err)
if test.wantExt == "" {
test.wantExt = ".sh"
}
wantScript := "\necho hi\n"
if test.wantExt == ".ps1" {
wantScript = "$ErrorActionPreference = 'stop'\necho hi\nif ((Test-Path -LiteralPath variable:/LASTEXITCODE)) { exit $LASTEXITCODE }"
}
assert.Equal(t, "workflow/1"+test.wantExt, name)
assert.Equal(t, wantScript, script)
assert.Equal(t, test.wantCmd, sr.cmd)
assert.Equal(t, test.env["DIR"], sr.WorkingDirectory)
cm.AssertExpectations(t)
})
}
} }
+119 -24
View File
@@ -7,12 +7,15 @@ package runner
import ( import (
"context" "context"
"errors" "errors"
"os"
"testing" "testing"
"gitea.com/gitea/runner/act/common" "gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model" "gitea.dev/actionslib/pkg/model"
log "github.com/sirupsen/logrus" log "github.com/sirupsen/logrus"
logrustest "github.com/sirupsen/logrus/hooks/test"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -157,21 +160,20 @@ func TestSetupEnv(t *testing.T) {
sm.On("getStepModel").Return(step) sm.On("getStepModel").Return(step)
sm.On("getEnv").Return(&env) sm.On("getEnv").Return(&env)
err := setupEnv(context.Background(), sm) require.NoError(t, setupEnv(context.Background(), sm))
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
// These are commit or system specific // These are commit or system specific
delete((env), "GITHUB_REF") delete(env, "GITHUB_REF")
delete((env), "GITHUB_REF_NAME") delete(env, "GITHUB_REF_NAME")
delete((env), "GITHUB_REF_TYPE") delete(env, "GITHUB_REF_TYPE")
delete((env), "GITHUB_SHA") delete(env, "GITHUB_SHA")
delete((env), "GITHUB_WORKSPACE") delete(env, "GITHUB_WORKSPACE")
delete((env), "GITHUB_REPOSITORY") delete(env, "GITHUB_REPOSITORY")
delete((env), "GITHUB_REPOSITORY_OWNER") delete(env, "GITHUB_REPOSITORY_OWNER")
delete((env), "GITHUB_ACTOR") delete(env, "GITHUB_ACTOR")
// Host-dependent, asserted in TestRunContextWithGithubEnvRunnerValues instead. // Host-dependent, asserted in TestRunContextWithGithubEnvRunnerValues instead.
delete((env), "RUNNER_NAME") delete(env, "RUNNER_NAME")
delete((env), "RUNNER_WORKSPACE") delete(env, "RUNNER_WORKSPACE")
assert.Equal(t, map[string]string{ assert.Equal(t, map[string]string{
"ACT": "true", "ACT": "true",
@@ -213,12 +215,7 @@ func TestIsStepEnabled(t *testing.T) {
return &stepRun{ return &stepRun{
RunContext: &RunContext{ RunContext: &RunContext{
Config: &Config{ Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }},
Workdir: ".",
Platforms: map[string]string{
"ubuntu-latest": "ubuntu-latest",
},
},
StepResults: map[string]*model.StepResult{}, StepResults: map[string]*model.StepResult{},
Env: map[string]string{}, Env: map[string]string{},
Run: &model.Run{ Run: &model.Run{
@@ -285,6 +282,13 @@ func TestIsStepEnabled(t *testing.T) {
Conclusion: model.StepStatusFailure, Conclusion: model.StepStatusFailure,
} }
assertObject.True(isStepEnabled(context.Background(), step.getStepModel().If.Value, step, stepStageMain)) assertObject.True(isStepEnabled(context.Background(), step.getStepModel().If.Value, step, stepStageMain))
// neither env nor the step's own with: values are inputs, at any stage
step = createTestStep(t, "if: inputs.forged")
step.getRunContext().Env["INPUT_FORGED"] = "leaked"
*step.getEnv() = map[string]string{"INPUT_FORGED": "leaked"}
assertObject.False(isStepEnabled(context.Background(), step.getStepModel().If.Value, step, stepStageMain))
assertObject.False(isStepEnabled(context.Background(), step.getStepModel().If.Value, step, stepStagePost))
} }
func TestIsContinueOnError(t *testing.T) { func TestIsContinueOnError(t *testing.T) {
@@ -295,12 +299,7 @@ func TestIsContinueOnError(t *testing.T) {
return &stepRun{ return &stepRun{
RunContext: &RunContext{ RunContext: &RunContext{
Config: &Config{ Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }},
Workdir: ".",
Platforms: map[string]string{
"ubuntu-latest": "ubuntu-latest",
},
},
StepResults: map[string]*model.StepResult{}, StepResults: map[string]*model.StepResult{},
Env: map[string]string{}, Env: map[string]string{},
Run: &model.Run{ Run: &model.Run{
@@ -350,6 +349,13 @@ func TestIsContinueOnError(t *testing.T) {
assertObject.False(continueOnError) assertObject.False(continueOnError)
assertObject.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act assertObject.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
// the step's own with: values are not inputs
step = createTestStep(t, "continue-on-error: ${{ inputs.forged }}")
*step.getEnv() = map[string]string{"INPUT_FORGED": "true"}
continueOnError, err = isContinueOnError(context.Background(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.False(continueOnError)
require.NoError(t, err)
// expression parse error // expression parse error
step = createTestStep(t, "continue-on-error: ${{ 'test' != test }}") step = createTestStep(t, "continue-on-error: ${{ 'test' != test }}")
continueOnError, err = isContinueOnError(context.Background(), step.getStepModel().RawContinueOnError, step, stepStageMain) continueOnError, err = isContinueOnError(context.Background(), step.getStepModel().RawContinueOnError, step, stepStageMain)
@@ -401,3 +407,92 @@ func TestRunStepExecutorDoesNotLeakRefusalToNextStep(t *testing.T) {
errB := runStepExecutor(stepB, stepStageMain, func(context.Context) error { return nil })(ctx) errB := runStepExecutor(stepB, stepStageMain, func(context.Context) error { return nil })(ctx)
require.NoError(t, errB) require.NoError(t, errB)
} }
func TestRunStepExecutorParity(t *testing.T) {
newStep := func(t *testing.T, stepModel *model.Step) *stepRun {
rc := createRunContext(t)
rc.JobContainer = &container.HostEnvironment{ActPath: t.TempDir()}
rc.ExprEval = rc.NewExpressionEvaluator(context.Background())
return &stepRun{RunContext: rc, Step: stepModel, env: map[string]string{}}
}
badExpression := "${{ 'test' != test }}"
for _, test := range []struct {
name, wantError string
step *model.Step
executor common.Executor
}{
{"condition error", "if-expression", &model.Step{ID: "condition", If: yaml.Node{Value: badExpression}}, noopExecutor},
{"continue-on-error expression error", "continue-on-error expression", &model.Step{ID: "continue", RawContinueOnError: badExpression}, common.NewErrorExecutor(assert.AnError)},
} {
t.Run(test.name, func(t *testing.T) {
step := newStep(t, test.step)
logger, hook := logrustest.NewNullLogger()
err := runStepExecutor(step, stepStageMain, test.executor)(common.WithLogger(context.Background(), logger))
require.ErrorContains(t, err, test.wantError)
assert.Equal(t, model.StepStatusFailure, step.RunContext.StepResults[test.step.ID].Conclusion)
assert.Equal(t, model.StepStatusFailure, hook.LastEntry().Data["stepResult"])
})
}
t.Run("file command error honors continue-on-error", func(t *testing.T) {
step := newStep(t, &model.Step{ID: "commands", RawContinueOnError: "true"})
logger, hook := logrustest.NewNullLogger()
err := runStepExecutor(step, stepStageMain, func(context.Context) error {
require.NoError(t, os.WriteFile(step.env["GITHUB_ENV"], []byte("GOOD=1\nmalformed\n"), 0o600))
require.NoError(t, os.WriteFile(step.env["GITHUB_OUTPUT"], []byte("kept=value\n"), 0o600))
require.NoError(t, os.WriteFile(step.env["GITHUB_STATE"], []byte("saved=value\n"), 0o600))
return nil
})(common.WithLogger(context.Background(), logger))
require.NoError(t, err)
result := step.RunContext.StepResults[step.Step.ID]
assert.Equal(t, model.StepStatusFailure, result.Outcome)
assert.Equal(t, model.StepStatusSuccess, result.Conclusion)
assert.Equal(t, model.StepStatusSuccess, hook.LastEntry().Data["stepResult"])
assert.Equal(t, "1", step.RunContext.Env["GOOD"])
assert.Equal(t, "value", result.Outputs["kept"])
assert.Equal(t, "value", step.RunContext.IntraActionState[step.Step.ID]["saved"])
for _, name := range []string{"GITHUB_ENV", "GITHUB_OUTPUT", "GITHUB_STATE", "GITHUB_PATH"} {
contents, readErr := os.ReadFile(step.env[name])
require.NoError(t, readErr)
assert.Empty(t, contents)
}
})
t.Run("composite child files do not leak", func(t *testing.T) {
outer := newStep(t, &model.Step{ID: "outer"})
childRC := &RunContext{
Config: outer.RunContext.Config, Run: outer.RunContext.Run, Env: map[string]string{}, StepResults: map[string]*model.StepResult{},
JobContainer: outer.RunContext.JobContainer, Parent: outer.RunContext,
}
childRC.ExprEval = childRC.NewExpressionEvaluator(context.Background())
child := &stepRun{RunContext: childRC, Step: &model.Step{ID: "child"}, env: map[string]string{}}
err := runStepExecutor(outer, stepStageMain, func(ctx context.Context) error {
require.NoError(t, runStepExecutor(child, stepStageMain, func(context.Context) error {
require.NoError(t, os.WriteFile(child.env["GITHUB_OUTPUT"], []byte("declared=child\nundeclared=leak\n"), 0o600))
require.NoError(t, os.WriteFile(child.env["GITHUB_STATE"], []byte("saved=child\n"), 0o600))
return nil
})(ctx))
outer.RunContext.setOutput(ctx, map[string]string{"name": "declared"}, "outer")
return nil
})(context.Background())
require.NoError(t, err)
assert.Equal(t, map[string]string{"declared": "outer"}, outer.RunContext.StepResults[outer.Step.ID].Outputs)
assert.Equal(t, map[string]string{"declared": "child", "undeclared": "leak"}, childRC.StepResults[child.Step.ID].Outputs)
assert.Empty(t, outer.RunContext.IntraActionState)
assert.Equal(t, "child", childRC.IntraActionState[child.Step.ID]["saved"])
})
t.Run("timeout must be positive", func(t *testing.T) {
exprEval := createRunContext(t).NewExpressionEvaluator(context.Background())
for timeout, wantDeadline := range map[string]bool{"-1": false, "0": false, "1": true} {
ctx, cancel := evaluateStepTimeout(context.Background(), exprEval, &model.Step{TimeoutMinutes: timeout})
_, hasDeadline := ctx.Deadline()
cancel()
assert.Equal(t, wantDeadline, hasDeadline, timeout)
}
})
}
@@ -1 +0,0 @@
ref: refs/heads/master
@@ -1,2 +0,0 @@
[core]
bare = true
-8
View File
@@ -1,8 +0,0 @@
name: checkout
on: push
jobs:
checkout:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
+6
View File
@@ -0,0 +1,6 @@
name: docker-proxy
description: Verify mounted Docker access through post steps
runs:
using: node24
main: index.js
post: index.js
+53
View File
@@ -0,0 +1,53 @@
const assert = require('node:assert/strict');
const {once} = require('node:events');
const fs = require('node:fs');
const http = require('node:http');
async function request(method, path, body) {
const req = http.request({
socketPath: '/var/run/docker.sock',
method,
path,
headers: {'Content-Type': 'application/json'},
signal: AbortSignal.timeout(10000),
});
req.end(JSON.stringify(body));
const [res] = await once(req, 'response');
req.on('error', (error) => res.destroy(error));
let data = '';
for await (const chunk of res.setEncoding('utf8')) {
data += chunk;
}
assert(res.statusCode >= 200 && res.statusCode < 300, `${method} ${path}: ${res.statusCode} ${data}`);
return data;
}
async function main() {
assert(fs.statSync('/var/run/docker.sock').isSocket(), 'Docker mount must be a socket');
assert.equal(await request('GET', '/_ping'), 'OK');
const api = `/v${JSON.parse(await request('GET', '/version')).ApiVersion}`;
const name = process.env.PROXY_TEST_RESOURCE;
const job = process.env.JOB_CONTAINER_NAME;
const post = process.env.STATE_post === 'true';
assert(name);
assert(job);
if (!post) {
await request('POST', `${api}/volumes/create`, {Name: name});
await request('POST', `${api}/networks/create`, {Name: name});
await request('POST', `${api}/networks/${name}/connect`, {Container: job});
fs.appendFileSync(process.env.GITHUB_STATE, 'post=true\n');
}
const label = process.env.PROXY_TEST_MODE === 'proxy' ? job : undefined;
assert.equal(JSON.parse(await request('GET', `${api}/volumes/${name}`)).Labels?.['com.gitea.runner.job'], label);
const network = JSON.parse(await request('GET', `${api}/networks/${name}`));
assert.equal(network.Labels?.['com.gitea.runner.job'], label);
assert(Object.values(network.Containers).some((container) => container.Name === job));
if (post) {
console.log('docker proxy post verified');
}
}
main().catch((error) => {
console.error(error);
process.exitCode = 1;
});
+8
View File
@@ -0,0 +1,8 @@
name: docker-proxy
on: push
jobs:
proxy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ./action
-21
View File
@@ -1,21 +0,0 @@
on:
workflow_dispatch:
inputs:
NAME:
description: "A random input name for the workflow"
type: string
required: true
SOME_VALUE:
description: "Some other input to pass"
type: string
required: true
jobs:
test:
name: Test
runs-on: ubuntu-latest
steps:
- name: Test with inputs
run: |
[ -z "${{ github.event.inputs.SOME_INPUT }}" ] && exit 1 || exit 0
+4 -1
View File
@@ -1,4 +1,4 @@
name: issue-597 name: issues-597-598
on: push on: push
@@ -13,6 +13,9 @@ jobs:
- name: My first true step - name: My first true step
if: ${{endsWith('Hello world', 'ld')}} if: ${{endsWith('Hello world', 'ld')}}
run: echo "Renst the Octocat" run: echo "Renst the Octocat"
- name: My second true step
if: "!endsWith('Hello world', 'od')"
run: echo "Renst the Octocat"
- name: My second false step - name: My second false step
if: "endsWith('Should not evaluate', 'o2')" if: "endsWith('Should not evaluate', 'o2')"
run: exit 1 run: exit 1
Loaded 100 of 162 files, more files were not shown because too many files have changed in this diff. Show more