Compare commits

...
6 Commits
Author SHA1 Message Date
3dbed63f55 fix: close the action header group before the action runs (#1345)
`uses:` steps and job hooks kept their "Run …" group open until they finished. Gitea doesn't nest groups, so the action's own first `::group::` closed the header and everything before it stayed hidden there, e.g. all of goreleaser's output in https://gitea.com/gitea/cli/actions/runs/1121069/jobs/1519275. The header group now closes right after its details, as in actions/runner.

The action header also matches actions/runner's otherwise now: it is titled by the `uses:` reference, lists the resolved inputs including `action.yml` defaults without empty ones, is shown for pre stages too, and post stages start with `Post job cleanup.`.

---------

Co-authored-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1345
Reviewed-by: silverwind <[email protected]>
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-10-26 17:42:16 +00:00
silverwindandbircni dafe8d8d0c fix: keep dry runs off the docker daemon and speed up the tests (#1349)
Without a docker daemon six `act/runner` tests failed instead of passing. A dry run still created and removed job networks, the job teardown dropped the dry-run flag along with every other context value, and the container mocks asked the daemon for the runner arch.

This also speeds up the tests and CI without dropping assertions. Most of the time went to git and docker round trips, which are now pinned, batched or coalesced, and the dind and e2e steps cache or overlap more of their setup. On CI, `act/runner` went from 77s to 55s and `test-dind` from 108s to 55s. Locally, `act/runner` without docker went from 53s to 7s.

Split out of https://gitea.com/gitea/runner/pulls/1348, which adds Windows CI on top. The go cache changes moved to https://gitea.com/gitea/runner/pulls/1350.

Written by Claude Code.

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1349
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-26 17:25:17 +00:00
Zettat123 8ae48ae033 fix: keep the shared action cache as extracted, re-fetch damaged trees (#1352)
The toolkit patch rewrote bundles in place in the shared action cache. Since https://gitea.com/gitea/runner/pulls/1316 trees are never refreshed, so a bundle left truncated failed every later job using it.

Patched bundles now only overwrite the job container's copy, so the runner no longer writes to an extracted tree.

Extraction records each file's size beside the tree, and a tree that no longer matches is fetched again, offline mode included, as a missing one always was. This also catches damage from outside the runner, such as storage losing a write, for a stat per file, a fraction of the copy each stage already makes. Trees from older runners have no record and download once more, except in offline mode, which keeps using them as before.

Generated with Claude Code.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1352
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: bircni <[email protected]>
Co-authored-by: Zettat123 <[email protected]>
2026-10-26 17:20:53 +00:00
Renovate Bot 8dd3776014 chore(deps): update dependencies (#1351)
Co-authored-by: Renovate Bot <[email protected]>
2026-10-26 07:06:33 +00:00
silverwind c6ca020281 ci: implement gitea's cache-seeder (#1350)
Like Gitea's CI, only a `cache-seeder` workflow on main saves the go caches, all other jobs only restore them. Caches are scoped per repository, so until now any pull request, forks included, could write a cache that other runs restore. The seeder compiles the tests without running them, so no test result is ever cached.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1350
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-24 17:04:51 +00:00
silverwind fe2a66dc25 fix: align runner behaviours (#1338)
Contains a number of small bug fixes where jobs behaved differently from GitHub or failed silently.

1. Hooks fail on a refused `::set-env::` or `::add-path::` and no longer inherit the last step's identity
1. Docker build, exec and wait errors are returned instead of swallowed, token-only registry credentials work, logs of `--tty` services decode
1. Env files split lines like GitHub on each OS, so a carriage return is kept on Linux and CRLF ends a line on Windows, empty heredoc names or delimiters are rejected
1. Post steps run after their pre step even when main was skipped, composites see undeclared inputs, `runs.args` wins without `with.args` being parsed
1. Docker actions skip evaluating defaults of inputs given in `with` and read `entryPoint` and `args` in any case, a lowercase `dockerfile` is found, undeclared inputs get GitHub's warning
1. A user-set `CI` is kept, `GITHUB_REPOSITORY_ID` and the other missing `GITHUB_*` variables are exported
1. `timeout-minutes` can read the step's own `env`, a step-level `ACTIONS_ALLOW_UNSECURE_COMMANDS: false` beats a job-level `true` while the runner's own opt-in still wins, `pwsh` quotes the script path
1. A step `env:` given as one expression such as `${{ fromJSON(matrix.env) }}` is applied, `hashFiles` fails with GitHub's messages on script errors and after 120 seconds, a job image without node still gets an empty hash
1. A step whose command files cannot be reset fails instead of importing stale ones, a timeout between steps fails the job instead of reporting success
1. One named volume can be mounted at several paths, host-mode PTY steps keep their last output when they fail, host scratch directories are removed when job setup fails
1. Cancelling a step right after it starts kills its whole process tree
1. Cache prefix restores pick the newest entry created within the same second, also on S3, cache commits with a wrong size are rejected
1. Artifact chunks are written at their offset, download URLs are escaped, listing a run without artifacts no longer crashes
1. Poll backoff reaches `fetch_interval_max`, proxy passwords are masked in more forms, secrets are also masked in step names and in their quoted form in the runner log, a failed final report is logged
1. On Windows hosts a differently cased unsecure-commands opt-in or proxy variable counts, and an early cancel no longer leaks a job object handle
1. Interactive `register` prompts for a missing token and stops on SIGTERM, `exec --list --detect-event` no longer panics without events, `config set` works on `cache.s3` keys

Reviewed-on: https://gitea.com/gitea/runner/pulls/1338
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-23 09:18:24 +00:00
110 changed files with 2427 additions and 574 deletions

No files matched your search

+5
View File
@@ -45,8 +45,13 @@ cpu.out
/bin
/dist
/.tmp
/.git/hooks
/.git/logs
/.git/rr-cache
/.env
/.runner
/.runner.lock
/config.yaml
/Dockerfile
.DS_Store
+29
View File
@@ -0,0 +1,29 @@
name: go-setup
description: Set up go and restore its caches, which only the cache-seeder workflow saves
runs:
using: composite
steps:
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
id: setup-go
with:
go-version-file: go.mod
check-latest: true
cache: false
- if: ${{ github.workflow == 'cache-seeder' }}
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
- if: ${{ github.workflow != 'cache-seeder' }}
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum', '.golangci.yml', 'Makefile', '.gitea/actions/go-setup/action.yml', '.gitea/workflows/cache-seeder.yml') }}
restore-keys: go-${{ runner.os }}-${{ runner.arch }}-${{ steps.setup-go.outputs.go-version }}-
+26
View File
@@ -0,0 +1,26 @@
name: cache-seeder
on:
workflow_dispatch:
push:
branches:
- main
paths:
- go.mod
- go.sum
- .golangci.yml
- Makefile
- .gitea/actions/go-setup/action.yml
- .gitea/workflows/cache-seeder.yml
jobs:
seed:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: ./.gitea/actions/go-setup
- run: make deps-tools lint build
- name: compile the test binaries without running them
run: |
make test GOTEST_FLAGS='-race -exec true'
go test -tags e2e -exec true ./e2e/...
+2 -2
View File
@@ -35,7 +35,7 @@ jobs:
go-version-file: "go.mod"
check-latest: true
- name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7
uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with:
distribution: goreleaser-pro
args: release --nightly
@@ -79,7 +79,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+2 -2
View File
@@ -35,7 +35,7 @@ jobs:
passphrase: ${{ secrets.PASSPHRASE }}
fingerprint: CC64B1DB67ABBEECAB24B6455FC346329753F4B0
- name: goreleaser
uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7
uses: goreleaser/goreleaser-action@a4f02ab5355a589de60d917a659ce617ed19b3de # v7
with:
distribution: goreleaser-pro
args: release
@@ -82,7 +82,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
- name: Login to DockerHub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+7 -11
View File
@@ -17,25 +17,24 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: 'go.mod'
check-latest: true
- uses: ./.gitea/actions/go-setup
- name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
# Pre-pull act/runner's two largest base images so a slow pull can't dominate `make test`;
# the rest (alpine/ubuntu) pull on demand, absorbed by the make-test -timeout. The host
# daemon retains them between runs, so this is usually a fast manifest re-check.
# daemon retains them between runs, and their digest pins skip the registry when it has them.
- name: pre-pull test images
env:
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:d6aa754f16b3197301076f047b5def2f02ea1dbbc2ca920407d46d7ec7f87b20 # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:df221db836e1754089190208cee7eeda94f233197056426eda74a43ab1abeac2 # renovate: datasource=docker
TEST_JOB_IMAGE: node:24-bookworm-slim@sha256:55fb5f5a527ea2c5c4a95f963e48ef90aa38c7b346054fdc8ee9957a2b55d205 # renovate: datasource=docker
TEST_SERVICE_IMAGE: nginx:alpine@sha256:a3554c0d0b48177b1413c26e01f695334428020d9df01243fbae3e11cfd17417 # renovate: datasource=docker
run: |
for image in "$TEST_JOB_IMAGE" "$TEST_SERVICE_IMAGE"; do
if [ "$(docker image inspect --format '{{.Os}}/{{.Architecture}}' "$image" 2>/dev/null)" != linux/amd64 ]; then
for attempt in 1 2 3; do
docker pull "$image" && break
[ "$attempt" = 3 ] || sleep 5
done
fi
docker tag "$image" "${image%@*}"
done
- name: lint
@@ -68,10 +67,7 @@ jobs:
- gitea/gitea:main-nightly
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version-file: 'go.mod'
check-latest: true
- uses: ./.gitea/actions/go-setup
- name: prepare anonymous docker config
run: mkdir -p "$DOCKER_CONFIG" && echo '{}' > "$DOCKER_CONFIG/config.json"
- name: e2e compatibility
+3 -1
View File
@@ -12,7 +12,9 @@ ENV GOPROXY=${GOPROXY:-}
COPY . /opt/src/runner
WORKDIR /opt/src/runner
RUN make clean && make build
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
make clean && make build
### DIND VARIANT
#
+3 -3
View File
@@ -4,7 +4,7 @@ GO ?= go
HAS_GO = $(shell hash $(GO) > /dev/null 2>&1 && echo "GO" || echo "NOGO" )
PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.9-immutable@sha256:87f42f37fa85b421664c30b8f5f9cfc23920057f5e5fcadf0b6f68d9edc6d415 # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:0e62e56b382ebf485bff1e51a05cbe27c708545a2e91425467fd31eb3e249521 # renovate: datasource=docker
E2E_JOB_IMAGE ?= docker.gitea.com/runner-images:ubuntu-latest@sha256:8856b200466e9a5af9cc0d4de4ca96f73547833bcdddf9a567189ff772c4d372 # renovate: datasource=docker
SERVICE_IMAGE ?= nginx:1.31.7-alpine@sha256:892bb2152464f0c5d8972593a5f5afaf8054552580b699d95f8ff4af08c2d33d # renovate: datasource=docker
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
E2E_K3S_IMAGE ?= rancher/k3s:v1.37.1-k3s1@sha256:ca7f37d993d82ef0dcdcfecb2e0e2618ea541dbaffc620c8cedebe01a82acd0d # renovate: datasource=docker
@@ -15,8 +15,8 @@ DOCKER_IMAGE ?= gitea/runner
DOCKER_TAG ?= nightly
DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]4.0 # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.8.0 # renovate: datasource=go
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected]5.0 # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/govulncheck@v1.9.0 # renovate: datasource=go
GOTEST_FLAGS ?= -race -timeout 20m -parallel 8
+32 -7
View File
@@ -15,6 +15,7 @@ import (
"encoding/json/v2"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/http/httputil"
@@ -710,9 +711,10 @@ func (h *Handler) reserve(w http.ResponseWriter, r *http.Request) {
}
defer db.Close()
now := time.Now().Unix()
cache.CreatedAt = now
cache.UsedAt = now
now := time.Now()
cache.CreatedAt = now.Unix()
cache.CreatedAtNanos = now.Nanosecond()
cache.UsedAt = now.Unix()
if err := h.insertCache(db, cache); err != nil {
h.responseJSON(w, r, 500, err)
return
@@ -778,6 +780,17 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
h.responseJSON(w, r, 400, err)
return
}
var request struct {
Size *int64 `json:"size"`
}
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
if err == nil && len(body) > 0 {
err = json.Unmarshal(body, &request)
}
if err != nil {
h.responseJSON(w, r, http.StatusBadRequest, err)
return
}
db, err := h.openDB()
if err != nil {
@@ -805,6 +818,14 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
return
}
if request.Size != nil {
if *request.Size < 0 || cache.Size >= 0 && cache.Size != *request.Size {
h.responseJSON(w, r, http.StatusBadRequest, errors.New("invalid cache size"))
return
}
cache.Size = *request.Size
}
db.Close()
if err := h.commitCache(cache); err != nil {
@@ -1067,7 +1088,7 @@ func findCache(db *cacheDB, repo string, keys []string, version string) (*Cache,
if err != nil {
return nil, fmt.Errorf("find cache: %w", err)
}
if cache := latest(caches, func(cache *Cache) int64 { return cache.CreatedAt }); cache != nil {
if cache := latest(caches, (*Cache).createdAtNano); cache != nil {
return cache, nil
}
}
@@ -1078,7 +1099,9 @@ func latest(caches []*Cache, stamp func(*Cache) int64) *Cache {
if len(caches) == 0 {
return nil
}
return slices.MaxFunc(caches, func(left, right *Cache) int { return cmp.Compare(stamp(left), stamp(right)) })
return slices.MaxFunc(caches, func(left, right *Cache) int {
return cmp.Or(cmp.Compare(stamp(left), stamp(right)), cmp.Compare(left.ID, right.ID))
})
}
// findExactCache returns the entry for exactly this key and version, or (nil, nil) if there is
@@ -1097,7 +1120,7 @@ func findExactCache(db *cacheDB, repo, key, version string, complete bool) (*Cac
}
stamp := func(cache *Cache) int64 { return cache.UsedAt }
if complete {
stamp = func(cache *Cache) int64 { return cache.CreatedAt }
stamp = (*Cache).createdAtNano
}
return latest(caches, stamp), nil
}
@@ -1267,7 +1290,9 @@ func (h *Handler) evictSuperseded(db *cacheDB) {
groups[group] = append(groups[group], cache)
}
for _, caches := range groups {
slices.SortFunc(caches, func(left, right *Cache) int { return cmp.Compare(left.CreatedAt, right.CreatedAt) })
slices.SortFunc(caches, func(left, right *Cache) int {
return cmp.Or(cmp.Compare(left.createdAtNano(), right.createdAtNano()), cmp.Compare(left.ID, right.ID))
})
for _, cache := range caches[:len(caches)-1] {
if inUse(cache) {
continue
+68
View File
@@ -15,6 +15,7 @@ import (
"net/http/httptest"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"testing/synctest"
@@ -47,10 +48,12 @@ func signArtifactURL(h *Handler, id int64) string {
}
func TestHandler_ExternalURL(t *testing.T) {
t.Parallel()
assert.Equal(t, "http://[2001:db8::1]:8080", (&Handler{outboundIP: "2001:db8::1", port: 8080}).ExternalURL())
}
func TestHandler_Upstream(t *testing.T) {
t.Parallel()
var upstream *httptest.Server
upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, http.MethodPut, r.Method)
@@ -100,6 +103,7 @@ func TestHandler_Upstream(t *testing.T) {
}
func TestHandler(t *testing.T) {
t.Parallel()
const version = "c19da02a2bd7e77277f1ac29ab45c09b7d46a4ee758284e26bb3045ad11d9d20"
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
@@ -302,6 +306,37 @@ func TestHandler(t *testing.T) {
}
})
t.Run("chunked commit validates size omitted at reservation", func(t *testing.T) {
for _, size := range []int64{5, 10} {
t.Run(strconv.FormatInt(size, 10), func(t *testing.T) {
id := reserveCache(t, testClient, base, strings.ToLower(t.Name()), version, 0)
req, err := http.NewRequest(http.MethodPatch, fmt.Sprintf("%s/caches/%d", base, id), strings.NewReader("cache"))
require.NoError(t, err)
req.Header.Set("Content-Range", "bytes 0-4/*")
resp, err := testClient.Do(req)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
resp, err = testClient.Post(fmt.Sprintf("%s/caches/%d", base, id), "application/json", io.MultiReader(strings.NewReader(fmt.Sprintf(`{"size":%d}`, size))))
require.NoError(t, err)
resp.Body.Close()
if size == 5 {
require.Equal(t, http.StatusOK, resp.StatusCode)
} else {
require.Equal(t, http.StatusInternalServerError, resp.StatusCode)
}
db, err := handler.openDB()
require.NoError(t, err)
defer db.Close()
cache, err := db.get(id)
require.NoError(t, err)
require.Equal(t, size == 5, cache.Complete)
})
}
})
t.Run("get with bad id", func(t *testing.T) {
resp, err := testClient.Get(base + "/artifacts/invalid_id")
require.NoError(t, err)
@@ -438,6 +473,7 @@ func uploadCacheNormally(t *testing.T, base, key, version string, content []byte
}
func TestHandler_gcCache(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir, Policy: Policy{Retention: testRetention}})
require.NoError(t, err)
@@ -540,6 +576,7 @@ func TestHandler_gcCache(t *testing.T) {
}
func TestHandler_evictPolicy(t *testing.T) {
t.Parallel()
now := time.Now()
stale := func(d time.Duration) int64 { return now.Add(-d).Unix() }
mib := func(n int64) int64 { return n * miB }
@@ -637,6 +674,7 @@ func TestHandler_evictPolicy(t *testing.T) {
}
func TestHandler_evictForFreeSpace(t *testing.T) {
t.Parallel()
free := func(n int64) func(string) (uint64, error) {
return func(string) (uint64, error) { return uint64(n), nil }
}
@@ -676,6 +714,7 @@ func TestHandler_evictForFreeSpace(t *testing.T) {
}
func TestHandler_SweepKeepsEntryWhenBlobSurvives(t *testing.T) {
t.Parallel()
cache := &Cache{Key: "stuck", Complete: true, UsedAt: time.Now().Add(-(testRetention + time.Hour)).Unix()}
handler := newTestHandler(t, Policy{Retention: testRetention}, cache)
@@ -690,6 +729,7 @@ func TestHandler_SweepKeepsEntryWhenBlobSurvives(t *testing.T) {
}
func TestHandler_FindProtectsFromEviction(t *testing.T) {
t.Parallel()
wanted := &Cache{Repo: testRepo, Key: "wanted", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-3 * time.Hour).Unix()}
other := &Cache{Repo: testRepo, Key: "other", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-2 * time.Hour).Unix()}
newest := &Cache{Repo: testRepo, Key: "newest", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-time.Hour).Unix()}
@@ -709,7 +749,23 @@ func TestHandler_FindProtectsFromEviction(t *testing.T) {
assert.Equal(t, []string{wanted.Key}, keptKeys(t, handler, []*Cache{wanted, other}), "the entry just promised to a job survives, the next least recently used goes instead")
}
func TestHandler_PrefixUsesNewestReservationWithinSecond(t *testing.T) {
t.Parallel()
created := time.Now().Add(-time.Hour).Unix()
older := &Cache{Repo: testRepo, Key: "deps-old", Version: "v", Complete: true, CreatedAt: created}
newer := &Cache{Repo: testRepo, Key: "deps-new", Version: "v", Complete: true, CreatedAt: created}
handler := newTestHandler(t, Policy{}, older, newer)
db, err := handler.openDB()
require.NoError(t, err)
defer db.Close()
cache, err := findCache(db, testRepo, []string{"deps-"}, "v")
require.NoError(t, err)
require.Equal(t, newer, cache)
}
func TestHandler_evictOnCommit(t *testing.T) {
t.Parallel()
full := &Cache{Repo: testRepo, Key: "full", Version: "v", Complete: true, Size: 4 * miB, UsedAt: time.Now().Add(-time.Hour).Unix()}
handler := newTestHandler(t, Policy{RepoSizeLimit: 4 * miB}, full)
@@ -719,6 +775,7 @@ func TestHandler_evictOnCommit(t *testing.T) {
}
func TestHandler_gcCacheInterval(t *testing.T) {
t.Parallel()
cache := &Cache{Key: "temp", UsedAt: time.Now().Add(-time.Hour).Unix()}
handler := newTestHandler(t, Policy{SweepInterval: 30 * time.Minute}, cache)
@@ -728,6 +785,7 @@ func TestHandler_gcCacheInterval(t *testing.T) {
}
func TestHandler_touchCacheSkipsFreshCompletedEntries(t *testing.T) {
t.Parallel()
fresh := &Cache{Repo: testRepo, Key: "fresh", Version: "v", Complete: true, UsedAt: time.Now().Unix()}
pending := &Cache{Repo: testRepo, Key: "pending", Version: "v", UsedAt: time.Now().Add(-time.Second).Unix()}
handler := newTestHandler(t, Policy{}, fresh, pending)
@@ -794,6 +852,7 @@ func writeBlob(t *testing.T, handler *Handler, id uint64) {
}
func TestHandler_RejectsBearer(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
@@ -830,6 +889,7 @@ func TestHandler_RejectsBearer(t *testing.T) {
}
func TestHandler_UnregisterRevokes(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
token string
@@ -871,6 +931,7 @@ func TestHandler_UnregisterRevokes(t *testing.T) {
}
func TestHandler_CrossRepoIsolation(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
@@ -921,6 +982,7 @@ func TestHandler_CrossRepoIsolation(t *testing.T) {
}
func TestHandler_ArtifactSignature(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
@@ -952,6 +1014,7 @@ func TestHandler_ArtifactSignature(t *testing.T) {
}
func TestHandler_SecretPersistsAcrossRestarts(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
first, err := StartHandler(Options{Dir: dir, OutboundIP: "127.0.0.1"})
@@ -968,6 +1031,7 @@ func TestHandler_SecretPersistsAcrossRestarts(t *testing.T) {
}
func TestHandler_SeesWritesOfOtherProcesses(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
insert := func(by *Handler, cache *Cache) {
db, err := by.openDB()
@@ -985,6 +1049,7 @@ func TestHandler_SeesWritesOfOtherProcesses(t *testing.T) {
}
func TestHandler_ArtifactSignatureDownload(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
@@ -1019,6 +1084,7 @@ func TestHandler_ArtifactSignatureDownload(t *testing.T) {
}
func TestHandler_GC_PerRepoDedup(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
@@ -1063,6 +1129,7 @@ func TestHandler_GC_PerRepoDedup(t *testing.T) {
}
func TestHandler_InternalAPI_Disabled(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
handler, err := StartHandler(Options{Dir: dir})
require.NoError(t, err)
@@ -1077,6 +1144,7 @@ func TestHandler_InternalAPI_Disabled(t *testing.T) {
}
func TestHandler_InternalAPI_AuthAndUsage(t *testing.T) {
t.Parallel()
dir := filepath.Join(t.TempDir(), "artifactcache")
const secret = "internal-secret"
handler, err := StartHandler(Options{Dir: dir, InternalSecret: secret})
+4 -3
View File
@@ -100,15 +100,16 @@ func (h *Handler) v2CreateCacheEntry(w http.ResponseWriter, r *http.Request) {
return
}
now := time.Now().Unix()
now := time.Now()
cache := &Cache{
Repo: cred.Repo,
Key: req.Key,
Version: req.Version,
Owner: owner,
Size: -1, // the size is only known at finalize time
CreatedAt: now,
UsedAt: now,
CreatedAt: now.Unix(),
CreatedAtNanos: now.Nanosecond(),
UsedAt: now.Unix(),
}
if err := h.insertCache(db, cache); err != nil {
h.twirpError(w, r, twirpInternal, err)
+5
View File
@@ -78,6 +78,7 @@ func saveV2(t *testing.T, handler *Handler, key, version string, content []byte)
}
func TestCacheServiceV2RoundTrip(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
content := []byte("the cached archive")
@@ -103,6 +104,7 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
}
func TestCacheServiceV2BlockUpload(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "blocks", "version": "v1"})
@@ -150,6 +152,7 @@ func TestCacheServiceV2BlockUpload(t *testing.T) {
}
func TestCacheServiceV2Lookups(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
saved, _ := saveV2(t, handler, "deps-abc", "v1", []byte("x"))
require.Equal(t, true, saved["ok"])
@@ -231,6 +234,7 @@ func TestCacheServiceV2Lookups(t *testing.T) {
}
func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
tests := []struct {
@@ -267,6 +271,7 @@ func TestCacheServiceV2RefusalsAreTwirp(t *testing.T) {
}
func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) {
t.Parallel()
handler := newTestHandler(t, Policy{})
content := []byte("the cached archive")
+7
View File
@@ -4,6 +4,8 @@
package artifactcache
import "time"
type Request struct {
Key string `json:"key" `
Version string `json:"version"`
@@ -20,4 +22,9 @@ type Cache struct {
Complete bool `json:"complete"`
UsedAt int64 `json:"usedAt"`
CreatedAt int64 `json:"createdAt"`
CreatedAtNanos int `json:"createdAtNanos,omitzero"`
}
func (cache *Cache) createdAtNano() int64 {
return cache.CreatedAt*int64(time.Second) + int64(cache.CreatedAtNanos)
}
+1
View File
@@ -17,6 +17,7 @@ import (
)
func TestFrontResultsService(t *testing.T) {
t.Parallel()
var gotHost, gotPath, gotProto string
gitea := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotHost, gotPath, gotProto = r.Host, r.URL.Path, r.Header.Get("X-Forwarded-Proto")
+23
View File
@@ -8,6 +8,7 @@ import (
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"sync/atomic"
@@ -19,6 +20,7 @@ import (
)
func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
t.Parallel()
var objects sync.Map
var uploads atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
@@ -51,14 +53,33 @@ func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
saver, restorer := startTestHandler(t, opts), startTestHandler(t, opts)
entry := map[string]any{"key": "deps", "version": "v1"}
started := time.Now()
finalized, _ := saveV2(t, saver, "deps", "v1", []byte("archive"))
require.Equal(t, true, finalized["ok"])
assert.EqualValues(t, 1, uploads.Load())
id, err := strconv.ParseUint(finalized["entry_id"].(string), 10, 64)
require.NoError(t, err)
cache, err := saver.s3.fetch(t.Context(), id)
require.NoError(t, err)
assert.WithinRange(t, time.Unix(cache.CreatedAt, int64(cache.CreatedAtNanos)), started, time.Now())
require.NoError(t, restorer.importMetadata(t.Context()))
found := v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", entry)
require.Equal(t, true, found["ok"])
assert.Equal(t, []byte("archive"), getURL(t, found["signed_download_url"].(string)))
for _, cache := range []*Cache{
{ID: 2, Repo: testRepo, Key: "deps-old", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt},
{ID: 1, Repo: testRepo, Key: "deps-new", Version: "v1", Complete: true, CreatedAt: cache.CreatedAt, CreatedAtNanos: 1},
} {
require.NoError(t, saver.s3.publish(cache))
objects.Store(saver.s3.blobKey(cache.ID), []byte(cache.Key))
}
require.NoError(t, restorer.importMetadata(t.Context()))
found = v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", map[string]any{"key": "deps-", "version": "v1"})
require.Equal(t, true, found["ok"])
assert.Equal(t, "deps-new", found["matched_key"])
assert.Equal(t, []byte("deps-new"), getURL(t, found["signed_download_url"].(string)))
objects.Clear()
require.NoError(t, restorer.importMetadata(t.Context()))
assert.Equal(t, true, v2Call(t, restorer, testClient, "CreateCacheEntry", entry)["ok"])
@@ -71,6 +92,7 @@ func (transport s3TestTransport) RoundTrip(req *http.Request) (*http.Response, e
}
func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) {
t.Parallel()
for _, failingPart := range []string{"", "2"} {
var parts, aborts atomic.Int32
var completion struct {
@@ -110,6 +132,7 @@ func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) {
}
func TestS3StorageSignsLikeAWS(t *testing.T) {
t.Parallel()
storage := &s3Storage{opts: S3Options{Region: "us-east-1", AccessKey: "AKIAIOSFODNN7EXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}}
req := httptest.NewRequest(http.MethodGet, "https://examplebucket.s3.amazonaws.com/?max-keys=2&prefix=J", nil)
req.Header.Set("X-Amz-Content-Sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
+20 -7
View File
@@ -13,6 +13,7 @@ import (
"io/fs"
"net"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
@@ -84,16 +85,25 @@ func uploads(router *http.ServeMux, baseDir string) {
if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil {
panic(err)
}
flags := os.O_CREATE | os.O_WRONLY | os.O_TRUNC
appendUpload := req.Header.Get("Content-Range")
if appendUpload != "" && !strings.HasPrefix(appendUpload, "bytes 0-") {
flags = os.O_CREATE | os.O_WRONLY | os.O_APPEND
var start int64
if contentRange := req.Header.Get("Content-Range"); contentRange != "" {
if _, err := fmt.Sscanf(contentRange, "bytes %d-", &start); err != nil || start < 0 {
http.Error(w, "invalid content range", http.StatusBadRequest)
return
}
}
flags := os.O_CREATE | os.O_WRONLY
if start == 0 {
flags |= os.O_TRUNC
}
file, err := os.OpenFile(safePath, flags, 0o644)
if err != nil {
panic(err)
}
defer file.Close()
if _, err := file.Seek(start, io.SeekStart); err != nil {
panic(err)
}
if req.Body == nil {
panic(errors.New("no body given"))
}
@@ -120,7 +130,7 @@ func downloads(router *http.ServeMux, baseDir string) {
runID := req.PathValue("runId")
entries, err := os.ReadDir(safeResolve(baseDir, runID))
if err != nil {
if err != nil && !os.IsNotExist(err) {
panic(err)
}
@@ -145,6 +155,9 @@ func downloads(router *http.ServeMux, baseDir string) {
var files []ContainerItem
err := filepath.WalkDir(safePath, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if !entry.IsDir() {
rel, err := filepath.Rel(safePath, path)
if err != nil {
@@ -157,12 +170,12 @@ func downloads(router *http.ServeMux, baseDir string) {
files = append(files, ContainerItem{
Path: filepath.ToSlash(filepath.Join(itemPath, rel)),
ItemType: "file",
ContentLocation: fmt.Sprintf("http://%s/artifact/%s/%s/%s", req.Host, container, itemPath, filepath.ToSlash(rel)),
ContentLocation: (&url.URL{Scheme: "http", Host: req.Host, Path: "/artifact/" + container + "/" + itemPath + "/" + filepath.ToSlash(rel)}).String(),
})
}
return nil
})
if err != nil {
if err != nil && !os.IsNotExist(err) {
panic(err)
}
+23 -5
View File
@@ -48,8 +48,17 @@ func TestArtifactFlow(t *testing.T) {
return resp.StatusCode, data
}
t.Run("no-artifacts", func(t *testing.T) {
status, data := request(t, http.MethodGet, baseURL+"/_apis/pipelines/workflows/0/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data))
var list NamedFileContainerResourceURLResponse
require.NoError(t, json.Unmarshal(data, &list))
require.Zero(t, list.Count)
require.Empty(t, list.Value)
})
t.Run("upload-and-download", func(t *testing.T) {
const runID, item, content = "1", "my-artifact/data.txt", "hello artifact\n"
const runID, item, content = "1", "my-artifact/data#1%25.txt", "hello artifact\n"
status, data := request(t, http.MethodPost, baseURL+"/_apis/pipelines/workflows/"+runID+"/artifacts", nil, nil)
require.Equal(t, http.StatusOK, status, string(data))
@@ -81,13 +90,13 @@ func TestArtifactFlow(t *testing.T) {
require.NoError(t, json.Unmarshal(data, &items))
require.Len(t, items.Value, 1)
require.Equal(t, "file", items.Value[0].ItemType)
require.Equal(t, "my-artifact/data.txt", items.Value[0].Path)
require.Equal(t, item, items.Value[0].Path)
status, data = request(t, http.MethodGet, items.Value[0].ContentLocation, nil, nil)
require.Equal(t, http.StatusOK, status)
require.Equal(t, content, string(data))
stored, err := os.ReadFile(filepath.Join(artifactPath, runID, "my-artifact", "data.txt"))
stored, err := os.ReadFile(filepath.Join(artifactPath, runID, filepath.FromSlash(item)))
require.NoError(t, err)
require.Equal(t, content, string(stored))
})
@@ -95,16 +104,25 @@ func TestArtifactFlow(t *testing.T) {
t.Run("content-range", func(t *testing.T) {
const rawURL = "/upload/4?itemPath=chunks.txt"
status, data := request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("first"),
http.Header{"Content-Range": []string{"bytes 0-4/11"}})
http.Header{"Content-Range": []string{"bytes 0-4/12"}})
require.Equal(t, http.StatusOK, status, string(data))
for range 2 {
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader("-second"),
http.Header{"Content-Range": []string{"bytes 5-11/11"}})
http.Header{"Content-Range": []string{"bytes 5-11/12"}})
require.Equal(t, http.StatusOK, status, string(data))
}
stored, err := os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err)
require.Equal(t, "first-second", string(stored))
status, data = request(t, http.MethodPut, baseURL+rawURL, strings.NewReader(""),
http.Header{"Content-Range": []string{"bytes 0--1/0"}})
require.Equal(t, http.StatusOK, status, string(data))
stored, err = os.ReadFile(filepath.Join(artifactPath, "4", "chunks.txt"))
require.NoError(t, err)
require.Empty(t, stored)
})
t.Run("gzip-roundtrip", func(t *testing.T) {
+4 -1
View File
@@ -164,7 +164,10 @@ func (e Executor) Finally(finally Executor) Executor {
err := e(ctx)
err2 := finally(ctx)
if err2 != nil {
return fmt.Errorf("error occurred running finally: %v (original error: %v)", err2, err)
if err != nil {
return fmt.Errorf("error occurred running finally: %w (original error: %w)", err2, err)
}
return fmt.Errorf("error occurred running finally: %w (original error: %v)", err2, err)
}
return err
}
+8 -3
View File
@@ -7,6 +7,7 @@ package common
import (
"context"
"errors"
"fmt"
"sync"
"sync/atomic"
"testing"
@@ -134,9 +135,13 @@ func TestExecutorConditionalsAndFinally(t *testing.T) {
}
func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
mainErr := errors.New("main failed")
finalErr := errors.New("cleanup failed")
for _, mainErr := range []error{context.Canceled, nil} {
err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context())
require.EqualError(t, err, "error occurred running finally: cleanup failed (original error: main failed)")
require.EqualError(t, err, fmt.Sprintf("error occurred running finally: cleanup failed (original error: %v)", mainErr))
require.ErrorIs(t, err, finalErr)
if mainErr != nil {
require.ErrorIs(t, err, mainErr)
}
}
}
+84 -14
View File
@@ -8,6 +8,7 @@ import (
"bytes"
"compress/gzip"
"context"
"encoding/json"
"errors"
"io"
"math/rand/v2"
@@ -28,6 +29,13 @@ import (
var actionFetches sync.Map
const treeSizes = ".sizes" // beside the tree rather than in it, which is copied into job containers as it is
type treeFile struct {
Name []byte // not a string, which JSON would turn into UTF-8, as git allows any bytes but NUL
Size int64
}
type actionFetch struct {
done chan struct{}
dir, sha string
@@ -86,9 +94,14 @@ func fetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, s
record := filepath.Join(input.Dir, "ref")
data, _ := os.ReadFile(record)
sha, etag, _ := strings.Cut(string(data), "\n")
if !isFullObjectID(sha) || !exists(tree(sha)) {
switch {
case !isFullObjectID(sha):
sha, etag = "", ""
} else if input.OfflineMode || isFullObjectID(input.Ref) {
case input.OfflineMode && exists(tree(sha)) && !exists(tree(sha)+treeSizes):
return tree(sha), sha, nil // from an older runner, with no sizes to check, so offline mode uses it as before
case !treeIntact(input.Dir, sha):
sha, etag = "", ""
case input.OfflineMode || isFullObjectID(input.Ref):
return tree(sha), sha, nil
}
@@ -173,7 +186,7 @@ func downloadArchiveOnce(ctx context.Context, client *http.Client, archive, toke
newEtag := response.Header.Get("ETag")
gz, err := gzip.NewReader(response.Body)
if err == nil {
sha, err = extractTree(gz, dir, !strings.HasSuffix(newEtag, `-false"`)) // Gitea's mark of an unprefixed archive
sha, err = extractTree(ctx, gz, dir, !strings.HasSuffix(newEtag, `-false"`)) // Gitea's mark of an unprefixed archive
}
var netErr net.Error
if errors.Is(err, io.ErrUnexpectedEOF) || errors.As(err, &netErr) {
@@ -191,18 +204,18 @@ func exportClone(ctx context.Context, input NewGitCloneExecutorInput) (string, e
clone := input
clone.Dir = filepath.Join(input.Dir, "git")
sha, _, err := resolveClone(ctx, clone)
if err != nil || exists(filepath.Join(input.Dir, sha)) {
if err != nil || treeIntact(input.Dir, sha) {
return sha, err
}
archive, err := runGit(ctx, clone.Dir, nil, "archive", "--format=tar", sha)
if err != nil {
return "", err
}
_, err = extractTree(bytes.NewReader(archive), input.Dir, false)
_, err = extractTree(ctx, bytes.NewReader(archive), input.Dir, false)
return sha, err
}
func extractTree(archive io.Reader, dir string, strip bool) (string, error) { // into dir/<commit of the pax header>, strip dropping the directory wrapping the files
func extractTree(ctx context.Context, archive io.Reader, dir string, strip bool) (string, error) { // into dir/<commit of the pax header>, strip dropping the directory wrapping the files
temp, err := os.MkdirTemp(dir, "tmp-")
if err != nil {
return "", err
@@ -215,6 +228,7 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
defer root.Close()
sha := ""
sizes := map[string]int64{}
reader := tar.NewReader(archive)
for {
header, err := reader.Next()
@@ -225,7 +239,7 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
return "", err
}
if header.Typeflag == tar.TypeXGlobalHeader {
if sha = header.PAXRecords["comment"]; isFullObjectID(sha) && exists(filepath.Join(dir, sha)) {
if sha = header.PAXRecords["comment"]; isFullObjectID(sha) && treeIntact(dir, sha) {
return sha, nil
}
continue
@@ -241,10 +255,10 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
case tar.TypeDir:
err = root.MkdirAll(name, 0o755)
case tar.TypeReg:
err = writeTreeFile(root, name, header.Mode, reader)
sizes[name], err = writeTreeFile(root, name, header.Mode, reader)
case tar.TypeSymlink:
if err = root.Symlink(header.Linkname, name); err != nil && runtime.GOOS == "windows" {
err = writeTreeFile(root, name, 0, strings.NewReader(header.Linkname))
sizes[name], err = writeTreeFile(root, name, 0, strings.NewReader(header.Linkname))
}
}
if err != nil {
@@ -257,20 +271,76 @@ func extractTree(archive io.Reader, dir string, strip bool) (string, error) { //
if err := root.Close(); err != nil {
return "", err
}
return sha, os.Rename(temp, filepath.Join(dir, sha))
return sha, installTree(ctx, dir, sha, temp, sizes)
}
func writeTreeFile(root *os.Root, name string, mode int64, content io.Reader) error {
var lockTree = AcquireCloneLock // replaced in tests to see installTree wait for a job holding the tree
// installTree moves temp into place as dir/sha, replacing a tree that failed treeIntact.
func installTree(ctx context.Context, dir, sha, temp string, sizes map[string]int64) error {
files := make([]treeFile, 0, len(sizes))
for name, size := range sizes {
files = append(files, treeFile{[]byte(name), size})
}
data, err := json.Marshal(files)
if err != nil {
return err
}
// Written before the swap: it describes the commit, not a copy of it, so it holds for whichever tree is left.
if err := os.WriteFile(filepath.Join(dir, sha+treeSizes), data, 0o600); err != nil { // a torn record fails treeIntact, so it needs no rename
return err
}
tree := filepath.Join(dir, sha)
if !exists(tree) {
return os.Rename(temp, tree)
}
stale, err := os.MkdirTemp(dir, "tmp-")
if err != nil {
return err
}
defer os.RemoveAll(stale) // after the unlock, as no job reads the old tree once it is moved
unlock, err := lockTree(ctx, tree) // jobs copying the old tree hold it
if err != nil {
return err
}
defer unlock()
if err := os.Rename(tree, filepath.Join(stale, sha)); err != nil {
return err
}
return os.Rename(temp, tree)
}
// treeIntact reports whether dir/sha still holds each file extracted into it at the size it was written.
// Only sizes are compared, not content: a stat per file is cheap enough for every job, hashing is not.
func treeIntact(dir, sha string) bool {
data, err := os.ReadFile(filepath.Join(dir, sha+treeSizes))
if err != nil || !exists(filepath.Join(dir, sha)) {
return false
}
var files []treeFile
if json.Unmarshal(data, &files) != nil {
return false
}
for _, file := range files {
info, err := os.Lstat(filepath.Join(dir, sha, filepath.FromSlash(string(file.Name))))
if err != nil || !info.Mode().IsRegular() || info.Size() != file.Size {
return false
}
}
return true
}
func writeTreeFile(root *os.Root, name string, mode int64, content io.Reader) (int64, error) {
perm := os.FileMode(0o644)
if mode&0o111 != 0 {
perm = 0o755
}
file, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm)
if err != nil {
return err
return 0, err
}
_, err = io.Copy(file, content)
return errors.Join(err, file.Close())
n, err := io.Copy(file, content)
return n, errors.Join(err, file.Close())
}
func exists(name string) bool {
+108 -11
View File
@@ -15,6 +15,7 @@ import (
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"sync"
"sync/atomic"
@@ -41,6 +42,7 @@ func TestFetchAction(t *testing.T) {
require.NoError(t, gitCmd("-C", remote, "add", "."))
require.NoError(t, gitCmd("-C", remote, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", remote, "tag", "v1"))
expectedSha := gitRevParse(t, remote, "main")
gitPath, err := exec.LookPath("git")
require.NoError(t, err)
@@ -48,6 +50,7 @@ func TestFetchAction(t *testing.T) {
var mu sync.Mutex
var archiveRequests []http.Header
var mirror *httptest.Server
archiveCache := map[string][]byte{}
requests, archives, failures := 0, "", 0
handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
mu.Lock()
@@ -84,7 +87,12 @@ func TestFetchAction(t *testing.T) {
if archives != "unprefixed" {
args = append(args, "--prefix=action/")
}
archive, _ := runGit(request.Context(), remote, nil, args...)
archiveKey := strings.Join(args, "\x00")
archive, ok := archiveCache[archiveKey]
if !ok {
archive, _ = runGit(request.Context(), remote, nil, args...)
archiveCache[archiveKey] = archive
}
writer.Header().Set("ETag", etag)
_, _ = writer.Write(archive)
})
@@ -107,7 +115,6 @@ func TestFetchAction(t *testing.T) {
}
t.Run("shares identical fetches and isolates cancellation, panics and differing inputs", func(t *testing.T) {
expectedSha := gitRevParse(t, remote, "main")
for _, scenario := range []string{"shared", "leader cancelled", "leader panics", "different token"} {
t.Run(scenario, func(t *testing.T) {
serve("prefixed", 0)
@@ -180,7 +187,7 @@ func TestFetchAction(t *testing.T) {
input := newInput("v1")
tree, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "v1"), sha)
assert.Equal(t, expectedSha, sha)
assert.Equal(t, filepath.Join(input.Dir, sha), tree)
assert.NoDirExists(t, filepath.Join(tree, ".git"))
info, err := os.Stat(filepath.Join(tree, "run.sh"))
@@ -196,9 +203,10 @@ func TestFetchAction(t *testing.T) {
require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved"))
require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1"))
expectedSha = gitRevParse(t, remote, "v1")
moved, movedSha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "v1"), movedSha)
assert.Equal(t, expectedSha, movedSha)
assert.NotEqual(t, tree, moved)
assert.FileExists(t, filepath.Join(tree, "action.yml"))
@@ -206,10 +214,17 @@ func TestFetchAction(t *testing.T) {
_, offlineSha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, movedSha, offlineSha)
require.NoError(t, os.Truncate(filepath.Join(moved, "action.yml"), 4)) // fetched again, offline or not
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err)
require.NoError(t, os.Remove(moved+treeSizes)) // from an older runner, used as it is offline
reused, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, moved, reused)
requests, headers := served()
assert.Equal(t, 3, requests)
require.Len(t, headers, 3)
assert.Equal(t, 4, requests)
require.Len(t, headers, 4)
assert.Equal(t, "Basic "+base64.StdEncoding.EncodeToString([]byte("token:secret")), headers[0].Get("Authorization"))
assert.Empty(t, headers[0].Get("If-None-Match"))
assert.Equal(t, fmt.Sprintf(`W/"%s-tar.gz-true"`, sha), headers[1].Get("If-None-Match"))
@@ -218,7 +233,7 @@ func TestFetchAction(t *testing.T) {
t.Run("downloads a pinned commit once and rejects a short one", func(t *testing.T) {
serve("prefixed", 0)
sha := gitRevParse(t, remote, "main")
sha := expectedSha
input := newInput(sha)
for range 2 {
tree, got, err := FetchAction(t.Context(), input)
@@ -233,12 +248,74 @@ func TestFetchAction(t *testing.T) {
assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
})
t.Run("downloads again over a tree whose files changed size or that has no record, not under a job copying it", func(t *testing.T) {
sha := gitRevParse(t, remote, "main")
for _, mode := range []string{"prefixed", "none"} {
serve(mode, 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.WriteFile(filepath.Join(tree, "trampoline.js"), nil, 0o644)) // the runner's own, no damage
_, _, err = FetchAction(t.Context(), input)
require.NoError(t, err, mode)
for _, damage := range []func() error{
func() error { return os.Truncate(yml, 4) },
func() error { return os.Remove(tree + treeSizes) },
} {
require.NoError(t, damage(), mode)
_, _, err := FetchAction(t.Context(), input)
require.NoError(t, err, mode)
body, err := os.ReadFile(yml)
require.NoError(t, err, mode)
assert.Equal(t, "name: action\n", string(body), mode)
}
_, headers := served()
assert.Len(t, headers, 3, mode)
}
serve("prefixed", 0)
input := newInput(sha)
tree, _, err := FetchAction(t.Context(), input)
require.NoError(t, err)
yml := filepath.Join(tree, "action.yml")
require.NoError(t, os.Truncate(yml, 4))
unlock, err := AcquireCloneLock(t.Context(), tree)
require.NoError(t, err)
release := sync.OnceFunc(unlock)
defer release()
waiting := make(chan struct{})
lockTree = func(ctx context.Context, dir string) (func(), error) {
close(waiting)
return AcquireCloneLock(ctx, dir)
}
t.Cleanup(func() { lockTree = AcquireCloneLock })
done := make(chan error, 1)
go func() {
_, _, err := FetchAction(t.Context(), input)
done <- err
}()
select {
case err := <-done:
t.Fatalf("FetchAction replaced the tree without waiting for the job holding it: %v", err)
case <-waiting:
}
info, err := os.Stat(yml)
require.NoError(t, err)
assert.Equal(t, int64(4), info.Size(), "the tree a job holds is left alone")
release()
require.NoError(t, <-done)
body, err := os.ReadFile(yml)
require.NoError(t, err)
assert.Equal(t, "name: action\n", string(body))
})
t.Run("retries a failing download and drops the token on a redirect to another origin", func(t *testing.T) {
serve("redirect", 2)
input := newInput("main")
_, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "main"), sha)
assert.Equal(t, expectedSha, sha)
assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
_, headers := served()
require.Len(t, headers, 4)
@@ -251,7 +328,7 @@ func TestFetchAction(t *testing.T) {
serve(mode, 0)
tree, sha, err := FetchAction(t.Context(), newInput("main"))
require.NoError(t, err, mode)
assert.Equal(t, gitRevParse(t, remote, "main"), sha, mode)
assert.Equal(t, expectedSha, sha, mode)
assert.FileExists(t, filepath.Join(tree, "action.yml"), mode)
assert.NoDirExists(t, filepath.Join(tree, ".git"), mode)
_, headers := served()
@@ -266,7 +343,7 @@ func TestFetchAction(t *testing.T) {
input.URL = "https://example.invalid/org/action"
_, sha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, gitRevParse(t, remote, "main"), sha)
assert.Equal(t, expectedSha, sha)
_, headers := served()
assert.Len(t, headers, 1)
assert.NoDirExists(t, filepath.Join(input.Dir, "git"))
@@ -306,10 +383,30 @@ func TestExtractTreeConfinesEntries(t *testing.T) {
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err := extractTree(&archive, dir, false)
_, err := extractTree(t.Context(), &archive, dir, false)
require.Error(t, err)
assert.NoFileExists(t, filepath.Join(dir, "escape"))
assert.NoFileExists(t, filepath.Join(outside, "escape"))
})
}
}
func TestTreeIntactFindsNamesThatAreNotUTF8(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("macOS and Windows store no name that is not UTF-8")
}
sha := strings.Repeat("a", 40)
var archive bytes.Buffer
writer := tar.NewWriter(&archive)
require.NoError(t, writer.WriteHeader(&tar.Header{Typeflag: tar.TypeXGlobalHeader, PAXRecords: map[string]string{"comment": sha}}))
require.NoError(t, writer.WriteHeader(&tar.Header{Name: "caf\xe9.js", Typeflag: tar.TypeReg, Mode: 0o644, Size: 1, Format: tar.FormatGNU}))
_, err := writer.Write([]byte("x"))
require.NoError(t, err)
require.NoError(t, writer.Close())
dir := t.TempDir()
_, err = extractTree(t.Context(), &archive, dir, false)
require.NoError(t, err)
require.FileExists(t, filepath.Join(dir, sha, "caf\xe9.js"))
assert.True(t, treeIntact(dir, sha))
}
+17 -2
View File
@@ -10,6 +10,7 @@ import (
"encoding/base64"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
@@ -320,11 +321,20 @@ func cachedRef(ref string) string {
func resolveRef(ctx context.Context, dir, ref string) (string, error) {
if !isFullObjectID(ref) {
for _, name := range []string{cachedRef(ref), "refs/tags/" + ref, "refs/remotes/origin/" + ref} {
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", name+"^{commit}"); err == nil {
revisions := []string{cachedRef(ref) + "^{commit}", "refs/tags/" + ref + "^{commit}", "refs/remotes/origin/" + ref + "^{commit}"}
if strings.ContainsAny(ref, "\x00\n") {
for _, revision := range revisions {
if sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", revision); err == nil {
return sha, nil
}
}
} else if output, err := runGitWithStdin(ctx, dir, nil, strings.NewReader(strings.Join(revisions, "\n")+"\n"), "cat-file", "--batch-check=%(objectname)"); err == nil {
for sha := range strings.SplitSeq(string(output), "\n") {
if isFullObjectID(sha) {
return sha, nil
}
}
}
}
sha, err := gitOutput(ctx, dir, "rev-parse", "--verify", ref+"^{commit}")
if err == nil && hexRefRegex.MatchString(ref) && len(sha) != len(ref) {
@@ -427,6 +437,10 @@ func ListFiles(ctx context.Context, dir string) ([]string, error) {
}
func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, args ...string) ([]byte, error) {
return runGitWithStdin(ctx, dir, input, nil, args...)
}
func runGitWithStdin(ctx context.Context, dir string, input *NewGitCloneExecutorInput, stdin io.Reader, args ...string) ([]byte, error) {
config := []string{"core.autocrlf=false", "core.eol=lf"} // keep action files LF
if input != nil {
config = append(config, RemoteConfig(input.Token, input.InsecureSkipTLS)...)
@@ -438,6 +452,7 @@ func runGit(ctx context.Context, dir string, input *NewGitCloneExecutorInput, ar
args = append([]string{"-C", dir}, args...)
}
cmd := exec.CommandContext(ctx, "git", args...)
cmd.Stdin = stdin
cmd.Env = slices.DeleteFunc(os.Environ(), func(entry string) bool {
name, _, _ := strings.Cut(entry, "=")
return slices.Contains(LocalEnvVars, name)
+37 -6
View File
@@ -29,6 +29,31 @@ import (
"github.com/stretchr/testify/require"
)
var gitFixtureDir string
func TestMain(m *testing.M) {
var err error
gitFixtureDir, err = os.MkdirTemp("", "runner-git-fixture-")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
code := 1
if err = gitCmd("init", "--quiet", "--template=", "--bare", "--initial-branch=main", filepath.Join(gitFixtureDir, "remote")); err == nil {
err = gitCmd("init", "--quiet", "--template=", "--initial-branch=main", filepath.Join(gitFixtureDir, "work"))
}
if err == nil {
code = m.Run()
} else {
fmt.Fprintln(os.Stderr, err)
}
if err := os.RemoveAll(gitFixtureDir); err != nil {
fmt.Fprintln(os.Stderr, err)
code = 1
}
os.Exit(code)
}
func TestFindGitSlug(t *testing.T) {
for _, testcase := range []struct {
remoteURL string
@@ -141,9 +166,8 @@ func TestGitFindRef(t *testing.T) {
} {
t.Run(name, func(t *testing.T) {
dir := filepath.Join(basedir, name)
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, gitCmd("-C", dir, "init", "--initial-branch=master"))
require.NoError(t, os.RemoveAll(filepath.Join(dir, ".git", "hooks")))
require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, os.WriteFile(filepath.Join(dir, ".git", "HEAD"), []byte("ref: refs/heads/master\n"), 0o644))
for _, command := range tt.commands {
require.NoError(t, gitCmd(append([]string{"-C", dir}, command...)...))
}
@@ -180,6 +204,7 @@ func TestGitCloneExecutor(t *testing.T) {
t.Setenv("GIT_CONFIG_GLOBAL", hostConfig)
t.Setenv("GIT_DIR", t.TempDir())
var clones sync.WaitGroup
for name, tt := range map[string]struct {
Err error
Ref string
@@ -207,6 +232,7 @@ func TestGitCloneExecutor(t *testing.T) {
Ref: fullSha[:4],
},
} {
clones.Go(func() {
t.Run(name, func(t *testing.T) {
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
@@ -231,7 +257,9 @@ func TestGitCloneExecutor(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, "#!/bin/sh\necho hi\n", string(script), "host core.autocrlf must not convert action files")
})
})
}
clones.Wait()
}
func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
@@ -476,9 +504,12 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
func newGitRemote(t *testing.T, messages ...string) (string, string) {
t.Helper()
remoteDir, workDir := t.TempDir(), t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, os.CopyFS(remoteDir, os.DirFS(filepath.Join(gitFixtureDir, "remote"))))
require.NoError(t, os.CopyFS(workDir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
config, err := os.ReadFile(filepath.Join(workDir, ".git", "config"))
require.NoError(t, err)
require.NoError(t, os.WriteFile(filepath.Join(workDir, ".git", "config"),
fmt.Appendf(config, "\n[remote \"origin\"]\n\turl = %s\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n", strconv.Quote(remoteDir)), 0o644))
for _, message := range messages {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
}
+4 -2
View File
@@ -74,8 +74,10 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Creating image from context dir '%s' with tag '%s' and platform '%s'", input.ContextDir, input.ImageTag, input.Platform)
resp, err := cli.ImageBuild(ctx, buildContext, options)
return logDockerResponse(logger, resp.Body, err != nil)
if err != nil {
return err
}
return logDockerResponse(logger, resp.Body, false)
}
}
+22
View File
@@ -8,13 +8,35 @@ package container
import (
"archive/tar"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
func TestDockerBuildReportsDaemonError(t *testing.T) {
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("API-Version", "1.47")
if strings.HasSuffix(request.URL.Path, "/build") {
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusInternalServerError)
_, _ = io.WriteString(writer, `{"message":"build rejected"}`)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
t.Setenv("DOCKER_CONFIG", t.TempDir())
t.Setenv("DOCKER_AUTH_CONFIG", "")
require.ErrorContains(t, NewDockerBuildExecutor(NewDockerBuildExecutorInput{
ImageTag: "action:test", Dockerfile: "Dockerfile", BuildContext: strings.NewReader(""),
})(t.Context()), "build rejected")
}
func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile(t *testing.T) {
dir := t.TempDir()
for name, content := range map[string]string{
+25 -23
View File
@@ -5,14 +5,17 @@
package container
import (
"bytes"
"context"
"fmt"
"os"
"os/exec"
"strings"
"testing"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
mobyclient "github.com/moby/moby/client"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -21,48 +24,47 @@ func init() {
logging.SetLevel(logging.LevelDebug)
}
// buildScratchImage builds a tiny empty image for the given platform locally (FROM scratch, no
// network or emulation since there is nothing to run) and returns its tag, removing it after
// the test.
func buildScratchImage(t *testing.T, platform string) string {
func loadScratchImage(t *testing.T, platform string) string {
t.Helper()
tag := fmt.Sprintf("act-test-exists-%s:latest", strings.TrimPrefix(platform, "linux/"))
cmd := exec.Command("docker", "build", "--platform", platform, "-t", tag, "-")
cmd.Stdin = strings.NewReader("FROM scratch\nLABEL act-test=1\n")
// Force BuildKit: it records the requested architecture in the image config for a
// FROM-scratch build, whereas the classic builder ignores --platform and tags it with the
// host arch, which would break the per-platform existence assertions below.
cmd.Env = append(os.Environ(), "DOCKER_BUILDKIT=1")
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
t.Cleanup(func() { _ = exec.Command("docker", "rmi", "-f", tag).Run() })
tag := fmt.Sprintf("act-test-exists-%s-%d:latest", strings.TrimPrefix(platform, "linux/"), time.Now().UnixNano())
var archive bytes.Buffer
require.NoError(t, writeFilesTar(t.Context(), &archive, 0, 0,
&FileEntry{Name: "config.json", Mode: 0o600, Body: fmt.Sprintf(`{"architecture":%q,"os":"linux","rootfs":{"type":"layers","diff_ids":[]}}`, strings.TrimPrefix(platform, "linux/"))},
&FileEntry{Name: "manifest.json", Mode: 0o600, Body: fmt.Sprintf(`[{"Config":"config.json","RepoTags":[%q],"Layers":[]}]`, tag)},
))
cli, err := GetDockerClient(t.Context())
require.NoError(t, err)
t.Cleanup(func() { _ = cli.Close() })
response, err := cli.ImageLoad(t.Context(), &archive)
require.NoError(t, err)
defer response.Close()
require.NoError(t, logDockerResponse(common.Logger(t.Context()), response, false))
t.Cleanup(func() {
_, _ = cli.ImageRemove(context.WithoutCancel(t.Context()), tag, mobyclient.ImageRemoveOptions{Force: true})
})
return tag
}
func TestImageExistsLocally(t *testing.T) {
t.Parallel()
requireDocker(t)
ctx := t.Context()
// a non-existent image is reported absent
missing, err := ImageExistsLocally(ctx, "library/alpine:this-random-tag-will-never-exist", "linux/amd64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, missing)
// Build tiny images for two architectures locally so per-platform existence can be checked
// offline (formerly pulled node:24-bookworm-slim for amd64 and arm64 over the network).
amd64Ref := buildScratchImage(t, "linux/amd64")
arm64Ref := buildScratchImage(t, "linux/arm64")
amd64Ref := loadScratchImage(t, "linux/amd64")
arm64Ref := loadScratchImage(t, "linux/arm64")
amd64Exists, err := ImageExistsLocally(ctx, amd64Ref, "linux/amd64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, amd64Exists)
// a non-host architecture image is detected for its own architecture
arm64Exists, err := ImageExistsLocally(ctx, arm64Ref, "linux/arm64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.True(t, arm64Exists)
// a present image is reported absent for a different platform
wrongPlatform, err := ImageExistsLocally(ctx, amd64Ref, "linux/arm64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, wrongPlatform)
+2 -1
View File
@@ -33,6 +33,7 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
defer dockerResponse.Close()
scanner := bufio.NewScanner(dockerResponse)
scanner.Buffer(nil, 64*1024*1024) // a build echoes long instructions in one record
for scanner.Scan() {
line := scanner.Bytes()
var msg dockerMessage
@@ -65,7 +66,7 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
}
}
return nil
return scanner.Err()
}
func writeLog(logger *logging.Logger, isError bool, format string, args ...any) {
+8 -8
View File
@@ -85,14 +85,14 @@ networks:
}
func NewDockerNetworkCreateExecutor(name string, opts NewDockerNetworkCreateExecutorInput) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
}
defer cli.Close()
return createDockerNetwork(ctx, cli, name, opts)
}
}).IfNot(common.Dryrun)
}
func createDockerNetwork(ctx context.Context, cli client.APIClient, name string, opts NewDockerNetworkCreateExecutorInput) error {
@@ -157,7 +157,7 @@ func isAddressPoolExhausted(err error) bool {
}
func NewDockerNetworkConnectExecutor(networkName, container string) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
@@ -167,7 +167,7 @@ func NewDockerNetworkConnectExecutor(networkName, container string) common.Execu
common.Logger(ctx).Warnf("cannot connect %s to network %s, so jobs on it cannot reach the cache: %v", container, networkName, err)
}
return nil
}
}).IfNot(common.Dryrun)
}
func connectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error {
@@ -182,14 +182,14 @@ func connectNetwork(ctx context.Context, cli client.APIClient, networkName, cont
}
func NewDockerNetworkDisconnectExecutor(networkName, container string) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
}
defer cli.Close()
return disconnectNetwork(ctx, cli, networkName, container)
}
}).IfNot(common.Dryrun)
}
func disconnectNetwork(ctx context.Context, cli client.APIClient, networkName, container string) error {
@@ -201,14 +201,14 @@ func disconnectNetwork(ctx context.Context, cli client.APIClient, networkName, c
}
func NewDockerNetworkRemoveExecutor(name string) common.Executor {
return func(ctx context.Context) error {
return common.Executor(func(ctx context.Context) error {
cli, err := GetDockerClient(ctx)
if err != nil {
return err
}
defer cli.Close()
return removeNetwork(ctx, cli, name)
}
}).IfNot(common.Dryrun)
}
func removeNetwork(ctx context.Context, cli client.APIClient, name string) error {
+54 -15
View File
@@ -48,13 +48,20 @@ var (
rawStreamPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers/[^/]+/attach|exec/[^/]+/start)$`)
)
var dockerProxyDirectories sync.Map
var dockerProxyDirectories, dockerProxyProbes sync.Map
type dockerProxyDirectory struct {
dir, daemonDir string
socket os.FileInfo
}
type dockerProxyProbe struct {
directory dockerProxyDirectory
done chan struct{}
seen bool
err error
}
func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if host := os.Getenv("DOCKER_HOST"); runtime.GOOS != "linux" || host != "" && !strings.HasPrefix(host, "unix://") {
return nil
@@ -79,16 +86,8 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
cached, _ := dockerProxyDirectories.Load(key)
directory, seen := cached.(dockerProxyDirectory)
seen = seen && os.SameFile(directory.socket, socket) // daemon replacement invalidates cached paths
dir, daemonDir := directory.dir, directory.daemonDir
if !seen {
dir, daemonDir = runnerContainerWorkdir(probeCtx, cli)
seen = daemonDir != ""
if !seen {
if dir, err = filepath.Abs(os.TempDir()); err == nil {
daemonDir = dir
seen, err = daemonSeesDir(probeCtx, cli, dir, daemonDir)
}
}
directory, seen, err = probeDockerProxyDirectory(probeCtx, cli, key, socket)
}
if err != nil {
common.Logger(ctx).Infof("docker proxy probe failed, jobs get the daemon socket directly: %v", err)
@@ -101,17 +100,51 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if ctx.Err() != nil {
return nil
}
proxy, err := StartDockerProxy(daemonSocket, dir, job)
proxy, err := StartDockerProxy(daemonSocket, directory.dir, job)
if err != nil {
dockerProxyDirectories.Delete(key)
common.Logger(ctx).Warnf("docker proxy not started, the job gets the daemon socket directly: %v", err)
return nil
}
dockerProxyDirectories.Store(key, dockerProxyDirectory{dir, daemonDir, socket})
proxy.Socket = daemonDir + strings.TrimPrefix(proxy.Socket, dir)
dockerProxyDirectories.Store(key, directory)
proxy.Socket = directory.daemonDir + strings.TrimPrefix(proxy.Socket, directory.dir)
return proxy
}
func probeDockerProxyDirectory(ctx context.Context, cli client.APIClient, key [3]string, socket os.FileInfo) (dockerProxyDirectory, bool, error) {
for {
probe := &dockerProxyProbe{directory: dockerProxyDirectory{socket: socket}, done: make(chan struct{})}
actual, loaded := dockerProxyProbes.LoadOrStore(key, probe)
if loaded {
probe, _ = actual.(*dockerProxyProbe)
if !os.SameFile(probe.directory.socket, socket) {
dockerProxyProbes.CompareAndDelete(key, probe)
continue
}
select {
case <-ctx.Done():
return dockerProxyDirectory{}, false, ctx.Err()
case <-probe.done:
}
if ctx.Err() == nil && (errors.Is(probe.err, context.Canceled) || errors.Is(probe.err, context.DeadlineExceeded)) {
continue
}
} else {
probe.directory.dir, probe.directory.daemonDir = runnerContainerWorkdir(ctx, cli)
probe.seen = probe.directory.daemonDir != ""
if !probe.seen {
if probe.directory.dir, probe.err = filepath.Abs(key[2]); probe.err == nil {
probe.directory.daemonDir = probe.directory.dir
probe.seen, probe.err = daemonSeesDir(ctx, cli, probe.directory.dir, probe.directory.daemonDir)
}
}
dockerProxyProbes.CompareAndDelete(key, probe)
close(probe.done)
}
return probe.directory, probe.seen, probe.err
}
}
// runnerContainerWorkdir looks the runner's container up by hostname to find the daemon's path to its working directory.
func runnerContainerWorkdir(ctx context.Context, cli client.APIClient) (workdir, daemonDir string) {
workdir, err := os.Getwd()
@@ -162,12 +195,18 @@ func daemonSeesDir(ctx context.Context, cli client.APIClient, dir, daemonDir str
if len(images.Items) == 0 {
return false, errors.New("no image available for the docker proxy probe")
}
created, err := cli.ContainerCreate(ctx, client.ContainerCreateOptions{
Config: &container.Config{Image: images.Items[0].ID, Cmd: []string{"true"}},
var created client.ContainerCreateResult
for _, image := range images.Items {
created, err = cli.ContainerCreate(ctx, client.ContainerCreateOptions{
Config: &container.Config{Image: image.ID, Cmd: []string{"true"}},
HostConfig: &container.HostConfig{Mounts: []mount.Mount{
{Type: mount.TypeBind, Source: daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}, // not the marker itself, podman creates a missing bind source where docker rejects it
}},
})
if !cerrdefs.IsNotFound(err) {
break
}
}
if cerrdefs.IsInvalidArgument(err) {
return false, nil
}
+144 -1
View File
@@ -22,6 +22,7 @@ import (
"strings"
"sync/atomic"
"testing"
"testing/synctest"
"time"
cerrdefs "github.com/containerd/errdefs"
@@ -371,6 +372,9 @@ func TestDockerProxyWithDaemon(t *testing.T) {
job := "proxy-test-" + strconv.FormatInt(time.Now().UnixNano(), 36)
proxy, err := StartDockerProxy(daemonSocketPath(t, direct), dir, job)
if os.Geteuid() != 0 && errors.Is(err, os.ErrPermission) {
t.Skipf("non-root runners that cannot take the daemon socket's ownership get it directly: %v", err)
}
require.NoError(t, err)
t.Cleanup(func() { _ = proxy.Close(context.Background()) })
viaProxy, err := mobyclient.New(mobyclient.WithHost("unix://" + proxy.Socket))
@@ -483,7 +487,7 @@ func (c *probeClient) ContainerInspect(context.Context, string, mobyclient.Conta
}
func (c *probeClient) ImageList(context.Context, mobyclient.ImageListOptions) (mobyclient.ImageListResult, error) {
return mobyclient.ImageListResult{Items: []image.Summary{{ID: "probe-image"}}}, nil
return mobyclient.ImageListResult{Items: []image.Summary{{ID: "removed-image"}, {ID: "probe-image"}}}, nil
}
func (c *probeClient) ContainerCreate(_ context.Context, opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
@@ -498,6 +502,139 @@ func (c *probeClient) ContainerStatPath(_ context.Context, _ string, opts mobycl
return mobyclient.ContainerStatPathResult{}, c.stat(opts.Path)
}
func TestDockerProxyDirectoryProbes(t *testing.T) {
type result struct {
seen bool
err error
}
for name, createErr := range map[string]error{"unavailable directory": nil, "daemon error": errors.New("probe failed")} {
t.Run(name, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
key := [3]string{"daemon", dir, dir}
release := make(chan struct{})
var calls, removals atomic.Int32
visible := false
cli := &probeClient{
create: func(_opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls.Add(1)
<-release
return mobyclient.ContainerCreateResult{ID: "probe"}, createErr
},
stat: func(_path string) error {
if visible {
return nil
} else {
return cerrdefs.ErrNotFound
}
},
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
removals.Add(1)
return mobyclient.ContainerRemoveResult{}, nil
},
}
results := make(chan result, 8)
for range cap(results) {
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
results <- result{seen, err}
}()
}
synctest.Wait()
assert.EqualValues(t, 1, calls.Load())
ctx, cancel := context.WithCancel(t.Context())
cancel()
_, seen, err := probeDockerProxyDirectory(ctx, cli, key, socket)
require.ErrorIs(t, err, context.Canceled)
assert.False(t, seen)
replacement, err := os.Stat(t.TempDir())
require.NoError(t, err)
replaced := make(chan result, 1)
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, replacement)
replaced <- result{seen, err}
}()
synctest.Wait()
assert.EqualValues(t, 2, calls.Load())
close(release)
for range cap(results) {
result := <-results
require.ErrorIs(t, result.err, createErr)
assert.False(t, result.seen)
}
replacementResult := <-replaced
require.ErrorIs(t, replacementResult.err, createErr)
assert.False(t, replacementResult.seen)
assert.EqualValues(t, 2, calls.Load())
if createErr == nil {
assert.EqualValues(t, 2, removals.Load())
} else {
assert.Zero(t, removals.Load())
}
createErr, visible = nil, true
directory, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
require.NoError(t, err)
assert.True(t, seen)
assert.Equal(t, dir, directory.dir)
assert.Equal(t, dir, directory.daemonDir)
assert.EqualValues(t, 3, calls.Load())
entries, err := os.ReadDir(dir)
require.NoError(t, err)
assert.Empty(t, entries)
})
})
}
t.Run("cancelled probe does not cancel another caller", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
dir := t.TempDir()
socket, err := os.Stat(t.TempDir())
require.NoError(t, err)
key := [3]string{"daemon", dir, dir}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
release := make(chan struct{})
calls := 0
cli := &probeClient{
create: func(_opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
calls++
if calls == 1 {
<-release
return mobyclient.ContainerCreateResult{}, ctx.Err()
} else {
return mobyclient.ContainerCreateResult{ID: "probe"}, nil
}
},
stat: func(_path string) error { return nil },
remove: func(_ctx context.Context, _id string, _opts mobyclient.ContainerRemoveOptions) (mobyclient.ContainerRemoveResult, error) {
return mobyclient.ContainerRemoveResult{}, nil
},
}
leader, follower := make(chan result, 1), make(chan result, 1)
go func() {
_, seen, err := probeDockerProxyDirectory(ctx, cli, key, socket)
leader <- result{seen, err}
}()
synctest.Wait()
go func() {
_, seen, err := probeDockerProxyDirectory(t.Context(), cli, key, socket)
follower <- result{seen, err}
}()
synctest.Wait()
assert.Equal(t, 1, calls)
cancel()
close(release)
leaderResult, followerResult := <-leader, <-follower
require.ErrorIs(t, leaderResult.err, context.Canceled)
assert.False(t, leaderResult.seen)
require.NoError(t, followerResult.err)
assert.True(t, followerResult.seen)
assert.Equal(t, 2, calls)
})
})
}
func TestDaemonSeesDir(t *testing.T) {
dir := t.TempDir()
markers := make(map[string]bool)
@@ -517,9 +654,14 @@ func TestDaemonSeesDir(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
removed := false
var attemptedImages []string
cli := &probeClient{
create: func(opts mobyclient.ContainerCreateOptions) (mobyclient.ContainerCreateResult, error) {
assert.Equal(t, []mount.Mount{{Type: mount.TypeBind, Source: testCase.daemonDir, Target: "/gitea-runner-probe", ReadOnly: true}}, opts.HostConfig.Mounts)
attemptedImages = append(attemptedImages, opts.Config.Image)
if opts.Config.Image == "removed-image" {
return mobyclient.ContainerCreateResult{}, cerrdefs.ErrNotFound
}
cancel()
if _, err := os.Stat(testCase.daemonDir); err != nil {
return mobyclient.ContainerCreateResult{}, cerrdefs.ErrInvalidArgument
@@ -547,6 +689,7 @@ func TestDaemonSeesDir(t *testing.T) {
}
seen, err := daemonSeesDir(ctx, cli, dir, testCase.daemonDir)
require.ErrorIs(t, err, testCase.removeErr)
assert.Equal(t, []string{"removed-image", "probe-image"}, attemptedImages)
assert.Equal(t, testCase.seen, seen)
assert.Equal(t, testCase.removed, removed)
entries, err := os.ReadDir(dir)
+1 -1
View File
@@ -114,7 +114,7 @@ func getImagePullOptions(ctx context.Context, input NewDockerPullExecutorInput)
if err != nil {
return imagePullOptions, err
}
if authConfig.Username == "" && authConfig.Password == "" {
if authConfig.Username == "" && authConfig.Password == "" && authConfig.IdentityToken == "" && authConfig.RegistryToken == "" {
return imagePullOptions, nil
}
logger.Info("using DockerAuthConfig authentication for docker pull")
+24
View File
@@ -6,12 +6,17 @@ package container
import (
"io"
"os"
"path/filepath"
"strings"
"testing"
"testing/iotest"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"github.com/moby/moby/api/pkg/authconfig"
"github.com/moby/moby/api/types/registry"
assert "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -64,6 +69,21 @@ func TestGetImagePullOptions(t *testing.T) {
})
assert.NoError(t, err, "Failed to create ImagePullOptions") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, "eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwicGFzc3dvcmQiOiJwYXNzd29yZFxuIiwic2VydmVyYWRkcmVzcyI6Imh0dHBzOi8vaW5kZXguZG9ja2VyLmlvL3YxLyJ9", options.RegistryAuth, "RegistryAuth should be taken from local docker config")
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"auths":{"identity.example.com":{"identitytoken":"identity-token"},"registry.example.com":{"registrytoken":"registry-token"}}}`), 0o600))
t.Setenv("DOCKER_CONFIG", dir)
for host, want := range map[string]registry.AuthConfig{
"identity.example.com": {IdentityToken: "identity-token", ServerAddress: "identity.example.com"},
"registry.example.com": {RegistryToken: "registry-token", ServerAddress: "registry.example.com"},
} {
options, err := getImagePullOptions(ctx, NewDockerPullExecutorInput{Image: host + "/app"})
require.NoError(t, err)
require.NotEmpty(t, options.RegistryAuth, host)
got, err := authconfig.Decode(options.RegistryAuth)
require.NoError(t, err)
assert.Equal(t, want, *got)
}
}
// A digest-pinned image is immutable, so its local copy is always current.
@@ -82,4 +102,8 @@ func TestLogDockerResponseError(t *testing.T) {
require.EqualError(t, streamErr(`{"error":"toomanyrequests: rate limit exceeded"}`), "toomanyrequests: rate limit exceeded")
require.EqualError(t, streamErr(`{"errorDetail":{"message":"unexpected EOF"}}`), "unexpected EOF")
require.NoError(t, streamErr(`{"status":"Downloading"}`))
require.NoError(t, streamErr(`{"stream":"`+strings.Repeat("x", 70000)+`"}`))
require.ErrorIs(t, logDockerResponse(logger, io.NopCloser(io.MultiReader(
strings.NewReader(`{"status":"Downloading"}`+"\n"), iotest.ErrReader(io.ErrUnexpectedEOF),
)), false), io.ErrUnexpectedEOF)
}
+35 -14
View File
@@ -205,18 +205,30 @@ func (cr *containerReference) DumpLogs(ctx context.Context) error {
if cr.id == "" {
return cr.missingContainerError("read its logs")
}
tty, err := cr.containerTTY(ctx)
if err != nil {
return err
}
logs, err := cr.cli.ContainerLogs(ctx, cr.id, client.ContainerLogsOptions{ShowStdout: true, ShowStderr: true})
if err != nil {
return err
}
defer logs.Close()
return cr.copyOutput(logs)
return cr.copyOutput(logs, tty)
}
func (cr *containerReference) containerTTY(ctx context.Context) (bool, error) {
inspect, err := cr.cli.ContainerInspect(ctx, cr.id, client.ContainerInspectOptions{})
if err != nil {
return false, fmt.Errorf("failed to inspect container: %w", err)
}
return inspect.Container.Config != nil && inspect.Container.Config.Tty, nil
}
// copyOutput writes a container stream to the writers the container was created with,
// demultiplexing it unless the container has a TTY, which sends a single raw stream.
func (cr *containerReference) copyOutput(stream io.Reader) error {
func (cr *containerReference) copyOutput(stream io.Reader, tty bool) error {
outWriter := cr.input.Stdout
if outWriter == nil {
outWriter = os.Stdout
@@ -227,7 +239,7 @@ func (cr *containerReference) copyOutput(stream io.Reader) error {
}
var err error
if !cr.input.AllocatePTY || os.Getenv("NORAW") != "" {
if !tty || os.Getenv("NORAW") != "" {
_, err = stdcopy.StdCopy(outWriter, errWriter, stream)
} else {
_, err = io.Copy(outWriter, stream)
@@ -296,7 +308,7 @@ func containerInfoFromInspect(inspect container.InspectResponse) *Info {
}
func (cr *containerReference) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(cr, srcPath, env).IfNot(common.Dryrun)
return parseEnvFile(cr, srcPath, env, false).IfNot(common.Dryrun)
}
func (cr *containerReference) UpdateFromImageEnv(env *map[string]string) common.Executor {
@@ -499,7 +511,7 @@ func (cr *containerReference) remove() common.Executor {
if idOrName == "" && cr.input != nil {
idOrName = cr.input.Name
}
if idOrName == "" {
if idOrName == "" || cr.cli == nil {
return nil
}
@@ -851,7 +863,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
cmdResponse := make(chan error, 1)
go func() {
cmdResponse <- cr.copyOutput(resp.Reader)
cmdResponse <- cr.copyOutput(resp.Reader, cr.input.AllocatePTY)
}()
select {
@@ -867,7 +879,10 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
// truncate the tail of the log. The goroutine exits once the hijacked
// stream is closed by resp.Close() in the caller's defer.
select {
case <-cmdResponse:
case err := <-cmdResponse:
if err != nil {
logger.Debugf("Failed to drain command output after cancellation: %v", err)
}
case <-time.After(drainGracePeriod):
logger.Warn("Timed out draining command output after cancellation")
}
@@ -876,11 +891,7 @@ func (cr *containerReference) waitForCommand(ctx context.Context, resp client.Hi
// from executing
return ctx.Err()
case err := <-cmdResponse:
if err != nil {
logger.Error(err)
}
return nil
return err
}
}
@@ -897,7 +908,7 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
}
logger.Debugf("Writing tarball %s from %s", tarFile.Name(), srcPath)
defer func() {
if err := tarFile.Close(); !errors.Is(err, os.ErrClosed) {
if err := tarFile.Close(); err != nil && !errors.Is(err, os.ErrClosed) {
logger.Error(err)
}
if err := os.Remove(tarFile.Name()); err != nil {
@@ -944,6 +955,10 @@ func (cr *containerReference) upload(ctx context.Context, dstPath string, conten
func (cr *containerReference) attach() common.Executor {
return func(ctx context.Context) error {
tty, err := cr.containerTTY(ctx)
if err != nil {
return err
}
out, err := cr.cli.ContainerAttach(ctx, cr.id, client.ContainerAttachOptions{
Stream: true,
Stdout: true,
@@ -954,9 +969,12 @@ func (cr *containerReference) attach() common.Executor {
}
done := make(chan struct{})
cr.attachDone = done
stopClose := context.AfterFunc(ctx, out.Close)
go func() {
defer close(done)
if copyErr := cr.copyOutput(out.Reader); copyErr != nil {
defer out.Close()
defer stopClose()
if copyErr := cr.copyOutput(out.Reader, tty); copyErr != nil && ctx.Err() == nil {
common.Logger(ctx).Error(copyErr)
}
}()
@@ -991,6 +1009,9 @@ func (cr *containerReference) wait() common.Executor {
return fmt.Errorf("failed to wait for container: %w", err)
}
case status := <-waitResult.Result:
if status.Error != nil {
return fmt.Errorf("failed to wait for container: %s", status.Error.Message)
}
statusCode = status.StatusCode
}
+87 -23
View File
@@ -92,6 +92,16 @@ func (m *mockDockerClient) ContainerAttach(ctx context.Context, containerID stri
return args.Get(0).(mobyclient.ContainerAttachResult), args.Error(1)
}
func (m *mockDockerClient) ContainerLogs(ctx context.Context, containerID string, opts mobyclient.ContainerLogsOptions) (mobyclient.ContainerLogsResult, error) {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerLogsResult), args.Error(1)
}
func (m *mockDockerClient) ContainerStart(ctx context.Context, containerID string, opts mobyclient.ContainerStartOptions) (mobyclient.ContainerStartResult, error) {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerStartResult), args.Error(1)
}
func (m *mockDockerClient) ContainerWait(ctx context.Context, containerID string, opts mobyclient.ContainerWaitOptions) mobyclient.ContainerWaitResult {
args := m.Called(ctx, containerID, opts)
return args.Get(0).(mobyclient.ContainerWaitResult)
@@ -158,6 +168,7 @@ func (r *interruptReader) Read(_ []byte) (int, error) {
type mockConn struct {
net.Conn
mock.Mock
closed atomic.Bool
}
func (m *mockConn) Write(b []byte) (n int, err error) {
@@ -166,6 +177,7 @@ func (m *mockConn) Write(b []byte) (n int, err error) {
}
func (m *mockConn) Close() (err error) {
m.closed.Store(true)
return nil
}
@@ -236,6 +248,21 @@ func TestDockerExecFailure(t *testing.T) {
client.AssertExpectations(t)
}
func TestDockerExecStreamFailure(t *testing.T) {
ctx := t.Context()
client := &mockDockerClient{}
client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).
Return(mobyclient.ExecCreateResult{ID: "id"}, nil)
client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).
Return(mobyclient.ExecAttachResult{
Conn: &mockConn{},
Reader: bufio.NewReader(bytes.NewReader(stdcopyFrame(stdcopy.Systemerr, "unable to start exec"))),
}, nil)
require.ErrorContains(t, (&containerReference{id: "123", cli: client, input: &NewContainerInput{}}).
exec([]string{"sh"}, nil, "", "")(ctx), "unable to start exec")
client.AssertExpectations(t)
}
// stdcopyFrame wraps payload in a single Docker multiplexed-stream frame, the
// format StdCopy expects: an 8-byte header (stream type + 4-byte big-endian
// length) followed by the payload.
@@ -248,60 +275,88 @@ func stdcopyFrame(stream stdcopy.StdType, payload string) []byte {
}
func TestDockerAttachFlushesTrailingLine(t *testing.T) {
for _, testcase := range []struct {
name string
tty bool
}{
{name: "multiplexed"},
{name: "tty from container options", tty: true},
} {
t.Run(testcase.name, func(t *testing.T) {
ctx := t.Context()
framed := bytes.NewBuffer(stdcopyFrame(stdcopy.Stdout, "line one\nlast line without newline"))
output := []byte("line one\nlast line without newline")
if !testcase.tty {
output = stdcopyFrame(stdcopy.Stdout, string(output))
}
var lines []string
logWriter := common.NewLineWriter(func(s string) bool {
lines = append(lines, s)
logWriter := common.NewLineWriter(func(line string) bool {
lines = append(lines, line)
return true
})
client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "123", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{Config: &container.Config{Tty: testcase.tty}}}, nil)
conn := &mockConn{}
client.On("ContainerAttach", ctx, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{
Conn: &mockConn{},
Reader: bufio.NewReader(framed),
Conn: conn,
Reader: bufio.NewReader(bytes.NewReader(output)),
}, nil)
client.On("ContainerLogs", ctx, "123", mobyclient.ContainerLogsOptions{ShowStdout: true, ShowStderr: true}).
Return(io.NopCloser(bytes.NewReader(output)), nil)
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 0}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Stdout: logWriter,
Stderr: logWriter,
},
}
cr := &containerReference{id: "123", cli: client, input: &NewContainerInput{Stdout: logWriter, Stderr: logWriter}}
require.NoError(t, cr.attach()(ctx))
require.NoError(t, cr.wait()(ctx))
assert.Equal(t, []string{"line one\n", "last line without newline"}, lines)
assert.True(t, conn.closed.Load())
require.NoError(t, cr.DumpLogs(ctx))
assert.Equal(t, []string{"line one\n", "last line without newline", "line one\n", "last line without newline"}, lines)
client.AssertExpectations(t)
})
}
}
func TestDockerAttachStreamClosesWithItsContext(t *testing.T) {
conn, peer := net.Pipe()
t.Cleanup(func() { _ = conn.Close() })
t.Cleanup(func() { _ = peer.Close() })
client := &mockDockerClient{}
client.On("ContainerInspect", mock.Anything, "123", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{Config: &container.Config{}}}, nil)
client.On("ContainerAttach", mock.Anything, "123", mock.AnythingOfType("client.ContainerAttachOptions")).
Return(mobyclient.ContainerAttachResult{HijackedResponse: mobyclient.NewHijackedResponse(conn, "")}, nil)
client.On("ContainerStart", mock.Anything, "123", mobyclient.ContainerStartOptions{}).
Return(mobyclient.ContainerStartResult{}, errors.New("container start failed"))
cr := &containerReference{id: "123", cli: client, input: &NewContainerInput{}}
ctx, cancel := context.WithCancel(t.Context())
require.ErrorContains(t, cr.Start(true)(ctx), "container start failed")
cancel()
<-cr.attachDone
client.AssertExpectations(t)
}
func TestDockerWaitFailure(t *testing.T) {
ctx := t.Context()
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 2}
client := &mockDockerClient{}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{id: "123", cli: client}
err := (&containerReference{id: "123", cli: client}).wait()(ctx)
err := cr.wait()(ctx)
var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(2), exitErr)
assert.Equal(t, "Process completed with exit code 2.", err.Error())
statusCh <- container.WaitResponse{Error: &container.WaitExitError{Message: "container wait failed"}}
require.EqualError(t, cr.wait()(ctx), "failed to wait for container: container wait failed")
client.AssertExpectations(t)
}
@@ -370,6 +425,13 @@ func TestRemoveWithoutIDUsesName(t *testing.T) {
client.AssertExpectations(t)
}
func TestRemovePreservesClientCreationError(t *testing.T) {
t.Setenv("DOCKER_HOST", "://")
require.ErrorContains(t, (&containerReference{input: &NewContainerInput{Name: "job-1"}}).
Remove()(t.Context()), "failed to connect to docker daemon")
}
func TestFindRevalidatesStaleID(t *testing.T) {
ctx := t.Context()
inspectOpts := mobyclient.ContainerInspectOptions{}
@@ -477,7 +539,8 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
{name: "sibling prefix", directory: true, destination: "/var/run/actor/action", extractionRoot: "/", entry: "var/run/actor/action/x"},
} {
t.Run(testcase.name, func(t *testing.T) {
ctx := t.Context()
hook := logtest.NewWithLevel(logging.LevelError)
ctx := common.WithLogger(t.Context(), logging.New(hook))
client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "stale", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound.WithMessage("gone"))
@@ -501,6 +564,7 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
require.NoError(t, cr.Copy(testcase.destination, &FileEntry{Name: "x", Mode: 0o644})(ctx))
}
assert.Equal(t, "fresh", cr.id)
assert.Empty(t, hook.AllRecords())
client.AssertExpectations(t)
})
}
+1
View File
@@ -30,6 +30,7 @@ func TestSSHArgs(t *testing.T) {
}
func TestDialCommandStreamsStdioAndReportsExit(t *testing.T) {
t.Parallel()
if _, err := exec.LookPath("sh"); err != nil {
t.Skip("needs a POSIX shell")
}
+19 -15
View File
@@ -225,12 +225,16 @@ func setupPty(cmd *exec.Cmd, cmdline string) (*os.File, *os.File, error) {
return ppty, tty, nil
}
func writeKeepAlive(ppty io.Writer) {
c := 1
var err error
for c == 1 && err == nil {
c, err = ppty.Write([]byte{4})
<-time.After(time.Second)
func writeKeepAlive(ctx context.Context, ppty io.Writer) {
for ctx.Err() == nil {
if count, err := ppty.Write([]byte{4}); count != 1 || err != nil {
return
}
select {
case <-ctx.Done():
return
case <-time.After(time.Second):
}
}
}
@@ -302,7 +306,7 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
defer finishLog()
_, _ = io.Copy(writer, ppty)
}()
go writeKeepAlive(ppty)
go writeKeepAlive(logctx, ppty)
}
if err := cmd.Start(); err != nil {
return err
@@ -318,21 +322,21 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
defer k.Close()
}
err = treeKill.Wait(ctx, cmd)
if err != nil {
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
return ExitCodeError(exitErr.ExitCode())
}
return err
}
if tty != nil {
writer.AutoStop.Store(true)
if _, err := tty.WriteString("\x04"); err != nil {
common.Logger(ctx).Debug("Failed to write EOT")
}
<-logctx.Done()
select {
case <-logctx.Done():
case <-ctx.Done(): // a cancelled step does not wait on output nobody consumes
}
ppty.Close()
ppty = nil
}
if exitErr, ok := errors.AsType[*exec.ExitError](err); ok {
return ExitCodeError(exitErr.ExitCode())
}
return err
}
@@ -355,7 +359,7 @@ func (e *HostEnvironment) ExecWithCmdLine(command []string, cmdline string, env
}
func (e *HostEnvironment) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(e, srcPath, env)
return parseEnvFile(e, srcPath, env, runtime.GOOS == "windows")
}
// removeAll is a var so tests can substitute a blocking stub.
+37 -3
View File
@@ -14,6 +14,7 @@ import (
"runtime"
"strings"
"testing"
"testing/synctest"
"time"
"gitea.com/gitea/runner/act/common"
@@ -79,20 +80,48 @@ func TestGetContainerArchive(t *testing.T) {
}
func TestHostEnvironmentExecExitCode(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
ctx := t.Context()
e := &HostEnvironment{Path: t.TempDir(), StdOut: io.Discard}
for name, allocatePTY := range map[string]bool{"without PTY": false, "with PTY": true} {
t.Run(name, func(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
reader, writer := io.Pipe()
defer reader.Close()
e := &HostEnvironment{Path: t.TempDir(), StdOut: writer, AllocatePTY: allocatePTY}
t.Cleanup(func() { assert.NoError(t, e.Remove()(t.Context())) })
err := e.Exec([]string{"sh", "-c", "exit 3"}, map[string]string{"PATH": os.Getenv("PATH")}, "", "")(ctx)
result := make(chan error, 1)
go func() {
defer writer.Close()
result <- e.Exec([]string{"sh", "-c", "printf 'failed\\n'; exit 3"}, map[string]string{"PATH": os.Getenv("PATH")}, "", "")(t.Context())
}()
first := make([]byte, 1)
_, err := io.ReadFull(reader, first)
require.NoError(t, err)
synctest.Wait()
select {
case err := <-result:
t.Fatalf("Exec returned before its output was consumed: %v", err)
default:
}
remaining, err := io.ReadAll(reader)
require.NoError(t, err)
err = <-result
var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(3), exitErr)
assert.Equal(t, "Process completed with exit code 3.", err.Error())
assert.Equal(t, "failed\n", string(first)+string(remaining))
})
})
}
}
func TestHostEnvironmentAllocatePTY(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
@@ -105,6 +134,7 @@ func TestHostEnvironmentAllocatePTY(t *testing.T) {
{name: "on", allocPTY: true, expect: "TTY"},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
buf := &bytes.Buffer{}
e := &HostEnvironment{
Path: t.TempDir(),
@@ -146,6 +176,7 @@ func TestHostEnvironmentRemoveWorkdir(t *testing.T) {
}
func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
release := make(chan struct{})
stubDone := make(chan struct{})
@@ -166,11 +197,13 @@ func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) {
err := removeAllWithContext(ctx, t.TempDir())
require.ErrorIs(t, err, context.DeadlineExceeded)
})
}
func TestHostEnvironmentRemoveDoesNotHang(t *testing.T) {
for _, phase := range []string{"CleanUp callback", "path removal"} {
t.Run(phase, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
origTimeout := hostCleanupTimeout
hostCleanupTimeout = 100 * time.Millisecond
t.Cleanup(func() { hostCleanupTimeout = origTimeout })
@@ -204,6 +237,7 @@ func TestHostEnvironmentRemoveDoesNotHang(t *testing.T) {
t.Fatalf("Remove() hung on a stuck %s", phase)
}
})
})
}
}
+1 -1
View File
@@ -500,7 +500,7 @@ func (c *kubernetesContainer) GetContainerArchive(ctx context.Context, srcPath s
}
func (c *kubernetesContainer) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
return parseEnvFile(c, srcPath, env)
return parseEnvFile(c, srcPath, env, false)
}
func (c *kubernetesContainer) UpdateFromImageEnv(env *map[string]string) common.Executor {
+10 -9
View File
@@ -89,7 +89,7 @@ func execFailure(t *testing.T, code int, status string) http.HandlerFunc {
}
}
func fakeKubernetes(t *testing.T, podStatus string) (string, *fakeCluster) {
func fakeKubernetes(t *testing.T, podStatus string, env ...string) (string, *fakeCluster) {
cluster := &fakeCluster{requests: map[string][]string{}}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer test-token" {
@@ -109,7 +109,7 @@ func fakeKubernetes(t *testing.T, podStatus string) (string, *fakeCluster) {
if failure := cluster.nextExecFailure(); failure != nil {
failure(w, r)
} else {
serveLocalExec(t, w, r)
serveLocalExec(t, w, r, env)
}
case strings.HasSuffix(r.URL.Path, "/log"):
_, _ = io.WriteString(w, "log of "+r.URL.Query().Get("container"))
@@ -132,7 +132,7 @@ users: [{name: test, user: {tokenFile: token}}]
return kubeconfig, cluster
}
func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) {
func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request, env []string) {
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{Subprotocols: []string{execProtocol}})
if err != nil {
t.Error(err)
@@ -154,7 +154,7 @@ func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) {
stdin.Write(frame[1:])
}
cmd := exec.Command(command[0], command[1:]...)
cmd.Env = append(os.Environ(), "COPYFILE_DISABLE=1")
cmd.Env = append(append(os.Environ(), "COPYFILE_DISABLE=1"), env...)
var stdout, stderr bytes.Buffer
cmd.Stdin, cmd.Stdout, cmd.Stderr = &stdin, &stdout, &stderr
status := `{"status":"Success"}`
@@ -183,15 +183,17 @@ func TestKubernetesServiceHostnames(t *testing.T) {
}
func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
t.Parallel()
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
ctx := t.Context()
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{},{}]},"status":{"phase":"Running","containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}},
{"name":"my-db","ready":true,"state":{"running":{}}}]}}`)
workdir, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err)
path := workdir + ":" + os.Getenv("PATH")
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{},{}]},"status":{"phase":"Running","containerStatuses":[
{"name":"job","ready":true,"state":{"running":{}}},
{"name":"my-db","ready":true,"state":{"running":{}}}]}}`, "PATH="+path)
options := KubernetesOptions{Kubeconfig: kubeconfig, RunnerUUID: "runner-uuid", PodTemplates: []map[string]any{
{"spec": map[string]any{
"automountServiceAccountToken": true,
@@ -252,7 +254,6 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/pods"][0], `"args":["postgres","-D","$$(PGDATA)"]`)
require.NoError(t, os.WriteFile(filepath.Join(workdir, "env"), []byte("#!/bin/sh\necho \"$@\" > \"$0-args\"\nexec /usr/bin/env \"$@\"\n"), 0o755))
t.Setenv("PATH", workdir+":"+os.Getenv("PATH"))
cluster.failExec(
execFailure(t, http.StatusInternalServerError, `{"message":"error dialing backend: remote error: tls: internal error"}`),
execFailure(t, 0, `{"status":"Failure","reason":"InternalError","message":"Internal error occurred: error sending request: remote error: tls: internal error"}`),
@@ -300,7 +301,7 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
pathEnv := map[string]string{}
require.NoError(t, job.UpdateFromImageEnv(&pathEnv)(ctx))
assert.Equal(t, os.Getenv("PATH"), pathEnv["PATH"])
assert.Equal(t, path, pathEnv["PATH"])
info, err := service.Inspect(ctx)
require.NoError(t, err)
+30 -4
View File
@@ -7,6 +7,7 @@ package container
import (
"archive/tar"
"bufio"
"bytes"
"context"
"fmt"
"io"
@@ -18,7 +19,8 @@ import (
"golang.org/x/text/transform"
)
func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Executor {
// parseEnvFile reads a GITHUB_ENV style file, splitting lines like GitHub's runner on Windows or elsewhere.
func parseEnvFile(e Container, srcPath string, env *map[string]string, windows bool) common.Executor {
localEnv := *env
return func(ctx context.Context) error {
envTar, err := e.GetContainerArchive(ctx, srcPath)
@@ -38,8 +40,22 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
s := bufio.NewScanner(decoded)
// Default 64 KiB max token size is too small for realistic env-file lines; allow up to 16 MiB.
s.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
scanned := 0
s.Split(func(data []byte, atEOF bool) (int, []byte, error) {
if index := bytes.IndexByte(data[scanned:], '\n'); index >= 0 {
advance := scanned + index + 1
scanned = 0
return advance, data[:advance], nil
}
scanned = len(data)
if atEOF && len(data) > 0 {
scanned = 0
return len(data), data, nil
}
return 0, nil, nil
})
for s.Scan() {
line := s.Text()
line := trimEnvLineEnding(s.Text(), windows)
// GitHub's runner ignores blank lines
if strings.TrimSpace(line) == "" {
continue
@@ -52,10 +68,13 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
case multiLineEnv != -1:
var multiLineEnvContent []string
multiLineEnvDelimiter := line[multiLineEnv+2:]
if multiLineEnv == 0 || multiLineEnvDelimiter == "" {
return fmt.Errorf("invalid format '%v', name and delimiter must not be empty", line)
}
delimiterFound := false
for s.Scan() {
content := s.Text()
if content == multiLineEnvDelimiter {
if trimEnvLineEnding(content, windows) == multiLineEnvDelimiter {
delimiterFound = true
break
}
@@ -67,7 +86,7 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
if !delimiterFound {
return fmt.Errorf("invalid format delimiter '%v' not found before end of file", multiLineEnvDelimiter)
}
localEnv[line[:multiLineEnv]] = strings.Join(multiLineEnvContent, "\n")
localEnv[line[:multiLineEnv]] = trimEnvLineEnding(strings.Join(multiLineEnvContent, ""), windows)
default:
return fmt.Errorf("invalid format '%v', expected a line with '=' or '<<'", line)
}
@@ -78,3 +97,10 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
return nil
}
}
func trimEnvLineEnding(line string, windows bool) string {
if windows && strings.HasSuffix(line, "\r\n") {
return line[:len(line)-2]
}
return strings.TrimSuffix(line, "\n")
}
+32 -7
View File
@@ -22,6 +22,7 @@ func newTestHostEnv(t *testing.T) (*HostEnvironment, string) {
}
func TestParseEnvFile(t *testing.T) {
t.Parallel()
largeValue := strings.Repeat("x", 2*1024*1024)
encodeUTF16 := func(endianness unicode.Endianness) string {
content, err := unicode.UTF16(endianness, unicode.UseBOM).NewEncoder().Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n"))
@@ -33,41 +34,65 @@ func TestParseEnvFile(t *testing.T) {
want map[string]string
}{
{"single line", "FOO=bar\nBAZ=qux\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"single line CRLF and final bare CR", "FOO=bar\r\nBAZ=qux\r", map[string]string{"FOO": "bar\r", "BAZ": "qux\r"}},
{"single line without final newline", "FOO=bar", map[string]string{"FOO": "bar"}},
{"blank lines", "\nFOO=bar\n\n \nBAZ=qux\n\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"multiline", "FOO<<EOF\nline1\nline2\nEOF\n", map[string]string{"FOO": "line1\nline2"}},
{"multiline without final newline", "FOO<<EOF\nline1\r\nline2\r\nEOF", map[string]string{"FOO": "line1\r\nline2\r"}},
{"empty multiline without final newline", "FOO<<EOF\nEOF", map[string]string{"FOO": ""}},
{"multiline blank lines", "FOO<<EOF\nline1\n\nline2\nEOF\n", map[string]string{"FOO": "line1\n\nline2"}},
{"multiline leading blank line", "FOO<<EOF\n\nline2\nEOF\n", map[string]string{"FOO": "\nline2"}},
{"multiline CRLF", "FOO<<EOF\r\nline1\r\nline2\r\nEOF\r\n", map[string]string{"FOO": "line1\r\nline2\r"}},
{"multiline mixed line endings", "FOO<<EOF\nline1\r\nline2\n\r\nEOF\n", map[string]string{"FOO": "line1\r\nline2\n\r"}},
{"large value within limit", "FOO<<EOF\n" + largeValue + "\nEOF\n", map[string]string{"FOO": largeValue}},
{"UTF8 BOM", "\xef\xbb\xbfFOO=bar\n", map[string]string{"FOO": "bar"}},
{"UTF16 little endian", encodeUTF16(unicode.LittleEndian), map[string]string{"FOO": "bar", "MULTI": "line1"}},
{"UTF16 big endian", encodeUTF16(unicode.BigEndian), map[string]string{"FOO": "bar", "MULTI": "line1"}},
{"UTF16 little endian", encodeUTF16(unicode.LittleEndian), map[string]string{"FOO": "bar\r", "MULTI": "line1\r"}},
{"UTF16 big endian", encodeUTF16(unicode.BigEndian), map[string]string{"FOO": "bar\r", "MULTI": "line1\r"}},
} {
t.Run(test.name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte(test.content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
require.NoError(t, parseEnvFile(e, envPath, &env, false)(t.Context()))
assert.Equal(t, test.want, env)
})
}
t.Run("Windows line endings", func(t *testing.T) {
hostEnv, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nline2\nEOF\r\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(hostEnv, envPath, &env, true)(t.Context()))
assert.Equal(t, map[string]string{"FOO": "bar", "MULTI": "line1\r\nline2"}, env)
})
}
func TestParseEnvFileLineExceedsBufferReportsScannerError(t *testing.T) {
t.Parallel()
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n"+strings.Repeat("x", 17*1024*1024)+"\nEOF\n"), 0o600))
env := map[string]string{}
err := parseEnvFile(e, envPath, &env)(t.Context())
err := parseEnvFile(e, envPath, &env, false)(t.Context())
require.ErrorIs(t, err, bufio.ErrTooLong)
assert.Contains(t, err.Error(), "reading env file")
}
func TestParseEnvFileMissingDelimiter(t *testing.T) {
func TestParseEnvFileInvalidHeredoc(t *testing.T) {
for name, content := range map[string]string{
"missing delimiter": "FOO<<EOF\nline1\nline2\n",
"empty name": "<<EOF\nline1\nEOF\n",
"empty delimiter": "FOO<<\nline1\n\n",
"delimiter with carriage return": "FOO<<EOF\nline1\nEOF\r\n",
} {
t.Run(name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\nline1\nline2\n"), 0o600))
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
env := map[string]string{}
err := parseEnvFile(e, envPath, &env)(t.Context())
err := parseEnvFile(e, envPath, &env, false)(t.Context())
require.Error(t, err)
assert.Contains(t, err.Error(), "delimiter")
assert.Empty(t, env)
})
}
}
+23 -6
View File
@@ -28,12 +28,12 @@ var (
type pinnedKey struct{}
// WithPinnedCheckout keeps a run's sha and ref fixed when HEAD moves.
func WithPinnedCheckout(ctx context.Context) context.Context {
// WithPinnedCheckout keeps a run's sha, ref and repository fixed when HEAD moves, in pins unless ctx has some.
func WithPinnedCheckout(ctx context.Context, pins *sync.Map) context.Context {
if ctx.Value(pinnedKey{}) != nil {
return ctx
}
return context.WithValue(ctx, pinnedKey{}, &sync.Map{})
return context.WithValue(ctx, pinnedKey{}, pins)
}
func pinned(ctx context.Context, key string, lookup func() (string, error)) (string, error) {
@@ -57,12 +57,29 @@ func SetRef(ctx context.Context, ghc *model.GithubContext, repoPath string) {
case "pull_request_target":
ghc.Ref = "refs/heads/" + ghc.BaseRef
case "pull_request", "pull_request_review", "pull_request_review_comment":
ghc.Ref = fmt.Sprintf("refs/pull/%.0f/merge", ghc.Event["number"])
number := ghc.Event["number"]
if number == nil {
number = model.NestedMapLookup(ghc.Event, "pull_request", "number")
}
ghc.Ref = fmt.Sprintf("refs/pull/%.0f/merge", number)
if ghc.EventName == "pull_request" && ghc.Event["action"] == "closed" && model.NestedMapLookup(ghc.Event, "pull_request", "merged") == true {
ghc.Ref = "refs/heads/" + ghc.BaseRef
}
case "deployment", "deployment_status":
ghc.Ref = model.AsString(model.NestedMapLookup(ghc.Event, "deployment", "ref"))
case "release":
ghc.Ref = "refs/tags/" + model.AsString(model.NestedMapLookup(ghc.Event, "release", "tag_name"))
case "push", "create", "workflow_dispatch":
case "create":
ghc.Ref = model.AsString(ghc.Event["ref"])
if ghc.Ref != "" && !strings.HasPrefix(ghc.Ref, "refs/") {
switch model.AsString(ghc.Event["ref_type"]) {
case "branch":
ghc.Ref = "refs/heads/" + ghc.Ref
case "tag":
ghc.Ref = "refs/tags/" + ghc.Ref
}
}
case "push", "workflow_dispatch":
ghc.Ref = model.AsString(ghc.Event["ref"])
default:
defaultBranch := model.AsString(model.NestedMapLookup(ghc.Event, "repository", "default_branch"))
@@ -131,7 +148,7 @@ func SetSha(ctx context.Context, ghc *model.GithubContext, repoPath string) {
// remote in repoPath when it is not set yet, and derives its owner.
func SetRepositoryAndOwner(ctx context.Context, ghc *model.GithubContext, repoPath string) {
if ghc.Repository == "" {
repo, err := findGithubRepo(ctx, repoPath)
repo, err := pinned(ctx, "repo:"+repoPath, func() (string, error) { return findGithubRepo(ctx, repoPath) })
if err != nil {
common.Logger(ctx).Warnf("unable to get git repo (repoPath: %v): %v", repoPath, err)
return
+50 -2
View File
@@ -7,6 +7,7 @@ package ghcontext
import (
"context"
"errors"
"sync"
"testing"
"gitea.dev/actionslib/pkg/model"
@@ -41,6 +42,30 @@ func TestSetRef(t *testing.T) {
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request_review",
event: map[string]any{"pull_request": map[string]any{"number": 1234.}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request",
event: map[string]any{"number": 1234., "action": "closed", "pull_request": map[string]any{"merged": true}},
ref: "refs/heads/master",
refName: "master",
},
{
eventName: "pull_request",
event: map[string]any{"number": 1234., "action": "closed", "pull_request": map[string]any{"merged": false}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "pull_request_review_comment",
event: map[string]any{"pull_request": map[string]any{"number": 1234.}},
ref: "refs/pull/1234/merge",
refName: "1234/merge",
},
{
eventName: "deployment",
event: map[string]any{
@@ -69,6 +94,24 @@ func TestSetRef(t *testing.T) {
ref: "refs/heads/somebranch",
refName: "somebranch",
},
{
eventName: "create",
event: map[string]any{"ref": "feature/new", "ref_type": "branch"},
ref: "refs/heads/feature/new",
refName: "feature/new",
},
{
eventName: "create",
event: map[string]any{"ref": "v1", "ref_type": "tag"},
ref: "refs/tags/v1",
refName: "v1",
},
{
eventName: "create",
event: map[string]any{"ref": "refs/tags/v1", "ref_type": "tag"},
ref: "refs/tags/v1",
refName: "v1",
},
{
eventName: "unknown",
event: map[string]any{
@@ -122,8 +165,10 @@ func TestSetRef(t *testing.T) {
func TestSetSha(t *testing.T) {
oldFindGitRef := findGitRef
oldFindGitRevision := findGitRevision
oldFindGithubRepo := findGithubRepo
defer func() { findGitRef = oldFindGitRef }()
defer func() { findGitRevision = oldFindGitRevision }()
defer func() { findGithubRepo = oldFindGithubRepo }()
findGitRevision = func(_ctx context.Context, _file string) (string, error) {
return "1234fakesha", nil
@@ -189,16 +234,19 @@ func TestSetSha(t *testing.T) {
})
}
t.Run("pinned checkout keeps the first sha and ref", func(t *testing.T) {
ctx := WithPinnedCheckout(t.Context())
t.Run("pinned checkout keeps the first sha, ref and repository", func(t *testing.T) {
ctx := WithPinnedCheckout(t.Context(), &sync.Map{})
for _, head := range []string{"first", "moved"} {
findGitRevision = func(context.Context, string) (string, error) { return head, nil }
findGitRef = func(context.Context, string) (string, error) { return "refs/heads/" + head, nil }
findGithubRepo = func(context.Context, string) (string, error) { return "owner/" + head, nil }
ghc := &model.GithubContext{Event: map[string]any{}}
SetSha(ctx, ghc, "/some/dir")
SetRef(ctx, ghc, "/some/dir")
SetRepositoryAndOwner(ctx, ghc, "/some/dir")
assert.Equal(t, "first", ghc.Sha)
assert.Equal(t, "refs/heads/first", ghc.Ref)
assert.Equal(t, "owner/first", ghc.Repository)
}
})
}
+95 -17
View File
@@ -19,6 +19,7 @@ import (
"path/filepath"
"regexp"
"runtime"
"slices"
"strings"
"gitea.com/gitea/runner/act/common"
@@ -74,18 +75,24 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
reader, closer, err = readFile("action.yaml")
addError("action.yaml", err)
if os.IsNotExist(err) {
_, closer, err := readFile("Dockerfile")
addError("Dockerfile", err)
dockerfile := "Dockerfile"
_, closer, err := readFile(dockerfile)
addError(dockerfile, err)
if os.IsNotExist(err) {
dockerfile = "dockerfile"
_, closer, err = readFile(dockerfile)
addError(dockerfile, err)
}
if err == nil {
closer.Close()
action := &model.Action{
Name: "(Synthetic)",
Runs: model.ActionRuns{
Using: "docker",
Image: "Dockerfile",
Image: dockerfile,
},
}
logger.Debugf("Using synthetic action %v for Dockerfile", action)
logger.Debugf("Using synthetic action %v for %s", action, dockerfile)
return action, nil
}
if val, ok := step.With["args"]; ok {
@@ -95,6 +102,7 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
action := &model.Action{
Name: "(Synthetic)",
Inputs: map[string]model.Input{
"args": {},
"cwd": {
Description: "(Actual working directory)",
Required: false,
@@ -146,12 +154,17 @@ func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actio
}
defer unlock()
if !rc.Config.NoActionPatch {
// Other jobs share the directory, so patch under the copy's lock.
defer patchActions(ctx, actionScriptPaths(actionDir, filepath.Join(actionDir, actionPath), step.getActionModel()))()
if err := rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", false, true)(ctx); err != nil { // an exported commit has no .git to apply gitignore with
return err
}
return rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", false, true)(ctx) // an exported commit has no .git to apply gitignore with
if rc.Config.NoActionPatch {
return nil
}
patched := patchedScripts(ctx, actionDir, actionScriptPaths(actionDir, filepath.Join(actionDir, actionPath), step.getActionModel()))
if len(patched) == 0 {
return nil
}
return rc.JobContainer.Copy(containerActionDirCopy, patched...)(ctx)
}
func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor {
@@ -172,6 +185,8 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
if err := populateEnvsFromInput(ctx, step.getEnv(), action, rc); err != nil {
return err
}
warnUnexpectedInputs(ctx, action, stepModel.With)
printRunActionHeader(ctx, stepModel, action, *step.getEnv(), rc)
actionName, containerActionDir := getContainerActionPaths(step)
@@ -301,10 +316,13 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
return err
}
correctArchExists, err := ContainerImageExistsLocally(ctx, image, rc.Config.ContainerArchitecture)
correctArchExists := anyArchExists
if anyArchExists && rc.Config.ContainerArchitecture != "" && rc.Config.ContainerArchitecture != "any" {
correctArchExists, err = ContainerImageExistsLocally(ctx, image, rc.Config.ContainerArchitecture)
if err != nil {
return err
}
}
if anyArchExists && !correctArchExists {
wasRemoved, err := container.RemoveImage(ctx, image, true, true)
@@ -350,14 +368,11 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
logger.Debugf("image '%s' for architecture '%s' already exists", image, rc.Config.ContainerArchitecture)
}
}
cmd, err := shellquote.Split(step.getStepModel().With["args"])
if err != nil {
return err
}
ee, err := evalDockerEnv(ctx, step, action)
if err != nil {
return err
}
var cmd []string
if action.Runs.Args != nil {
// a fresh slice, the manifest is evaluated again for every stage
cmd = make([]string, len(action.Runs.Args))
@@ -366,6 +381,11 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
return fmt.Errorf("unable to interpolate runs.args: %w", err)
}
}
} else {
cmd, err = shellquote.Split(stepInput(step.getStepModel(), "args"))
if err != nil {
return err
}
}
entrypoint, err := dockerEntrypoint(step, stage)
if err != nil {
@@ -395,7 +415,7 @@ func dockerEntrypoint(step actionStep, stage stepStage) ([]string, error) {
default:
entrypoint = runs.Entrypoint
if entrypoint == "" {
if fields := strings.Fields(step.getStepModel().With["entrypoint"]); len(fields) > 0 {
if fields := strings.Fields(stepInput(step.getStepModel(), "entrypoint")); len(fields) > 0 {
return fields, nil
}
}
@@ -413,9 +433,15 @@ func evalDockerEnv(ctx context.Context, step step, action *model.Action) (*expre
var err error
inputs := make(map[string]string)
providedInputs := make(map[string]bool, len(step.getStepModel().With))
for key := range step.getStepModel().With {
providedInputs[strings.ToLower(key)] = true
}
eval := rc.NewExpressionEvaluator(ctx)
// Set Defaults
for k, input := range action.Inputs {
if providedInputs[strings.ToLower(k)] {
continue
}
if inputs[k], err = eval.Interpolate(ctx, input.Default); err != nil {
return nil, fmt.Errorf("unable to interpolate the default of input %s: %w", k, err)
}
@@ -479,10 +505,14 @@ func populateEnvsFromSavedState(env *map[string]string, step actionStep, rc *Run
}
}
func inputEnvKey(inputID string) string {
return "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_")
}
func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *model.Action, rc *RunContext) error {
eval := rc.NewExpressionEvaluator(ctx)
for inputID, input := range action.Inputs {
envKey := "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_")
envKey := inputEnvKey(inputID)
if _, ok := (*env)[envKey]; !ok {
var err error
if (*env)[envKey], err = eval.Interpolate(ctx, input.Default); err != nil {
@@ -493,6 +523,47 @@ func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *
return nil
}
// lookupFold finds name in values, ignoring case where the environment does, as GitHub's inputs always do.
func lookupFold[V any](values map[string]V, name string, caseInsensitive bool) (V, bool) {
if value, ok := values[name]; ok || !caseInsensitive {
return value, ok
}
for key, value := range values {
if strings.EqualFold(key, name) {
return value, true
}
}
var zero V
return zero, false
}
func stepInput(step *model.Step, name string) string {
value, _ := lookupFold(step.With, name, true)
return value
}
func warnUnexpectedInputs(ctx context.Context, action *model.Action, inputs map[string]string) {
docker := action.Runs.Using.IsDocker()
var unexpectedInputs []string
for input := range inputs {
_, declared := lookupFold(action.Inputs, input, true)
containerInput := docker && (strings.EqualFold(input, "entryPoint") || strings.EqualFold(input, "args"))
if !declared && !containerInput {
unexpectedInputs = append(unexpectedInputs, input)
}
}
if len(unexpectedInputs) == 0 {
return
}
validInputs := slices.Sorted(maps.Keys(action.Inputs))
if docker {
validInputs = append([]string{"entryPoint", "args"}, validInputs...)
}
slices.Sort(unexpectedInputs)
common.Logger(ctx).With(rawOutputField, true).Warnf("##[warning]%s", EscapeCommandData(fmt.Sprintf(
"Unexpected input(s) '%s', valid inputs are ['%s']", strings.Join(unexpectedInputs, "', '"), strings.Join(validInputs, "', '"))))
}
func getContainerActionPaths(step actionStep) (string, string) {
rc := step.getRunContext()
if sar, ok := step.(*stepActionRemote); ok { // a copy per action, as its repository's directory is shared
@@ -585,6 +656,7 @@ func runPreStep(step actionStep) common.Executor {
return err
}
}
printRunActionHeader(ctx, step.getStepModel(), action, *step.getEnv(), rc)
switch {
case x.IsNode():
if err := maybeCopyToActionDir(ctx, step, actionDir, actionPath, containerActionDir); err != nil {
@@ -633,6 +705,9 @@ func runPreStep(step actionStep) common.Executor {
func shouldRunPostStep(step actionStep) common.Conditional {
return func(ctx context.Context) bool {
if remote, ok := step.(*stepActionRemote); ok && remote.preStepStarted && !remote.action.Runs.Using.IsComposite() {
return true
}
log := common.Logger(ctx)
stepResults := step.getRunContext().StepResults
stepResult := stepResults[step.getStepModel().ID]
@@ -680,6 +755,9 @@ func runPostStep(step actionStep) common.Executor {
actionDir, actionPath, _, containerActionDir := actionStagePaths(step)
x := action.Runs.Using
if !x.IsComposite() { // a composite's children print their own
logger.With(rawOutputField, true).Info("Post job cleanup.")
}
switch {
case x.IsNode():
+6 -1
View File
@@ -31,7 +31,7 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
ee := parent.NewStepExpressionEvaluator(ctx, step)
for inputID, input := range step.getActionModel().Inputs {
envKey := "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_")
envKey := inputEnvKey(inputID)
// lookup if key is defined in the step but the already
// evaluated value from the environment
@@ -52,6 +52,11 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
}
}
}
for envKey, value := range (&model.Step{With: step.getStepModel().With}).GetEnv() {
if _, ok := env[envKey]; !ok {
env[envKey] = value
}
}
gh := step.getGithubContext(ctx)
env["GITHUB_ACTION_REPOSITORY"] = gh.ActionRepository
env["GITHUB_ACTION_REF"] = gh.ActionRef
+18 -9
View File
@@ -7,7 +7,10 @@ import (
"context"
"testing"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
@@ -17,10 +20,11 @@ import (
func TestCompositeActionParity(t *testing.T) {
t.Run("inherits contexts without leaking inputs", func(t *testing.T) {
ctx := t.Context()
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
strategy := &model.Strategy{MaxParallelString: "3", RawMatrix: yaml.Node{Kind: yaml.MappingNode}}
parent := &RunContext{
Config: &Config{},
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Matrix: map[string]any{"os": "linux"},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Name: "workflow", Jobs: map[string]*model.Job{"job": {Strategy: strategy}}}},
JobContainer: &jobContainerMock{},
@@ -28,21 +32,26 @@ func TestCompositeActionParity(t *testing.T) {
jobIndex: 1,
jobTotal: 4,
}
composite, err := newCompositeRunContext(ctx, parent, &stepActionRemote{
Step: &model.Step{With: map[string]string{"SHARED": "outer"}},
step := &stepActionLocal{
Step: &model.Step{Uses: "./action", With: map[string]string{"SHARED": "outer", "extra": "literal${{ github.job }}"}},
RunContext: parent,
action: &model.Action{Inputs: map[string]model.Input{"shared": {Default: "outer-default"}}},
action: &model.Action{Inputs: map[string]model.Input{"shared": {Default: "outer-default"}}, Runs: model.ActionRuns{Using: "composite"}},
env: map[string]string{"INPUT_SHARED": "outer"},
}, "/action")
}
require.NoError(t, runActionImpl(step, "/action", nil)(ctx))
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, "##[warning]Unexpected input(s) 'extra', valid inputs are ['shared']", hook.LastRecord().Message)
composite, err := step.getCompositeRunContext(ctx)
require.NoError(t, err)
assert.Same(t, strategy, composite.Run.Job().Strategy)
assert.True(t, composite.IsHostEnv())
interpolated, err := composite.NewExpressionEvaluator(ctx).Interpolate(ctx,
"${{ matrix.os }}|${{ strategy.max-parallel }}|${{ strategy.job-index }}|${{ inputs.shared }}")
"${{ matrix.os }}|${{ strategy.max-parallel }}|${{ strategy.job-index }}|${{ inputs.shared }}|${{ inputs.extra }}")
require.NoError(t, err)
assert.Equal(t, "linux|3|1|outer", interpolated)
assert.Equal(t, "linux|3|1|outer|literal${{ github.job }}", interpolated)
assert.NotContains(t, composite.Env, "INPUT_SHARED")
assert.NotContains(t, composite.Env, "INPUT_EXTRA")
nestedEnv := composite.GetEnv()
require.NoError(t, populateEnvsFromInput(ctx, &nestedEnv, &model.Action{Inputs: map[string]model.Input{"shared": {Default: "inner-default"}}}, composite))
@@ -54,7 +63,7 @@ func TestCompositeActionParity(t *testing.T) {
return "", "", assert.AnError
})
rc := &RunContext{
Config: &Config{GitHubInstance: "github.com", ActionCacheDir: t.TempDir()},
Config: &Config{GitHubInstance: "github.com", ActionCacheDir: t.TempDir(), PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{},
}
+164 -13
View File
@@ -8,6 +8,7 @@ import (
"context"
"io"
"io/fs"
"net/http"
"os"
"os/exec"
"path/filepath"
@@ -19,6 +20,8 @@ import (
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model"
@@ -85,6 +88,19 @@ runs:
},
},
},
{
name: "readLowercaseDockerfile",
step: &model.Step{},
filename: "dockerfile",
fileContent: "FROM ubuntu:20.04",
expected: &model.Action{
Name: "(Synthetic)",
Runs: model.ActionRuns{
Using: "docker",
Image: "dockerfile",
},
},
},
{
name: "readWithArgs",
step: &model.Step{
@@ -95,6 +111,7 @@ runs:
expected: &model.Action{
Name: "(Synthetic)",
Inputs: map[string]model.Input{
"args": {},
"cwd": {
Description: "(Actual working directory)",
Required: false,
@@ -147,14 +164,33 @@ func TestActionRunner(t *testing.T) {
for _, tt := range []struct {
name string
inputs map[string]model.Input
with map[string]string
state map[string]string
expectedEnv map[string]string
warning string
}{
{
name: "with-input",
inputs: map[string]model.Input{"key": {Default: "default value"}},
expectedEnv: map[string]string{"INPUT_KEY": "default value"},
},
{
name: "declared input ignores case",
inputs: map[string]model.Input{"key": {Default: "default value"}},
with: map[string]string{"KEY": "provided value"},
expectedEnv: map[string]string{"INPUT_KEY": "provided value"},
},
{
name: "unexpected inputs warn once across stages",
inputs: map[string]model.Input{"key": {}},
with: map[string]string{"KEY": "provided value", "UNKNOWN": "value", "other%0A\r\n": "value"},
warning: "##[warning]Unexpected input(s) 'UNKNOWN', 'other%250A%0D%0A', valid inputs are ['key']",
},
{
name: "no declared inputs",
with: map[string]string{"unknown": "value"},
warning: "##[warning]Unexpected input(s) 'unknown', valid inputs are ['']",
},
{
name: "restore-saved-state",
state: map[string]string{"name": "state value"},
@@ -162,6 +198,8 @@ func TestActionRunner(t *testing.T) {
},
} {
t.Run(tt.name, func(t *testing.T) {
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
cm := &containerMock{}
containerActionDir := "/var/run/act/actions/" + model.UsesHash("/org/repo/path@ref") + "/"
cm.On("CopyDir", containerActionDir, "dir/", false, true).Return(func(_ctx context.Context) error { return nil })
@@ -175,21 +213,36 @@ func TestActionRunner(t *testing.T) {
return true
})
cm.On("Exec", []string{"node", "--preserve-symlinks-main", containerActionDir + "path"}, envMatcher, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", nodeActionCommand(containerActionDir+"path/pre.js"), mock.Anything, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", nodeActionCommand(containerActionDir+"path/post.js"), envMatcher, "", "").Return(func(_ctx context.Context) error { return nil })
remoteAction := mustNewRemoteAction(t, "org/repo/path@ref")
step := &stepActionRemote{
Step: &model.Step{ID: "step", Uses: "org/repo/path@ref"},
Step: &model.Step{ID: "step", Uses: "org/repo/path@ref", With: tt.with},
RunContext: &RunContext{
Config: &Config{},
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
IntraActionState: map[string]map[string]string{"step": tt.state},
JobContainer: cm,
},
action: &model.Action{Inputs: tt.inputs, Runs: model.ActionRuns{Using: "node16"}},
env: map[string]string{},
action: &model.Action{Inputs: tt.inputs, Runs: model.ActionRuns{Using: "node16", Pre: "pre.js", Post: "post.js"}},
actionDir: "dir",
env: (&model.Step{With: tt.with}).GetEnv(),
remoteAction: remoteAction,
}
require.NoError(t, runActionImpl(step, "dir", remoteAction)(t.Context()))
require.NoError(t, runPreStep(step)(ctx))
assert.Empty(t, hook.AllRecords())
require.NoError(t, runActionImpl(step, "dir", remoteAction)(ctx))
require.NoError(t, runPostStep(step)(ctx))
if tt.warning == "" {
assert.Empty(t, hook.AllRecords())
} else {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, tt.warning, hook.LastRecord().Message)
rawOutput, ok := hook.LastRecord().Attr(rawOutputField)
require.True(t, ok)
assert.Equal(t, true, rawOutput)
}
cm.AssertExpectations(t)
})
}
@@ -376,7 +429,7 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "remote/action@v1", With: map[string]string{}},
RunContext: &RunContext{
Config: &Config{},
Config: &Config{Env: testCheckoutEnv()},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
@@ -414,7 +467,68 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
cancel()
releaseOnce()
<-done
assert.ErrorIs(t, <-done, context.Canceled)
}
func TestExecAsDockerImageChecks(t *testing.T) {
originalExists, originalBuild, originalContainer := ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer
t.Cleanup(func() {
ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer = originalExists, originalBuild, originalContainer
})
ContainerNewContainer = func(*container.NewContainerInput) container.ExecutionsEnvironment { return fakeContainer{} }
for _, testcase := range []struct {
name, platform string
exists, matches, rebuild bool
wantBuild, wantRemove bool
wantChecks []string
}{
{name: "missing image", platform: "linux/arm64", wantBuild: true, wantChecks: []string{"any"}},
{name: "unspecified platform", exists: true, matches: true, wantChecks: []string{"any"}},
{name: "any platform", platform: "any", exists: true, matches: true, wantChecks: []string{"any"}},
{name: "matching platform", platform: "linux/arm64", exists: true, matches: true, wantChecks: []string{"any", "linux/arm64"}},
{name: "different platform", platform: "linux/arm64", exists: true, wantBuild: true, wantRemove: true, wantChecks: []string{"any", "linux/arm64"}},
{name: "forced rebuild", exists: true, matches: true, rebuild: true, wantBuild: true, wantChecks: []string{"any"}},
} {
t.Run(testcase.name, func(t *testing.T) {
var checks []string
ContainerImageExistsLocally = func(_ctx context.Context, _image, platform string) (bool, error) {
checks = append(checks, platform)
return testcase.exists && (platform == "any" || testcase.matches), nil
}
var built, removed bool
ContainerNewDockerBuildExecutor = func(container.NewDockerBuildExecutorInput) common.Executor {
return func(context.Context) error { built = true; return nil }
}
fakeDockerDaemon(t, func(writer http.ResponseWriter, request *http.Request) {
if request.Method == http.MethodDelete {
removed = true
_, _ = io.WriteString(writer, `[]`)
} else {
_, _ = io.WriteString(writer, `{"Architecture":"aarch64"}`)
}
})
step := &stepActionRemote{
Step: &model.Step{ID: "action"},
RunContext: &RunContext{
Config: &Config{
PresetGitHubContext: &model.GithubContext{},
ContainerArchitecture: testcase.platform,
ForceRebuild: testcase.rebuild,
},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{},
},
action: &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "Dockerfile"}},
env: map[string]string{},
}
dir := t.TempDir()
require.NoError(t, execAsDocker(t.Context(), step, "action", dir, dir, false, stepStageMain))
assert.Equal(t, testcase.wantChecks, checks)
assert.Equal(t, testcase.wantBuild, built)
assert.Equal(t, testcase.wantRemove, removed)
})
}
}
func TestDockerActionImageTag(t *testing.T) {
@@ -468,15 +582,20 @@ func TestDockerActionImageTag(t *testing.T) {
func TestExecAsDockerStageEntrypoint(t *testing.T) {
orig := ContainerNewContainer
defer func() { ContainerNewContainer = orig }()
fakeDockerDaemon(t, func(writer http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(writer, `{"Architecture":"x86_64"}`)
})
for _, tc := range []struct {
name string
stage stepStage
with map[string]string
inputs map[string]model.Input
runs model.ActionRuns
env map[string]string
wantCmd []string
wantEntrypoint []string
warning string
}{
{
name: "main stage prefers manifest values",
@@ -491,17 +610,36 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
wantEntrypoint: []string{"main.sh"},
},
{
name: "main stage uses caller fallbacks",
name: "main stage uses caller fallbacks in any case and warns about undeclared inputs",
stage: stepStageMain,
with: map[string]string{"args": "caller --flag", "entrypoint": "input.sh --verbose"},
with: map[string]string{"ARGS": "caller --flag", "entryPoint": "input.sh --verbose", "unknown": "value"},
inputs: map[string]model.Input{"message": {}},
runs: model.ActionRuns{Env: map[string]string{"ACTION_ONLY": "manifest"}},
wantCmd: []string{"caller", "--flag"},
wantEntrypoint: []string{"input.sh", "--verbose"},
warning: "##[warning]Unexpected input(s) 'unknown', valid inputs are ['entryPoint', 'args', 'message']",
},
{
name: "manifest args keep caller input quotes literal",
stage: stepStageMain,
with: map[string]string{"args": "it's literal"},
runs: model.ActionRuns{Args: []string{"${{ inputs.args }}"}},
env: map[string]string{"INPUT_ARGS": "it's literal"},
wantCmd: []string{"it's literal"},
},
{
name: "caller input skips its default expression",
stage: stepStageMain,
with: map[string]string{"MESSAGE": "provided"},
inputs: map[string]model.Input{"message": {Default: "${{ fromJSON('') }}"}},
runs: model.ActionRuns{Args: []string{"${{ inputs.message }}"}},
env: map[string]string{"INPUT_MESSAGE": "provided"},
wantCmd: []string{"provided"},
},
{
name: "explicit empty manifest args suppress caller args",
stage: stepStageMain,
with: map[string]string{"args": "caller"},
with: map[string]string{"args": "it's ignored"},
runs: model.ActionRuns{Args: []string{}},
wantCmd: []string{},
},
@@ -527,6 +665,8 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
},
} {
t.Run(tc.name, func(t *testing.T) {
hook := logtest.NewWithLevel(logging.LevelWarn)
ctx := common.WithLogger(t.Context(), logging.New(hook))
tc.runs.Using, tc.runs.Image = "docker", "docker://node:14"
cm := &containerMock{}
var input *container.NewContainerInput
@@ -538,12 +678,13 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
step := &stepActionRemote{
Step: &model.Step{ID: "1", Uses: "org/action@v1", With: tc.with},
RunContext: &RunContext{
Config: &Config{},
Config: &Config{PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: cm,
},
action: &model.Action{Runs: tc.runs},
action: &model.Action{Inputs: tc.inputs, Runs: tc.runs},
env: mergeMaps(tc.env),
remoteAction: mustNewRemoteAction(t, "org/action@v1"),
}
cm.On("Pull", false).Return(func(context.Context) error { return nil })
@@ -552,7 +693,17 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
cm.On("Start", true).Return(func(context.Context) error { return nil })
cm.On("Close").Return(func(context.Context) error { return nil })
require.NoError(t, execAsDocker(t.Context(), step, "action", t.TempDir(), t.TempDir(), false, tc.stage))
if tc.stage == stepStageMain {
require.NoError(t, runActionImpl(step, t.TempDir(), step.remoteAction)(ctx))
} else {
require.NoError(t, execAsDocker(ctx, step, "action", t.TempDir(), t.TempDir(), false, tc.stage))
}
if tc.warning == "" {
assert.Empty(t, hook.AllRecords())
} else {
require.Len(t, hook.AllRecords(), 1)
assert.Equal(t, tc.warning, hook.LastRecord().Message)
}
require.NotNil(t, input)
assert.Equal(t, tc.wantCmd, input.Cmd)
assert.Equal(t, tc.wantEntrypoint, input.Entrypoint)
+4
View File
@@ -101,9 +101,13 @@ func TestMainStepsExecutorRunsAlwaysStepsAfterInterruption(t *testing.T) {
assert.Equal(t, []string{"step1", "always-step"}, ran)
require.NoError(t, laterStepCtxErr)
assert.Equal(t, wantError == context.Canceled, rc.jobCancelled)
setJobResult(t.Context(), rc, rc, true)
if wantError == context.DeadlineExceeded {
assert.True(t, rc.jobFailed)
assert.Equal(t, "failure", rc.getJobContext().Status)
assert.Equal(t, "failure", rc.Run.Job().Result)
} else {
assert.Equal(t, "cancelled", rc.Run.Job().Result)
}
})
}
+24 -3
View File
@@ -8,6 +8,8 @@ import (
"cmp"
"context"
"fmt"
"maps"
"os"
"strings"
"unicode"
@@ -158,11 +160,18 @@ const allowUnsecureCommandsVar = "ACTIONS_ALLOW_UNSECURE_COMMANDS"
// not run, recording the error that fails the step. GitHub disabled both because a step that
// echoes untrusted content can use them to set NODE_OPTIONS or PATH for every later step.
func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string) bool {
if isTruthyEnv(rc.currentStepEnv()[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.Env[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.GlobalEnv[allowUnsecureCommandsVar]) {
if isTruthyEnv(os.Getenv(allowUnsecureCommandsVar)) { // the runner's own opt-in wins, as on GitHub
return false
}
caseInsensitive := rc.JobContainer != nil && rc.JobContainer.IsEnvironmentCaseInsensitive()
for _, env := range []map[string]string{rc.currentStepEnv(), rc.Env, rc.GlobalEnv} {
if value, ok := lookupFold(env, allowUnsecureCommandsVar, caseInsensitive); ok {
if isTruthyEnv(value) {
return false
}
break
}
}
// The step executor logs the failure itself, so keep this line's wording distinct.
common.Logger(ctx).With(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(fmt.Sprintf(
@@ -192,6 +201,18 @@ func (rc *RunContext) takeUnsecureCommandError() error {
return err
}
// runWithCommands runs executor judging its workflow commands by env, failing it on a refused one.
func (rc *RunContext) runWithCommands(ctx context.Context, env map[string]string, executor common.Executor) error {
rc.setCurrentStepEnv(maps.Clone(env)) // the executor keeps writing to env while the command handler reads it
defer rc.setCurrentStepEnv(nil)
_ = rc.takeUnsecureCommandError() // a refusal from before belongs to no step
err := executor(ctx)
if refusal := rc.takeUnsecureCommandError(); err == nil {
return refusal
}
return err
}
func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg string, fromCommand bool) {
name := kvPairs["name"]
if strings.EqualFold(name, "NODE_OPTIONS") {
+9 -3
View File
@@ -138,19 +138,26 @@ func TestUnsecureCommands(t *testing.T) {
tests := []struct {
name string
jobEnv map[string]string
globalEnv map[string]string
stepEnv map[string]string
processEnv string
caseInsensitive bool
optedIn bool
}{
{name: "refused with no opt-in"},
// GitHub reads the opt-in with bool.TryParse, so "1" is not one.
{name: "refused for a value bool.TryParse rejects", jobEnv: map[string]string{allowUnsecureCommandsVar: "1"}},
{name: "opted in through the step environment", stepEnv: map[string]string{allowUnsecureCommandsVar: "true"}, optedIn: true},
{name: "opted in through the job environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "TRUE"}, optedIn: true},
{name: "step opt-out overrides the job and global environment", jobEnv: map[string]string{allowUnsecureCommandsVar: "true"}, globalEnv: map[string]string{allowUnsecureCommandsVar: "true"}, stepEnv: map[string]string{allowUnsecureCommandsVar: "false"}},
{name: "the runner's own opt-in overrides a step opt-out", stepEnv: map[string]string{allowUnsecureCommandsVar: "false"}, processEnv: "true", optedIn: true},
{name: "a differently cased opt-in counts where env names are case-insensitive", jobEnv: map[string]string{"Actions_Allow_Unsecure_Commands": "true"}, caseInsensitive: true, optedIn: true},
{name: "a differently cased opt-in is ignored where env names are case-sensitive", jobEnv: map[string]string{"Actions_Allow_Unsecure_Commands": "true"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Setenv(allowUnsecureCommandsVar, tt.processEnv)
a := assert.New(t)
rc := &RunContext{Config: &Config{}, Env: tt.jobEnv}
rc := &RunContext{Config: &Config{}, Env: tt.jobEnv, GlobalEnv: tt.globalEnv, JobContainer: fakeContainer{caseInsensitive: tt.caseInsensitive}}
rc.setCurrentStepEnv(tt.stepEnv)
handler := rc.commandHandler(t.Context())
@@ -160,7 +167,6 @@ func TestUnsecureCommands(t *testing.T) {
if !tt.optedIn {
a.Empty(rc.Env["x"])
a.Empty(rc.ExtraPath)
// The refusal fails the step that produced it, once.
require.ErrorContains(t, rc.takeUnsecureCommandError(), "set-env")
a.NoError(rc.takeUnsecureCommandError())
return
Loaded 50 of 110 files, more files were not shown because too many files have changed in this diff. Show more