Compare commits

...
9 Commits
Author SHA1 Message Date
bircni c8b16f4792 enhance: add builtin:download-artifact (#1344)
Co-authored-by: bircni <[email protected]>
2026-11-06 03:32:59 +00:00
bircni 02446620df enhance: add builtin:upload-artifact (#1343)
Co-authored-by: bircni <[email protected]>
2026-11-06 03:17:49 +00:00
8c8950d9ef refactor(container): stream copies into the job and archives out of the host (#1340)
Adds `CopyTarStream` to the job environments. It extracts directories and files into the job as they are written, kept below the destination and owned by the job user along with any directories it creates, so built-in actions can put large content in place without buffering it. The host's `GetContainerArchive` now streams instead of building the archive in memory.

---------

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1340
Reviewed-by: Lunny Xiao <[email protected]>
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-06 02:22:40 +00:00
bircniandsilverwind b88d111cc5 feat: port the path patterns of @actions/glob (#1342)
Built-in actions that select files need to match what the JavaScript actions match. This ports @actions/glob 0.6.1 and the minimatch 3 subset it uses, as bundled in `actions/upload-artifact` v7. It only answers questions about paths; walking the files stays with the caller. Only malformed bracket expressions and some non-ASCII case folding on Windows can match differently.

Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1342
Reviewed-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
2026-11-06 01:13:07 +00:00
silverwind bb6d793cb9 fix: make container.sweep opt-in (#1389)
Sweeping removed every container a job started through its Docker proxy, which silently tore down services that jobs deploy to the runner's own Docker daemon. Deploying from a runner on the target host is common, so `container.sweep` now defaults to `false` and CI-only runners can opt in with `container.sweep: true`.

Fixes https://gitea.com/gitea/runner/issues/1385

Reviewed-on: https://gitea.com/gitea/runner/pulls/1389
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-05 13:50:46 +00:00
silverwind 95ef17d022 fix: prevent masking gaps in workflow command logs (#1384)
Mask partially escaped workflow command output and complete one-character values while preserving literal output and stream line endings.

Fixes: https://gitea.com/gitea/runner/issues/300
Reviewed-on: https://gitea.com/gitea/runner/pulls/1384
Reviewed-by: bircni <[email protected]>
2026-11-05 08:01:44 +00:00
Lunny Xiaoandsilverwind ee03687688 fix: upgrade Alpine packages in the runner images (#1388)
All images run `apk upgrade` before installing packages, so they pick up Alpine fixes their base images lag on. `docker:29.9.2-dind` shipped libexpat 2.9.0-r0 while Alpine 3.24 provides 2.10.0-r0 with the security fixes, and both `docker:29.9.3-dind` and `alpine:3.24` still ship apk-tools 3.0.8-r0 where 3.0.9-r0 is available. The dind bases are also bumped to `docker:29.9.3`, which drops several vulnerable bundled binaries' findings.

Verified by building the targets locally: libexpat is 2.10.0-r0 (was 2.9.0-r0), apk-tools is 3.0.9-r0 (was 3.0.8-r0), and Trivy reports no Alpine package findings on the dind images.

Assisted-by: Codet:claude
Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: silverwind <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1388
Reviewed-by: silverwind <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
2026-11-05 07:45:27 +00:00
silverwind 30e600d8f6 test: speed up (#1386)
Reduce redundant fixture setup, Git subprocesses, and large data copies while preserving test coverage.

Local macOS measurements against main:

| Measurement | Before | After |
| --- | ---: | ---: |
| Git package time, median of 3 alternating runs | 4.09 s | 3.48 s (-15%) |
| Allocations across 18 action-bundle cases | 489 MiB | 251 MiB (-49%) |
| Traced Git starts per checkout table run | 234 | 175 |

Reviewed-on: https://gitea.com/gitea/runner/pulls/1386
Reviewed-by: bircni <[email protected]>
2026-11-05 07:33:17 +00:00
silverwind 664978c819 test: fix windows path test (#1387)
`TestCopyWorkflowFilesHandsNodeTheRunnerCAUnlessTheJobOrImageSetsIt` from https://gitea.com/gitea/runner/pulls/1366 fails on Windows. `HostEnvironment.GetActPath` turns backslashes into slashes there, so `NODE_EXTRA_CA_CERTS` gets the slash form while the test expects the raw temp dir path. The runner behaviour is correct, only the expectation is wrong.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1387
Reviewed-by: Lunny Xiao <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-11-05 07:14:56 +00:00
76 changed files with 5042 additions and 382 deletions

No files matched your search

+7 -4
View File
@@ -23,11 +23,14 @@ depending on the prefix:
covers both forms: it escapes `%` first, so a literal `%3B` becomes `%253B` that the extra
`##[…]` rules cannot match, and a raw `;` or `]` is never decoded. It is also what makes
multi-line work, `\n` becomes `%0A` and the UI turns it back into a line break.
- **Forwarding a command from step output?** Leave the payload alone, it arrived escaped and is
decoded once. Decoding here double-decodes and destroys multi-line.
- **Forwarding a command from step output?** Preserve its escaping unless masking changes it.
Decoding for display happens in the UI.
- **No prefix?** Do not escape, and split multi-line values into one row each.
- **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the
encoded forms too.
- **Interpolating a secret?** Masking runs after escaping. Recognised command payloads are
normalised before masking to cover partially escaped values. Their original escaping is
preserved when no replacement occurs. `AppendSecretMasker` also registers encoded forms for
other output. Complete nonblank one-character values are masked, while one-byte fragments
of multiline values are skipped.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the command
handler folds an annotation's location into its decoded message and escapes the result again.
+7 -5
View File
@@ -19,14 +19,15 @@ RUN --mount=type=cache,target=/go/pkg/mod \
### DIND VARIANT
#
#
FROM docker:29.9.2-dind AS dind
FROM docker:29.9.3-dind AS dind
ARG VERSION=dev
LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}"
RUN apk add --no-cache s6 bash git tzdata nftables
# upgrade first: the docker base image can lag behind Alpine security fixes
RUN apk upgrade --no-cache && apk add --no-cache s6 bash git tzdata nftables
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh
@@ -41,7 +42,7 @@ ENTRYPOINT ["s6-svscan","/etc/s6"]
### DIND-ROOTLESS VARIANT
#
#
FROM docker:29.9.2-dind-rootless AS dind-rootless
FROM docker:29.9.3-dind-rootless AS dind-rootless
ARG VERSION=dev
@@ -49,7 +50,8 @@ LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}"
USER root
RUN apk add --no-cache s6 bash git tzdata nftables
# upgrade first: the docker base image can lag behind Alpine security fixes
RUN apk upgrade --no-cache && apk add --no-cache s6 bash git tzdata nftables
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh
@@ -75,7 +77,7 @@ ARG VERSION=dev
LABEL org.opencontainers.image.source="https://gitea.com/gitea/runner"
LABEL org.opencontainers.image.version="${VERSION}"
RUN apk add --no-cache tini bash git tzdata
RUN apk upgrade --no-cache && apk add --no-cache tini bash git tzdata
COPY --from=builder /opt/src/runner/gitea-runner /usr/local/bin/gitea-runner
COPY scripts/run.sh /usr/local/bin/run.sh
+2 -2
View File
@@ -243,11 +243,11 @@ volumes:
- ${GITEA_DOCKER_WORKSPACE:-.}/data:/app/data
```
In most Linux container jobs, the containers, networks and volumes a job creates are removed when it ends, `docker compose` services included. Set `container.sweep: false` if jobs deploy services to the runner's Docker daemon.
Set `container.sweep: true` to remove the containers, networks and volumes most Linux container jobs create when they end, `docker compose` services included. Leave it off if jobs deploy services to the runner's Docker daemon.
### Built-in actions (`uses: builtin:<name>`)
`uses: builtin:checkout` checks out a repository without downloading `actions/checkout` or needing Node in the job image, see [docs/builtin-actions.md](docs/builtin-actions.md).
`uses: builtin:checkout`, `builtin:upload-artifact` and `builtin:download-artifact` replace their `actions/` counterparts without an action download or Node in the job image, see [docs/builtin-actions.md](docs/builtin-actions.md).
### Caching (`actions/cache`)
+10 -7
View File
@@ -851,6 +851,9 @@ func TestHandler_touchCacheSkipsFreshCompletedEntries(t *testing.T) {
func newTestHandler(t *testing.T, policy Policy, entries ...*Cache) *Handler {
t.Helper()
handler := startTestHandler(t, Options{Policy: policy})
if len(entries) == 0 {
return handler
}
db, err := handler.openDB()
require.NoError(t, err)
@@ -940,6 +943,10 @@ func TestHandler_RejectsBearer(t *testing.T) {
func TestHandler_UnregisterRevokes(t *testing.T) {
t.Parallel()
handler, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache")})
require.NoError(t, err)
defer handler.Close()
for _, tc := range []struct {
name string
token string
@@ -950,9 +957,6 @@ func TestHandler_UnregisterRevokes(t *testing.T) {
{"reference counted", "shared", "v", 2},
} {
t.Run(tc.name, func(t *testing.T) {
handler, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache")})
require.NoError(t, err)
defer handler.Close()
revokers := make([]func(), tc.registrations)
for i := range revokers {
revokers[i] = handler.RegisterJob(tc.token, JobCredential{Repo: testRepo})
@@ -1049,9 +1053,8 @@ func TestHandler_ArtifactSignature(t *testing.T) {
base := handler.ExternalURL() + apiPath
good := signArtifactURL(handler, 1)
other, err := StartHandler(Options{Dir: filepath.Join(t.TempDir(), "artifactcache2")})
require.NoError(t, err)
defer other.Close()
secret := bytes.Clone(handler.secret)
secret[0] ^= 1
for _, tc := range []struct {
name string
@@ -1060,7 +1063,7 @@ func TestHandler_ArtifactSignature(t *testing.T) {
{"missing signature", fmt.Sprintf("%s/artifacts/%d", base, 1)},
{"tampered signature", good[:len(good)-4] + "dead"},
{"expired signature", handler.signedArtifactURL(JobCredential{}, 1, time.Now().Add(-time.Second))},
{"signature from a different server", base + strings.SplitN(signArtifactURL(other, 1), apiPath, 2)[1]},
{"signature from a different server", base + strings.SplitN(signArtifactURL(&Handler{secret: secret}, 1), apiPath, 2)[1]},
} {
t.Run(tc.name, func(t *testing.T) {
resp, err := testClient.Get(tc.url)
+1 -1
View File
@@ -218,7 +218,7 @@ func Serve(ctx context.Context, artifactPath, addr, port string) context.CancelF
logger.Debugf("Artifacts base path '%s'", artifactPath)
uploads(router, artifactPath)
downloads(router, artifactPath)
registerV4(router, artifactPath)
RegisterV4(router, artifactPath)
server := &http.Server{
Addr: net.JoinHostPort(addr, port),
+2 -2
View File
@@ -28,7 +28,7 @@ func TestArtifactFlow(t *testing.T) {
router := http.NewServeMux()
uploads(router, artifactPath)
downloads(router, artifactPath)
registerV4(router, artifactPath)
RegisterV4(router, artifactPath)
server := httptest.NewServer(router)
defer server.Close()
@@ -256,7 +256,7 @@ func TestArtifactFlow(t *testing.T) {
require.Equal(t, "attachment; filename*=UTF-8''50%25%20art%40v1", header.Get("Content-Disposition"))
rerun := http.NewServeMux()
registerV4(rerun, artifactPath)
RegisterV4(rerun, artifactPath)
recorder := httptest.NewRecorder()
rerun.ServeHTTP(recorder, httptest.NewRequest(http.MethodPost, artifactServicePath+"/CreateArtifact", strings.NewReader("{"+artifactRef+"}")))
require.Equal(t, http.StatusOK, recorder.Code, recorder.Body.String())
+1 -1
View File
@@ -35,7 +35,7 @@ const (
maxTwirpBytes = 1 << 20
)
func registerV4(router *http.ServeMux, baseDir string) {
func RegisterV4(router *http.ServeMux, baseDir string) {
server := &serverV4{dir: filepath.Join(baseDir, "v4"), lastID: time.Now().UnixMilli(), created: map[string]bool{}} // IDs stay unique across invocations sharing baseDir
router.HandleFunc("POST "+artifactServicePath+"/CreateArtifact", twirp(server.createArtifact))
router.HandleFunc("POST "+artifactServicePath+"/FinalizeArtifact", twirp(server.finalizeArtifact))
+14 -3
View File
@@ -36,7 +36,7 @@ func TestFetchAction(t *testing.T) {
root := t.TempDir()
remote := filepath.Join(root, "org", "action")
require.NoError(t, gitCmd("init", "--initial-branch=main", remote))
require.NoError(t, os.CopyFS(remote, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, os.WriteFile(filepath.Join(remote, "action.yml"), []byte("name: action\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(remote, "run.sh"), []byte("#!/bin/sh\n"), 0o755))
require.NoError(t, os.Symlink("action.yml", filepath.Join(remote, "link")))
@@ -53,6 +53,7 @@ func TestFetchAction(t *testing.T) {
var archiveRequests []http.Header
var mirror *httptest.Server
archiveCache := map[string][]byte{}
revisions := map[string]string{}
requests, archives, failures := 0, "", 0
handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
mu.Lock()
@@ -79,7 +80,14 @@ func TestFetchAction(t *testing.T) {
http.Redirect(writer, request, mirror.URL+request.URL.Path, http.StatusFound)
return
}
sha, _ := gitOutput(request.Context(), remote, "rev-parse", strings.TrimSuffix(ref, ".tar.gz"))
sha, ok := revisions[ref]
if !ok {
var err error
sha, err = gitOutput(request.Context(), remote, "rev-parse", strings.TrimSuffix(ref, ".tar.gz"))
if err == nil {
revisions[ref] = sha
}
}
etag := fmt.Sprintf(`W/"%s-tar.gz-%t"`, sha, archives != "unprefixed")
if request.Header.Get("If-None-Match") == etag {
writer.WriteHeader(http.StatusNotModified)
@@ -208,6 +216,9 @@ func TestFetchAction(t *testing.T) {
require.NoError(t, gitCmd("-C", remote, "commit", "--allow-empty", "-m", "moved"))
require.NoError(t, gitCmd("-C", remote, "tag", "--force", "v1"))
expectedSha = gitRevParse(t, remote, "v1")
mu.Lock()
clear(revisions)
mu.Unlock()
moved, movedSha, err := FetchAction(t.Context(), input)
require.NoError(t, err)
assert.Equal(t, expectedSha, movedSha)
@@ -253,7 +264,7 @@ func TestFetchAction(t *testing.T) {
})
t.Run("downloads again over a tree whose files changed size or that has no record, not under a job copying it", func(t *testing.T) {
sha := gitRevParse(t, remote, "main")
sha := expectedSha
for _, mode := range []string{"prefixed", "none"} {
serve(mode, 0)
input := newInput(sha)
+25 -31
View File
@@ -112,8 +112,7 @@ func TestFindGitMetadataOfSHA256RepositoryUsesOrigin(t *testing.T) {
t.Parallel()
basedir := t.TempDir()
const remoteURL = "https://github.com/owner/repo.git"
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, os.RemoveAll(filepath.Join(basedir, ".git", "hooks")))
require.NoError(t, gitCmd("init", "--template=", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init"))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL))
require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git"))
@@ -205,16 +204,13 @@ func TestGitCloneExecutor(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh", ".gitattributes"))
require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "tag", "v2"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v2"))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "act-fails"))
fullSha := gitRevParse(t, workDir, "main")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:refs/heads/"+fullSha[:4]))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", "v2", "act-fails", "main:refs/heads/"+fullSha[:4]))
hostConfig := filepath.Join(t.TempDir(), "gitconfig")
require.NoError(t, os.WriteFile(hostConfig, []byte("[core]\n\tautocrlf = true\n\teol = crlf\n[url "+strconv.Quote(remoteDir)+"]\n\tinsteadOf = https://example.invalid/action\n"), 0o644))
@@ -282,10 +278,10 @@ func TestGitCloneExecutor(t *testing.T) {
func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
t.Parallel()
createRemote := func(message string) string {
remoteDir, workDir := newGitRemote(t, message)
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
return remoteDir
dir := t.TempDir()
require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", message))
return dir
}
oldRemoteDir := createRemote("old-action")
@@ -405,31 +401,30 @@ func TestGitCloneExecutorShallow(t *testing.T) {
remoteDir, workDir := newGitRemote(t, "c1", "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v1"))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", "v1"))
shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") }
for name, ref := range map[string]string{"branch is cloned shallowly": "main", "tag is cloned shallowly": "v1"} {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: ref, Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present")
assert.Equal(t, "c3", gitHeadSubject(t, dir))
assert.Equal(t, []string{"c3"}, gitSubjects(t, dir), "only the tip commit should be present")
})
}
unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string {
remote := filepath.Join(t.TempDir(), "remote.git")
require.NoError(t, gitCmd("clone", "--bare", remoteDir, remote))
require.NoError(t, gitCmd("-C", remote, "config", "uploadpack.allowAnySHA1InWant", strconv.FormatBool(allowUnadvertised)))
require.NoError(t, gitCmd("clone", "--bare", "--template=", "--config", "uploadpack.allowAnySHA1InWant="+strconv.FormatBool(allowUnadvertised), remoteDir, remote))
return remote
}
t.Run("commit hash resolves when the remote accepts a direct shallow fetch", func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: unadvertisedRemote(t, false), Ref: unadvertisedSHA, Dir: dir, Depth: 1,
@@ -438,6 +433,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
})
t.Run("commit hash is fetched shallowly, moved to another hash, then reused without reaching the remote", func(t *testing.T) {
t.Parallel()
remote := unadvertisedRemote(t, true)
dir := t.TempDir()
cloneAt := func(ref string) error {
@@ -448,7 +444,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
require.NoError(t, cloneAt(unadvertisedSHA))
assert.FileExists(t, shallowMarker(dir))
assert.Equal(t, 1, gitRevCount(t, dir))
assert.Len(t, gitSubjects(t, dir), 1)
assert.Equal(t, unadvertisedSHA, gitRevParse(t, dir, "HEAD"))
olderSHA := gitRevParse(t, workDir, "HEAD~2")
@@ -461,6 +457,8 @@ func TestGitCloneExecutorShallow(t *testing.T) {
})
t.Run("moving hexadecimal-named branch updates while staying shallow", func(t *testing.T) {
t.Parallel()
remoteDir, workDir := newGitRemote(t, "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef"))
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{
@@ -473,19 +471,17 @@ func TestGitCloneExecutorShallow(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:deadbeef"))
require.NoError(t, clone(t.Context()))
assert.Equal(t, "c4", gitHeadSubject(t, dir), "reused shallow clone should update to the new tip")
assert.Equal(t, []string{"c4"}, gitSubjects(t, dir), "reused shallow clone should update to the new tip")
assert.FileExists(t, shallowMarker(dir), "repo should remain shallow after update")
assert.Equal(t, 1, gitRevCount(t, dir))
})
t.Run("sha256 full hash is cloned shallowly, reused without the remote, and its 40-character prefix is short", func(t *testing.T) {
t.Parallel()
remote := filepath.Join(t.TempDir(), "remote.git")
require.NoError(t, gitCmd("init", "--bare", "--object-format=sha256", "--initial-branch=main", remote))
work := t.TempDir()
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", work))
require.NoError(t, gitCmd("-C", work, "remote", "add", "origin", remote))
require.NoError(t, gitCmd("init", "--template=", "--object-format=sha256", "--initial-branch=main", work))
require.NoError(t, gitCmd("-C", work, "commit", "--allow-empty", "-m", "pinned"))
require.NoError(t, gitCmd("-C", work, "push", "origin", "main"))
require.NoError(t, gitCmd("clone", "--bare", "--template=", work, remote))
sha := gitRevParse(t, work, "HEAD")
dir := t.TempDir()
clone := NewGitCloneExecutor(NewGitCloneExecutorInput{URL: remote, Ref: sha, Dir: dir, Depth: 1})
@@ -504,11 +500,11 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
pinned := gitRevParse(t, workDir, "HEAD")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy"))
require.NoError(t, gitCmd("-C", workDir, "tag", pinned))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", pinned))
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main", pinned))
for name, depth := range map[string]int{"shallow": 1, "full clone": 0} {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: pinned, Dir: dir, Depth: depth,
@@ -540,13 +536,11 @@ func gitRevParse(t *testing.T, dir, rev string) string {
return strings.TrimSpace(string(out))
}
func gitRevCount(t *testing.T, dir string) int {
func gitSubjects(t *testing.T, dir string) []string {
t.Helper()
out, err := exec.Command("git", "-C", dir, "rev-list", "--count", "HEAD").Output()
out, err := exec.Command("git", "-C", dir, "log", "--format=%s").Output()
require.NoError(t, err)
n, err := strconv.Atoi(strings.TrimSpace(string(out)))
require.NoError(t, err)
return n
return strings.Split(strings.TrimSpace(string(out)), "\n")
}
func gitHeadSubject(t *testing.T, dir string) string {
@@ -680,7 +674,7 @@ func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
})
dir := filepath.Join(t.TempDir(), "cached-action")
require.NoError(t, gitCmd("init", dir))
require.NoError(t, os.CopyFS(dir, os.DirFS(filepath.Join(gitFixtureDir, "work"))))
require.NoError(t, gitCmd("-C", dir, "remote", "add", "origin", server.URL))
ctx, cancel := context.WithCancel(t.Context())
+35
View File
@@ -5,6 +5,7 @@
package container
import (
"bytes"
"context"
"errors"
"fmt"
@@ -30,6 +31,38 @@ func (e ExitCodeError) Error() string {
return fmt.Sprintf("Process completed with exit code %d.", int(e))
}
type commandOutput struct {
*io.PipeReader
cancel context.CancelFunc
done chan struct{}
err error
}
func (output *commandOutput) Close() error {
_, _ = io.Copy(io.Discard, output.PipeReader)
output.PipeReader.Close()
output.cancel()
<-output.done
return output.err
}
func streamOutput(ctx context.Context, run func(context.Context, io.Writer, io.Writer) error) io.ReadCloser {
ctx, cancel := context.WithCancel(ctx)
reader, writer := io.Pipe()
output := &commandOutput{PipeReader: reader, cancel: cancel, done: make(chan struct{})}
go func() {
defer close(output.done)
defer cancel()
var stderr bytes.Buffer
output.err = run(ctx, writer, &stderr)
if output.err != nil && strings.TrimSpace(stderr.String()) != "" {
output.err = fmt.Errorf("%w: %s", output.err, strings.TrimSpace(stderr.String()))
}
writer.CloseWithError(output.err)
}()
return output
}
// NewContainerInput the input for the New function
type NewContainerInput struct {
Image string
@@ -130,7 +163,9 @@ type Container interface {
Create(capAdd, capDrop []string) common.Executor
Copy(destPath string, files ...*FileEntry) common.Executor
CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor
ExecOutput(ctx context.Context, command []string) (io.ReadCloser, error)
GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error)
CopyTarStream(ctx context.Context, destPath string, write func(*TarWriter) error) error
Inspect(ctx context.Context) (*Info, error)
DumpLogs(ctx context.Context) error
Pull(forcePull bool) common.Executor
+1 -1
View File
@@ -42,11 +42,11 @@ func TestDockerBuildLabelsImageWithItsTagAndReportsDaemonError(t *testing.T) {
func TestCreateBuildContextFollowsDockerignoreAndSymlinkedRootButKeepsDockerfile(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.Mkdir(filepath.Join(dir, "build"), 0o755))
for name, content := range map[string]string{
".dockerignore": "*.log\nbuild\n!build/keep.txt\nDockerfile", "Dockerfile": "FROM scratch", "a.txt": "a",
"a.log": "log", "build/drop.txt": "drop", "build/keep.txt": "keep",
} {
require.NoError(t, os.MkdirAll(filepath.Join(dir, filepath.Dir(name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
}
require.NoError(t, os.Mkdir(filepath.Join(dir, "empty"), 0o755))
+80 -4
View File
@@ -146,6 +146,47 @@ func (cr *containerReference) CopyDir(destPath, srcPath string, useGitIgnore, sk
).IfNot(common.Dryrun)
}
func (cr *containerReference) ExecOutput(ctx context.Context, command []string) (io.ReadCloser, error) {
if common.Dryrun(ctx) {
return nil, errors.New("DRYRUN is not supported in ExecOutput")
}
if err := cr.connect()(ctx); err != nil {
return nil, err
}
if err := cr.find()(ctx); err != nil {
return nil, err
}
if cr.id == "" {
return nil, cr.missingContainerError("exec %v", command)
}
return streamOutput(ctx, func(ctx context.Context, stdout, stderr io.Writer) error {
created, err := cr.cli.ExecCreate(ctx, cr.id, client.ExecCreateOptions{
Cmd: command, AttachStdout: true, AttachStderr: true,
})
if err != nil {
return err
}
attached, err := cr.cli.ExecAttach(ctx, created.ID, client.ExecAttachOptions{})
if err != nil {
return err
}
defer attached.Close()
stop := context.AfterFunc(ctx, func() { attached.Close() })
defer stop()
if _, err := stdcopy.StdCopy(stdout, stderr, attached.Reader); err != nil {
return err
}
inspected, err := cr.cli.ExecInspect(ctx, created.ID, client.ExecInspectOptions{})
if err != nil {
return err
}
if inspected.ExitCode != 0 {
return ExitCodeError(inspected.ExitCode)
}
return nil
}), nil
}
func (cr *containerReference) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
if common.Dryrun(ctx) {
return nil, errors.New("DRYRUN is not supported in GetContainerArchive")
@@ -938,10 +979,7 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
logger.Error(err)
}
}()
extractionRoot := "/"
if actPath := cr.GetActPath(); strings.HasPrefix(dstPath+"/", actPath+"/") {
extractionRoot = actPath // keeps tar entries from crossing the image's /var/run symlink
}
extractionRoot := cr.extractionRoot(dstPath)
if err := writeDirTar(ctx, tarFile, path.Join("/", strings.TrimPrefix(dstPath, extractionRoot)), srcPath, useGitIgnore, skipGitDir, cr.UID, cr.GID, cr.markExecutable(ctx)); err != nil {
return err
}
@@ -954,6 +992,44 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
}
}
func (cr *containerReference) extractionRoot(dstPath string) string {
if actPath := cr.GetActPath(); strings.HasPrefix(dstPath+"/", actPath+"/") {
return actPath // keeps tar entries from crossing the image's /var/run symlink
}
return "/"
}
func (cr *containerReference) CopyTarStream(ctx context.Context, destPath string, write func(*TarWriter) error) error {
if common.Dryrun(ctx) {
return nil
}
if err := cr.connect()(ctx); err != nil {
return err
}
if err := cr.find()(ctx); err != nil {
return err
}
if cr.id == "" {
return cr.missingContainerError("copy to %s", destPath)
}
destPath = path.Clean(destPath)
root := cr.extractionRoot(destPath)
rel := func(fullPath string) string { return strings.TrimPrefix(strings.TrimPrefix(fullPath, root), "/") }
var parents []string // sent as entries so they belong to the job user, the daemon creates missing ones as root
for dir := destPath; dir != root && dir != "/" && dir != "."; dir = path.Dir(dir) {
if _, err := cr.cli.ContainerStatPath(ctx, cr.id, client.ContainerStatPathOptions{Path: dir}); err == nil {
break
} else if !cerrdefs.IsNotFound(err) {
return fmt.Errorf("stat %s: %w", dir, err)
}
parents = append(parents, rel(dir))
}
slices.Reverse(parents)
content := pipeTar(cr.UID, cr.GID, rel(destPath), parents, write)
defer content.Close()
return cr.upload(ctx, root, content)
}
func (cr *containerReference) copyContent(dstPath string, files ...*FileEntry) common.Executor {
return func(ctx context.Context) error {
if cr.id == "" {
+76 -5
View File
@@ -122,6 +122,11 @@ func (m *mockDockerClient) CopyToContainer(ctx context.Context, id string, optio
return args.Get(0).(mobyclient.CopyToContainerResult), args.Error(1)
}
func (m *mockDockerClient) ContainerStatPath(ctx context.Context, id string, options mobyclient.ContainerStatPathOptions) (mobyclient.ContainerStatPathResult, error) {
args := m.Called(ctx, id, options)
return args.Get(0).(mobyclient.ContainerStatPathResult), args.Error(1)
}
func (m *mockDockerClient) ContainerInspect(ctx context.Context, id string, opts mobyclient.ContainerInspectOptions) (mobyclient.ContainerInspectResult, error) {
args := m.Called(ctx, id, opts)
return args.Get(0).(mobyclient.ContainerInspectResult), args.Error(1)
@@ -255,6 +260,28 @@ func TestDockerExecFailure(t *testing.T) {
assert.Equal(t, ExitCodeError(1), exitErr)
assert.Equal(t, "Process completed with exit code 1.", err.Error())
client.On("ContainerInspect", ctx, "123", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, nil)
client.On("ExecCreate", mock.Anything, "123", mobyclient.ExecCreateOptions{
Cmd: []string{"printf", `a\b`}, AttachStdout: true, AttachStderr: true,
}).Return(mobyclient.ExecCreateResult{ID: "output"}, nil)
client.On("ExecAttach", mock.Anything, "output", mobyclient.ExecAttachOptions{}).
Return(mobyclient.ExecAttachResult{
Conn: &mockConn{},
Reader: bufio.NewReader(bytes.NewReader(append(stdcopyFrame(stdcopy.Stdout, "a\x00b"), stdcopyFrame(stdcopy.Stderr, " failed \n")...))),
}, nil)
client.On("ExecInspect", mock.Anything, "output", mobyclient.ExecInspectOptions{}).
Return(mobyclient.ExecInspectResult{ExitCode: 3}, nil)
output, err := cr.ExecOutput(ctx, []string{"printf", `a\b`})
require.NoError(t, err)
content, err := io.ReadAll(output)
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(3), exitErr)
assert.Equal(t, "Process completed with exit code 3.: failed", err.Error())
assert.Equal(t, "a\x00b", string(content))
require.ErrorAs(t, output.Close(), &exitErr)
assert.Equal(t, ExitCodeError(3), exitErr)
client.AssertExpectations(t)
}
@@ -528,6 +555,8 @@ func TestRejectsMissingContainer(t *testing.T) {
check("exec", cr.exec([]string{"echo"}, nil, "", "")(ctx))
_, err := cr.GetContainerArchive(ctx, "/var/run/act/x")
check("GetContainerArchive", err)
_, err = cr.ExecOutput(ctx, []string{"true"})
check("ExecOutput", err)
_, err = cr.Inspect(ctx)
check("Inspect", err)
@@ -539,7 +568,50 @@ func TestRejectsMissingContainer(t *testing.T) {
check("Inspect after removal", err)
}
func TestCopyTarStreamOwnsMissingParentsAndRejectsEscapes(t *testing.T) {
ctx := t.Context()
client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "job", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{ID: "job", Name: "/job"}}, nil)
stat := func(p string, err error) {
client.On("ContainerStatPath", ctx, "job", mobyclient.ContainerStatPathOptions{Path: p}).
Return(mobyclient.ContainerStatPathResult{}, err).Once()
}
stat("/w/out/a", cerrdefs.ErrNotFound)
stat("/w/out", cerrdefs.ErrNotFound)
stat("/w", nil)
var entries []string
client.On("CopyToContainer", ctx, "job", mock.Anything).Run(func(args mock.Arguments) {
opts := args.Get(2).(mobyclient.CopyToContainerOptions)
assert.Equal(t, "/", opts.DestinationPath)
reader := tar.NewReader(opts.Content)
for {
header, err := reader.Next()
if errors.Is(err, io.EOF) {
return
}
require.NoError(t, err)
assert.Equal(t, []any{1001, 1002, ""}, []any{header.Uid, header.Gid, header.Uname}, header.Name)
entries = append(entries, header.Name)
}
}).Return(mobyclient.CopyToContainerResult{}, nil).Once()
cr := &containerReference{id: "job", cli: client, input: &NewContainerInput{Name: "job"}, UID: 1001, GID: 1002}
require.NoError(t, cr.CopyTarStream(ctx, "/w/out/a/", func(tw *TarWriter) error {
return tw.WriteHeader(&tar.Header{Typeflag: tar.TypeReg, Name: "dir/f.txt", Mode: 0o644, Uname: "nobody"})
}))
assert.Equal(t, []string{"w/out/", "w/out/a/", "w/out/a/dir/", "w/out/a/dir/f.txt"}, entries)
client.AssertExpectations(t)
require.ErrorContains(t, writeTarStream(io.Discard, 0, 0, "w/out/a", nil, func(tw *TarWriter) error {
return tw.WriteHeader(&tar.Header{Typeflag: tar.TypeReg, Name: "../../../etc/passwd"})
}), "escapes")
}
func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
source := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(source, "x"), nil, 0o644))
for _, testcase := range []struct {
name, destination, extractionRoot, entry string
directory bool
@@ -567,8 +639,6 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
cr := &containerReference{id: "stale", cli: client, input: &NewContainerInput{Name: "job-1"}}
if testcase.directory {
source := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(source, "x"), nil, 0o644))
require.NoError(t, cr.CopyDir(testcase.destination, source+string(filepath.Separator), false, true)(ctx))
} else {
require.NoError(t, cr.Copy(testcase.destination, &FileEntry{Name: "x", Mode: 0o644})(ctx))
@@ -1108,6 +1178,10 @@ func TestConnectTakesDaemonOSFromPing(t *testing.T) {
}
func TestContainerCopyModes(t *testing.T) {
source := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(source, "run.sh"), []byte("#!/bin/sh\n"), 0o666))
require.NoError(t, os.Chmod(filepath.Join(source, "run.sh"), 0o666))
for _, testcase := range []struct {
hostOS, serverOS string
expected int64
@@ -1117,9 +1191,6 @@ func TestContainerCopyModes(t *testing.T) {
{"linux", "linux", 0o666},
} {
t.Run(testcase.hostOS+"-"+testcase.serverOS, func(t *testing.T) {
source := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(source, "run.sh"), []byte("#!/bin/sh\n"), 0o666))
require.NoError(t, os.Chmod(filepath.Join(source, "run.sh"), 0o666))
file := &FileEntry{Name: "run.sh", Mode: 0o600, Body: "#!/bin/sh\n"}
original := hostOS
hostOS = testcase.hostOS
+89 -25
View File
@@ -6,7 +6,6 @@ package container
import (
"archive/tar"
"bytes"
"cmp"
"context"
"errors"
@@ -17,6 +16,7 @@ import (
"os/exec"
"path/filepath"
"runtime"
"slices"
"strings"
"sync"
"sync/atomic"
@@ -116,18 +116,39 @@ func (e *HostEnvironment) Inspect(_ context.Context) (*Info, error) {
return &Info{Health: HealthNone, Ports: map[string]string{}}, nil
}
func (e *HostEnvironment) ExecOutput(ctx context.Context, command []string) (io.ReadCloser, error) {
return streamOutput(ctx, func(ctx context.Context, stdout, stderr io.Writer) error {
cmd := exec.CommandContext(ctx, command[0], command[1:]...)
cmd.Dir, cmd.Stdout, cmd.Stderr = e.Workdir, stdout, stderr
if err := cmd.Run(); err != nil {
if exit, ok := errors.AsType[*exec.ExitError](err); ok {
return ExitCodeError(exit.ExitCode())
}
return err
}
return nil
}), nil
}
func (e *HostEnvironment) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
buf := &bytes.Buffer{}
tw := tar.NewWriter(buf)
defer tw.Close()
srcPath = filepath.Clean(srcPath)
fi, err := os.Lstat(srcPath)
if err != nil {
return nil, err
}
tc := &filecollector.TarCollector{
TarWriter: tw,
}
reader, writer := io.Pipe()
go func() {
tw := tar.NewWriter(writer)
err := writeHostArchive(ctx, &filecollector.TarCollector{TarWriter: tw}, srcPath, fi)
if err == nil {
err = tw.Close()
}
writer.CloseWithError(err)
}()
return startedArchive(reader)
}
func writeHostArchive(ctx context.Context, tc *filecollector.TarCollector, srcPath string, fi fs.FileInfo) error {
if fi.IsDir() {
srcPrefix := srcPath
if !strings.HasSuffix(srcPrefix, string(filepath.Separator)) {
@@ -138,31 +159,74 @@ func (e *HostEnvironment) GetContainerArchive(ctx context.Context, srcPath strin
SrcPrefix: srcPrefix,
Handler: tc,
}
err = filepath.Walk(srcPath, fc.CollectFiles(ctx))
if err != nil {
return nil, err
return filepath.Walk(srcPath, fc.CollectFiles(ctx))
}
var f io.ReadCloser
var linkname string
var err error
if fi.Mode()&fs.ModeSymlink != 0 {
if linkname, err = os.Readlink(srcPath); err != nil {
return err
}
} else {
var f io.ReadCloser
var linkname string
if fi.Mode()&fs.ModeSymlink != 0 {
linkname, err = os.Readlink(srcPath)
if err != nil {
return nil, err
}
if f, err = os.Open(srcPath); err != nil {
return err
}
defer f.Close()
}
return tc.WriteFile(fi.Name(), fi, linkname, f)
}
func (e *HostEnvironment) CopyTarStream(ctx context.Context, destPath string, write func(*TarWriter) error) error {
if err := os.MkdirAll(destPath, 0o777); err != nil {
return err
}
root, err := os.OpenRoot(destPath)
if err != nil {
return err
}
defer root.Close()
content := pipeTar(0, 0, "", nil, write)
defer content.Close()
tr := tar.NewReader(content)
var dirs []*tar.Header
for {
if err := ctx.Err(); err != nil {
return err
}
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
break
} else if err != nil {
return err
}
hdr.Name = filepath.FromSlash(strings.TrimSuffix(hdr.Name, "/"))
if hdr.Typeflag == tar.TypeDir {
err = root.MkdirAll(hdr.Name, 0o777) // modes apply after the contents, so read-only directories can be filled
dirs = append(dirs, hdr)
} else {
f, err = os.Open(srcPath)
if err != nil {
return nil, err
}
defer f.Close()
err = writeRootFile(root, hdr.Name, hdr.FileInfo().Mode().Perm(), tr)
}
err := tc.WriteFile(fi.Name(), fi, linkname, f)
if err != nil {
return nil, err
return err
}
}
return io.NopCloser(buf), nil
for _, dir := range slices.Backward(dirs) {
if err := root.Chmod(dir.Name, dir.FileInfo().Mode().Perm()); err != nil {
return err
}
}
return nil
}
func writeRootFile(root *os.Root, name string, mode fs.FileMode, body io.Reader) error {
_ = root.Remove(name) // replaces read-only files and applies mode, as the daemon and tar do
file, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, mode)
if err != nil {
return err
}
_, err = io.Copy(file, body)
return errors.Join(err, file.Close())
}
func (e *HostEnvironment) Pull(_ bool) common.Executor {
+69 -6
View File
@@ -8,6 +8,7 @@ import (
"archive/tar"
"bytes"
"context"
"fmt"
"io"
"os"
"path/filepath"
@@ -32,6 +33,11 @@ func TestActionPlatformNames(t *testing.T) {
}
func TestCopyDir(t *testing.T) {
src := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.MkdirAll(filepath.Join(src, ".git", "objects"), 0o700))
require.NoError(t, os.WriteFile(filepath.Join(src, ".git", "objects", "pack"), []byte("x"), 0o600))
require.NoError(t, os.WriteFile(filepath.Join(src, "action.yml"), []byte("name: x\n"), 0o600))
for _, test := range []struct {
name string
useGitIgnore, skipGitDir bool
@@ -41,11 +47,6 @@ func TestCopyDir(t *testing.T) {
{name: "unfiltered outside a git repository", useGitIgnore: true},
} {
t.Run(test.name, func(t *testing.T) {
src := filepath.Join(t.TempDir(), "action")
require.NoError(t, os.MkdirAll(filepath.Join(src, ".git", "objects"), 0o700))
require.NoError(t, os.WriteFile(filepath.Join(src, ".git", "objects", "pack"), []byte("x"), 0o600))
require.NoError(t, os.WriteFile(filepath.Join(src, "action.yml"), []byte("name: x\n"), 0o600))
dest := filepath.Join(t.TempDir(), "dest")
require.NoError(t, (&HostEnvironment{}).CopyDir(dest, src+string(filepath.Separator), test.useGitIgnore, test.skipGitDir)(t.Context()))
@@ -76,7 +77,54 @@ func TestGetContainerArchive(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, expectedContent, content)
_, err = reader.Next()
assert.ErrorIs(t, err, io.EOF)
require.ErrorIs(t, err, io.EOF)
require.NoError(t, archive.Close())
_, err = e.GetContainerArchive(ctx, filepath.Join(e.Path, "missing"))
assert.ErrorIs(t, err, os.ErrNotExist)
}
func TestHostCopyTarStream(t *testing.T) {
dest := filepath.Join(t.TempDir(), "out")
write := func(name string, mode int64, body string) func(*TarWriter) error {
return func(tw *TarWriter) error {
if err := tw.WriteHeader(&tar.Header{Typeflag: tar.TypeReg, Name: name, Mode: mode, Size: int64(len(body))}); err != nil {
return err
}
_, err := tw.Write([]byte(body))
return err
}
}
e := &HostEnvironment{}
require.NoError(t, e.CopyTarStream(t.Context(), dest, write("dir/run.sh", 0o755, "#!/bin/sh\n")))
content, err := os.ReadFile(filepath.Join(dest, "dir", "run.sh"))
require.NoError(t, err)
assert.Equal(t, "#!/bin/sh\n", string(content))
if runtime.GOOS != "windows" {
require.NoError(t, os.Chmod(filepath.Join(dest, "dir", "run.sh"), 0o444))
require.NoError(t, e.CopyTarStream(t.Context(), dest, func(tw *TarWriter) error {
if err := tw.WriteHeader(&tar.Header{Typeflag: tar.TypeDir, Name: "dir/", Mode: 0o555}); err != nil {
return err
}
return write("dir/run.sh", 0o755, "")(tw)
}))
t.Cleanup(func() { require.NoError(t, os.Chmod(filepath.Join(dest, "dir"), 0o755)) })
for name, mode := range map[string]os.FileMode{"dir": 0o555, filepath.Join("dir", "run.sh"): 0o755} {
info, err := os.Stat(filepath.Join(dest, name))
require.NoError(t, err)
assert.Equal(t, mode, info.Mode().Perm(), name)
}
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(dest, "link")))
require.ErrorContains(t, e.CopyTarStream(t.Context(), dest, write("link/x", 0o644, "x")), "escapes")
}
require.ErrorContains(t, e.CopyTarStream(t.Context(), dest, write("../escape", 0o644, "x")), "escapes")
assert.NoFileExists(t, filepath.Join(filepath.Dir(dest), "escape"))
require.Error(t, e.CopyTarStream(t.Context(), dest, func(_tw *TarWriter) error { return fmt.Errorf("read body: %w", io.EOF) }))
canceled, cancel := context.WithCancel(t.Context())
cancel()
require.ErrorIs(t, e.CopyTarStream(canceled, dest, write("late", 0o644, "x")), context.Canceled)
assert.NoFileExists(t, filepath.Join(dest, "late"))
}
func TestHostEnvironmentExecExitCode(t *testing.T) {
@@ -84,6 +132,21 @@ func TestHostEnvironmentExecExitCode(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
for _, read := range []bool{true, false} {
output, err := (&HostEnvironment{AllocatePTY: true}).ExecOutput(t.Context(), []string{"sh", "-c", `test ! -t 1; printf 'out\000'; printf ' failed \n' >&2; exit 3`})
require.NoError(t, err)
if read {
content, err := io.ReadAll(output)
var exit ExitCodeError
require.ErrorAs(t, err, &exit)
assert.Equal(t, ExitCodeError(3), exit)
assert.Equal(t, "out\x00", string(content))
assert.Equal(t, "Process completed with exit code 3.: failed", err.Error())
}
var exit ExitCodeError
require.ErrorAs(t, output.Close(), &exit)
assert.Equal(t, ExitCodeError(3), exit)
}
for name, allocatePTY := range map[string]bool{"without PTY": false, "with PTY": true} {
t.Run(name, func(t *testing.T) {
t.Parallel()
+13 -18
View File
@@ -4,7 +4,6 @@
package container
import (
"bufio"
"bytes"
"cmp"
"compress/gzip"
@@ -554,6 +553,10 @@ func (c *kubernetesContainer) Copy(destPath string, files ...*FileEntry) common.
}
}
func (c *kubernetesContainer) CopyTarStream(ctx context.Context, destPath string, write func(*TarWriter) error) error {
return c.extract(ctx, destPath, true, func(writer io.Writer) error { return writeTarStream(writer, 0, 0, "", nil, write) }) // streams carry large content like artifacts
}
func (c *kubernetesContainer) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
return func(ctx context.Context) error {
return c.extract(ctx, destPath, true, func(writer io.Writer) error { // directories are actions and workspaces, megabytes of compressible sources
@@ -562,26 +565,18 @@ func (c *kubernetesContainer) CopyDir(destPath, srcPath string, useGitIgnore, sk
}
}
// GetContainerArchive streams the archive, checking the path first as tar writes an empty archive for a missing one.
func (c *kubernetesContainer) ExecOutput(ctx context.Context, command []string) (io.ReadCloser, error) {
return streamOutput(ctx, func(ctx context.Context, stdout, stderr io.Writer) error {
return c.pod.client.exec(ctx, c.pod.name, c.name, command, nil, stdout, stderr)
}), nil
}
func (c *kubernetesContainer) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
reader, writer := io.Pipe()
go func() {
var stderr bytes.Buffer
err := c.pod.client.exec(ctx, c.pod.name, c.name, []string{"sh", "-c", `test -e "$1" -o -L "$1" && exec tar -cf - -C "$2" "$3"`, "sh", srcPath, path.Dir(srcPath), path.Base(srcPath)}, nil, writer, &stderr)
if err != nil {
err = fmt.Errorf("archive %s: %w: %s", srcPath, err, strings.TrimSpace(stderr.String()))
}
writer.CloseWithError(err)
}()
buffered := bufio.NewReader(reader)
if _, err := buffered.Peek(1); err != nil {
reader.Close()
reader, err := c.ExecOutput(ctx, []string{"sh", "-c", `test -e "$1" -o -L "$1" && exec tar -cf - -C "$2" "$3"`, "sh", srcPath, path.Dir(srcPath), path.Base(srcPath)})
if err != nil {
return nil, err
}
return struct {
io.Reader
io.Closer
}{buffered, reader}, nil
return startedArchive(reader)
}
func (c *kubernetesContainer) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
+24 -1
View File
@@ -273,6 +273,17 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
assert.Equal(t, "hunter2\nit's\nme\n5\n"+workdir+"\n", jobLog.String())
assert.Equal(t, ExitCodeError(3), job.Exec([]string{"sh", "-c", "echo failing >&2; exit 3"}, nil, "", "")(ctx))
assert.Contains(t, jobLog.String(), "failing\n")
output, err := job.ExecOutput(ctx, []string{"sh", "-c", `printf 'out\000'; printf ' failed \n' >&2; exit 3`})
require.NoError(t, err)
content, err := io.ReadAll(output)
var exit ExitCodeError
require.ErrorAs(t, err, &exit)
assert.Equal(t, ExitCodeError(3), exit)
assert.Contains(t, err.Error(), ": failed")
assert.Equal(t, "out\x00", string(content))
require.ErrorAs(t, output.Close(), &exit)
assert.Equal(t, ExitCodeError(3), exit)
for _, query := range cluster.requests["exec"] {
assert.NotContains(t, query, "hunter2")
}
@@ -285,7 +296,7 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
reader := tar.NewReader(archive)
header, err := reader.Next()
require.NoError(t, err)
content, err := io.ReadAll(reader)
content, err = io.ReadAll(reader)
require.NoError(t, err)
assert.Equal(t, "event.json", header.Name)
assert.JSONEq(t, `{"ref":"main"}`, string(content))
@@ -309,6 +320,18 @@ func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
assert.Contains(t, lastExec(), "command="+url.QueryEscape(workdir+"/act/actions/sha/"))
assert.Contains(t, lastExec(), url.QueryEscape("gzip -dc |"))
require.NoError(t, job.CopyTarStream(ctx, workdir+"/download/new", func(tw *TarWriter) error {
if err := tw.WriteHeader(&tar.Header{Typeflag: tar.TypeReg, Name: "dir/data.txt", Mode: 0o644, Size: 7}); err != nil {
return err
}
_, err := tw.Write([]byte("payload"))
return err
}))
downloaded, err := os.ReadFile(workdir + "/download/new/dir/data.txt")
require.NoError(t, err)
assert.Equal(t, "payload", string(downloaded))
assert.Contains(t, lastExec(), url.QueryEscape("gzip -dc |"))
pathEnv := map[string]string{}
require.NoError(t, job.UpdateFromImageEnv(&pathEnv)(ctx))
assert.Equal(t, path, pathEnv["PATH"])
+17 -15
View File
@@ -5,6 +5,7 @@ package container
import (
"bufio"
"bytes"
"os"
"path/filepath"
"strings"
@@ -15,14 +16,9 @@ import (
"golang.org/x/text/encoding/unicode"
)
func newTestHostEnv(t *testing.T) (*HostEnvironment, string) {
t.Helper()
e := &HostEnvironment{Path: t.TempDir()}
return e, filepath.Join(e.Path, "envfile")
}
func TestParseEnvFile(t *testing.T) {
t.Parallel()
hostEnv := &HostEnvironment{Path: t.TempDir()}
largeValue := strings.Repeat("x", 2*1024*1024)
encodeUTF16 := func(endianness unicode.Endianness) string {
content, err := unicode.UTF16(endianness, unicode.UseBOM).NewEncoder().Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n"))
@@ -51,15 +47,15 @@ func TestParseEnvFile(t *testing.T) {
{"UTF16 big endian", encodeUTF16(unicode.BigEndian), map[string]string{"FOO": "bar\r", "MULTI": "line1\r"}},
} {
t.Run(test.name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
envPath := filepath.Join(hostEnv.Path, test.name)
require.NoError(t, os.WriteFile(envPath, []byte(test.content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env, false)(t.Context()))
require.NoError(t, parseEnvFile(hostEnv, envPath, &env, false)(t.Context()))
assert.Equal(t, test.want, env)
})
}
t.Run("ordered entries preserve repeated names", func(t *testing.T) {
hostEnv, envPath := newTestHostEnv(t)
envPath := filepath.Join(hostEnv.Path, "ordered")
require.NoError(t, os.WriteFile(envPath, []byte("FOO=first\r\nfoo<<EOF\r\nmiddle\r\nEOF\r\nFOO=last\r\n"), 0o600))
var entries [][2]string
require.NoError(t, ParseEnvFile(hostEnv, envPath, func(name, value string) {
@@ -70,7 +66,7 @@ func TestParseEnvFile(t *testing.T) {
}, entries)
})
t.Run("Windows line endings", func(t *testing.T) {
hostEnv, envPath := newTestHostEnv(t)
envPath := filepath.Join(hostEnv.Path, "windows")
require.NoError(t, os.WriteFile(envPath, []byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nline2\nEOF\r\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(hostEnv, envPath, &env, true)(t.Context()))
@@ -80,16 +76,22 @@ func TestParseEnvFile(t *testing.T) {
func TestParseEnvFileLineExceedsBufferReportsScannerError(t *testing.T) {
t.Parallel()
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n"+strings.Repeat("x", 17*1024*1024)+"\nEOF\n"), 0o600))
hostEnv := &HostEnvironment{Path: t.TempDir()}
envPath := filepath.Join(hostEnv.Path, "envfile")
const header, footer = "FOO<<EOF\n", "\nEOF\n"
content := bytes.Repeat([]byte("x"), len(header)+17*1024*1024+len(footer))
copy(content, header)
copy(content[len(content)-len(footer):], footer)
require.NoError(t, os.WriteFile(envPath, content, 0o600))
env := map[string]string{}
err := parseEnvFile(e, envPath, &env, false)(t.Context())
err := parseEnvFile(hostEnv, envPath, &env, false)(t.Context())
require.ErrorIs(t, err, bufio.ErrTooLong)
assert.Contains(t, err.Error(), "reading env file")
}
func TestParseEnvFileInvalidHeredoc(t *testing.T) {
hostEnv := &HostEnvironment{Path: t.TempDir()}
for name, content := range map[string]string{
"missing delimiter": "FOO<<EOF\nline1\nline2\n",
"empty name": "<<EOF\nline1\nEOF\n",
@@ -97,11 +99,11 @@ func TestParseEnvFileInvalidHeredoc(t *testing.T) {
"delimiter with carriage return": "FOO<<EOF\nline1\nEOF\r\n",
} {
t.Run(name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
envPath := filepath.Join(hostEnv.Path, name)
require.NoError(t, os.WriteFile(envPath, []byte("APPLIED=before-error\n"+content), 0o600))
env := map[string]string{}
err := parseEnvFile(e, envPath, &env, false)(t.Context())
err := parseEnvFile(hostEnv, envPath, &env, false)(t.Context())
require.Error(t, err)
assert.Contains(t, err.Error(), "delimiter")
assert.Equal(t, map[string]string{"APPLIED": "before-error"}, env)
+83
View File
@@ -6,8 +6,13 @@ package container
import (
"archive/tar"
"bufio"
"context"
"errors"
"fmt"
"io"
"io/fs"
"path"
"path/filepath"
"strings"
@@ -15,6 +20,84 @@ import (
"gitea.com/gitea/runner/act/filecollector"
)
// TarWriter takes directories and files named relative to the destination, owned by the job user.
type TarWriter struct {
writer *tar.Writer
uid, gid int
prefix string
dirs map[string]bool
}
func (w *TarWriter) Write(data []byte) (int, error) {
return w.writer.Write(data)
}
func (w *TarWriter) WriteHeader(hdr *tar.Header) error {
name := strings.TrimSuffix(hdr.Name, "/")
if !fs.ValidPath(name) {
return fmt.Errorf("tar entry %q escapes the destination", hdr.Name)
}
if hdr.Typeflag != tar.TypeDir && hdr.Typeflag != tar.TypeReg {
return fmt.Errorf("tar entry %q: unsupported type %q", hdr.Name, hdr.Typeflag)
}
for index := range len(name) {
if dir := name[:index]; name[index] == '/' && !w.dirs[dir] {
if err := w.write(&tar.Header{Typeflag: tar.TypeDir, Name: path.Join(w.prefix, dir), Mode: 0o755}); err != nil {
return err
}
w.dirs[dir] = true
}
}
if hdr.Typeflag == tar.TypeDir {
w.dirs[name] = true
}
entry := *hdr
entry.Name = path.Join(w.prefix, name)
return w.write(&entry)
}
func (w *TarWriter) write(hdr *tar.Header) error {
hdr.Uid, hdr.Gid, hdr.Uname, hdr.Gname = w.uid, w.gid, "", "" // tar running as root prefers names over ids
if hdr.Typeflag == tar.TypeDir {
hdr.Name += "/"
}
return w.writer.WriteHeader(hdr)
}
func writeTarStream(dst io.Writer, uid, gid int, prefix string, parents []string, write func(*TarWriter) error) error {
tw := &TarWriter{writer: tar.NewWriter(dst), uid: uid, gid: gid, prefix: prefix, dirs: map[string]bool{}}
for _, name := range parents {
if err := tw.write(&tar.Header{Typeflag: tar.TypeDir, Name: name, Mode: 0o755}); err != nil {
return err
}
}
if err := write(tw); errors.Is(err, io.EOF) {
return fmt.Errorf("tar stream: %v", err) // %v, as a wrapped io.EOF reads as the archive's end
} else if err != nil {
return err
}
return tw.writer.Close()
}
func pipeTar(uid, gid int, prefix string, parents []string, write func(*TarWriter) error) *io.PipeReader {
reader, writer := io.Pipe()
go func() { writer.CloseWithError(writeTarStream(writer, uid, gid, prefix, parents, write)) }()
return reader
}
// startedArchive returns reader once its first byte arrives, so failing to start archiving errors here, not on read.
func startedArchive(reader io.ReadCloser) (io.ReadCloser, error) {
buffered := bufio.NewReader(reader)
if _, err := buffered.Peek(1); err != nil {
reader.Close()
return nil, err
}
return struct {
io.Reader
io.Closer
}{buffered, reader}, nil
}
func writeFilesTar(ctx context.Context, w io.Writer, uid, gid int, files ...*FileEntry) error {
tw := tar.NewWriter(w)
for _, file := range files {
+9 -7
View File
@@ -28,10 +28,12 @@ func gitCmd(t *testing.T, args ...string) {
}
func TestCollectFilesKeepsWhatGitTracksOrDoesNotIgnore(t *testing.T) {
repoDir := filepath.Join(t.TempDir(), "mygitrepo")
for _, dir := range []string{repoDir, filepath.Join(repoDir, "sub"), filepath.Join(repoDir, "nested")} {
gitCmd(t, "init", "-q", dir)
repoDir := t.TempDir()
gitCmd(t, "init", "-q", "--template=", repoDir)
for _, dir := range []string{"sub", "nested"} {
require.NoError(t, os.CopyFS(filepath.Join(repoDir, dir, ".git"), os.DirFS(filepath.Join(repoDir, ".git"))))
}
require.NoError(t, os.Mkdir(filepath.Join(repoDir, "ign"), 0o755))
for name, body := range map[string]string{
".gitignore": ".*\nign/\n",
".env": "",
@@ -43,15 +45,15 @@ func TestCollectFilesKeepsWhatGitTracksOrDoesNotIgnore(t *testing.T) {
"sub/kept": "",
"nested/kept": "",
} {
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(repoDir, name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(repoDir, name), []byte(body), 0o644))
}
require.NoError(t, os.Mkdir(filepath.Join(repoDir, "uninitialized"), 0o755))
gitCmd(t, "-C", repoDir, "add", "-f", ".gitignore", "Renamed", "ign/tracked")
require.NoError(t, os.Rename(filepath.Join(repoDir, "Renamed"), filepath.Join(repoDir, "renamed")))
for _, gitlink := range []string{"sub", "uninitialized"} {
gitCmd(t, "-C", repoDir, "update-index", "--add", "--cacheinfo", "160000,4b825dc642cb6eb9a060e54bf8d69288fbee4904,"+gitlink)
}
gitCmd(t, "-C", repoDir, "update-index", "--add",
"--cacheinfo", "160000,4b825dc642cb6eb9a060e54bf8d69288fbee4904,sub",
"--cacheinfo", "160000,4b825dc642cb6eb9a060e54bf8d69288fbee4904,uninitialized",
)
expected := map[string]string{".gitignore": "", "ign/tracked": "", "nested/kept": "", "renamed": "", "sub/.gitignore": "", "sub/kept": ""}
if runtime.GOOS != "windows" {
require.NoError(t, os.Chmod(filepath.Join(repoDir, "renamed"), 0o777|os.ModeSticky))
+21 -1
View File
@@ -11,6 +11,8 @@ import (
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.com/gitea/runner/internal/pkg/action"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
@@ -92,7 +94,7 @@ func TestCompositeActionParity(t *testing.T) {
return "", "", assert.AnError
})
rc := &RunContext{
Config: &Config{GitHubInstance: "github.com", ActionCacheDir: t.TempDir(), PresetGitHubContext: &model.GithubContext{}},
Config: &Config{GitHubInstance: "github.com", PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
JobContainer: &jobContainerMock{},
}
@@ -182,3 +184,21 @@ func newTestCompositeRunContext(t *testing.T, parent *RunContext) *RunContext {
require.NoError(t, err)
return composite
}
func TestBuiltinActionGetsLowercaseInputsAndTheJobTokenInComposites(t *testing.T) {
job := makeTestRC(t, "runs-on: ubuntu-latest")
job.Config.Secrets = map[string]string{"ACTIONS_RUNTIME_TOKEN": "daemon-token"}
job.JobContainer = &container.HostEnvironment{Path: t.TempDir(), ActPath: t.TempDir(), Workdir: t.TempDir()}
composite := newTestCompositeRunContext(t, job)
composite.JobContainer = job.JobContainer
for _, rc := range []*RunContext{job, composite} {
var got *action.Context
sab := &stepActionBuiltin{Step: &model.Step{ID: "x", With: map[string]string{"Name": "x"}}, RunContext: rc, run: func(_ctx context.Context, c *action.Context) error {
got = c
return nil
}}
require.NoError(t, sab.main()(t.Context()))
assert.Equal(t, "daemon-token", got.RuntimeToken)
assert.Equal(t, map[string]string{"name": "x"}, got.Inputs)
}
}
+17 -13
View File
@@ -249,6 +249,7 @@ func TestActionRunner(t *testing.T) {
}
func TestGoActionUsesLinuxExecutablePaths(t *testing.T) {
actionRoot := t.TempDir()
for _, stage := range []string{"main", "pre", "post"} {
t.Run(stage, func(t *testing.T) {
cm := &containerMock{}
@@ -264,7 +265,7 @@ func TestGoActionUsesLinuxExecutablePaths(t *testing.T) {
JobContainer: cm,
},
action: &model.Action{Runs: model.ActionRuns{Using: model.ActionRunsUsingGo, Main: "main.go", Pre: "pre.go", Post: "post.go"}},
actionDir: t.TempDir(),
actionDir: filepath.Join(actionRoot, stage),
env: map[string]string{},
remoteAction: mustNewRemoteAction(t, "org/repo@ref"),
}
@@ -504,11 +505,21 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
}
func TestExecAsDockerImageChecks(t *testing.T) {
actionRoot := t.TempDir()
originalExists, originalBuild, originalContainer := ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer
t.Cleanup(func() {
ContainerImageExistsLocally, ContainerNewDockerBuildExecutor, ContainerNewContainer = originalExists, originalBuild, originalContainer
})
ContainerNewContainer = func(*container.NewContainerInput) container.ExecutionsEnvironment { return fakeContainer{} }
var removed bool
fakeDockerDaemon(t, func(writer http.ResponseWriter, request *http.Request) {
if request.Method == http.MethodDelete {
removed = true
_, _ = io.WriteString(writer, `[]`)
} else {
_, _ = io.WriteString(writer, `{"Architecture":"aarch64"}`)
}
})
for _, testcase := range []struct {
name, platform string
@@ -529,18 +540,11 @@ func TestExecAsDockerImageChecks(t *testing.T) {
checks = append(checks, platform)
return testcase.exists && (platform == "any" || testcase.matches), nil
}
var built, removed bool
var built bool
removed = false
ContainerNewDockerBuildExecutor = func(container.NewDockerBuildExecutorInput) common.Executor {
return func(context.Context) error { built = true; return nil }
}
fakeDockerDaemon(t, func(writer http.ResponseWriter, request *http.Request) {
if request.Method == http.MethodDelete {
removed = true
_, _ = io.WriteString(writer, `[]`)
} else {
_, _ = io.WriteString(writer, `{"Architecture":"aarch64"}`)
}
})
step := &stepActionRemote{
Step: &model.Step{ID: "action"},
RunContext: &RunContext{
@@ -555,7 +559,7 @@ func TestExecAsDockerImageChecks(t *testing.T) {
action: &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "Dockerfile"}},
env: map[string]string{},
}
dir := t.TempDir()
dir := filepath.Join(actionRoot, testcase.name)
require.NoError(t, execAsDocker(t.Context(), step, "action", dir, dir, false, stepStageMain))
assert.Equal(t, testcase.wantChecks, checks)
assert.Equal(t, testcase.wantBuild, built)
@@ -727,9 +731,9 @@ func TestExecAsDockerStageEntrypoint(t *testing.T) {
cm.On("Close").Return(func(context.Context) error { return nil })
if tc.stage == stepStageMain {
require.NoError(t, runActionImpl(step, t.TempDir(), step.remoteAction)(ctx))
require.NoError(t, runActionImpl(step, "", step.remoteAction)(ctx))
} else {
require.NoError(t, execAsDocker(ctx, step, "action", t.TempDir(), t.TempDir(), false, tc.stage))
require.NoError(t, execAsDocker(ctx, step, "action", "", "", false, tc.stage))
}
if tc.warning == "" {
assert.Empty(t, hook.AllRecords())
+5
View File
@@ -193,6 +193,11 @@ func TestCommandLogWriter(t *testing.T) {
log []string
masks []string
}{
{
name: "mask commands decode modern and legacy escapes once",
steps: [][]string{{"::add-mask::%25", "##[add-mask]alpha beta%3Bgamma%5Ddelta%250A"}},
masks: []string{"%", "alpha beta;gamma]delta%0A"},
},
{
name: "applied commands are hidden unless echoed",
steps: [][]string{{
+10
View File
@@ -66,6 +66,12 @@ func (cm *containerMock) Exec(command []string, env map[string]string, user, wor
return cm.Called(command, env, user, workdir).Get(0).(func(context.Context) error)
}
func (cm *containerMock) ExecOutput(ctx context.Context, command []string) (io.ReadCloser, error) {
args := cm.Called(ctx, command)
output, _ := args.Get(0).(io.ReadCloser)
return output, args.Error(1)
}
func (cm *containerMock) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
args := cm.Called(ctx, srcPath)
err, _ := args.Get(1).(error)
@@ -82,3 +88,7 @@ func (cm *containerMock) Inspect(ctx context.Context) (*container.Info, error) {
err, _ := args.Get(1).(error)
return info, err
}
func (cm *containerMock) CopyTarStream(ctx context.Context, destPath string, write func(*container.TarWriter) error) error {
return cm.Called(ctx, destPath, write).Error(0)
}
@@ -49,6 +49,7 @@ func TestDockerProxyMountedJob(t *testing.T) {
ContainerNamePrefix: resourceName,
ContainerDaemonSocket: dockerClient.DaemonHost(),
ContainerMaxLifetime: 2 * time.Minute,
Sweep: true,
ContainerOptions: "-v /usr/local/bin/docker:/usr/local/bin/docker:ro -v /usr/local/libexec/docker/cli-plugins:/usr/local/libexec/docker/cli-plugins:ro",
ValidVolumes: []string{"/usr/local/bin/docker", "/usr/local/libexec/docker/cli-plugins"},
Env: map[string]string{"PROXY_TEST_RESOURCE": resourceName, "PROXY_TEST_MODE": mode, "PROXY_TEST_IMAGE": baseImage, "COMPOSE_PROJECT_NAME": resourceName},
+2 -3
View File
@@ -412,11 +412,10 @@ func TestGetEvaluatorInputsBoolean(t *testing.T) {
}
for _, eventName := range []string{"workflow_call", "workflow_dispatch"} {
wf, err := model.ReadWorkflow(strings.NewReader(workflow(eventName)))
require.NoError(t, err)
for _, table := range tables {
t.Run(eventName+"/"+table.name, func(t *testing.T) {
wf, err := model.ReadWorkflow(strings.NewReader(workflow(eventName)))
require.NoError(t, err)
rc := &RunContext{
Config: &Config{Workdir: "."},
Run: &model.Run{JobID: "job1", Workflow: wf},
+4 -3
View File
@@ -993,6 +993,7 @@ func tarArchive(t *testing.T, entries ...tarEntry) []byte {
buf := &bytes.Buffer{}
tw := tar.NewWriter(buf)
for _, entry := range entries {
buf.Grow(len(entry.body) + 3*512)
typeflag := entry.typeflag
if typeflag == 0 {
typeflag = tar.TypeReg
@@ -1009,7 +1010,7 @@ func tarArchive(t *testing.T, entries ...tarEntry) []byte {
}
require.NoError(t, tw.WriteHeader(header))
if typeflag == tar.TypeReg {
_, err := tw.Write([]byte(entry.body))
_, err := io.CopyN(tw, strings.NewReader(entry.body), int64(len(entry.body)))
require.NoError(t, err)
}
}
@@ -1149,12 +1150,12 @@ func TestTryUploadJobSummaryMasksSecrets(t *testing.T) {
}))
defer server.Close()
content, err := testcase.encoding.NewEncoder().Bytes([]byte(testcase.content))
content, err := testcase.encoding.NewEncoder().String(testcase.content)
require.NoError(t, err)
cm := &containerMock{}
cm.On("GetContainerArchive", mock.Anything, "/var/run/act/workflow/step-summary-0.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{
name: "step-summary-0.md", body: string(content),
name: "step-summary-0.md", body: content,
}))),
nil,
).Once()
+5
View File
@@ -71,6 +71,11 @@ func (c *hookContainer) UpdateFromEnv(_ string, env *map[string]string) common.E
}
}
func (c *hookContainer) ExecOutput(_ctx context.Context, command []string) (io.ReadCloser, error) {
c.cmd = command
return io.NopCloser(bytes.NewBufferString(c.output)), c.err
}
func (c *hookContainer) GetContainerArchive(context.Context, string) (io.ReadCloser, error) {
return io.NopCloser(bytes.NewReader(c.pathTar)), nil
}
+37 -9
View File
@@ -263,29 +263,27 @@ func AppendSecretMaskers(oldnew []string, secrets map[string]string) []string {
return oldnew
}
// AppendSecretMasker registers v and each of its lines, as GitHub does.
// AppendSecretMasker skips one-byte line fragments to avoid masking punctuation from structured secrets.
func AppendSecretMasker(oldnew []string, v string) []string {
ret := appendMaskedValue(oldnew, v)
for l := range strings.FieldsFuncSeq(v, func(r rune) bool { return r == '\r' || r == '\n' }) {
ret = appendMaskedValue(ret, strings.TrimSpace(l))
if line := strings.TrimSpace(l); len(line) > 1 {
ret = appendMaskedValue(ret, line)
}
}
return ret
}
// appendMaskedValue registers one value and every shape it takes on its way into a log.
func appendMaskedValue(ret []string, v string) []string {
// formatted JSON secrets could otherwise mask {,[,],} everywhere
if len(strings.TrimSpace(v)) <= 1 || slices.Contains(ret, v) {
if strings.TrimSpace(v) == "" || slices.Contains(ret, v) {
return ret
}
ret = append(ret, v, "***")
// command data reaches the log escaped, so "pass%word" also arrives as "pass%25word"
if strings.ContainsAny(v, "%\r\n") {
ret = append(ret, EscapeCommandData(v), "***")
}
for _, encode := range secretValueEncoders {
encoded := encode(v)
// An encoding that leaves the value unchanged is already masked above.
if encoded == v || len(encoded) <= 1 || slices.Contains(ret, encoded) {
continue
}
@@ -319,7 +317,7 @@ func valueMasker(insecureSecrets bool, oldnew []string) entryProcessor {
masks := Masks(ctx)
if len(*masks) == 0 {
r.Message = defReplacer.Replace(r.Message)
r.Message = MaskLog(r.Message, defReplacer)
return r
}
@@ -339,12 +337,42 @@ func valueMasker(insecureSecrets bool, oldnew []string) entryProcessor {
cmasker := replacer
mu.Unlock()
r.Message = cmasker.Replace(r.Message)
r.Message = MaskLog(r.Message, cmasker)
return r
}
}
// MaskLog decodes command data before masking to cover partially escaped values.
func MaskLog(message string, replacer *strings.Replacer) string {
var command, data string
var found bool
unescape := UnescapeCommandData
if strings.HasPrefix(message, "##[") {
command, data, found = strings.Cut(message[3:], "]")
unescape = UnescapeLegacyCommand
} else if strings.HasPrefix(message, "::") {
command, data, found = strings.Cut(message[2:], "::")
if name, _, properties := strings.Cut(command, " "); properties {
switch name {
case "error", "warning", "notice", "debug":
command = name
}
}
}
switch command {
case "error", "warning", "notice", "debug", "group", "endgroup", "add-matcher":
if found {
trimmedData := strings.TrimRight(data, "\r\n")
normalized := message[:len(message)-len(data)] + EscapeCommandData(unescape(trimmedData)) + data[len(trimmedData):]
if masked := replacer.Replace(normalized); masked != normalized {
return masked
}
}
}
return replacer.Replace(message)
}
type maskedHandler struct {
logging.Handler
masker entryProcessor
+61 -4
View File
@@ -28,6 +28,7 @@ func TestValueMasker(t *testing.T) {
secrets map[string]string
masks []string
disallowed []string
want string
}{
{
name: "Multiline Private Key",
@@ -49,6 +50,60 @@ func TestValueMasker(t *testing.T) {
secrets: map[string]string{"TOKEN": "pass%word"},
disallowed: []string{"pass%25word"},
},
{
name: "one-character mask",
lines: "%",
masks: []string{"%"},
want: "***",
},
{
name: "fully escaped legacy secret",
lines: "##[group]alpha beta%3Bgamma%5Ddelta",
secrets: map[string]string{"TOKEN": "alpha beta;gamma]delta"},
want: "##[group]***",
},
{
name: "partially escaped legacy mask",
lines: "##[group]alpha beta%3Bgamma]delta",
masks: []string{"alpha beta;gamma]delta"},
want: "##[group]***",
},
{
name: "partially escaped modern mask preserves unrelated escapes",
lines: "::group::alpha%25beta%gamma tail%250A%0A",
masks: []string{"alpha%beta%gamma"},
want: "::group::*** tail%250A%0A",
},
{
name: "command escapes are decoded once",
lines: "##[group]alpha%253Bbeta%5Dgamma",
masks: []string{"alpha;beta]gamma"},
want: "##[group]alpha%253Bbeta%5Dgamma",
},
{
name: "plain output preserves escapes",
lines: "alpha beta%3Bgamma%5Ddelta",
masks: []string{"alpha beta;gamma]delta"},
want: "alpha beta%3Bgamma%5Ddelta",
},
{
name: "command display preserves literal escapes around a mask",
lines: "##[command]alpha tail%3Bvalue%5D",
masks: []string{"alpha"},
want: "##[command]*** tail%3Bvalue%5D",
},
{
name: "unknown modern commands preserve literal escapes",
lines: "::unknown::alpha tail%25value",
masks: []string{"alpha"},
want: "::unknown::*** tail%25value",
},
{
name: "masks can span command headers and data",
lines: "::group::hidden other",
masks: []string{"group::hidden", "other"},
want: "::*** ***",
},
}
for _, entry := range table {
t.Run(entry.name, func(t *testing.T) {
@@ -56,6 +111,9 @@ func TestValueMasker(t *testing.T) {
masker := valueMasker(false, AppendSecretMaskers(nil, entry.secrets))
for line := range strings.SplitSeq(entry.lines, "\n") {
lentry := masker(ctx, logging.Record{Message: line})
if entry.want != "" {
assert.Equal(t, entry.want, lentry.Message)
}
for _, line := range entry.disallowed {
assert.NotContains(t, lentry.Message, line)
}
@@ -139,6 +197,7 @@ func TestValueMaskerEncodedMasks(t *testing.T) {
{"encoded: " + base64.StdEncoding.EncodeToString([]byte(" s3cr3t ")), "encoded: ***"},
{"encoded: " + base64.StdEncoding.EncodeToString([]byte("s3cr3t")), "encoded: ***"},
{"encoded: " + base64.StdEncoding.EncodeToString([]byte("first")), "encoded: ***"},
{"##[group]s3cr3t value\r\n", "##[group]***\r\n"},
} {
entry := masker(WithMasks(t.Context(), &masks), logging.Record{Message: tc.line})
assert.Equal(t, tc.want, entry.Message, tc.line)
@@ -194,17 +253,15 @@ func TestValueMaskerCachedReplacerSeesNewMasks(t *testing.T) {
}
func TestAppendSecretMaskerSkipsUselessEncodings(t *testing.T) {
// A token with no character an escape would touch only gains its base64 forms:
// JSON, query and path escaping all leave it unchanged.
pairs := AppendSecretMasker(nil, "plaintoken")
assert.Equal(t, []string{
"plaintoken", "***", "cGxhaW50b2tlbg==", "***", "bGFpbnRva2Vu", "***", "YWludG9rZW4=", "***",
"aW50b2tl", "***", "YWludG9r", "***", "bGFpbnRv", "***",
}, pairs)
// Too short to mask.
assert.Empty(t, AppendSecretMasker(nil, "x"))
assert.Contains(t, AppendSecretMasker(nil, "x"), "x")
assert.Empty(t, AppendSecretMasker(nil, " \t"))
assert.NotContains(t, AppendSecretMasker(nil, "{\nvalue\n}"), "{")
assert.NotContains(t, AppendSecretMasker(nil, `"123456"`), "123456")
}
+7 -6
View File
@@ -4,6 +4,7 @@
package runner
import (
"bytes"
"context"
"encoding/base64"
"io"
@@ -209,14 +210,14 @@ func TestPatchedBundleAcrossActions(t *testing.T) {
assert.Equal(t, data, out, "an untouched bundle must come back byte for byte")
return
}
assert.NotContains(t, string(out), ".LOCALHOST", "a copy of the gate was missed")
assert.False(t, bytes.Contains(out, []byte(".LOCALHOST")), "a copy of the gate was missed")
if !strings.Contains(string(data), CacheServiceV2Env) {
if !bytes.Contains(data, []byte(CacheServiceV2Env)) {
return
}
assert.NotZero(t, strings.Count(string(out), "(process.env."+cacheURLEnv+"||process.env"),
assert.NotZero(t, bytes.Count(out, []byte("(process.env."+cacheURLEnv+"||process.env")),
"the cache service URL was not redirected")
assert.Equal(t, strings.Count(string(data), resultsURLEnv), strings.Count(string(out), resultsURLEnv),
assert.Equal(t, bytes.Count(data, []byte(resultsURLEnv)), bytes.Count(out, []byte(resultsURLEnv)),
"a read of the results URL was lost, it must stay as the fallback")
})
}
@@ -311,18 +312,18 @@ func TestCodeSplitActionFindsTheCacheService(t *testing.T) {
requireHostTools(t, "node", "rustc")
dir := tempDirPath(t)
require.NoError(t, os.Mkdir(filepath.Join(dir, "dist"), 0o755))
for _, name := range []string{
"action.yml", "dist/restore.js", "dist/save.js", "dist/cleanup-BWEbZ6YT.js",
"dist/cache-CTfQoCTL.js", "dist/cache-DC26rUaF.js", "dist/state-cjs-CkvFUaNy.js",
} {
body, err := os.ReadFile(bundleFromGitHub(t, "Swatinem/rust-cache", "v2.9.2", name))
require.NoError(t, err)
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), body, 0o600))
}
body, err := os.ReadFile(filepath.Join(dir, "action.yml"))
require.NoError(t, err)
action, err := model.ReadAction(strings.NewReader(string(body)))
action, err := model.ReadAction(bytes.NewReader(body))
require.NoError(t, err)
inputs := map[string]string{"lookup-only": "true", "add-rust-environment-hash-key": "false", "cache-bin": "false"}
for name, input := range action.Inputs {
+14 -12
View File
@@ -4,6 +4,7 @@
package runner
import (
"bytes"
"context"
"os"
"os/exec"
@@ -182,9 +183,9 @@ func TestPatchedBundleSurvivesInsideAStringLiteral(t *testing.T) {
out, patched := patchedBundle([]byte(embedded))
require.True(t, patched)
file := filepath.Join(t.TempDir(), "bundle.js")
require.NoError(t, os.WriteFile(file, out, 0o600))
checked, err := exec.CommandContext(t.Context(), "node", "--check", file).CombinedOutput()
cmd := exec.CommandContext(t.Context(), "node", "--check", "-")
cmd.Stdin = bytes.NewReader(out)
checked, err := cmd.CombinedOutput()
require.NoError(t, err, "%s", checked)
}
@@ -256,6 +257,14 @@ func TestActionScriptPaths(t *testing.T) {
}
func TestPatchActionsAtTheContainerCopy(t *testing.T) {
actionDir := t.TempDir()
script := filepath.Join(actionDir, "dist", "cache.js")
require.NoError(t, os.MkdirAll(filepath.Dir(script), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(filepath.Dir(script), "index.js"), []byte("require('./cache.js')"), 0o600))
require.NoError(t, os.WriteFile(script, []byte(gateTSC), 0o600))
info, err := os.Stat(script)
require.NoError(t, err)
copiedBundles := func(t *testing.T, noPatch bool) ([]string, []*container.FileEntry, int64) {
t.Helper()
@@ -264,19 +273,12 @@ func TestPatchActionsAtTheContainerCopy(t *testing.T) {
Step: &model.Step{Uses: "owner/repo/sub@v1"},
remoteAction: &remoteAction{Org: "owner", Repo: "repo", Path: "sub", Ref: "v1"},
action: &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "../dist/index.js"}},
actionDir: t.TempDir(),
actionDir: actionDir,
RunContext: &RunContext{
Config: &Config{ActionCacheDir: t.TempDir(), NoActionPatch: noPatch},
Config: &Config{NoActionPatch: noPatch},
JobContainer: cm,
},
}
script := filepath.Join(sar.actionDir, "dist", "cache.js")
require.NoError(t, os.MkdirAll(filepath.Dir(script), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(filepath.Dir(script), "index.js"), []byte("require('./cache.js')"), 0o600))
require.NoError(t, os.WriteFile(script, []byte(gateTSC), 0o600))
info, err := os.Stat(script)
require.NoError(t, err)
var calls []string
var copied []*container.FileEntry
cm.On("CopyDir", "/var/run/act/actions/repo/", sar.actionDir+"/", false, true).Return(func(context.Context) error {
+8 -16
View File
@@ -33,24 +33,17 @@ func TestReusableWorkflowCachedBranchRefRefreshes(t *testing.T) {
}
remoteDir := t.TempDir()
gitMust(t, "", "init", "--bare", "--initial-branch=master", remoteDir)
workDir := t.TempDir()
gitMust(t, "", "clone", remoteDir, workDir)
gitMust(t, workDir, "config", "user.email", "test@test")
gitMust(t, workDir, "config", "user.name", "test")
gitMust(t, workDir, "checkout", "-b", "master")
gitMust(t, "", "init", "--template=", "--initial-branch=master", remoteDir)
const workflowPath = ".gitea/workflows/reusable.yml"
tmpl := func(tag string) string {
return "name: reusable\non:\n workflow_call:\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - run: echo " + tag + "\n"
}
require.NoError(t, os.MkdirAll(filepath.Join(workDir, ".gitea/workflows"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(workDir, workflowPath), []byte(tmpl("v1")), 0o644))
gitMust(t, workDir, "add", workflowPath)
gitMust(t, workDir, "commit", "-m", "v1")
gitMust(t, workDir, "push", "-u", "origin", "master")
require.NoError(t, os.MkdirAll(filepath.Join(remoteDir, ".gitea/workflows"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(remoteDir, workflowPath), []byte(tmpl("v1")), 0o644))
gitMust(t, remoteDir, "add", workflowPath)
gitMust(t, remoteDir, "commit", "-m", "v1")
rc := &RunContext{
Config: &Config{Env: testCheckoutEnv()},
@@ -70,9 +63,8 @@ func TestReusableWorkflowCachedBranchRefRefreshes(t *testing.T) {
require.Equal(t, tmpl("v1"), string(got))
// Branch tip moves; cache key (cacheDir) does not.
require.NoError(t, os.WriteFile(filepath.Join(workDir, workflowPath), []byte(tmpl("v2")), 0o644))
gitMust(t, workDir, "commit", "-am", "v2")
gitMust(t, workDir, "push", "origin", "master")
require.NoError(t, os.WriteFile(filepath.Join(remoteDir, workflowPath), []byte(tmpl("v2")), 0o644))
gitMust(t, remoteDir, "commit", "-am", "v2")
require.NoError(t, cloneRemoteReusableWorkflow(rc, remoteDir, "master", cacheDir)(t.Context()))
got, err = os.ReadFile(filepath.Join(cacheDir, workflowPath))
@@ -214,7 +206,7 @@ func TestInstanceAuthAndTransportOnlyApplyToTheInstance(t *testing.T) {
func gitMust(t *testing.T, dir string, args ...string) {
t.Helper()
cmd := exec.Command("git", args...)
cmd := exec.Command("git", append([]string{"-c", "user.name=test", "-c", "user.email=test@test"}, args...)...)
if dir != "" {
cmd.Dir = dir
}
+1 -1
View File
@@ -810,7 +810,7 @@ func (rc *RunContext) cleanupJobResources(networkName string, createAndDeleteNet
errs = append(errs, runAll(ctx, containers))
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), jobCleanupTimeout)
defer cancel()
if !common.Dryrun(ctx) && !rc.Config.NoSweep && (preclean || rc.hadDockerProxy) {
if !common.Dryrun(ctx) && rc.Config.Sweep && (preclean || rc.hadDockerProxy) {
errs = append(errs, container.RemoveDockerJobResources(ctx, rc.jobContainerName())) // after the containers, whose endpoints would block removing the job networks
}
var rest []common.Executor
+8 -8
View File
@@ -725,7 +725,7 @@ func TestCleanupJobResources(t *testing.T) {
service.On("Remove").Return(removeAlongside(serviceRemoving, jobRemoving, removeError)).Once()
service.On("Close").Return(func(context.Context) error { return closeError }).Once()
rc := &RunContext{
Config: &Config{CacheContainer: "cache-container", NoSweep: name == "kept proxy"},
Config: &Config{CacheContainer: "cache-container", Sweep: name != "kept proxy"},
Run: &model.Run{Workflow: &model.Workflow{Name: "wf"}, JobID: "job"},
JobContainer: job,
serviceContainers: []*serviceContainer{{name: "svc", container: service}},
@@ -1490,7 +1490,7 @@ func TestCopyWorkflowFilesHandsNodeTheRunnerCAUnlessTheJobOrImageSetsIt(t *testi
return env
}
assert.Equal(t, actPath+"/ca.crt", stepEnv(caFile, nil, nil, nil)["NODE_EXTRA_CA_CERTS"])
assert.Equal(t, filepath.ToSlash(actPath)+"/ca.crt", stepEnv(caFile, nil, nil, nil)["NODE_EXTRA_CA_CERTS"])
ca, err := os.ReadFile(filepath.Join(actPath, "ca.crt"))
require.NoError(t, err)
assert.Equal(t, "PEM\n", string(ca))
@@ -1666,21 +1666,21 @@ func TestRunContextUpdateExtraPath(t *testing.T) {
longPath := strings.Repeat("x", 64*1024+1)
for _, testcase := range []struct {
name string
content []byte
content string
want string
wantErr bool
}{
{name: "UTF-8 BOM", content: []byte("\xef\xbb\xbf/utf8/tools\n"), want: "/utf8/tools"},
{name: "UTF-16 BOM", content: []byte{0xff, 0xfe, 'C', 0, ':', 0, '\\', 0, 't', 0, 'o', 0, 'o', 0, 'l', 0, 's', 0, '\n', 0}, want: `C:\tools`},
{name: "over 64 KiB", content: []byte(longPath + "\n"), want: longPath},
{name: "over 16 MiB", content: []byte(strings.Repeat("x", 16*1024*1024+1)), wantErr: true},
{name: "UTF-8 BOM", content: "\xef\xbb\xbf/utf8/tools\n", want: "/utf8/tools"},
{name: "UTF-16 BOM", content: "\xff\xfeC\x00:\x00\\\x00t\x00o\x00o\x00l\x00s\x00\n\x00", want: `C:\tools`},
{name: "over 64 KiB", content: longPath + "\n", want: longPath},
{name: "over 16 MiB", content: strings.Repeat("x", 16*1024*1024+1), wantErr: true},
} {
t.Run(testcase.name, func(t *testing.T) {
logger := logging.New(logging.DiscardHandler)
ctx := common.WithLogger(t.Context(), logger)
jobContainer := &containerMock{}
jobContainer.On("GetContainerArchive", mock.Anything, "/github/path").
Return(io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "path", body: string(testcase.content)}))), nil).Once()
Return(io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "path", body: testcase.content}))), nil).Once()
defer jobContainer.AssertExpectations(t)
rc := &RunContext{JobContainer: jobContainer}
+2 -1
View File
@@ -81,6 +81,7 @@ type Config struct {
PlatformPicker func(labels []string) string
KubernetesPicker func(labels []string) container.KubernetesOptions
InstanceTransport http.RoundTripper
ResultsURL string // the results service as the runner reaches it, never the job's ACTIONS_RESULTS_URL, which a step can rewrite
JobLoggerLevel *logging.Level // the level of job logger
ValidVolumes []string // only volumes (and bind mounts) in this slice can be mounted on the job container or service containers
ContainerVolumes []string // mounted into every job, service and docker action container
@@ -91,7 +92,7 @@ type Config struct {
MaxParallel int // max parallel jobs to run across all workflows (0 = no limit, uses CPU count)
AllocatePTY bool // allocate a pseudo-TTY for each step's process
ServiceReadyTimeout time.Duration // how long a job waits for its service containers to report healthy (0 uses the default)
NoSweep bool // keep the containers, networks and volumes a job creates through its docker proxy
Sweep bool // remove the containers, networks and volumes a job creates through its docker proxy
RunnerName string // name this runner registered with, reported as `runner.name`, defaults to the hostname
JobStartedHook string // script run inside the job environment before the job's first step; ACTIONS_RUNNER_HOOK_JOB_STARTED is read from Env when empty
JobCompletedHook string // script run inside the job environment after the job's last step; ACTIONS_RUNNER_HOOK_JOB_COMPLETED is read from Env when empty
+21 -2
View File
@@ -4,17 +4,24 @@
package runner
import (
"cmp"
"context"
"fmt"
"strings"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/internal/pkg/action"
"gitea.com/gitea/runner/internal/pkg/action/checkout"
"gitea.com/gitea/runner/internal/pkg/action/downloadartifact"
"gitea.com/gitea/runner/internal/pkg/action/uploadartifact"
"gitea.dev/actionslib/pkg/model"
)
var builtinActions = map[string]func(context.Context, *action.Context) error{
"checkout": checkout.Main,
"checkout": checkout.Main,
"download-artifact": downloadartifact.Main,
"upload-artifact": uploadartifact.Main,
}
type stepActionBuiltin struct {
@@ -34,13 +41,18 @@ func (sab *stepActionBuiltin) main() common.Executor {
}
printRunActionHeader(ctx, sab.Step, nil, sab.env, sab.RunContext)
rawLogger := common.Logger(ctx).With(rawOutputField, true)
rc := sab.RunContext
rc.ApplyExtraPath(ctx, &sab.env)
inputs := make(map[string]string, len(sab.Step.With))
for name, value := range sab.Step.With {
inputs[strings.ToLower(name)] = value // input names are case-insensitive on GitHub
}
return sab.run(ctx, &action.Context{
Container: rc.JobContainer,
Github: getStepGithubContext(ctx, sab),
Inputs: sab.Step.With,
Inputs: inputs,
HostWorkdir: rc.Config.Workdir,
Workspace: rc.JobContainer.ToContainerPath(rc.Config.Workdir),
BindWorkdir: rc.Config.BindWorkdir,
@@ -51,6 +63,13 @@ func (sab *stepActionBuiltin) main() common.Executor {
ClientKeyFile: rc.Config.ClientKeyFile,
CAFile: rc.jobCAFile(),
Env: sab.env,
ResultsURL: rc.Config.ResultsURL,
RuntimeToken: cmp.Or(rc.runtimeToken, rc.topLevelRunContext().Config.Secrets["ACTIONS_RUNTIME_TOKEN"]), // composite contexts have no secrets
Transport: rc.Config.InstanceTransport,
Infof: rawLogger.Infof,
Warning: func(format string, args ...any) {
rawLogger.Infof("%s", formatAnnotation("warning", nil, fmt.Sprintf(format, args...)))
},
})
})
}
+3 -9
View File
@@ -35,14 +35,8 @@ func setFetch(t *testing.T, fetch func(context.Context, git.NewGitCloneExecutorI
func TestShortSHAActionRejected(t *testing.T) {
actionRoot := t.TempDir()
repo := filepath.Join(actionRoot, "actions", "hello-world-docker-action")
require.NoError(t, os.MkdirAll(repo, 0o755))
gitMust(t, "", "init", "--initial-branch=main", repo)
gitMust(t, repo, "config", "user.email", "test@test")
gitMust(t, repo, "config", "user.name", "test")
require.NoError(t, os.WriteFile(filepath.Join(repo, "action.yml"),
[]byte("name: hello\nruns:\n using: node24\n main: index.js\n"), 0o644))
gitMust(t, repo, "add", ".")
gitMust(t, repo, "commit", "-m", "initial")
gitMust(t, "", "init", "--template=", "--initial-branch=main", repo)
gitMust(t, repo, "commit", "--allow-empty", "-m", "initial")
output, err := exec.Command("git", "-C", repo, "rev-parse", "--short=7", "HEAD").Output()
require.NoError(t, err)
@@ -393,7 +387,7 @@ func TestStepActionRemotePreResolvesDownloadedCommit(t *testing.T) {
[]byte("name: setup-go\nruns:\n using: node20\n main: index.js\n"), 0o600))
for _, args := range [][]string{
{"init", "--initial-branch=main", actionDir},
{"init", "--template=", "--initial-branch=main", actionDir},
{"-C", actionDir, "add", "action.yml"},
{"-C", actionDir, "-c", "user.name=runner", "-c", "[email protected]", "-c", "commit.gpgsign=false", "commit", "-m", "action"},
} {
+46
View File
@@ -4,6 +4,8 @@ Built-in actions ship with the runner, so a job needs no action download and no
```yaml
- uses: builtin:checkout
- uses: builtin:upload-artifact
- uses: builtin:download-artifact
```
## checkout
@@ -37,3 +39,47 @@ A branch is checked out as a local branch tracking `origin`, a tag or commit as
Persisted credentials end with the job. `token` is only sent to `github.server_url`, while `ssh-key` is offered to any SSH host, as with `actions/checkout`. The runner's `client_cert_file` is only offered to `github.server_url` and only during the step, so later steps do not get it. Its `ca_file` replaces the job image's CA bundle for `github.server_url`, in later steps too when credentials persist. Without `ssh-key`, submodule URLs of the form `git@<server host>:` are fetched from `github.server_url` with `token`. With `ssh-key`, the repository is fetched from the SSH URL in the event's `repository.ssh_url`, or from the server's host on port 22.
With `gitea-runner exec`, checking out the workflow's own repository copies your local working directory, unless `--no-skip-checkout` or `ssh-key` is set.
## upload-artifact
Works like `actions/upload-artifact` v7 with these inputs. Any other input logs a warning, as on GitHub. It needs Gitea 1.22.4 or later, and 1.26 for `archive: false`.
| Input | Default |
| --- | --- |
| `name` | `artifact`, ignored with `archive: false`, which names the artifact after the file |
| `path` | required, files, directories or patterns, one per line, `!` excludes |
| `if-no-files-found` | `warn`, also `error` or `ignore` |
| `retention-days` | the server's default, at most the repository's retention limit |
| `compression-level` | `6`, from `0` for none to `9` |
| `overwrite` | `false`, `true` deletes an artifact of the same name first |
| `include-hidden-files` | `false`, `true` uploads files and directories whose name starts with `.` |
| `archive` | `true`, `false` uploads a single file as is |
Patterns, symlinks and the artifact's root directory follow `actions/upload-artifact`. `ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY` sets upload concurrency (default `5`), and `ACTIONS_ARTIFACT_UPLOAD_TIMEOUT_MS` sets the upload stall timeout (default `300000`). The step outputs `artifact-id`, `artifact-digest` and `artifact-url`. On Gitea 1.26 and later, the `artifact-url` is the artifact's link on the run page for this attempt, where `actions/upload-artifact` gives a link only GitHub serves.
Container jobs list the search paths in the job container and transfer only the selected files, gzipped when the image has gzip, requiring `sh`, `find` and `tar` in the image. On Kubernetes these files still pass through the API server, unlike `actions/upload-artifact`, which uploads from the pod.
## download-artifact
Works like `actions/download-artifact` v8 with these inputs. Any other input logs a warning, as on GitHub.
| Input | Default |
| --- | --- |
| `name` | empty, every artifact of the run |
| `artifact-ids` | empty, comma-separated IDs instead of `name` |
| `path` | workspace root, `~` is the job user's home |
| `pattern` | empty, a glob the artifact names must match, without the extglobs and brace ranges of minimatch, ignored with `name` or `artifact-ids` |
| `merge-multiple` | `false`, several artifacts go to `path/<name>`, `true` puts them all in `path` |
| `github-token` | empty, a token to download from another run or repository |
| `repository` | `github.repository`, with `github-token` |
| `run-id` | `github.run_id`, with `github-token` |
| `skip-decompress` | `false`, `true` keeps a zipped artifact as `<name>.zip` |
| `digest-mismatch` | `error`, also `ignore`, `info` or `warn` |
ZIP entries become regular files with current modification times, so files lose their executable bit and symlinks become files containing their targets. `digest-mismatch` only applies when the server reports a digest, which Gitea does not. The step outputs `download-path`.
The runner downloads each artifact to a temporary file on the runner because the zip is read out of order, then streams the files into the job container. On Kubernetes, the stream goes through the API server, gzipped when the image has gzip, or over the pod network with `kubernetes.transfer.mode` set to `auto` or `direct`, while `actions/download-artifact` downloads from the pod directly.
`github-token` needs Gitea 1.24 or later and is only sent to the instance the runner is registered with. Downloads from another run paginate through its artifacts and skip expired ones, where `actions/download-artifact` sees only Gitea's first 30 and fails on an expired newest copy. `run-id` must be positive, since Gitea treats nonpositive IDs as a request for artifacts from every run. Artifact names and ZIP entries that would leave the destination are rejected.
With `gitea-runner exec`, both actions use the local artifact server of `--artifact-server-path`, which serves no other runs for `github-token`.
+49
View File
@@ -26,3 +26,52 @@ jobs:
name: e2e-artifact-${{ github.run_id }}
path: got
- run: grep -q artifact-payload got/data.txt
builtin-upload:
runs-on: e2e-artifact
steps:
- run: |
mkdir -p out/sub
echo "builtin-payload" > out/data.txt
echo "nested" > out/sub/nested.txt
echo "hidden" > out/.hidden
echo "raw" > report.txt
- id: zipped
uses: builtin:upload-artifact
with:
name: e2e-builtin-${{ github.run_id }}
path: out
- run: test -n "${{ steps.zipped.outputs.artifact-id }}" && test -n "${{ steps.zipped.outputs.artifact-digest }}"
- uses: builtin:upload-artifact
with:
path: report.txt
archive: false
js-downloads-builtin:
needs: builtin-upload
runs-on: e2e-artifact
steps:
- uses: actions/download-artifact@v8
with:
name: e2e-builtin-${{ github.run_id }}
path: got
- run: grep -q builtin-payload got/data.txt && grep -q nested got/sub/nested.txt && test ! -e got/.hidden
- uses: actions/download-artifact@v8
with:
name: report.txt
- run: grep -q raw report.txt
builtin-downloads-js:
needs: upload
runs-on: e2e-artifact
steps:
- id: own
uses: builtin:download-artifact
with:
name: e2e-artifact-${{ github.run_id }}
path: got
- run: grep -q artifact-payload got/data.txt && test "${{ steps.own.outputs.download-path }}" = "$GITHUB_WORKSPACE/got"
- uses: builtin:download-artifact
with:
name: e2e-artifact-${{ github.run_id }}
path: rest
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- run: grep -q artifact-payload rest/data.txt
+5 -4
View File
@@ -128,6 +128,11 @@ func (*daemonTestService) UpdateLog(_ context.Context, req *runnerv1.UpdateLogRe
func TestDaemonRetriesDeclareOutagesAndRemovesOnlyConsumedEphemeralRegistration(t *testing.T) {
defer func(delay time.Duration) { declareRetryDelay = delay }(declareRetryDelay)
declareRetryDelay = 0
dir := t.TempDir()
regFile := filepath.Join(dir, "registration.json")
configFile := filepath.Join(dir, "config.yaml")
require.NoError(t, os.WriteFile(configFile, []byte("runner:\n file: "+regFile+"\n idle_cleanup_interval: 0s\ncache:\n enabled: false\n"), 0o600))
for _, tc := range []struct {
name string
ephemeral, beforeTask bool
@@ -154,10 +159,6 @@ func TestDaemonRetriesDeclareOutagesAndRemovesOnlyConsumedEphemeralRegistration(
connecthttp.Mount(mux, rpcServer, connecthttp.WithReadMaxBytes(0))
server := httptest.NewServer(http.StripPrefix("/api/actions", mux))
defer server.Close()
dir := t.TempDir()
regFile := filepath.Join(dir, "registration.json")
configFile := filepath.Join(dir, "config.yaml")
require.NoError(t, os.WriteFile(configFile, []byte("runner:\n file: "+regFile+"\n idle_cleanup_interval: 0s\ncache:\n enabled: false\n"), 0o600))
require.NoError(t, config.SaveRegistration(regFile, &config.Registration{Address: server.URL, UUID: "test", Name: "test", Token: "test", Labels: []string{"host:host"}, Ephemeral: tc.ephemeral}))
err := runDaemon(ctx, &daemonArgs{Once: true}, &configFile)(nil, nil)
if tc.declareErr == nil && tc.fetchErr == nil || tc.beforeTask {
+6 -1
View File
@@ -209,6 +209,10 @@ func parseKVAndFile(rawKVs []string, filePath string) map[string]string {
return result
}
func (i *executeArgs) artifactServerURL() string {
return "http://" + net.JoinHostPort(i.artifactServerAddr, i.artifactServerPort)
}
// Workdir returns path to workdir
func (i *executeArgs) Workdir() string {
return i.resolve(".")
@@ -382,6 +386,7 @@ func (i *executeArgs) runnerConfig(eventName string, env, proxyEnv map[string]st
ArtifactServerPath: i.artifactServerPath,
ArtifactServerPort: i.artifactServerPort,
ArtifactServerAddr: i.artifactServerAddr,
ResultsURL: i.artifactServerURL(),
NoSkipCheckout: i.noSkipCheckout,
EventName: eventName,
EventJSON: eventJSON,
@@ -513,7 +518,7 @@ func runExec(ctx context.Context, execArgs *executeArgs) func(cmd *cobra.Command
token := cmp.Or(inheritedToken, artifacts.LocalRuntimeToken(workflowRun, maxLifetime))
config.RuntimeTokens[run.Workflow] = token
config.ExtraMasks = append(config.ExtraMasks, token)
handler.RegisterJob(token, artifactcache.JobCredential{Repo: "__local/__exec", Results: "http://" + net.JoinHostPort(execArgs.artifactServerAddr, execArgs.artifactServerPort)})
handler.RegisterJob(token, artifactcache.JobCredential{Repo: "__local/__exec", Results: execArgs.artifactServerURL()})
}
}
+27 -35
View File
@@ -75,40 +75,35 @@ func TestExecuteArgsLoadInputs(t *testing.T) {
}
func TestExecuteArgsEventJSON(t *testing.T) {
payload := func(content string) string {
path := filepath.Join(t.TempDir(), "event.json")
require.NoError(t, os.WriteFile(path, []byte(content), 0o600))
return path
}
dir := t.TempDir()
tests := []struct {
name string
eventpath string
inputs []string
expected string
name string
payload string
inputs []string
expected string
}{
{
name: "keeps a payload without inputs to merge",
eventpath: payload(`{"inputs": {"kept": "from_event"}}`),
expected: `{"inputs": {"kept": "from_event"}}`,
name: "keeps a payload without inputs to merge",
payload: `{"inputs": {"kept": "from_event"}}`,
expected: `{"inputs": {"kept": "from_event"}}`,
},
{
name: "overrides the event inputs it names, keeping the rest",
eventpath: payload(`{"inputs": {"kept": "from_event", "named": "from_event"}}`),
inputs: []string{"named=from_cli"},
expected: `{"inputs": {"kept": "from_event", "named": "from_cli"}}`,
name: "overrides the event inputs it names, keeping the rest",
payload: `{"inputs": {"kept": "from_event", "named": "from_event"}}`,
inputs: []string{"named=from_cli"},
expected: `{"inputs": {"kept": "from_event", "named": "from_cli"}}`,
},
{
name: "adds inputs to a payload carrying none",
eventpath: payload(`{"repository": {"name": "test-repo"}}`),
inputs: []string{"flag=val"},
expected: `{"repository": {"name": "test-repo"}, "inputs": {"flag": "val"}}`,
name: "adds inputs to a payload carrying none",
payload: `{"repository": {"name": "test-repo"}}`,
inputs: []string{"flag=val"},
expected: `{"repository": {"name": "test-repo"}, "inputs": {"flag": "val"}}`,
},
{
name: "handles a null payload",
eventpath: payload("null"),
inputs: []string{"flag=val"},
expected: `{"inputs": {"flag": "val"}}`,
name: "handles a null payload",
payload: "null",
inputs: []string{"flag=val"},
expected: `{"inputs": {"flag": "val"}}`,
},
{
name: "builds a payload without an event file",
@@ -119,7 +114,12 @@ func TestExecuteArgsEventJSON(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
eventJSON, err := (&executeArgs{eventpath: tt.eventpath, inputs: tt.inputs}).eventJSON()
var eventpath string
if tt.payload != "" {
eventpath = filepath.Join(dir, tt.name+".json")
require.NoError(t, os.WriteFile(eventpath, []byte(tt.payload), 0o600))
}
eventJSON, err := (&executeArgs{eventpath: eventpath, inputs: tt.inputs}).eventJSON()
require.NoError(t, err)
require.JSONEq(t, tt.expected, eventJSON)
})
@@ -275,21 +275,13 @@ func workflowForList(name, file, event, jobID, jobName string) *model.Workflow {
return &model.Workflow{
Name: name,
File: file,
RawOn: rawOnNode(event),
RawOn: yaml.Node{Kind: yaml.ScalarNode, Tag: "!!str", Value: event},
Jobs: map[string]*model.Job{
jobID: {Name: jobName},
},
}
}
func rawOnNode(event string) yaml.Node {
var node yaml.Node
if err := yaml.Unmarshal([]byte(event), &node); err != nil {
panic(err)
}
return *node.Content[0]
}
func TestExecuteArgsLoadEnvsResultsOrigin(t *testing.T) {
t.Setenv("ACTIONS_RESULTS_URL", "") // the default is only supplied when nothing else does
handler, err := artifactcache.StartHandler(artifactcache.Options{Dir: t.TempDir(), OutboundIP: "127.0.0.1"})
+3 -14
View File
@@ -27,16 +27,9 @@ func TestRunPostTaskScriptSkippedWhenEmpty(t *testing.T) {
r := &Runner{
cfg: &config.Config{},
}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
taskCtx, err := structpb.NewStruct(map[string]any{})
require.NoError(t, err)
task := &runnerv1.Task{Id: 1, Context: taskCtx}
reporter := report.NewReporter(ctx, cancel, nil, task, r.cfg)
require.NotPanics(t, func() {
r.runPostTaskScript(ctx, reporter, task, "/workspace/owner/repo")
r.runPostTaskScript(t.Context(), nil, nil, "/workspace/owner/repo")
})
}
@@ -59,9 +52,7 @@ func TestRunPostTaskScriptNonZeroExitDoesNotPanic(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
taskCtx, err := structpb.NewStruct(map[string]any{})
require.NoError(t, err)
task := &runnerv1.Task{Id: 1, Context: taskCtx}
task := &runnerv1.Task{Id: 1, Context: &structpb.Struct{}}
reporter := report.NewReporter(ctx, cancel, nil, task, cfg)
require.NotPanics(t, func() {
@@ -146,9 +137,7 @@ func TestRunPostTaskScriptIntegration(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
taskCtx, err := structpb.NewStruct(map[string]any{})
require.NoError(t, err)
task := &runnerv1.Task{Id: 11, Context: taskCtx}
task := &runnerv1.Task{Id: 11, Context: &structpb.Struct{}}
reporter := report.NewReporter(ctx, cancel, nil, task, cfg)
setReporterJobResult(t, reporter, runnerv1.Result_RESULT_SUCCESS)
+2 -1
View File
@@ -612,7 +612,7 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
},
ContainerOptions: r.cfg.Container.Options,
ServiceReadyTimeout: r.cfg.Container.ServiceReadyTimeout,
NoSweep: r.cfg.Container.Sweep != nil && !*r.cfg.Container.Sweep,
Sweep: r.cfg.Container.Sweep,
ContainerDaemonSocket: r.cfg.Container.DockerHost,
Privileged: r.cfg.Container.Privileged,
DefaultActionInstance: r.getDefaultActionsURL(task),
@@ -629,6 +629,7 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
ClientKeyFile: r.cfg.Runner.ClientKeyFile,
CAFile: r.cfg.Runner.CAFile,
InstanceTransport: r.instanceTransport,
ResultsURL: strings.TrimSuffix(r.client.Address(), "/"),
RunnerName: r.name,
KubernetesPicker: r.kubernetesOptions,
}
+3 -6
View File
@@ -21,8 +21,7 @@ import (
func TestRunnerCleanupStaleTaskDirs(t *testing.T) {
now := time.Date(2026, time.April, 29, 20, 0, 0, 0, time.UTC)
workdirRoot := filepath.Join(t.TempDir(), "workspace")
require.NoError(t, os.MkdirAll(workdirRoot, 0o700))
workdirRoot := t.TempDir()
oldTask := filepath.Join(workdirRoot, "1001")
freshTask := filepath.Join(workdirRoot, "1002")
@@ -59,8 +58,7 @@ func TestRunnerCleanupStaleTaskDirs(t *testing.T) {
// be reclaimed, while the shared tool_cache and operator data are preserved.
func TestRunnerOnIdleCleansStaleHostScratchDirs(t *testing.T) {
now := time.Date(2026, time.April, 29, 20, 0, 0, 0, time.UTC)
hostRoot := filepath.Join(t.TempDir(), "act")
require.NoError(t, os.MkdirAll(hostRoot, 0o700))
hostRoot := t.TempDir()
staleScratch := filepath.Join(hostRoot, "0123456789abcdef") // 16 hex
freshScratch := filepath.Join(hostRoot, "fedcba9876543210")
@@ -118,8 +116,7 @@ func TestRunnerCleanupStaleTaskDirsMissingRoot(t *testing.T) {
func TestRunnerCleanupStaleTaskDirsHonorsContext(t *testing.T) {
now := time.Date(2026, time.April, 29, 20, 0, 0, 0, time.UTC)
workdirRoot := filepath.Join(t.TempDir(), "workspace")
require.NoError(t, os.MkdirAll(workdirRoot, 0o700))
workdirRoot := t.TempDir()
for i := 1001; i <= 1003; i++ {
dir := filepath.Join(workdirRoot, strconv.Itoa(i))
+2 -1
View File
@@ -96,6 +96,7 @@ func TestSetupLinesInputs(t *testing.T) {
}
func TestPrettyOSName(t *testing.T) {
dir := t.TempDir()
for _, tt := range []struct {
name string
osRelease string
@@ -109,7 +110,7 @@ func TestPrettyOSName(t *testing.T) {
{"missing file", "", "", true},
} {
t.Run(tt.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "os-release")
path := filepath.Join(dir, tt.name)
if !tt.missing {
require.NoError(t, os.WriteFile(path, []byte(tt.osRelease), 0o600))
}
+147
View File
@@ -5,6 +5,19 @@
package action
import (
"context"
"errors"
"fmt"
"net/http"
"os"
"path"
"path/filepath"
"runtime"
"slices"
"strconv"
"strings"
"unicode"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/model"
@@ -24,4 +37,138 @@ type Context struct {
ClientKeyFile string
CAFile string // the runner CA as seen inside Container
Env map[string]string
ResultsURL string // the results service as the runner reaches it
RuntimeToken string
Transport http.RoundTripper // carries the instance's TLS settings, nil for the default
Infof func(format string, args ...any)
Warning func(format string, args ...any)
}
func (c *Context) ApplyDefaults(pairs ...string) {
var valid, unexpected []string
for i := 0; i+1 < len(pairs); i += 2 {
valid = append(valid, pairs[i])
if _, set := c.Inputs[pairs[i]]; !set && pairs[i+1] != "" {
c.Inputs[pairs[i]] = pairs[i+1]
}
}
for name := range c.Inputs {
if !slices.Contains(valid, name) {
unexpected = append(unexpected, name)
}
}
if len(unexpected) > 0 {
slices.Sort(unexpected)
c.Warning("Unexpected input(s) '%s', valid inputs are ['%s']", strings.Join(unexpected, "', '"), strings.Join(valid, "', '"))
}
}
// Input is the trimmed input, as @actions/core getInput returns it.
func (c *Context) Input(name string) string {
return strings.TrimSpace(c.Inputs[name])
}
// BoolInput accepts what @actions/core getBooleanInput does.
func (c *Context) BoolInput(name string) (bool, error) {
switch c.Input(name) {
case "true", "True", "TRUE":
return true, nil
case "false", "False", "FALSE":
return false, nil
}
return false, Failf("Input does not meet YAML 1.2 \"Core Schema\" specification: %s\nSupport boolean input list: `true | True | TRUE | false | False | FALSE`", name)
}
func (c *Context) SetOutputs(ctx context.Context, pairs ...string) error {
output := c.Env["GITHUB_OUTPUT"]
if output == "" {
return nil
}
var body strings.Builder
for i := 0; i+1 < len(pairs); i += 2 {
if strings.ContainsAny(pairs[i+1], "\r\n") {
return fmt.Errorf("output %s spans several lines", pairs[i])
}
body.WriteString(pairs[i] + "=" + pairs[i+1] + "\n")
}
return c.Container.Copy(c.Dir(output), &container.FileEntry{Name: c.Base(output), Mode: 0o666, Body: body.String()})(ctx)
}
func (c *Context) Host() bool {
_, host := c.Container.(*container.HostEnvironment)
return host
}
func (c *Context) Windows() bool {
return c.Host() && runtime.GOOS == "windows"
}
func (c *Context) Join(elem ...string) string {
if c.Windows() {
return filepath.Join(elem...)
}
return path.Join(elem...)
}
func (c *Context) Dir(p string) string {
if c.Windows() {
return filepath.Dir(p)
}
return path.Dir(p)
}
func (c *Context) Base(p string) string {
if c.Windows() {
return filepath.Base(p)
}
return path.Base(p)
}
func (c *Context) Resolve(p string) string {
if c.Windows() && filepath.IsAbs(p) || !c.Windows() && path.IsAbs(p) {
return c.Join(p)
}
return c.Join(c.Workspace, p)
}
// Home is the job user's home directory, what os.homedir() returns to a JavaScript action.
func (c *Context) Home(ctx context.Context) (string, error) {
if home := c.Env["HOME"]; home != "" {
return home, nil
}
if home := c.Env["USERPROFILE"]; c.Windows() && home != "" {
return home, nil
}
if c.Host() {
return os.UserHomeDir()
}
var out strings.Builder
stdout, stderr := c.Container.ReplaceLogWriter(&out, &out)
err := c.Container.Exec([]string{"sh", "-c", `printf %s "$HOME"`}, nil, "", "")(ctx)
c.Container.ReplaceLogWriter(stdout, stderr)
if err != nil {
return "", fmt.Errorf("cannot expand ~ without HOME: %w", err)
}
if out.Len() == 0 {
return "", errors.New("cannot expand ~, the job user has no HOME")
}
return out.String(), nil
}
// Failf is fmt.Errorf for messages worded as the JavaScript action words them, which the error string lint rejects.
func Failf(format string, args ...any) error {
return fmt.Errorf(format, args...)
}
// ParseInt reads a leading integer the way JavaScript's parseInt does.
func ParseInt(s string) (int, error) {
s = strings.TrimLeftFunc(s, unicode.IsSpace)
end := 0
if end < len(s) && (s[end] == '-' || s[end] == '+') {
end++
}
for end < len(s) && s[end] >= '0' && s[end] <= '9' {
end++
}
return strconv.Atoi(s[:end])
}
+184
View File
@@ -0,0 +1,184 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package artifact
import (
"bytes"
"context"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"sync/atomic"
"time"
"golang.org/x/sync/errgroup"
)
// BlockSize is the toolkit's upload chunk.
const BlockSize = 8 << 20
type Uploaded struct {
Size int64
Digest string // sha256:<hex> for Finalize
}
// Upload puts body to a signed URL as parallel Azure blocks, failing when upload progress stalls for stall.
func (c *Client) Upload(ctx context.Context, signedURL, contentType string, body io.Reader, concurrency int, stall time.Duration) (result Uploaded, err error) {
ctx, cancel := context.WithCancelCause(ctx)
defer func() {
if ctx.Err() != nil {
err = context.Cause(ctx)
}
cancel(nil)
}()
stall = max(stall, time.Millisecond)
timer := time.AfterFunc(stall, func() { cancel(errors.New("upload progress stalled")) })
defer timer.Stop()
hash := sha256.New()
reader := io.TeeReader(body, hash)
group, blockCtx := errgroup.WithContext(ctx)
if closer, ok := body.(io.ReadCloser); ok {
stop := context.AfterFunc(blockCtx, func() { closer.Close() })
defer stop()
}
buffers := make(chan []byte, concurrency)
prefix := rand.Text()
var ids []string
var uploaded atomic.Int64
for index := 0; ; index++ {
var buf []byte
if index < concurrency {
buf = make([]byte, BlockSize)
} else {
select {
case buf = <-buffers:
case <-blockCtx.Done():
return Uploaded{}, group.Wait()
}
}
n, readErr := io.ReadFull(reader, buf)
if errors.Is(readErr, io.EOF) {
break
} else if readErr != nil && !errors.Is(readErr, io.ErrUnexpectedEOF) {
if blockCtx.Err() == nil {
cancel(readErr)
}
if err := group.Wait(); err != nil {
return Uploaded{}, err
}
return Uploaded{}, readErr
}
id := base64.StdEncoding.EncodeToString(fmt.Appendf(nil, "%s-%08d", prefix, index)) // fixed width as Azure needs, unique as Gitea before 1.25.5 shares a run's blocks
ids = append(ids, id)
group.Go(func() error {
defer func() { buffers <- buf }()
if err := c.put(blockCtx, signedURL+"&comp=block&blockid="+url.QueryEscape(id), "x-ms-blob-type", "BlockBlob", buf[:n]); err != nil {
return err
}
timer.Reset(stall)
c.Logf("Uploaded bytes %d", uploaded.Add(int64(n)))
return nil
})
if readErr != nil {
break
}
}
if err := group.Wait(); err != nil {
return Uploaded{}, err
}
var list strings.Builder
list.WriteString("<BlockList>")
for _, id := range ids {
list.WriteString("<Latest>" + id + "</Latest>")
}
list.WriteString("</BlockList>")
if err := c.put(ctx, signedURL+"&comp=blocklist", "x-ms-blob-content-type", contentType, []byte(list.String())); err != nil {
return Uploaded{}, err
}
return Uploaded{Size: uploaded.Load(), Digest: "sha256:" + hex.EncodeToString(hash.Sum(nil))}, nil
}
func (c *Client) put(ctx context.Context, signedURL, header, value string, body []byte) error {
resp, err := c.Retry.do(ctx, c.Logf, "blob upload", func() (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodPut, signedURL, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set(header, value)
resp, err := c.blob.Do(req)
return resp, scrub(err)
})
if err != nil {
return err
}
return resp.Body.Close()
}
// Download GETs a signed URL, a read failing when nothing arrives for stall.
func (c *Client) Download(ctx context.Context, signedURL string, stall time.Duration) (*http.Response, error) {
ctx, cancel := context.WithCancelCause(ctx)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, signedURL, nil)
if err != nil {
cancel(nil)
return nil, err
}
resp, err := c.blob.Do(req)
if err != nil {
cancel(nil)
return nil, scrub(err)
}
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
cancel(nil)
return nil, fmt.Errorf("unexpected HTTP response from blob storage: %d %s", resp.StatusCode, http.StatusText(resp.StatusCode))
}
resp.Body = &stallReader{body: resp.Body, timer: time.AfterFunc(stall, func() {
cancel(fmt.Errorf("blob storage chunk did not respond in %s", stall))
}), stall: stall, cancel: cancel, ctx: ctx}
return resp, nil
}
func scrub(err error) error { // the signature of a signed URL is a credential
if urlErr, ok := errors.AsType[*url.Error](err); ok {
if parsed, parseErr := url.Parse(urlErr.URL); parseErr == nil {
parsed.RawQuery = ""
urlErr.URL = parsed.String()
} else {
urlErr.URL = "<unparsable URL>"
}
}
return err
}
type stallReader struct {
body io.ReadCloser
timer *time.Timer
stall time.Duration
cancel context.CancelCauseFunc
ctx context.Context
}
func (r *stallReader) Read(p []byte) (int, error) {
n, err := r.body.Read(p)
if n > 0 {
r.timer.Reset(r.stall)
}
if err != nil && !errors.Is(err, io.EOF) && context.Cause(r.ctx) != nil {
err = context.Cause(r.ctx)
}
return n, err
}
func (r *stallReader) Close() error {
r.timer.Stop()
r.cancel(nil)
return r.body.Close()
}
+264
View File
@@ -0,0 +1,264 @@
// Copyright 2026 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
// Package artifact talks to the v4 results service of the job's run.
package artifact
import (
"bytes"
"cmp"
"context"
"encoding/base64"
"encoding/json/v2"
"errors"
"fmt"
"io"
"math"
"net/http"
"net/url"
"strconv"
"strings"
"time"
)
type Artifact struct {
ID int64 `json:"databaseId,string"`
Name string `json:"name"`
Size int64 `json:"size,string"`
Digest string `json:"digest"`
}
type Retry struct {
Attempts int
Base time.Duration
Multiplier float64
MaxWait time.Duration
}
var DefaultRetry = Retry{Attempts: 5, Base: 8 * time.Second, Multiplier: 1.5, MaxWait: 2 * time.Minute}
type Client struct {
base *url.URL
token string
ids backendIDsMessage
api, blob *http.Client
Retry Retry
Logf func(format string, args ...any)
}
// New reaches base through transport, which carries the instance's TLS settings, and signed URLs elsewhere directly.
func New(base, token string, transport http.RoundTripper) (*Client, error) {
if base == "" {
return nil, errors.New("no artifact service is configured for this runner")
}
parsed, err := url.Parse(base)
if err != nil {
return nil, err
}
if token == "" {
return nil, errors.New("ACTIONS_RUNTIME_TOKEN is not set")
}
ids, err := backendIDs(token)
if err != nil {
return nil, err
}
return &Client{
base: parsed, token: token, ids: ids,
api: &http.Client{Transport: transport, Timeout: 3 * time.Minute},
blob: &http.Client{Transport: blobTransport{instance: parsed.Host, transport: transport}},
Retry: DefaultRetry,
Logf: func(string, ...any) {},
}, nil
}
// backendIDs reads the unverified scp claim of a runtime token, as the toolkit does.
func backendIDs(token string) (backendIDsMessage, error) {
parts := strings.Split(token, ".")
if len(parts) != 3 {
return backendIDsMessage{}, errors.New("the runtime token is not a JWT, artifacts v4 need Gitea 1.22 or later")
}
payload, err := base64.RawURLEncoding.DecodeString(strings.TrimRight(parts[1], "="))
if err != nil {
return backendIDsMessage{}, err
}
var claims struct {
Scp string `json:"scp"`
}
if err := json.Unmarshal(payload, &claims); err != nil {
return backendIDsMessage{}, err
}
for scope := range strings.FieldsSeq(claims.Scp) {
if fields := strings.Split(scope, ":"); fields[0] == "Actions.Results" && len(fields) == 3 {
return backendIDsMessage{fields[1], fields[2]}, nil
}
}
return backendIDsMessage{}, errors.New("the runtime token has no Actions.Results scope")
}
type blobTransport struct { // the instance's TLS settings only for the instance, as a signed URL can point at object storage
instance string
transport http.RoundTripper
}
func (t blobTransport) RoundTrip(req *http.Request) (*http.Response, error) {
if req.URL.Host == t.instance && t.transport != nil {
return t.transport.RoundTrip(req)
}
return http.DefaultTransport.RoundTrip(req)
}
type StatusError struct {
Status int
Msg string
}
func (e *StatusError) Error() string {
if e.Msg == "" {
return fmt.Sprintf("(%d) %s", e.Status, http.StatusText(e.Status))
}
return fmt.Sprintf("(%d) %s: %s", e.Status, http.StatusText(e.Status), e.Msg)
}
func IsNotFound(err error) bool {
var status *StatusError
return errors.As(err, &status) && status.Status == http.StatusNotFound
}
func (r Retry) do(ctx context.Context, logf func(string, ...any), what string, send func() (*http.Response, error)) (*http.Response, error) {
var waited time.Duration
for attempt := 0; ; attempt++ {
resp, err := send()
var wait time.Duration
if err == nil {
if resp.StatusCode >= 200 && resp.StatusCode < 400 {
return resp, nil
}
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
err = &StatusError{Status: resp.StatusCode, Msg: twirpMessage(body)}
switch resp.StatusCode {
case http.StatusTooManyRequests, http.StatusInternalServerError, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout:
default:
return nil, err
}
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 && resp.StatusCode == http.StatusTooManyRequests {
wait = time.Duration(seconds) * time.Second
}
}
if ctx.Err() != nil {
return nil, ctx.Err()
}
if attempt+1 >= r.Attempts {
return nil, fmt.Errorf("%s failed after %d attempts: %w", what, r.Attempts, err)
}
if wait == 0 {
wait = time.Duration(float64(r.Base) * math.Pow(r.Multiplier, float64(attempt)))
}
if waited+wait > r.MaxWait {
return nil, fmt.Errorf("%s: a retry wait of %s would exceed %s: %w", what, wait, r.MaxWait, err)
}
logf("Attempt %d of %d failed with error: %v. Retrying request in %d ms...", attempt+1, r.Attempts, err, wait.Milliseconds())
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(wait):
}
waited += wait
}
}
// twirpMessage is the msg of a twirp error, the message of a Gitea API one, or the body Gitea answers in plain text.
func twirpMessage(body []byte) string {
var twirpErr struct {
Msg string `json:"msg"`
Message string `json:"message"`
}
if json.Unmarshal(body, &twirpErr) == nil && cmp.Or(twirpErr.Msg, twirpErr.Message) != "" {
return cmp.Or(twirpErr.Msg, twirpErr.Message)
}
return strings.TrimSpace(string(body))
}
type ( // the ArtifactService messages in the protobuf JSON mapping, requests with the proto names the toolkit sends
backendIDsMessage struct {
WorkflowRunBackendID string `json:"workflow_run_backend_id"`
WorkflowJobRunBackendID string `json:"workflow_job_run_backend_id"`
}
namedRequest struct {
backendIDsMessage
Name string `json:"name"`
}
createRequest struct {
namedRequest
ExpiresAt *time.Time `json:"expires_at,omitempty"`
Version int32 `json:"version"`
MimeType string `json:"mime_type,omitempty"` // Gitea before 1.26 rejects the field
}
finalizeRequest struct {
namedRequest
Size int64 `json:"size,string"`
Hash string `json:"hash"`
}
listRequest struct {
backendIDsMessage
NameFilter string `json:"name_filter,omitempty"`
}
idResponse struct {
ArtifactID int64 `json:"artifactId,string"`
}
)
func call[Resp any](ctx context.Context, c *Client, method string, req any) (Resp, error) {
var resp Resp
body, err := json.Marshal(req)
if err != nil {
return resp, err
}
endpoint := c.base.JoinPath("/twirp/github.actions.results.api.v1.ArtifactService", method).String()
httpResp, err := c.Retry.do(ctx, c.Logf, method, func() (*http.Response, error) {
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return nil, err
}
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Authorization", "Bearer "+c.token)
return c.api.Do(httpReq)
})
if err != nil {
return resp, fmt.Errorf("failed to %s: %w", method, err)
}
defer httpResp.Body.Close()
err = json.UnmarshalRead(httpResp.Body, &resp)
return resp, err
}
func (c *Client) Create(ctx context.Context, name, mimeType string, expiresAt *time.Time) (string, error) {
resp, err := call[struct {
SignedUploadURL string `json:"signedUploadUrl"`
}](ctx, c, "CreateArtifact", createRequest{namedRequest{c.ids, name}, expiresAt, 7, mimeType})
return resp.SignedUploadURL, err
}
func (c *Client) Finalize(ctx context.Context, name string, size int64, digest string) (int64, error) {
resp, err := call[idResponse](ctx, c, "FinalizeArtifact", finalizeRequest{namedRequest{c.ids, name}, size, digest})
return resp.ArtifactID, err
}
func (c *Client) List(ctx context.Context, name string) ([]Artifact, error) {
resp, err := call[struct {
Artifacts []Artifact `json:"artifacts"`
}](ctx, c, "ListArtifacts", listRequest{c.ids, name})
return resp.Artifacts, err
}
func (c *Client) SignedURL(ctx context.Context, name string) (string, error) {
resp, err := call[struct {
SignedURL string `json:"signedUrl"`
}](ctx, c, "GetSignedArtifactURL", namedRequest{c.ids, name})
return resp.SignedURL, err
}
func (c *Client) Delete(ctx context.Context, name string) (int64, error) {
resp, err := call[idResponse](ctx, c, "DeleteArtifact", namedRequest{c.ids, name})
return resp.ArtifactID, err
}
Loaded 50 of 76 files, more files were not shown because too many files have changed in this diff. Show more