Compare commits

..
6 Commits
Author SHA1 Message Date
silverwind d882311ec6 refactor: remove dead code and consolidate tests (#1335)
Removes dead code and redundant logic and consolidates tests, so there is less to read and maintain. No behavior change.

1. **act/runner**: inline single-use wrappers, compile regexes once instead of per call, embed the trampoline as a string, drop the unused legacy result of `parseActionCommand`
2. **act/container**: `sanitizeConfig` and `mergeContainerConfigs` stop returning the configs they already mutate in place, drop never-set option fields and the unused `user` parameter of `ExecWithCmdLine`
3. **act/common and cache server**: drop the unused `Trace`/`Tracef` loggers and the short SHA of `FindGitRevision`, `RegisterJob` releases through `RevokeJob`
4. **daemon, poller and reporter**: one shutdown wait for both poll modes, drop an unreachable fetch timeout branch, guard the reporter's last report time with its existing lock instead of an atomic
5. **build**: drop unused Makefile and release workflow variables, collapse the goreleaser ignore list to the same build targets
6. **tests**: fold near-duplicate tests into tables with shared setup

Throughout, hand-rolled logic gives way to `cmp.Or`, `strings.Cut`, `slices` and `min`/`max`.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1335
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-22 19:41:10 +00:00
silverwind 923a0bfb59 refactor: remove bolthold and gofakes3 deps (#1336)
1. Replace bolthold with direct bbolt access in the cache server. bolthold has not changed since March 2024 and kept `encoding/gob` in the binary although the store was already JSON. The on-disk layout stays the same, so existing caches keep working, and the index buckets bolthold left behind are dropped on the first sweep. Queries now run against an in-memory copy of the entries that is only re-read after another process wrote to the store, so lookups and sweeps no longer decode the whole store.

2. Replace gofakes3 with a small in-memory S3 fake in the one test that used it, which removes aws-sdk-go-v2, afero, mgo and x/tools from the module graph.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1336
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-22 16:28:49 +00:00
silverwind 6f0d4f1b48 refactor: migrate to connect-go v2 (#1325)
Migrates the runner to `connectrpc.com/connect/v2` by pinning actionslib v1.5.0, which carries https://gitea.com/gitea/actionslib/pulls/28. The wire behaviour towards Gitea is unchanged and works against Gitea servers on connect v1 and v2. The client keeps v1's unlimited response size, and the error codes and messages the runner checks stay the same.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1325
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-21 16:10:58 +00:00
silverwindandbircni 430f7bb2cf fix: handle workflow commands like GitHub's runner (#1333)
Commands the runner applies itself, like the `::set-output::` older actions still print, showed up raw in the job log. The reporter also re-parsed every line with one job-wide stop-commands and echo state, while act keeps one command handler per step, stage, composite child and hook, as GitHub does. A `::stop-commands::` left active in one of them kept the reporter suppressed for the others, so their `::add-mask::` values were never registered, and post steps logged command lines twice.

- act's command handler decides what reaches the log, as GitHub's runner does: applied commands only under `ACTIONS_STEP_DEBUG` or `::echo::on`, `::stop-commands::` and its resume line shown with long tokens masked
- Masks go straight to the reporter, which no longer parses command lines
- One parser follows GitHub's rules: leading whitespace, `##[...]` anywhere in a line, registered commands only
- Service logs honour only `::add-mask::`

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1333
Reviewed-by: bircni <[email protected]>
Co-authored-by: silverwind <[email protected]>
2026-10-21 16:03:40 +00:00
silverwind befb1b4bb9 fix: improve the actions toolkit patching (#1328)
Patch the scripts beside each node action entrypoint too, so code-split bundles like rust-cache v2.9.2, setup-java v6 and setup-deno v2 reach the v2 cache API. Entrypoints may point anywhere in the action's repository, as `gradle/actions/setup-gradle` does with `../dist`.

The `.LOCALHOST` match is now anchored on the `.GHE.COM` test before it, so another library's uppercase localhost check is never rewritten.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1328
Reviewed-by: Lunny Xiao <[email protected]>
2026-10-21 05:26:32 +00:00
silverwind 1a7f09224f perf: speed up job setup, action preparation and the cache server (#1327)
- Look up containers by name instead of listing all of them
- On Linux, cache the docker proxy discovery instead of starting a probe container per job
- Download a job's actions concurrently and share identical fetches between concurrent jobs
- Cache `runner.arch` per run and poll service health every 250ms
- Cache server: queue on an in-process lock instead of polling the database file lock, upload S3 objects up to 64MiB in one request and larger ones in 4 concurrent parts

| `exec`, median of 5 | Before | After |
|---|---:|---:|
| 1 step | 1.70s | 0.87s |
| 10 steps | 2.84s | 1.97s |
| Service + step | 3.78s | 2.18s |
| checkout + Node action | 3.64s | 2.52s |

Reviewed-on: https://gitea.com/gitea/runner/pulls/1327
Reviewed-by: Zettat123 <[email protected]>
2026-10-21 05:12:54 +00:00
131 changed files with 4063 additions and 7778 deletions

No files matched your search

-1
View File
@@ -11,7 +11,6 @@ on:
env:
DOCKER_ORG: gitea
DOCKER_LATEST: nightly
jobs:
goreleaser:
-1
View File
@@ -69,7 +69,6 @@ jobs:
image: catthehacker/ubuntu:act-latest
env:
DOCKER_ORG: gitea
DOCKER_LATEST: latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
-20
View File
@@ -28,38 +28,18 @@ builds:
ignore:
- goos: darwin
goarch: arm
- goos: darwin
goarch: ppc64le
- goos: darwin
goarch: s390x
- goos: windows
goarch: ppc64le
- goos: windows
goarch: s390x
- goos: windows
goarch: arm
goarm: "5"
- goos: windows
goarch: arm
goarm: "6"
- goos: windows
goarch: arm
goarm: "7"
- goos: windows
goarch: arm64
- goos: freebsd
goarch: ppc64le
- goos: freebsd
goarch: s390x
- goos: freebsd
goarch: arm
goarm: "5"
- goos: freebsd
goarch: arm
goarm: "6"
- goos: freebsd
goarch: arm
goarm: "7"
- goos: freebsd
goarch: arm64
flags:
+2 -2
View File
@@ -28,8 +28,8 @@ depending on the prefix:
- **No prefix?** Do not escape, and split multi-line values into one row each.
- **Interpolating a secret?** Masking runs after escaping, so `AppendSecretMasker` registers the
encoded forms too.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the reporter
decodes exactly those two when folding a location into an annotation.
- Command *properties* also escape `%3A` and `%2C`, which the UI never decodes, so the command
handler folds an annotation's location into its decoded message and escapes the result again.
## Logging
-4
View File
@@ -14,7 +14,6 @@ GOFILES := $(shell find . -type f -name "*.go" -o -name "go.mod" ! -name "genera
DOCKER_IMAGE ?= gitea/runner
DOCKER_TAG ?= nightly
DOCKER_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)
DOCKER_ROOTLESS_REF := $(DOCKER_IMAGE):$(DOCKER_TAG)-dind-rootless
GOLANGCI_LINT_PACKAGE ?= github.com/golangci/golangci-lint/v2/cmd/[email protected] # renovate: datasource=go
GOVULNCHECK_PACKAGE ?= golang.org/x/vuln/cmd/[email protected] # renovate: datasource=go
@@ -34,15 +33,12 @@ endif
ifeq ($(OS), Windows_NT)
GOFLAGS := -v -buildmode=exe
EXECUTABLE ?= $(EXECUTABLE).exe
GO_ENV_WINDOWS := set GOOS=windows&&
else ifeq ($(OS), Windows)
GOFLAGS := -v -buildmode=exe
EXECUTABLE ?= $(EXECUTABLE).exe
GO_ENV_WINDOWS := set GOOS=windows&&
else
GOFLAGS := -v
EXECUTABLE ?= $(EXECUTABLE)
GO_ENV_WINDOWS := GOOS=windows
endif
+270 -146
View File
@@ -32,8 +32,8 @@ import (
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/internal/pkg/disk"
"github.com/timshannon/bolthold"
"go.etcd.io/bbolt"
berrors "go.etcd.io/bbolt/errors"
)
const (
@@ -44,6 +44,8 @@ const (
// Short enough that a leaked URL is near-worthless; long enough to let the
// @actions/cache client download a big blob that was returned from /cache.
artifactURLTTL = 10 * time.Minute
dbOpenTimeout = 5 * time.Second
)
type credKey struct{}
@@ -79,12 +81,14 @@ type Handler struct {
dir string
storage cacheStorage
s3 *s3Storage
router *http.ServeMux
listener net.Listener
port int
server *http.Server
logger *logging.Logger
dbLock chan struct{} // queues store openers without polling bolt's file lock
entries cacheEntries // guarded by dbLock
gcing atomic.Bool
gcAt time.Time
@@ -142,6 +146,7 @@ func StartHandler(opts Options) (*Handler, error) {
internalSecret: opts.InternalSecret,
policy: opts.Policy.withDefaults(),
freeDisk: disk.FreeBytes,
dbLock: make(chan struct{}, 1),
}
if logger == nil {
@@ -194,11 +199,9 @@ func StartHandler(opts Options) (*Handler, error) {
h.storage = h.s3
}
secret, err := loadOrCreateSecret(dir)
if err != nil {
if h.secret, err = loadOrCreateSecret(dir); err != nil {
return nil, err
}
h.secret = secret
router := http.NewServeMux()
router.HandleFunc("GET "+apiPath+"/cache", h.bearerAuth(h.find))
@@ -217,8 +220,6 @@ func StartHandler(opts Options) (*Handler, error) {
h.registerV2Routes(router)
router.HandleFunc(artifactServicePath, h.forwardOrNotFound)
h.router = router
h.gcCache()
if err := h.serve(opts.Port, router); err != nil {
@@ -327,16 +328,7 @@ func (h *Handler) RegisterJob(token string, cred JobCredential) func() {
}
}
h.credMu.Unlock()
return func() {
h.credMu.Lock()
if entry, ok := h.creds[token]; ok {
entry.refs--
if entry.refs <= 0 {
delete(h.creds, token)
}
}
h.credMu.Unlock()
}
return func() { h.RevokeJob(token) }
}
// RevokeJob explicitly revokes one registration of token, mirroring one call
@@ -399,10 +391,7 @@ func (h *Handler) Close() error {
}
var retErr error
if h.server != nil {
err := h.server.Close()
if err != nil {
retErr = err
}
retErr = h.server.Close()
h.server = nil
}
if h.listener != nil {
@@ -418,16 +407,170 @@ func (h *Handler) Close() error {
return retErr
}
func (h *Handler) openDB() (*bolthold.Store, error) {
return bolthold.Open(filepath.Join(h.dir, "bolt.db"), 0o644, &bolthold.Options{
Encoder: func(value any) ([]byte, error) { return json.Marshal(value) },
Decoder: func(data []byte, value any) error { return json.Unmarshal(data, value) },
Options: &bbolt.Options{
Timeout: 5 * time.Second,
NoGrowSync: bbolt.DefaultOptions.NoGrowSync,
FreelistType: bbolt.DefaultOptions.FreelistType,
},
// cacheBucket holds each entry as JSON under its decimal ID, the layout bolthold wrote, so older stores stay readable.
var cacheBucket = []byte("Cache")
var errCacheNotFound = errors.New("cache not found")
type cacheDB struct {
bolt *bbolt.DB
close func() error
entries *cacheEntries
}
// cacheEntries mirrors the bucket as of transaction tx, so queries decode it again only after another process wrote.
type cacheEntries struct {
tx int
byID map[uint64]*Cache
}
func (db *cacheDB) Close() error {
return db.close()
}
func cacheKey(id uint64) []byte {
return strconv.AppendUint(nil, id, 10)
}
func (db *cacheDB) get(id uint64) (*Cache, error) {
if err := db.bolt.View(db.load); err != nil {
return nil, err
}
cache, ok := db.entries.byID[id]
if !ok {
return nil, errCacheNotFound
}
copied := *cache
return &copied, nil
}
// find returns copies of the matching entries in ID order, so ties break the same way every time.
func (db *cacheDB) find(match func(*Cache) bool) ([]*Cache, error) {
if err := db.bolt.View(db.load); err != nil {
return nil, err
}
var caches []*Cache
for _, cache := range db.entries.byID {
if match(cache) {
copied := *cache
caches = append(caches, &copied)
}
}
slices.SortFunc(caches, func(left, right *Cache) int { return cmp.Compare(left.ID, right.ID) })
return caches, nil
}
func (db *cacheDB) load(tx *bbolt.Tx) error {
if db.entries.byID != nil && db.entries.tx == tx.ID() {
return nil
}
byID := make(map[uint64]*Cache)
if bucket := tx.Bucket(cacheBucket); bucket != nil {
if err := bucket.ForEach(func(key, data []byte) (err error) {
cache := &Cache{}
if err := json.Unmarshal(data, cache); err != nil {
return err
}
cache.ID, err = strconv.ParseUint(string(key), 10, 64) // an interrupted legacy insert left "id":0 in the value
byID[cache.ID] = cache
return err
}); err != nil {
return err
}
}
db.entries.byID, db.entries.tx = byID, tx.ID()
return nil
}
// update keeps the mirror in step with its own write, and drops it when another process wrote in between or the write rolled back.
func (db *cacheDB) update(fn func(*bbolt.Bucket) error) error {
err := db.bolt.Update(func(tx *bbolt.Tx) error {
if db.entries.tx != tx.ID()-1 {
db.entries.byID = nil
}
db.entries.tx = tx.ID()
bucket, err := tx.CreateBucketIfNotExists(cacheBucket)
if err != nil {
return err
}
return fn(bucket)
})
if err != nil {
db.entries.byID = nil
}
return err
}
func (db *cacheDB) put(bucket *bbolt.Bucket, cache *Cache) error {
data, err := json.Marshal(cache)
if err != nil {
return err
}
if db.entries.byID != nil {
copied := *cache
db.entries.byID[cache.ID] = &copied
}
return bucket.Put(cacheKey(cache.ID), data)
}
func (db *cacheDB) delete(bucket *bbolt.Bucket, id uint64) error {
delete(db.entries.byID, id)
return bucket.Delete(cacheKey(id))
}
// dropLegacyIndexes deletes the index buckets bolthold kept, all created together and never recreated, checking first so later runs do not write.
func (db *cacheDB) dropLegacyIndexes() error {
var found bool
_ = db.bolt.View(func(tx *bbolt.Tx) error {
found = tx.Bucket([]byte("_index:Cache:Key")) != nil
return nil
})
if !found {
return nil
}
return db.bolt.Update(func(tx *bbolt.Tx) error {
for _, field := range []string{"Repo", "Key", "Version", "Complete", "UsedAt", "CreatedAt"} {
if err := tx.DeleteBucket([]byte("_index:Cache:" + field)); err != nil && !errors.Is(err, berrors.ErrBucketNotFound) {
return err
}
}
return nil
})
}
// save rewrites a stored entry, failing once a sweep has removed it so a late commit cannot resurrect it.
func (db *cacheDB) save(cache *Cache) error {
return db.update(func(bucket *bbolt.Bucket) error {
if bucket.Get(cacheKey(cache.ID)) == nil {
return errCacheNotFound
}
return db.put(bucket, cache)
})
}
func (h *Handler) openDB() (store *cacheDB, err error) {
select {
case h.dbLock <- struct{}{}:
case <-time.After(dbOpenTimeout):
return nil, berrors.ErrTimeout
}
defer func() {
if store == nil { // only an opened store hands the lock to Close, even when Open panics
<-h.dbLock
}
}()
db, err := bbolt.Open(filepath.Join(h.dir, "bolt.db"), 0o644, &bbolt.Options{
Timeout: dbOpenTimeout,
NoGrowSync: bbolt.DefaultOptions.NoGrowSync,
FreelistType: bbolt.DefaultOptions.FreelistType,
})
if err != nil {
return nil, err
}
return &cacheDB{bolt: db, entries: &h.entries, close: sync.OnceValue(func() error {
defer func() { <-h.dbLock }()
return db.Close()
})}, nil
}
// syncMetadata mirrors the bucket into the store until ctx ends, so no request waits on S3 for it.
@@ -452,7 +595,7 @@ func (h *Handler) importMetadata(ctx context.Context) error {
if err != nil {
return err
}
local := h.caches(db, bolthold.Where("Complete").Eq(true).Index("Complete"))
local := h.caches(db, func(cache *Cache) bool { return cache.Complete })
db.Close()
published, err := h.s3.published(ctx)
@@ -480,15 +623,15 @@ func (h *Handler) importMetadata(ctx context.Context) error {
return err
}
defer db.Close()
return db.Bolt().Update(func(tx *bbolt.Tx) error {
return db.update(func(bucket *bbolt.Bucket) error {
for _, cache := range added {
if err := db.TxUpsert(tx, cache.ID, cache); err != nil {
if err := db.put(bucket, cache); err != nil {
return err
}
}
for _, cache := range gone {
_ = h.s3.Storage.Remove(cache.ID) // a blob kept on disk before the runner used S3
if err := db.TxDelete(tx, cache.ID, cache); err != nil && !errors.Is(err, bolthold.ErrNotFound) {
if err := db.delete(bucket, cache.ID); err != nil {
return err
}
}
@@ -528,7 +671,7 @@ func (h *Handler) find(w http.ResponseWriter, r *http.Request) {
// lookupCache returns the entry to restore for these keys, or (nil, nil) when there is none:
// either nothing matched, or the match had lost its blob to a prune, in which case the dangling
// entry is dropped on the way out.
func (h *Handler) lookupCache(db *bolthold.Store, repo string, keys []string, version string) (*Cache, error) {
func (h *Handler) lookupCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) {
cache, err := findCache(db, repo, keys, version)
if err != nil || cache == nil {
return nil, err
@@ -588,15 +731,15 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
return
}
cache := &Cache{}
db, err := h.openDB()
if err != nil {
h.responseJSON(w, r, 500, err)
return
}
defer db.Close()
if err := db.Get(id, cache); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
cache, err := db.get(uint64(id))
if err != nil {
if errors.Is(err, errCacheNotFound) {
h.responseJSON(w, r, 400, fmt.Errorf("cache %d: not reserved", id))
return
}
@@ -614,7 +757,7 @@ func (h *Handler) upload(w http.ResponseWriter, r *http.Request) {
return
}
db.Close()
start, _, err := parseContentRange(r.Header.Get("Content-Range"))
start, err := parseContentRange(r.Header.Get("Content-Range"))
if err != nil {
h.responseJSON(w, r, 400, err)
return
@@ -636,15 +779,15 @@ func (h *Handler) commit(w http.ResponseWriter, r *http.Request) {
return
}
cache := &Cache{}
db, err := h.openDB()
if err != nil {
h.responseJSON(w, r, 500, err)
return
}
defer db.Close()
if err := db.Get(id, cache); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
cache, err := db.get(uint64(id))
if err != nil {
if errors.Is(err, errCacheNotFound) {
h.responseJSON(w, r, 400, fmt.Errorf("cache %d: not reserved", id))
return
}
@@ -696,7 +839,7 @@ func (h *Handler) commitCache(cache *Cache) error {
return err
}
defer db.Close()
if err := db.Update(cache.ID, cache); err != nil {
if err := db.save(cache); err != nil {
return err
}
if h.s3 != nil {
@@ -881,10 +1024,8 @@ func bearerToken(r *http.Request) string {
}
func credFromContext(ctx context.Context) JobCredential {
if cred, ok := ctx.Value(credKey{}).(JobCredential); ok {
return cred
}
return JobCredential{}
cred, _ := ctx.Value(credKey{}).(JobCredential)
return cred
}
// computeSignature signs a URL for one cache entry and expiry. purpose is mixed into the
@@ -910,8 +1051,7 @@ func (h *Handler) signedArtifactURL(cred JobCredential, cacheID uint64, exp time
}
// if not found, return (nil, nil) instead of an error.
func findCache(db *bolthold.Store, repo string, keys []string, version string) (*Cache, error) {
cache := &Cache{}
func findCache(db *cacheDB, repo string, keys []string, version string) (*Cache, error) {
for _, prefix := range keys {
// if a key in the list matches exactly, don't return partial matches
exact, err := findExactCache(db, repo, prefix, version, true)
@@ -921,22 +1061,26 @@ func findCache(db *bolthold.Store, repo string, keys []string, version string) (
if exact != nil {
return exact, nil
}
if err := db.FindOne(cache,
bolthold.Where("Repo").Eq(repo).Index("Repo").
And("Key").MatchFunc(func(key string) (bool, error) { return strings.HasPrefix(key, prefix), nil }).
And("Version").Eq(version).
And("Complete").Eq(true).
SortBy("CreatedAt").Reverse()); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
continue
}
caches, err := db.find(func(cache *Cache) bool {
return cache.Repo == repo && strings.HasPrefix(cache.Key, prefix) && cache.Version == version && cache.Complete
})
if err != nil {
return nil, fmt.Errorf("find cache: %w", err)
}
return cache, nil
if cache := latest(caches, func(cache *Cache) int64 { return cache.CreatedAt }); cache != nil {
return cache, nil
}
}
return nil, nil //nolint:nilnil // pre-existing issue from nektos/act
}
func latest(caches []*Cache, stamp func(*Cache) int64) *Cache {
if len(caches) == 0 {
return nil
}
return slices.MaxFunc(caches, func(left, right *Cache) int { return cmp.Compare(stamp(left), stamp(right)) })
}
// findExactCache returns the entry for exactly this key and version, or (nil, nil) if there is
// none. Unlike findCache it never falls back to a prefix (restore-key) match, which is what both
// its callers need: a new key that is only a prefix of an existing key is not the same entry.
@@ -944,43 +1088,33 @@ func findCache(db *bolthold.Store, repo string, keys []string, version string) (
// A completed entry is the one to restore, sorted by when it was written. An incomplete one is a
// reservation being uploaded to, sorted by when it was last written to, because the upload route
// touches UsedAt on every part.
func findExactCache(db *bolthold.Store, repo, key, version string, complete bool) (*Cache, error) {
sortBy := "UsedAt"
if complete {
sortBy = "CreatedAt"
}
cache := &Cache{}
err := db.FindOne(cache,
bolthold.Where("Key").Eq(key).Index("Key").
And("Repo").Eq(repo).
And("Version").Eq(version).
And("Complete").Eq(complete).
SortBy(sortBy).Reverse())
if errors.Is(err, bolthold.ErrNotFound) {
return nil, nil //nolint:nilnil // absence is not an error here
}
func findExactCache(db *cacheDB, repo, key, version string, complete bool) (*Cache, error) {
caches, err := db.find(func(cache *Cache) bool {
return cache.Key == key && cache.Repo == repo && cache.Version == version && cache.Complete == complete
})
if err != nil {
return nil, fmt.Errorf("find cache: %w", err)
}
return cache, nil
stamp := func(cache *Cache) int64 { return cache.UsedAt }
if complete {
stamp = func(cache *Cache) int64 { return cache.CreatedAt }
}
return latest(caches, stamp), nil
}
func (h *Handler) insertCache(db *bolthold.Store, cache *Cache) error {
if h.s3 != nil {
var id [8]byte
_, _ = rand.Read(id[:])
cache.ID = binary.BigEndian.Uint64(id[:]) >> 11 // IDs must be unique across runners, and v1 clients parse an ID as a JS number
return db.Insert(cache.ID, cache)
}
return db.Bolt().Update(func(tx *bbolt.Tx) error {
if err := db.TxInsert(tx, bolthold.NextSequence(), cache); err != nil {
func (h *Handler) insertCache(db *cacheDB, cache *Cache) error {
return db.update(func(bucket *bbolt.Bucket) (err error) {
if h.s3 != nil {
var id [8]byte
_, _ = rand.Read(id[:])
cache.ID = binary.BigEndian.Uint64(id[:]) >> 11 // IDs must be unique across runners, and v1 clients parse an ID as a JS number
} else if cache.ID, err = bucket.NextSequence(); err != nil {
return fmt.Errorf("insert cache: %w", err)
}
// write back id to db
if err := db.TxUpdate(tx, cache.ID, cache); err != nil {
return fmt.Errorf("write back id to db: %w", err)
if bucket.Get(cacheKey(cache.ID)) != nil {
return fmt.Errorf("insert cache: id %d is taken", cache.ID)
}
return nil
return db.put(bucket, cache)
})
}
@@ -996,11 +1130,11 @@ func (h *Handler) touchCache(id uint64, requireIncomplete bool) error {
}
defer db.Close()
cache := &Cache{}
if err := db.Get(id, cache); err != nil {
if errors.Is(err, bolthold.ErrNotFound) {
return nil
}
cache, err := db.get(id)
if errors.Is(err, errCacheNotFound) {
return nil
}
if err != nil {
return err
}
if requireIncomplete && cache.Complete {
@@ -1010,7 +1144,7 @@ func (h *Handler) touchCache(id uint64, requireIncomplete bool) error {
return nil
}
cache.UsedAt = time.Now().Unix()
return db.Update(cache.ID, cache)
return db.save(cache)
}
const (
@@ -1056,9 +1190,6 @@ func (p Policy) withDefaults() Policy {
}
func (h *Handler) gcCache() {
if h.gcing.Load() {
return
}
if !h.gcing.CompareAndSwap(false, true) {
return
}
@@ -1077,6 +1208,9 @@ func (h *Handler) gcCache() {
}
defer db.Close()
if err := db.dropLegacyIndexes(); err != nil {
h.logger.Warnf("drop legacy indexes: %v", err)
}
h.evictIncomplete(db)
if h.s3 != nil {
return // lifecycle rules on the bucket expire its entries
@@ -1089,7 +1223,7 @@ func (h *Handler) gcCache() {
// evictForFreeSpace bounds the volume itself, so it also covers bytes the cache never
// accounted for.
func (h *Handler) evictForFreeSpace(db *bolthold.Store) {
func (h *Handler) evictForFreeSpace(db *cacheDB) {
free, err := h.freeDisk(h.dir)
if err != nil {
h.logger.Debugf("free disk check: %v", err) // unsupported platform, treat as unavailable rather than full
@@ -1109,38 +1243,31 @@ func (h *Handler) evictForFreeSpace(db *bolthold.Store) {
}
// evictIncomplete removes uploads that stopped part way, which are most likely broken.
func (h *Handler) evictIncomplete(db *bolthold.Store) {
h.sweep(db, bolthold.
Where("UsedAt").Lt(time.Now().Add(-uploadStallTimeout).Unix()).
And("Complete").Eq(false).
Index("UsedAt"))
func (h *Handler) evictIncomplete(db *cacheDB) {
stalled := time.Now().Add(-uploadStallTimeout).Unix()
h.sweep(db, func(cache *Cache) bool { return cache.UsedAt < stalled && !cache.Complete })
}
func (h *Handler) evictExpired(db *bolthold.Store) {
func (h *Handler) evictExpired(db *cacheDB) {
if h.policy.Retention <= 0 {
return
}
// Never below inUseGrace, or a short retention would outrun a signed URL already issued.
window := max(h.policy.Retention+touchStale, inUseGrace)
h.sweep(db, bolthold.Where("UsedAt").Lt(time.Now().Add(-window).Unix()).Index("UsedAt"))
expired := time.Now().Add(-max(h.policy.Retention+touchStale, inUseGrace)).Unix()
h.sweep(db, func(cache *Cache) bool { return cache.UsedAt < expired })
}
// evictSuperseded removes entries a newer one with the same key and version replaced. The
// aggregation includes Repo so two repos sharing a (key, version) do not evict each other.
func (h *Handler) evictSuperseded(db *bolthold.Store) {
results, err := db.FindAggregate(&Cache{}, bolthold.Where("Complete").Eq(true).Index("Complete"), "Repo", "Key", "Version")
if err != nil {
h.logger.Warnf("find aggregate caches: %v", err)
return
// grouping includes Repo so two repos sharing a (key, version) do not evict each other.
func (h *Handler) evictSuperseded(db *cacheDB) {
type entry struct{ repo, key, version string }
groups := make(map[entry][]*Cache)
for _, cache := range h.caches(db, func(cache *Cache) bool { return cache.Complete }) {
group := entry{cache.Repo, cache.Key, cache.Version}
groups[group] = append(groups[group], cache)
}
var caches []*Cache
for _, result := range results {
if result.Count() <= 1 {
continue
}
result.Sort("CreatedAt")
caches = caches[:0]
result.Reduction(&caches)
for _, caches := range groups {
slices.SortFunc(caches, func(left, right *Cache) int { return cmp.Compare(left.CreatedAt, right.CreatedAt) })
for _, cache := range caches[:len(caches)-1] {
if inUse(cache) {
continue
@@ -1152,7 +1279,7 @@ func (h *Handler) evictSuperseded(db *bolthold.Store) {
// evictOversized applies the per-repository limit, then the whole-store one. Only completed
// entries count, since only those carry a size measured at commit rather than claimed.
func (h *Handler) evictOversized(db *bolthold.Store) {
func (h *Handler) evictOversized(db *cacheDB) {
if h.policy.RepoSizeLimit > 0 {
byRepo := make(map[string][]*Cache)
for _, cache := range h.completedByUse(db) {
@@ -1167,7 +1294,7 @@ func (h *Handler) evictOversized(db *bolthold.Store) {
// evictTotal caps the store as a whole. It re-queries because the per-repo pass may have
// deleted rows an earlier result still holds.
func (h *Handler) evictTotal(db *bolthold.Store) {
func (h *Handler) evictTotal(db *cacheDB) {
if h.policy.SizeLimit <= 0 {
return
}
@@ -1175,15 +1302,15 @@ func (h *Handler) evictTotal(db *bolthold.Store) {
}
// evictRepo reclaims space when a commit pushes a repo over, rather than at the next sweep.
func (h *Handler) evictRepo(db *bolthold.Store, repo string) {
func (h *Handler) evictRepo(db *cacheDB, repo string) {
if h.policy.RepoSizeLimit <= 0 {
return
}
h.evictTo(db, h.cachesByUse(db, bolthold.Where("Repo").Eq(repo).And("Complete").Eq(true).Index("Repo")), h.policy.RepoSizeLimit, "repository "+repo)
h.evictTo(db, h.cachesByUse(db, func(cache *Cache) bool { return cache.Repo == repo && cache.Complete }), h.policy.RepoSizeLimit, "repository "+repo)
}
// evictTo deletes until caches fit limit. caches must be ordered by UsedAt ascending.
func (h *Handler) evictTo(db *bolthold.Store, caches []*Cache, limit int64, scope string) {
func (h *Handler) evictTo(db *cacheDB, caches []*Cache, limit int64, scope string) {
// An entry bigger than the limit never fits, so it goes on its own account instead of
// dragging every neighbour out first and then following them next sweep.
fits := caches[:0]
@@ -1230,41 +1357,39 @@ func touchNeeded(cache *Cache) bool {
// touch stamps UsedAt through the caller's store, a bolt write on the read path. It cannot
// go through touchCache, which opens its own store and would block on the exclusive lock
// for as long as the caller holds one.
func (h *Handler) touch(db *bolthold.Store, cache *Cache) {
func (h *Handler) touch(db *cacheDB, cache *Cache) {
if !touchNeeded(cache) {
return
}
cache.UsedAt = time.Now().Unix()
if err := db.Update(cache.ID, cache); err != nil {
if err := db.save(cache); err != nil {
h.logger.Warnf("touch cache: %v", err)
}
}
func (h *Handler) sweep(db *bolthold.Store, query *bolthold.Query) {
for _, cache := range h.caches(db, query) {
func (h *Handler) sweep(db *cacheDB, match func(*Cache) bool) {
for _, cache := range h.caches(db, match) {
h.deleteCache(db, cache)
}
}
func (h *Handler) caches(db *bolthold.Store, query *bolthold.Query) []*Cache {
var caches []*Cache
if err := db.Find(&caches, query); err != nil {
func (h *Handler) caches(db *cacheDB, match func(*Cache) bool) []*Cache {
caches, err := db.find(match)
if err != nil {
h.logger.Warnf("find caches: %v", err)
}
return caches
}
// cachesByUse returns matches least recently accessed first, sorting here rather than with
// bolthold's SortBy, which reflects over every field it compares.
func (h *Handler) cachesByUse(db *bolthold.Store, query *bolthold.Query) []*Cache {
caches := h.caches(db, query)
func (h *Handler) cachesByUse(db *cacheDB, match func(*Cache) bool) []*Cache {
caches := h.caches(db, match)
slices.SortFunc(caches, func(a, b *Cache) int { return cmp.Compare(a.UsedAt, b.UsedAt) })
return caches
}
// completedByUse returns every entry the size limits count, least recently accessed first.
func (h *Handler) completedByUse(db *bolthold.Store) []*Cache {
return h.cachesByUse(db, bolthold.Where("Complete").Eq(true).Index("Complete"))
func (h *Handler) completedByUse(db *cacheDB) []*Cache {
return h.cachesByUse(db, func(cache *Cache) bool { return cache.Complete })
}
func totalSize(caches []*Cache) int64 {
@@ -1277,7 +1402,7 @@ func totalSize(caches []*Cache) int64 {
// deleteCache drops an entry and its bytes, reporting whether it went fully. The blob goes
// first, so a failed unlink leaves the row for the next sweep instead of orphaning bytes.
func (h *Handler) deleteCache(db *bolthold.Store, cache *Cache) bool {
func (h *Handler) deleteCache(db *cacheDB, cache *Cache) bool {
storage := h.storage
if h.s3 != nil && !cache.Complete {
storage = h.s3.Storage // an upload not yet committed has nothing in the bucket
@@ -1286,7 +1411,7 @@ func (h *Handler) deleteCache(db *bolthold.Store, cache *Cache) bool {
h.logger.Warnf("remove cache blob: %v", err)
return false
}
if err := db.Delete(cache.ID, cache); err != nil {
if err := db.update(func(bucket *bbolt.Bucket) error { return db.delete(bucket, cache.ID) }); err != nil {
h.logger.Warnf("delete cache: %v", err)
return false
}
@@ -1311,18 +1436,17 @@ func (h *Handler) responseJSON(w http.ResponseWriter, r *http.Request, code int,
_, _ = w.Write(data)
}
func parseContentRange(s string) (int64, int64, error) {
func parseContentRange(s string) (int64, error) {
// support the format like "bytes 11-22/*" only
s, _, _ = strings.Cut(strings.TrimPrefix(s, "bytes "), "/")
s1, s2, _ := strings.Cut(s, "-")
start, err := strconv.ParseInt(s1, 10, 64)
if err != nil {
return 0, 0, fmt.Errorf("parse %q: %w", s, err)
return 0, fmt.Errorf("parse %q: %w", s, err)
}
stop, err := strconv.ParseInt(s2, 10, 64)
if err != nil {
return 0, 0, fmt.Errorf("parse %q: %w", s, err)
if _, err := strconv.ParseInt(s2, 10, 64); err != nil {
return 0, fmt.Errorf("parse %q: %w", s, err)
}
return start, stop, nil
return start, nil
}
File diff suppressed because it is too large. Load diff
+4 -36
View File
@@ -19,8 +19,6 @@ import (
"github.com/stretchr/testify/require"
)
// v2Call posts a twirp request to the cache service and returns the decoded response.
// Field names are the proto ones, which is what the toolkit's client sends.
func v2Call(t *testing.T, handler *Handler, client *http.Client, method string, request any) map[string]any {
t.Helper()
@@ -37,7 +35,6 @@ func v2Call(t *testing.T, handler *Handler, client *http.Client, method string,
return got
}
// putBlob uploads to a signed URL and returns the status, so a test can assert a refusal.
func putBlob(t *testing.T, url string, content []byte) int {
t.Helper()
@@ -47,8 +44,6 @@ func putBlob(t *testing.T, url string, content []byte) int {
require.NoError(t, err)
defer resp.Body.Close()
if resp.StatusCode == http.StatusCreated {
// The Azure SDK client dereferences this header without checking, so a blob upload that
// omits it panics the caller rather than failing it.
require.NotEmpty(t, resp.Header.Get("x-ms-request-id"))
}
return resp.StatusCode
@@ -67,8 +62,6 @@ func getURL(t *testing.T, url string) []byte {
return body
}
// saveV2 runs the reserve/upload/finalize sequence and returns the finalize response along
// with the upload URL it used.
func saveV2(t *testing.T, handler *Handler, key, version string, content []byte) (finalized map[string]any, uploadURL string) {
t.Helper()
@@ -84,9 +77,6 @@ func saveV2(t *testing.T, handler *Handler, key, version string, content []byte)
}), uploadURL
}
// The whole round trip an actions/cache v2 client makes, plus the guarantees on the signed
// URLs it is handed: unsigned requests are refused, an upload URL cannot be replayed to read
// or to replace a finalized entry.
func TestCacheServiceV2RoundTrip(t *testing.T) {
handler := newTestHandler(t, Policy{})
content := []byte("the cached archive")
@@ -98,10 +88,8 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
require.Equal(t, true, finalized["ok"])
assert.NotEmpty(t, finalized["entry_id"])
// The upload URL outlives the finalize call, so replaying it must not poison the entry,
// and it is an upload URL only: nothing reads a blob back through it.
assert.Equal(t, http.StatusBadRequest, putBlob(t, uploadURL, []byte("poisoned")))
resp, err := http.Get(uploadURL) //nolint:noctx // the URL is the server under test
resp, err := http.Get(uploadURL)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode)
@@ -114,8 +102,6 @@ func TestCacheServiceV2RoundTrip(t *testing.T) {
assert.Equal(t, content, getURL(t, downloadURL))
}
// A large archive is staged as blocks and only put in order by the final block list, so
// blocks that arrive out of order must still be assembled the way the client asked.
func TestCacheServiceV2BlockUpload(t *testing.T) {
handler := newTestHandler(t, Policy{})
@@ -130,7 +116,6 @@ func TestCacheServiceV2BlockUpload(t *testing.T) {
blocks[blockID] = []byte(part)
order = append(order, blockID)
}
// Upload in an order that is not the block list order.
for _, blockID := range []string{order[2], order[0], order[1]} {
require.Equal(t, http.StatusCreated, putBlob(t, uploadURL+"&comp=block&blockid="+blockID, blocks[blockID]))
}
@@ -174,8 +159,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"])
})
// The toolkit serialises with the proto field names; the camelCase spellings of the same
// proto JSON mapping are accepted alongside them.
for _, field := range []string{"restore_keys", "restoreKeys"} {
t.Run("restore keys match by prefix, spelled "+field, func(t *testing.T) {
got := v2Call(t, handler, testClient, "GetCacheEntryDownloadURL", map[string]any{
@@ -191,9 +174,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, again["ok"])
})
// A key that is only a prefix of an existing one is a different entry, so the
// reservation check must be exact and not a restore-key prefix match, or the shorter
// key would be reported as existing and silently never saved.
t.Run("a prefix of an existing key is still reserved", func(t *testing.T) {
reserved := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "deps", "version": "v1"})
require.Equal(t, true, reserved["ok"])
@@ -219,7 +199,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"])
})
// The size the client declares is what Commit validates the assembled archive against.
t.Run("finalizing with the wrong size is not ok", func(t *testing.T) {
created := v2Call(t, handler, testClient, "CreateCacheEntry", map[string]any{"key": "wrong-size", "version": "v1"})
require.Equal(t, http.StatusCreated, putBlob(t, created["signed_upload_url"].(string), []byte("four")))
@@ -230,8 +209,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.Equal(t, false, got["ok"])
})
// Both API versions are served from one store, so an entry written through v2 is a hit for
// a v1 client asking for the same key and version.
t.Run("a v1 client sees an entry written through v2", func(t *testing.T) {
resp, err := testClient.Get(fmt.Sprintf("%s%s/cache?keys=deps-abc&version=v1", handler.ExternalURL(), apiPath))
require.NoError(t, err)
@@ -244,7 +221,6 @@ func TestCacheServiceV2Lookups(t *testing.T) {
assert.NotEmpty(t, got["archiveLocation"])
})
// The cache of one repository must stay invisible to another, as it does for the v1 API.
t.Run("another repository sees nothing", func(t *testing.T) {
handler.RegisterJob("other-runtime-token", JobCredential{Repo: "other/repo"})
otherClient := &http.Client{Transport: &bearerTransport{token: "other-runtime-token"}}
@@ -321,20 +297,12 @@ func TestCacheServiceV2ConcurrentSaveAndRetries(t *testing.T) {
"another job's upload is not this job's to finalize")
base := handler.ExternalURL() + apiPath
reserveV1, err := json.Marshal(&Request{Key: "v1-made", Version: "v1", Size: int64(len(content))})
require.NoError(t, err)
resp, err := testClient.Post(base+"/caches", "application/json", bytes.NewReader(reserveV1))
require.NoError(t, err)
var madeByV1 struct {
CacheID uint64 `json:"cacheId"`
}
require.NoError(t, json.UnmarshalRead(resp.Body, &madeByV1))
resp.Body.Close()
madeByV1 := reserveCache(t, testClient, base, "v1-made", "v1", int64(len(content)))
upload, err := http.NewRequestWithContext(t.Context(), http.MethodPatch,
fmt.Sprintf("%s/caches/%d", base, madeByV1.CacheID), bytes.NewReader(content))
fmt.Sprintf("%s/caches/%d", base, madeByV1), bytes.NewReader(content))
require.NoError(t, err)
upload.Header.Set("Content-Range", fmt.Sprintf("bytes 0-%d/*", len(content)-1))
resp, err = testClient.Do(upload)
resp, err := testClient.Do(upload)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
+7 -7
View File
@@ -11,13 +11,13 @@ type Request struct {
}
type Cache struct {
ID uint64 `json:"id" boltholdKey:"ID"`
Repo string `json:"repo" boltholdIndex:"Repo"`
Key string `json:"key" boltholdIndex:"Key"`
Version string `json:"version" boltholdIndex:"Version"`
ID uint64 `json:"id"`
Repo string `json:"repo"`
Key string `json:"key"`
Version string `json:"version"`
Size int64 `json:"cacheSize"`
Owner string `json:"owner"` // hashed bearer, so a job's own retry is not another job's reservation
Complete bool `json:"complete" boltholdIndex:"Complete"`
UsedAt int64 `json:"usedAt" boltholdIndex:"UsedAt"`
CreatedAt int64 `json:"createdAt" boltholdIndex:"CreatedAt"`
Complete bool `json:"complete"`
UsedAt int64 `json:"usedAt"`
CreatedAt int64 `json:"createdAt"`
}
+2 -8
View File
@@ -4,6 +4,7 @@
package artifactcache
import (
"cmp"
"crypto/tls"
"errors"
"fmt"
@@ -74,14 +75,7 @@ func (h *Handler) forwardOrNotFound(w http.ResponseWriter, r *http.Request) {
// A protocol switch loses the bearer or the agent, and an untrusted instance needs skipped verification.
func (h *Handler) mustProxy(r *http.Request, cred JobCredential, target *url.URL) bool {
if cred.InsecureTLS && target.Scheme == "https" {
return true
}
scheme := r.Header.Get("X-Forwarded-Proto")
if scheme == "" {
scheme = "http"
}
return scheme != target.Scheme
return cred.InsecureTLS && target.Scheme == "https" || cmp.Or(r.Header.Get("X-Forwarded-Proto"), "http") != target.Scheme
}
func mergeQuery(target, request string) string {
+32 -10
View File
@@ -24,6 +24,10 @@ import (
"strconv"
"strings"
"time"
"gitea.com/gitea/runner/act/common"
"golang.org/x/sync/errgroup"
)
// S3Options configures the bucket that stores completed cache entries.
@@ -47,7 +51,7 @@ type cacheStorage interface {
Remove(uint64) error
}
const s3PartSize = 5 << 30 // the largest part S3 accepts
const s3PartSize = 64 << 20
// s3Storage stages uploads on disk and keeps completed entries and their metadata in a bucket.
type s3Storage struct {
@@ -69,6 +73,7 @@ func newS3Storage(staging *Storage, opts S3Options) (*s3Storage, error) {
}
transport, _ := http.DefaultTransport.(*http.Transport)
transport = transport.Clone()
transport.MaxIdleConnsPerHost = 32 // 4 parts each for the 8 jobs a runner commonly runs
transport.ResponseHeaderTimeout = time.Minute // a stalled endpoint must not hold the store forever
pathStyle := !strings.HasSuffix(endpoint.Hostname(), ".amazonaws.com")
if opts.PathStyle != nil {
@@ -209,26 +214,43 @@ func (s *s3Storage) Commit(id uint64, size int64) (int64, error) {
return written, nil
}
// upload always goes in parts, the only way S3 takes an object over 5 GiB.
func (s *s3Storage) upload(ctx context.Context, key string, file io.ReaderAt, size int64) error {
func (s *s3Storage) upload(ctx context.Context, key string, file io.ReaderAt, size int64) (uploadErr error) {
if size <= s3PartSize {
_, err := s.call(ctx, http.MethodPut, key, nil, io.NewSectionReader(file, 0, size), nil)
return err
}
var created struct {
UploadID string `xml:"UploadId"`
}
if _, err := s.call(ctx, http.MethodPost, key, url.Values{"uploads": {""}}, nil, &created); err != nil {
return err
}
defer func() {
if uploadErr != nil && created.UploadID != "" {
abortCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer cancel()
_, _ = s.call(abortCtx, http.MethodDelete, key, url.Values{"uploadId": {created.UploadID}}, nil, nil)
}
}()
var completion struct {
XMLName xml.Name `xml:"CompleteMultipartUpload"`
Parts []s3Part `xml:"Part"`
}
for offset := int64(0); offset == 0 || offset < size; offset += s3PartSize {
number, length := len(completion.Parts)+1, min(s3PartSize, size-offset)
query := url.Values{"partNumber": {strconv.Itoa(number)}, "uploadId": {created.UploadID}}
header, err := s.call(ctx, http.MethodPut, key, query, io.NewSectionReader(file, offset, length), nil)
if err != nil {
partSize := max(int64(s3PartSize), (size-1)/10000+1)
completion.Parts = make([]s3Part, (size-1)/partSize+1)
workers, workerCtx := errgroup.WithContext(ctx)
workers.SetLimit(4)
for part := range completion.Parts {
workers.Go(func() (err error) {
defer common.RecoverPanic(&err)
offset := int64(part) * partSize
header, err := s.call(workerCtx, http.MethodPut, key, url.Values{"partNumber": {strconv.Itoa(part + 1)}, "uploadId": {created.UploadID}}, io.NewSectionReader(file, offset, min(partSize, size-offset)), nil)
completion.Parts[part] = s3Part{part + 1, header.Get("ETag")}
return err
}
completion.Parts = append(completion.Parts, s3Part{number, header.Get("ETag")})
})
}
if err := workers.Wait(); err != nil {
return err
}
body, err := xml.Marshal(completion)
if err != nil {
+79 -7
View File
@@ -4,21 +4,48 @@
package artifactcache
import (
"encoding/xml"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
backend := s3mem.New()
require.NoError(t, backend.CreateBucket("cache"))
server := httptest.NewServer(gofakes3.New(backend).Server())
var objects sync.Map
var uploads atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
key := strings.TrimPrefix(request.URL.Path, "/cache/")
switch {
case request.Method == http.MethodPut:
if strings.Contains(key, "/blobs/") {
uploads.Add(1)
}
body, err := io.ReadAll(request.Body)
assert.NoError(t, err)
objects.Store(key, body)
case request.URL.Query().Has("list-type"):
_, _ = io.WriteString(writer, "<ListBucketResult>")
objects.Range(func(stored, _ any) bool {
_, _ = io.WriteString(writer, "<Contents><Key>"+stored.(string)+"</Key></Contents>")
return true
})
_, _ = io.WriteString(writer, "</ListBucketResult>")
default:
if body, ok := objects.Load(key); ok {
_, _ = writer.Write(body.([]byte))
} else {
writer.WriteHeader(http.StatusNotFound)
}
}
}))
t.Cleanup(server.Close)
opts := Options{S3: &S3Options{Endpoint: server.URL, Bucket: "cache", Prefix: "runners", AccessKey: "key", SecretKey: "secret"}}
saver, restorer := startTestHandler(t, opts), startTestHandler(t, opts)
@@ -26,17 +53,62 @@ func TestHandler_S3SharesEntriesAndDropsExpiredOnes(t *testing.T) {
finalized, _ := saveV2(t, saver, "deps", "v1", []byte("archive"))
require.Equal(t, true, finalized["ok"])
assert.EqualValues(t, 1, uploads.Load())
require.NoError(t, restorer.importMetadata(t.Context()))
found := v2Call(t, restorer, testClient, "GetCacheEntryDownloadURL", entry)
require.Equal(t, true, found["ok"])
assert.Equal(t, []byte("archive"), getURL(t, found["signed_download_url"].(string)))
require.NoError(t, backend.ForceDeleteBucket("cache"))
require.NoError(t, backend.CreateBucket("cache"))
objects.Clear()
require.NoError(t, restorer.importMetadata(t.Context()))
assert.Equal(t, true, v2Call(t, restorer, testClient, "CreateCacheEntry", entry)["ok"])
}
type s3TestTransport func(*http.Request) (*http.Response, error)
func (transport s3TestTransport) RoundTrip(req *http.Request) (*http.Response, error) {
return transport(req)
}
func TestS3StorageUploadsLargeObjectsInPartsAndAbortsOnFailure(t *testing.T) {
for _, failingPart := range []string{"", "2"} {
var parts, aborts atomic.Int32
var completion struct {
Parts []s3Part `xml:"Part"`
}
storage := &s3Storage{bucketURL: httptest.NewRequest(http.MethodGet, "http://cache.local/bucket", nil).URL}
storage.client = &http.Client{Transport: s3TestTransport(func(req *http.Request) (*http.Response, error) {
query, response := req.URL.Query(), &http.Response{StatusCode: http.StatusOK, Header: make(http.Header), Body: http.NoBody}
switch {
case query.Has("uploads"):
response.Body = io.NopCloser(strings.NewReader("<InitiateMultipartUploadResult><UploadId>upload</UploadId></InitiateMultipartUploadResult>"))
case req.Method == http.MethodPut:
parts.Add(1)
response.Header.Set("ETag", "etag-"+query.Get("partNumber"))
if query.Get("partNumber") == failingPart {
response.StatusCode, response.Body = http.StatusBadRequest, io.NopCloser(strings.NewReader("<Error><Code>InvalidPart</Code></Error>"))
}
case req.Method == http.MethodPost:
require.NoError(t, xml.NewDecoder(req.Body).Decode(&completion))
response.Body = io.NopCloser(strings.NewReader("<CompleteMultipartUploadResult/>"))
case req.Method == http.MethodDelete:
aborts.Add(1)
}
return response, nil
})}
err := storage.upload(t.Context(), "blobs/1", strings.NewReader(""), 2*int64(s3PartSize)+1)
assert.EqualValues(t, 3, parts.Load())
if failingPart == "" {
require.NoError(t, err)
assert.Equal(t, []s3Part{{1, "etag-1"}, {2, "etag-2"}, {3, "etag-3"}}, completion.Parts)
assert.Zero(t, aborts.Load())
} else {
require.ErrorContains(t, err, "InvalidPart")
assert.EqualValues(t, 1, aborts.Load())
}
}
}
func TestS3StorageSignsLikeAWS(t *testing.T) {
storage := &s3Storage{opts: S3Options{Region: "us-east-1", AccessKey: "AKIAIOSFODNN7EXAMPLE", SecretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}}
req := httptest.NewRequest(http.MethodGet, "https://examplebucket.s3.amazonaws.com/?max-keys=2&prefix=J", nil)
+5 -10
View File
@@ -32,13 +32,11 @@ func NewStorage(rootDir string) (*Storage, error) {
}
func (s *Storage) Exist(id uint64) (bool, error) {
name := s.filename(id)
if _, err := os.Stat(name); os.IsNotExist(err) {
_, err := os.Stat(s.filename(id))
if os.IsNotExist(err) {
return false, nil
} else if err != nil {
return false, err
}
return true, nil
return err == nil, err
}
func (s *Storage) Write(id uint64, offset int64, reader io.Reader) error {
@@ -92,9 +90,7 @@ func (s *Storage) OrderBlocks(id uint64, blockIDs []string) error {
}
func (s *Storage) Commit(id uint64, size int64) (int64, error) {
defer func() {
_ = os.RemoveAll(s.tempDir(id))
}()
defer func() { _ = os.RemoveAll(s.tempDir(id)) }()
name := s.filename(id)
tempNames, err := s.tempNames(id)
@@ -153,8 +149,7 @@ func assemble(name string, tempNames []string) (int64, error) {
}
func (s *Storage) Serve(w http.ResponseWriter, r *http.Request, id uint64) {
name := s.filename(id)
http.ServeFile(w, r, name)
http.ServeFile(w, r, s.filename(id))
}
// Remove deletes an entry's blob and any staged parts. It reports failure so the caller can
+4 -11
View File
@@ -79,8 +79,7 @@ func uploads(router *http.ServeMux, baseDir string) {
itemPath += gzipExtension
}
safeRunPath := safeResolve(baseDir, req.PathValue("runId"))
safePath := safeResolve(safeRunPath, itemPath)
safePath := safeResolve(safeResolve(baseDir, req.PathValue("runId")), itemPath)
if err := os.MkdirAll(filepath.Dir(safePath), os.ModePerm); err != nil {
panic(err)
@@ -120,9 +119,7 @@ func downloads(router *http.ServeMux, baseDir string) {
router.HandleFunc("GET /_apis/pipelines/workflows/{runId}/artifacts", func(w http.ResponseWriter, req *http.Request) {
runID := req.PathValue("runId")
safePath := safeResolve(baseDir, runID)
entries, err := os.ReadDir(safePath)
entries, err := os.ReadDir(safeResolve(baseDir, runID))
if err != nil {
panic(err)
}
@@ -156,15 +153,11 @@ func downloads(router *http.ServeMux, baseDir string) {
// if it was upload as gzip
rel = strings.TrimSuffix(rel, gzipExtension)
path := filepath.Join(itemPath, rel)
rel = filepath.ToSlash(rel)
path = filepath.ToSlash(path)
files = append(files, ContainerItem{
Path: path,
Path: filepath.ToSlash(filepath.Join(itemPath, rel)),
ItemType: "file",
ContentLocation: fmt.Sprintf("http://%s/artifact/%s/%s/%s", req.Host, container, itemPath, rel),
ContentLocation: fmt.Sprintf("http://%s/artifact/%s/%s/%s", req.Host, container, itemPath, filepath.ToSlash(rel)),
})
}
return nil
+2 -7
View File
@@ -14,13 +14,8 @@ const dryrunContextKeyVal = dryrunContextKey("dryrun")
// Dryrun returns true if the current context is dryrun
func Dryrun(ctx context.Context) bool {
val := ctx.Value(dryrunContextKeyVal)
if val != nil {
if dryrun, ok := val.(bool); ok {
return dryrun
}
}
return false
dryrun, _ := ctx.Value(dryrunContextKeyVal).(bool)
return dryrun
}
// WithDryrun adds a value to the context for dryrun
+15 -22
View File
@@ -21,8 +21,7 @@ type Conditional func(ctx context.Context) bool
// NewInfoExecutor is an executor that logs messages
func NewInfoExecutor(format string, args ...any) Executor {
return func(ctx context.Context) error {
logger := Logger(ctx)
logger.Infof(format, args...)
Logger(ctx).Infof(format, args...)
return nil
}
}
@@ -30,8 +29,7 @@ func NewInfoExecutor(format string, args ...any) Executor {
// NewDebugExecutor is an executor that logs messages
func NewDebugExecutor(format string, args ...any) Executor {
return func(ctx context.Context) error {
logger := Logger(ctx)
logger.Debugf(format, args...)
Logger(ctx).Debugf(format, args...)
return nil
}
}
@@ -39,9 +37,7 @@ func NewDebugExecutor(format string, args ...any) Executor {
// NewPipelineExecutor creates a new executor from a series of other executors
func NewPipelineExecutor(executors ...Executor) Executor {
if len(executors) == 0 {
return func(ctx context.Context) error {
return nil
}
return NewErrorExecutor(nil)
}
var rtn Executor
for _, executor := range executors {
@@ -61,6 +57,13 @@ func NewErrorExecutor(err error) Executor {
}
}
func RecoverPanic(err *error) { // defer directly in workers to keep stacks in host logs
if recovered := recover(); recovered != nil {
logging.Errorf("panic: %v\n%s", recovered, debug.Stack())
*err = fmt.Errorf("panic: %v", recovered)
}
}
// NewParallelExecutor creates a new executor from a parallel of other executors
func NewParallelExecutor(parallel int, executors ...Executor) Executor {
if len(executors) == 0 {
@@ -80,8 +83,8 @@ func NewParallelExecutor(parallel int, executors ...Executor) Executor {
logging.Infof("NewParallelExecutor: Creating %d workers for %d executors", parallel, len(executors))
for i := 0; i < parallel; i++ {
go func(workerID int, work <-chan Executor, errs chan<- error) {
for workerID := range parallel {
go func() {
logging.Debugf("Worker %d started", workerID)
taskCount := 0
for executor := range work {
@@ -91,17 +94,12 @@ func NewParallelExecutor(parallel int, executors ...Executor) Executor {
// goroutine which would leave the runner process hung.
// https://gitea.com/gitea/runner/issues/371
errs <- func() (err error) {
defer func() {
if r := recover(); r != nil {
logging.Errorf("panic in executor: %v\n%s", r, debug.Stack())
err = fmt.Errorf("panic: %v", r)
}
}()
defer RecoverPanic(&err)
return executor(ctx)
}()
}
logging.Debugf("Worker %d finished (%d tasks executed)", workerID, taskCount)
}(i, work, errs)
}()
}
for i := range executors {
@@ -150,12 +148,7 @@ func (e Executor) If(conditional Conditional) Executor {
// IfNot only runs this executor if conditional is true
func (e Executor) IfNot(conditional Conditional) Executor {
return func(ctx context.Context) error {
if !conditional(ctx) {
return e(ctx)
}
return nil
}
return e.If(func(ctx context.Context) bool { return !conditional(ctx) })
}
// IfBool only runs this executor if conditional is true
+54 -100
View File
@@ -7,8 +7,6 @@ package common
import (
"context"
"errors"
"reflect"
"strings"
"sync"
"sync/atomic"
"testing"
@@ -18,42 +16,25 @@ import (
)
func TestNewWorkflow(t *testing.T) {
assert := assert.New(t)
ctx := t.Context()
require.NoError(t, NewPipelineExecutor()(ctx))
require.EqualError(t, NewErrorExecutor(errors.New("test error"))(ctx), "test error")
// empty
emptyWorkflow := NewPipelineExecutor()
assert.NoError(emptyWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
// error case
errorWorkflow := NewErrorExecutor(errors.New("test error"))
assert.Error(errorWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
// multiple success case
runcount := 0
successWorkflow := NewPipelineExecutor(
func(ctx context.Context) error {
runcount++
return nil
},
func(ctx context.Context) error {
runcount++
return nil
})
assert.NoError(successWorkflow(ctx)) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(2, runcount)
increment := func(_ctx context.Context) error {
runcount++
return nil
}
require.NoError(t, NewPipelineExecutor(increment, increment)(ctx))
assert.Equal(t, 2, runcount)
}
// concurrencyProbe returns an executor recording the peak number of concurrent copies. Copies
// block until wantActive are in flight so the peak is exact without sleeping, and later copies
// find the gate already open so the last one still finishes with no partner left.
func concurrencyProbe(wantActive int32) (exec Executor, count, maxActive *atomic.Int32) {
var counted, active, peak atomic.Int32
var once sync.Once
reached := make(chan struct{})
return func(ctx context.Context) error {
return func(_ctx context.Context) error {
counted.Add(1)
running := active.Add(1)
for {
@@ -72,18 +53,14 @@ func concurrencyProbe(wantActive int32) (exec Executor, count, maxActive *atomic
}
func TestNewParallelExecutor(t *testing.T) {
ctx := t.Context()
exec, count, maxActive := concurrencyProbe(2)
require.NoError(t, NewParallelExecutor(2, exec, exec, exec)(ctx))
assert.Equal(t, int32(3), count.Load(), "should run all 3 executors")
assert.Equal(t, int32(2), maxActive.Load(), "should run at most 2 executors in parallel")
// parallelism below 1 falls back to a single worker
exec, count, maxActive = concurrencyProbe(1)
require.NoError(t, NewParallelExecutor(0, exec, exec, exec)(ctx))
assert.Equal(t, int32(3), count.Load(), "should run all 3 executors")
assert.Equal(t, int32(1), maxActive.Load(), "should run at most 1 executor in parallel")
for name, parallel := range map[string]int{"parallel": 2, "below minimum": 0} {
t.Run(name, func(t *testing.T) {
exec, count, maxActive := concurrencyProbe(int32(max(1, parallel)))
require.NoError(t, NewParallelExecutor(parallel, exec, exec, exec)(t.Context()))
assert.Equal(t, int32(3), count.Load())
assert.Equal(t, int32(max(1, parallel)), maxActive.Load())
})
}
}
func TestNewParallelExecutorEmpty(t *testing.T) {
@@ -99,66 +76,49 @@ func TestNewParallelExecutorEmpty(t *testing.T) {
assert.ErrorIs(err, context.Canceled)
}
func TestNewParallelExecutorFailed(t *testing.T) {
assert := assert.New(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
count := 0
errorWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count++
return errors.New("fake error")
})
err := NewParallelExecutor(1, errorWorkflow)(ctx)
assert.Equal(1, count)
assert.ErrorIs(context.Canceled, err)
}
func TestNewParallelExecutorCanceled(t *testing.T) {
assert := assert.New(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
errExpected := errors.New("fake error")
var count atomic.Int32
successWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count.Add(1)
return nil
})
errorWorkflow := NewPipelineExecutor(func(ctx context.Context) error {
count.Add(1)
return errExpected
})
err := NewParallelExecutor(3, errorWorkflow, successWorkflow, successWorkflow)(ctx)
assert.Equal(int32(3), count.Load())
assert.Error(errExpected, err) //nolint:testifylint // pre-existing issue from nektos/act
}
func TestNewParallelExecutorRunsRemainingAfterFailure(t *testing.T) {
var successCount atomic.Int32
executors := make([]Executor, 5)
for i := range executors {
executors[i] = func(ctx context.Context) error {
if i == 2 {
return errors.New("fake error")
func TestNewParallelExecutorRunsAllTasksAfterFailureOrCancellation(t *testing.T) {
for name, tc := range map[string]struct {
canceled bool
parallel, tasks int
}{
"failure": {false, 2, 5},
"canceled": {true, 2, 5},
"canceled single task": {true, 1, 1},
} {
t.Run(name, func(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
if tc.canceled {
cancel()
}
var calls atomic.Int32
executors := make([]Executor, tc.tasks)
for i := range executors {
executors[i] = func(_ctx context.Context) error {
calls.Add(1)
if i == tc.tasks/2 {
return errors.New("fake error")
}
return nil
}
}
successCount.Add(1)
return nil
}
}
require.Error(t, NewParallelExecutor(2, executors...)(t.Context()))
assert.Equal(t, int32(4), successCount.Load(), "a failing executor must not stop the others")
err := NewParallelExecutor(tc.parallel, executors...)(ctx)
if tc.canceled {
require.ErrorIs(t, err, context.Canceled)
} else {
require.EqualError(t, err, "fake error")
}
assert.Equal(t, int32(tc.tasks), calls.Load())
})
}
}
func TestExecutorConditionalsAndFinally(t *testing.T) {
ctx := t.Context()
var calls []string
record := func(name string) Executor {
return func(ctx context.Context) error {
return func(_ctx context.Context) error {
calls = append(calls, name)
return nil
}
@@ -170,10 +130,7 @@ func TestExecutorConditionalsAndFinally(t *testing.T) {
require.NoError(t, record("if-bool").IfBool(true)(ctx))
require.NoError(t, record("main").Finally(record("finally"))(ctx))
want := []string{"if-true", "if-not", "if-bool", "main", "finally"}
if !reflect.DeepEqual(calls, want) {
t.Fatalf("calls = %v, want %v", calls, want)
}
assert.Equal(t, []string{"if-true", "if-not", "if-bool", "main", "finally"}, calls)
}
func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
@@ -181,8 +138,5 @@ func TestExecutorFinallyReturnsFinallyErrorWithOriginal(t *testing.T) {
finalErr := errors.New("cleanup failed")
err := NewErrorExecutor(mainErr).Finally(NewErrorExecutor(finalErr))(t.Context())
require.Error(t, err)
if !strings.Contains(err.Error(), "cleanup failed") || !strings.Contains(err.Error(), "main failed") {
t.Fatalf("finally error = %q, want both cleanup and original error", err)
}
require.EqualError(t, err, "error occurred running finally: cleanup failed (original error: main failed)")
}
+41
View File
@@ -16,14 +16,25 @@ import (
"net/url"
"os"
"path/filepath"
"reflect"
"runtime"
"strconv"
"strings"
"sync"
"time"
"gitea.com/gitea/runner/act/common"
)
var actionFetches sync.Map
type actionFetch struct {
done chan struct{}
dir, sha string
err error
retry bool // the leader was cancelled or panicked, so followers fetch themselves
}
var archiveRetryDelay = func(retryAfter string) time.Duration { // as actions/runner backs off
if seconds, err := strconv.Atoi(retryAfter); err == nil {
return min(max(time.Duration(seconds)*time.Second, 10*time.Second), 10*time.Minute)
@@ -32,6 +43,36 @@ var archiveRetryDelay = func(retryAfter string) time.Duration { // as actions/ru
}
func FetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, string, error) { // input.Dir is one clone URL and ref's, holding a directory per commit
if !reflect.ValueOf(input).Comparable() {
return fetchAction(ctx, input)
}
for ctx.Err() == nil {
call := &actionFetch{done: make(chan struct{}), retry: true}
previous, loaded := actionFetches.LoadOrStore(input, call)
if !loaded {
func() {
defer func() { actionFetches.Delete(input); close(call.done) }()
call.dir, call.sha, call.err = fetchAction(ctx, input)
call.retry = ctx.Err() != nil
}()
if call.retry {
continue
}
return call.dir, call.sha, call.err
}
call, _ = previous.(*actionFetch)
select {
case <-ctx.Done():
case <-call.done:
if !call.retry && ctx.Err() == nil {
return call.dir, call.sha, call.err
}
}
}
return "", "", ctx.Err()
}
func fetchAction(ctx context.Context, input NewGitCloneExecutorInput) (string, string, error) {
unlock, err := AcquireCloneLock(ctx, input.Dir)
if err != nil {
return "", "", err
+90
View File
@@ -6,6 +6,7 @@ package git
import (
"archive/tar"
"bytes"
"context"
"encoding/base64"
"fmt"
"net/http"
@@ -16,9 +17,12 @@ import (
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"gitea.com/gitea/runner/act/common"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -102,6 +106,75 @@ func TestFetchAction(t *testing.T) {
return NewGitCloneExecutorInput{URL: server.URL + "/org/action", Ref: ref, Dir: t.TempDir(), Token: "secret"}
}
t.Run("shares identical fetches and isolates cancellation, panics and differing inputs", func(t *testing.T) {
expectedSha := gitRevParse(t, remote, "main")
for _, scenario := range []string{"shared", "leader cancelled", "leader panics", "different token"} {
t.Run(scenario, func(t *testing.T) {
serve("prefixed", 0)
input := newInput("main")
started, release := make(chan struct{}), make(chan struct{})
releaseFetches := sync.OnceFunc(func() { close(release) })
t.Cleanup(releaseFetches)
var once sync.Once
var panicked atomic.Bool
transport := actionFetchTransport(func(request *http.Request) (*http.Response, error) {
once.Do(func() { close(started) })
select {
case <-request.Context().Done():
return nil, request.Context().Err()
case <-release:
}
if scenario == "leader panics" && panicked.CompareAndSwap(false, true) {
panic("broken archive")
}
return http.DefaultTransport.RoundTrip(request)
})
input.Transport = &transport
leaderCtx, cancelLeader := context.WithCancel(t.Context())
defer cancelLeader()
waiting := &actionFetchContext{Context: t.Context(), waiting: make(chan struct{})}
followerInput := input
expectedRequests := 1
if scenario == "different token" {
followerInput.Token, expectedRequests = "other", 2
}
leaderDone, followerDone := make(chan error, 1), make(chan error, 1)
fetch := func(ctx context.Context, input NewGitCloneExecutorInput, done chan<- error) {
var err error
defer func() { done <- err }()
defer common.RecoverPanic(&err)
var tree, sha string
if tree, sha, err = FetchAction(ctx, input); err == nil {
assert.Equal(t, expectedSha, sha)
assert.Equal(t, filepath.Join(input.Dir, sha), tree)
}
}
go fetch(leaderCtx, input, leaderDone)
<-started
go fetch(waiting, followerInput, followerDone)
<-waiting.waiting
if scenario == "leader cancelled" {
cancelLeader()
require.ErrorIs(t, <-leaderDone, context.Canceled)
}
releaseFetches()
switch scenario {
case "leader cancelled":
case "leader panics":
require.ErrorContains(t, <-leaderDone, "panic: broken archive")
default:
require.NoError(t, <-leaderDone)
}
require.NoError(t, <-followerDone)
_, headers := served()
assert.Len(t, headers, expectedRequests)
if scenario == "different token" {
assert.NotEqual(t, headers[0].Get("Authorization"), headers[1].Get("Authorization"))
}
})
}
})
t.Run("downloads, revalidates and follows a moved tag", func(t *testing.T) {
serve("prefixed", 0)
input := newInput("v1")
@@ -200,6 +273,23 @@ func TestFetchAction(t *testing.T) {
})
}
type actionFetchTransport func(*http.Request) (*http.Response, error)
func (transport actionFetchTransport) RoundTrip(request *http.Request) (*http.Response, error) {
return transport(request)
}
type actionFetchContext struct {
context.Context
waiting chan struct{}
once sync.Once
}
func (ctx *actionFetchContext) Done() <-chan struct{} {
ctx.once.Do(func() { close(ctx.waiting) })
return ctx.Context.Done()
}
func TestExtractTreeConfinesEntries(t *testing.T) {
outside := t.TempDir()
for name, entries := range map[string][]*tar.Header{
+6 -6
View File
@@ -79,25 +79,25 @@ func (e *Error) Unwrap() error { return e.err }
func (e *Error) Commit() string { return e.commit }
// FindGitRevision get the current git revision
func FindGitRevision(ctx context.Context, file string) (shortSHA, sha string, err error) {
func FindGitRevision(ctx context.Context, file string) (string, error) {
logger := common.Logger(ctx)
sha, err = gitOutput(ctx, file, "rev-parse", "--verify", "HEAD^{commit}")
sha, err := gitOutput(ctx, file, "rev-parse", "--verify", "HEAD^{commit}")
if err != nil {
logger.WithError(err).Error("path", file, "not located inside a git repository")
return "", "", err
return "", err
}
if len(sha) < 7 {
return "", "", errors.New("HEAD could not be resolved")
return "", errors.New("HEAD could not be resolved")
}
logger.Debugf("Found revision: %s", sha)
return sha[:7], sha, nil
return sha, nil
}
// FindGitRef get the current git ref
func FindGitRef(ctx context.Context, file string) (string, error) {
logger := common.Logger(ctx)
logger.Debugf("Loading revision from git directory")
_, revision, err := FindGitRevision(ctx, file)
revision, err := FindGitRevision(ctx, file)
if err != nil {
return "", err
}
+67 -150
View File
@@ -67,42 +67,20 @@ func TestErrorWrapsCommitAndCause(t *testing.T) {
require.Equal(t, "abc123", err.Commit())
}
func cleanGitHooks(dir string) error {
hooksDir := filepath.Join(dir, ".git", "hooks")
files, err := os.ReadDir(hooksDir)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
for _, f := range files {
if f.IsDir() {
continue
}
relName := filepath.Join(hooksDir, f.Name())
if err := os.Remove(relName); err != nil {
return err
}
}
return nil
}
func TestFindGitMetadataOfSHA256RepositoryUsesOrigin(t *testing.T) {
t.Parallel()
basedir := t.TempDir()
const remoteURL = "https://github.com/owner/repo.git"
require.NoError(t, gitCmd("init", "--object-format=sha256", "--initial-branch=main", basedir))
require.NoError(t, cleanGitHooks(basedir))
require.NoError(t, os.RemoveAll(filepath.Join(basedir, ".git", "hooks")))
require.NoError(t, gitCmd("-C", basedir, "commit", "--allow-empty", "-m", "init"))
require.NoError(t, gitCmd("-C", basedir, "remote", "add", "origin", remoteURL))
require.NoError(t, gitCmd("-C", basedir, "config", "--add", "remote.origin.url", "https://github.com/other/repo.git"))
shortSHA, sha, err := FindGitRevision(t.Context(), basedir)
sha, err := FindGitRevision(t.Context(), basedir)
require.NoError(t, err)
require.Equal(t, gitRevParse(t, basedir, "HEAD"), sha)
require.Len(t, sha, 64)
require.Equal(t, sha[:7], shortSHA)
ref, err := FindGitRef(t.Context(), basedir)
require.NoError(t, err)
@@ -122,89 +100,66 @@ func TestGitFindRef(t *testing.T) {
basedir := t.TempDir()
for name, tt := range map[string]struct {
Prepare func(t *testing.T, dir string)
Assert func(t *testing.T, ref string, err error)
commands [][]string
ref string
}{
"new_repo": {
Prepare: func(t *testing.T, dir string) {},
Assert: func(t *testing.T, ref string, err error) {
require.Error(t, err)
},
},
"new_repo": {},
"new_repo_with_commit": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/master", ref)
},
commands: [][]string{{"commit", "--allow-empty", "-m", "msg"}},
ref: "refs/heads/master",
},
"current_head_is_tag": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "commit msg"))
require.NoError(t, gitCmd("-C", dir, "tag", "v1.2.3"))
require.NoError(t, gitCmd("-C", dir, "checkout", "v1.2.3"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/tags/v1.2.3", ref)
commands: [][]string{
{"commit", "--allow-empty", "-m", "commit msg"},
{"tag", "v1.2.3"},
{"checkout", "v1.2.3"},
},
ref: "refs/tags/v1.2.3",
},
"current_head_is_same_as_tag": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "1.4.2 release"))
require.NoError(t, gitCmd("-C", dir, "tag", "v1.4.2"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/tags/v1.4.2", ref)
commands: [][]string{
{"commit", "--allow-empty", "-m", "1.4.2 release"},
{"tag", "v1.4.2"},
},
ref: "refs/tags/v1.4.2",
},
"current_head_is_not_tag": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg"))
require.NoError(t, gitCmd("-C", dir, "tag", "v1.4.2"))
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg2"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/master", ref)
commands: [][]string{
{"commit", "--allow-empty", "-m", "msg"},
{"tag", "v1.4.2"},
{"commit", "--allow-empty", "-m", "msg2"},
},
ref: "refs/heads/master",
},
"current_head_is_another_branch": {
Prepare: func(t *testing.T, dir string) {
require.NoError(t, gitCmd("-C", dir, "checkout", "-b", "mybranch"))
require.NoError(t, gitCmd("-C", dir, "commit", "--allow-empty", "-m", "msg"))
},
Assert: func(t *testing.T, ref string, err error) {
require.NoError(t, err)
require.Equal(t, "refs/heads/mybranch", ref)
commands: [][]string{
{"checkout", "-b", "mybranch"},
{"commit", "--allow-empty", "-m", "msg"},
},
ref: "refs/heads/mybranch",
},
} {
t.Run(name, func(t *testing.T) {
dir := filepath.Join(basedir, name)
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, gitCmd("-C", dir, "init", "--initial-branch=master"))
require.NoError(t, cleanGitHooks(dir))
tt.Prepare(t, dir)
require.NoError(t, os.RemoveAll(filepath.Join(dir, ".git", "hooks")))
for _, command := range tt.commands {
require.NoError(t, gitCmd(append([]string{"-C", dir}, command...)...))
}
ref, err := FindGitRef(t.Context(), dir)
tt.Assert(t, ref, err)
if tt.ref == "" {
require.Error(t, err)
} else {
require.NoError(t, err)
require.Equal(t, tt.ref, ref)
}
})
}
}
func TestGitCloneExecutor(t *testing.T) {
// Build a local bare "remote" so this runs offline and fast. The cases below mirror
// the tag/branch/sha/short-sha ref paths the executor handles, formerly exercised by
// cloning actions/checkout and anchore/scan-action over the network.
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
remoteDir, workDir := newGitRemote(t)
require.NoError(t, os.WriteFile(filepath.Join(workDir, "action.sh"), []byte("#!/bin/sh\necho hi\n"), 0o755))
require.NoError(t, gitCmd("-C", workDir, "add", "action.sh"))
require.NoError(t, gitCmd("-C", workDir, "commit", "-m", "initial"))
@@ -212,14 +167,11 @@ func TestGitCloneExecutor(t *testing.T) {
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "v2"))
// A branch with a dash in the name (mirrors the historical scan-action@act-fails case).
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "act-fails"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "branch-commit"))
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "act-fails"))
out, err := exec.Command("git", "-C", workDir, "rev-parse", "main").Output()
require.NoError(t, err)
fullSha := strings.TrimSpace(string(out))
fullSha := gitRevParse(t, workDir, "main")
require.NoError(t, gitCmd("-C", workDir, "push", "origin", "main:refs/heads/"+fullSha[:4]))
@@ -265,12 +217,12 @@ func TestGitCloneExecutor(t *testing.T) {
err := clone(t.Context())
if tt.Err != nil {
assert.Error(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.Error(t, err)
assert.Equal(t, tt.Err, err)
assert.Equal(t, tt.Err, clone(t.Context()), "a retry on the cache keeps the error")
return
}
assert.Empty(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
marker := filepath.Join(dir, "marker")
require.NoError(t, os.WriteFile(marker, nil, 0o644))
require.NoError(t, clone(t.Context()))
@@ -285,13 +237,7 @@ func TestGitCloneExecutor(t *testing.T) {
func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
t.Parallel()
createRemote := func(message string) string {
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
remoteDir, workDir := newGitRemote(t, message)
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
return remoteDir
@@ -341,13 +287,7 @@ func TestGitCloneExecutorReclonesWhenOriginURLChanges(t *testing.T) {
func TestGitCloneExecutorFollowsRemoteRefChanges(t *testing.T) {
t.Parallel()
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "initial"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "feature-1"))
remoteDir, workDir := newGitRemote(t, "initial", "feature-1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
dir := t.TempDir()
@@ -380,16 +320,9 @@ func TestGitCloneExecutorFollowsRemoteRefChanges(t *testing.T) {
func TestGitCloneExecutorOfflineMode(t *testing.T) {
t.Parallel()
// Build a local "remote" with a single commit on main.
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "initial"))
remoteDir, workDir := newGitRemote(t, "initial")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
// Prime the cache with an online clone of main.
cacheDir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir,
@@ -398,8 +331,6 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
})(t.Context()))
t.Run("cached branch resolves without fetching", func(t *testing.T) {
// Offline reuse of a cached branch must succeed even though ResolveRevision(input.Ref)
// finds no local refs/heads/<ref>.
err := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir,
Ref: "main",
@@ -414,7 +345,6 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
})
t.Run("unresolvable cached ref returns error", func(t *testing.T) {
// The ref was never cached; offline mode cannot resolve it and must return an error.
err := NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir,
Ref: "never-fetched",
@@ -427,15 +357,7 @@ func TestGitCloneExecutorOfflineMode(t *testing.T) {
func TestGitCloneExecutorShallow(t *testing.T) {
t.Parallel()
// Build a local "remote" with several commits on main plus a tag, so a full clone would pull noticeably more history than a shallow one.
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
for _, m := range []string{"c1", "c2", "c3"} {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", m))
}
remoteDir, workDir := newGitRemote(t, "c1", "c2", "c3")
require.NoError(t, gitCmd("-C", workDir, "tag", "v1"))
unadvertisedSHA := gitRevParse(t, workDir, "HEAD~1")
require.NoError(t, gitCmd("-C", workDir, "push", "-u", "origin", "main"))
@@ -443,25 +365,17 @@ func TestGitCloneExecutorShallow(t *testing.T) {
shallowMarker := func(dir string) string { return filepath.Join(dir, ".git", "shallow") }
t.Run("branch is cloned shallowly", func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: "main", Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present")
assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
t.Run("tag is cloned shallowly", func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: "v1", Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir))
assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
for name, ref := range map[string]string{"branch is cloned shallowly": "main", "tag is cloned shallowly": "v1"} {
t.Run(name, func(t *testing.T) {
dir := t.TempDir()
require.NoError(t, NewGitCloneExecutor(NewGitCloneExecutorInput{
URL: remoteDir, Ref: ref, Dir: dir, Depth: 1,
})(t.Context()))
assert.FileExists(t, shallowMarker(dir), "clone should be shallow")
assert.Equal(t, 1, gitRevCount(t, dir), "only the tip commit should be present")
assert.Equal(t, "c3", gitHeadSubject(t, dir))
})
}
unadvertisedRemote := func(t *testing.T, allowUnadvertised bool) string {
remote := filepath.Join(t.TempDir(), "remote.git")
@@ -541,12 +455,7 @@ func TestGitCloneExecutorShallow(t *testing.T) {
func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
t.Parallel()
remoteDir := t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
workDir := t.TempDir()
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "pinned"))
remoteDir, workDir := newGitRemote(t, "pinned")
pinned := gitRevParse(t, workDir, "HEAD")
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", "decoy"))
require.NoError(t, gitCmd("-C", workDir, "tag", pinned))
@@ -564,6 +473,18 @@ func TestGitCloneExecutorPinnedHashIgnoresShadowingRef(t *testing.T) {
}
}
func newGitRemote(t *testing.T, messages ...string) (string, string) {
t.Helper()
remoteDir, workDir := t.TempDir(), t.TempDir()
require.NoError(t, gitCmd("init", "--bare", "--initial-branch=main", remoteDir))
require.NoError(t, gitCmd("clone", remoteDir, workDir))
require.NoError(t, gitCmd("-C", workDir, "checkout", "-b", "main"))
for _, message := range messages {
require.NoError(t, gitCmd("-C", workDir, "commit", "--allow-empty", "-m", message))
}
return remoteDir, workDir
}
func gitRevParse(t *testing.T, dir, rev string) string {
t.Helper()
out, err := exec.Command("git", "-C", dir, "rev-parse", rev).Output()
@@ -591,9 +512,6 @@ func gitCmd(args ...string) error {
cmd := exec.Command("git", args...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
// Inject a deterministic identity and ignore the host's global/system config so commits
// succeed regardless of the host having no user.name/user.email (e.g. CI, GITHUB_ACTIONS
// unset) or a global commit.gpgsign, and without mutating the developer's ~/.gitconfig.
cmd.Env = append(os.Environ(),
"GIT_AUTHOR_NAME=Unit Test",
"[email protected]",
@@ -695,7 +613,6 @@ func TestAcquireCloneLock(t *testing.T) {
})
}
// An unresponsive remote must not pin a job: the refresh has to be interruptible.
func TestNewGitCloneExecutorFetchSendsTokenAndHonoursContext(t *testing.T) {
t.Parallel()
block := make(chan struct{})
+3 -9
View File
@@ -14,13 +14,8 @@ const jobErrorContextKeyVal = jobErrorContextKey("job.error")
// JobError returns the job error for current context if any
func JobError(ctx context.Context) error {
val := ctx.Value(jobErrorContextKeyVal)
if val != nil {
if container, ok := val.(map[string]error); ok {
return container["error"]
}
}
return nil
container, _ := ctx.Value(jobErrorContextKeyVal).(map[string]error)
return container["error"]
}
func SetJobError(ctx context.Context, err error) {
@@ -31,6 +26,5 @@ func SetJobError(ctx context.Context, err error) {
// WithJobErrorContainer adds a value to the context as a container for an error
func WithJobErrorContainer(ctx context.Context) context.Context {
container := map[string]error{}
return context.WithValue(ctx, jobErrorContextKeyVal, container)
return context.WithValue(ctx, jobErrorContextKeyVal, map[string]error{})
}
+9 -21
View File
@@ -26,9 +26,7 @@ type lineWriter struct {
// NewLineWriter creates a new instance of a line writer
func NewLineWriter(handlers ...LineHandler) io.Writer {
w := new(lineWriter)
w.handlers = handlers
return w
return &lineWriter{handlers: handlers}
}
// FlushWriter flushes w if it implements Flusher. It is a no-op otherwise, so
@@ -39,24 +37,15 @@ func FlushWriter(w io.Writer) {
}
}
func (lw *lineWriter) Write(p []byte) (n int, err error) {
pBuf := bytes.NewBuffer(p)
written := 0
for {
line, err := pBuf.ReadString('\n')
w, _ := lw.buffer.WriteString(line)
written += w
if err != nil {
if err == io.EOF {
break
}
return written, err
func (lw *lineWriter) Write(p []byte) (int, error) {
for line := range bytes.SplitAfterSeq(p, []byte{'\n'}) {
_, _ = lw.buffer.Write(line)
if len(line) > 0 && line[len(line)-1] == '\n' {
lw.handleLine(lw.buffer.String())
lw.buffer.Reset()
}
lw.handleLine(lw.buffer.String())
lw.buffer.Reset()
}
return written, nil
return len(p), nil
}
// Flush emits any buffered, not-yet-newline-terminated content as a final line.
@@ -72,8 +61,7 @@ func (lw *lineWriter) Flush() {
func (lw *lineWriter) handleLine(line string) {
for _, h := range lw.handlers {
ok := h(line)
if !ok {
if !h(line) {
break
}
}
+12 -34
View File
@@ -9,24 +9,24 @@ import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLineWriter(t *testing.T) {
lines := make([]string, 0)
lineHandler := func(s string) bool {
func TestLineWriterBuffersAndFlushesLines(t *testing.T) {
var lines []string
lineWriter := NewLineWriter(func(s string) bool {
lines = append(lines, s)
return true
}
lineWriter := NewLineWriter(lineHandler)
})
assert := assert.New(t)
write := func(s string) {
n, err := lineWriter.Write([]byte(s))
assert.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(len(s), n, s)
}
write("")
write("hello")
write(" ")
write("world!!\nextra")
@@ -34,39 +34,17 @@ func TestLineWriter(t *testing.T) {
write(" line\n")
write("no newline here...")
assert.Len(lines, 4)
assert.Equal("hello world!!\n", lines[0])
assert.Equal("extra line\n", lines[1])
assert.Equal(" and another\n", lines[2])
assert.Equal("last line\n", lines[3])
}
expected := []string{"hello world!!\n", "extra line\n", " and another\n", "last line\n"}
assert.Equal(expected, lines)
func TestLineWriterFlush(t *testing.T) {
lines := make([]string, 0)
lineHandler := func(s string) bool {
lines = append(lines, s)
return true
}
lineWriter := NewLineWriter(lineHandler)
assert := assert.New(t)
_, err := lineWriter.Write([]byte("complete line\npartial line without newline"))
assert.NoError(err) //nolint:testifylint // pre-existing pattern from nektos/act
// Only the newline-terminated line is emitted before flushing.
assert.Equal([]string{"complete line\n"}, lines)
// Flushing emits the buffered, not-yet-terminated trailing line.
FlushWriter(lineWriter)
assert.Equal([]string{"complete line\n", "partial line without newline"}, lines)
expected = append(expected, "no newline here...")
assert.Equal(expected, lines)
// Flushing again is a no-op: nothing is buffered.
FlushWriter(lineWriter)
assert.Len(lines, 2)
assert.Equal(expected, lines)
}
func TestFlushWriterIgnoresNonFlusher(t *testing.T) {
// FlushWriter must be a safe no-op for writers that do not buffer lines.
assert.NotPanics(t, func() { FlushWriter(io.Discard) })
}
+2 -4
View File
@@ -34,10 +34,8 @@ const loggerHandlerKeyVal = loggerHandlerKey("logging.Handler")
// LoggerHandler returns the extra handler the job logger sends its records to, or nil.
// It affects the job logger, not the global logger.
func LoggerHandler(ctx context.Context) logging.Handler {
if h, ok := ctx.Value(loggerHandlerKeyVal).(logging.Handler); ok {
return h
}
return nil
h, _ := ctx.Value(loggerHandlerKeyVal).(logging.Handler)
return h
}
// WithLoggerHandler adds the extra handler for the job logger to the context
+2 -5
View File
@@ -9,6 +9,7 @@ import (
"fmt"
"io"
"log/slog"
"maps"
"reflect"
"runtime"
"slices"
@@ -94,11 +95,7 @@ func (h *consoleHandler) Handle(_ context.Context, r slog.Record) error {
}
prefixFieldClashes(data, caller != nil)
keys := make([]string, 0, len(data))
for k := range data {
keys = append(keys, k)
}
slices.Sort(keys)
keys := slices.Sorted(maps.Keys(data))
var b bytes.Buffer
if terminal && !opts.DisableColors {
-2
View File
@@ -73,12 +73,10 @@ func Enabled(level Level) bool { return Default().Enabled(level) }
func With(args ...any) *Logger { return Default().With(args...) }
func WithError(err error) *Logger { return Default().WithError(err) }
func Tracef(format string, args ...any) { Default().logf(LevelTrace, format, args) }
func Debugf(format string, args ...any) { Default().logf(LevelDebug, format, args) }
func Infof(format string, args ...any) { Default().logf(LevelInfo, format, args) }
func Warnf(format string, args ...any) { Default().logf(LevelWarn, format, args) }
func Errorf(format string, args ...any) { Default().logf(LevelError, format, args) }
func Trace(args ...any) { Default().log(LevelTrace, args) }
func Debug(args ...any) { Default().log(LevelDebug, args) }
func Info(args ...any) { Default().log(LevelInfo, args) }
func Warn(args ...any) { Default().log(LevelWarn, args) }
-1
View File
@@ -43,7 +43,6 @@ func (l *Logger) Enabled(level Level) bool {
return l.h.Enabled(l.ctx, level)
}
func (l *Logger) Tracef(format string, args ...any) { l.logf(LevelTrace, format, args) }
func (l *Logger) Debugf(format string, args ...any) { l.logf(LevelDebug, format, args) }
func (l *Logger) Infof(format string, args ...any) { l.logf(LevelInfo, format, args) }
func (l *Logger) Warnf(format string, args ...any) { l.logf(LevelWarn, format, args) }
+3 -7
View File
@@ -38,6 +38,7 @@ func (c countingStringer) String() string { *c.n++; return "x" }
func TestLogger(t *testing.T) {
h := logtest.NewWithLevel(logging.LevelTrace)
assert.Nil(t, h.LastRecord())
l := logging.New(h)
l.With("job", "j1").WithError(errors.New("boom")).Warnf("step %d", 2)
@@ -53,12 +54,9 @@ func TestLogger(t *testing.T) {
assert.Equal(t, "j1", job)
errv, _ := recs[0].Attr("error")
require.EqualError(t, errv.(error), "boom")
assert.Equal(t, "a1 2b", recs[1].Message) // fmt.Sprint spacing
assert.Equal(t, "a1 2b", recs[1].Message)
bad, _ := recs[2].Attr("!BADKEY")
assert.Equal(t, 7, bad)
h.Reset()
assert.Nil(t, h.LastRecord())
}
func TestLoggerSkipsFormattingWhenDisabled(t *testing.T) {
@@ -117,10 +115,8 @@ func TestDefaultLevel(t *testing.T) {
func TestSlogHandler(t *testing.T) {
var buf strings.Builder
slogger := slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
l := logging.New(logging.NewSlogHandler(slogger.Handler())).With("job", "j1")
l := logging.New(logging.NewSlogHandler(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}))).With("job", "j1")
l.Tracef("dropped")
assert.False(t, l.Enabled(logging.LevelTrace))
l.WithError(errors.New("boom")).Errorf("fail %s", "x")
-6
View File
@@ -72,12 +72,6 @@ func (h *Handler) LastRecord() *logging.Record {
return &r
}
func (h *Handler) Reset() {
h.store.mu.Lock()
defer h.store.mu.Unlock()
h.store.records = nil
}
// SetDefault makes the process-wide Logger record into a new Handler until the test ends.
func SetDefault(t testing.TB) *Handler {
t.Helper()
+8 -12
View File
@@ -12,7 +12,7 @@ import (
type interfaceIP struct {
net.IP
net.Interface
Name string
}
// GetOutboundIP returns an outbound IP address of this machine.
@@ -43,7 +43,7 @@ func GetOutboundIP() net.IP {
ip = v.IP
}
if ip.IsGlobalUnicast() {
ips = append(ips, interfaceIP{ip, i})
ips = append(ips, interfaceIP{ip, i.Name})
}
}
}
@@ -59,29 +59,25 @@ func preferredIP(ips []interfaceIP) net.IP {
return nil
}
sort.Slice(ips, func(i, j int) bool {
ifi := ips[i].Interface
ifj := ips[j].Interface
first, second := ips[i], ips[j]
// ethernet is preferred
if vi, vj := strings.HasPrefix(ifi.Name, "e"), strings.HasPrefix(ifj.Name, "e"); vi != vj {
if vi, vj := strings.HasPrefix(first.Name, "e"), strings.HasPrefix(second.Name, "e"); vi != vj {
return vi
}
ipi := ips[i].IP
ipj := ips[j].IP
// IPv4 is preferred
if vi, vj := ipi.To4() != nil, ipj.To4() != nil; vi != vj {
if vi, vj := first.To4() != nil, second.To4() != nil; vi != vj {
return vi
}
// en0 is preferred to en1
if ifi.Name != ifj.Name {
return ifi.Name < ifj.Name
if first.Name != second.Name {
return first.Name < second.Name
}
// fallback
return ipi.String() < ipj.String()
return first.String() < second.String()
})
return ips[0].IP
}
+2 -7
View File
@@ -46,9 +46,8 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Building image from '%v'", input.ContextDir)
tags := []string{input.ImageTag}
options := client.ImageBuildOptions{
Tags: tags,
Tags: []string{input.ImageTag},
Remove: true,
AuthConfigs: LoadDockerAuthConfigs(ctx),
Dockerfile: input.Dockerfile,
@@ -76,11 +75,7 @@ func NewDockerBuildExecutor(input NewDockerBuildExecutorInput) common.Executor {
logger.Debugf("Creating image from context dir '%s' with tag '%s' and platform '%s'", input.ContextDir, input.ImageTag, input.Platform)
resp, err := cli.ImageBuild(ctx, buildContext, options)
err = logDockerResponse(logger, resp.Body, err != nil)
if err != nil {
return err
}
return nil
return logDockerResponse(logger, resp.Body, err != nil)
}
}
+16 -54
View File
@@ -4,12 +4,12 @@
//go:build !(WITHOUT_DOCKER || !(linux || darwin || windows || netbsd))
// This file is exact copy of https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts.go with:
// This file is based on https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts.go with:
// * appended with license information
// * regexp and ParseVolume in place of the import-restricted internal/lazyregexp and internal/volumespec
// * cdiDeviceName in place of the container-device-interface parser's IsQualifiedName
// * jsontext in place of json.Compact for the seccomp profile
// * invalidParameter from the package's errors.go, and convertPortSet/convertPortMap for the callers in docker_run.go
// * invalidParameter from the package's errors.go, and convertPortSet/convertPortMap for nat port types
//
// docker/cli is licensed under the Apache License, Version 2.0.
// See DOCKER_LICENSE for the full license text.
@@ -154,9 +154,6 @@ type containerOptions struct {
init bool
umask opts.UmaskOpt
annotations *opts.MapOpts
Image string
Args []string
}
// addFlags adds all command line flags that will be used by parse to the FlagSet
@@ -422,12 +419,7 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
tmpfs[k] = v
}
var runCmd, entrypoint []string
if len(copts.Args) > 0 {
runCmd = copts.Args
}
var entrypoint []string
if copts.entrypoint != "" {
entrypoint = []string{copts.entrypoint}
} else if flags.Changed("entrypoint") {
@@ -448,36 +440,15 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
if err != nil {
return nil, err
}
portBindings := network.PortMap{}
for port, bindings := range natPortBindings {
p, err := network.ParsePort(string(port))
if err != nil {
return nil, err
}
portBindings[p] = []network.PortBinding{}
for _, b := range bindings {
var hostIP netip.Addr
if b.HostIP != "" {
hostIP, err = netip.ParseAddr(b.HostIP)
if err != nil {
return nil, err
}
}
portBindings[p] = append(portBindings[p], network.PortBinding{
HostIP: hostIP,
HostPort: b.HostPort,
})
}
portBindings, err := convertPortMap(natPortBindings)
if err != nil {
return nil, err
}
// Add published ports as exposed ports.
exposedPorts := network.PortSet{}
for port := range ports {
p, err := network.ParsePort(string(port))
if err != nil {
return nil, err
}
exposedPorts[p] = struct{}{}
exposedPorts, err := convertPortSet(ports)
if err != nil {
return nil, err
}
// Merge in exposed ports to the map of published ports
@@ -618,11 +589,10 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
deviceRequests := copts.gpus.Value()
if len(cdiDeviceNames) > 0 {
cdiDeviceRequest := container.DeviceRequest{
deviceRequests = append(deviceRequests, container.DeviceRequest{
Driver: "cdi",
DeviceIDs: cdiDeviceNames,
}
deviceRequests = append(deviceRequests, cdiDeviceRequest)
})
}
resources := container.Resources{
@@ -668,8 +638,6 @@ func parse(flags *pflag.FlagSet, copts *containerOptions, serverOS string) (*con
AttachStdout: attachStdout,
AttachStderr: attachStderr,
Env: envVariables,
Cmd: runCmd,
Image: copts.Image,
Volumes: volumes,
Entrypoint: entrypoint,
WorkingDir: copts.workingDir,
@@ -810,10 +778,8 @@ func parseNetworkOpts(copts *containerOptions) (map[string]*network.EndpointSett
// For backward compatibility: if no custom options are provided for the network,
// and only a single network is specified, omit the endpoint-configuration
// on the client (the daemon will still create it when creating the container)
if i == 0 && len(copts.netMode.Value()) == 1 {
if ep == nil || reflect.ValueOf(*ep).IsZero() {
continue
}
if i == 0 && len(copts.netMode.Value()) == 1 && reflect.ValueOf(*ep).IsZero() {
continue
}
endpoints[n.Target] = ep
}
@@ -844,14 +810,12 @@ func applyContainerOptions(n *opts.NetworkAttachmentOpts, copts *containerOption
return invalidParameter(errors.New("conflicting options: cannot specify both --link-local-ip and per-network link-local IP addresses"))
}
if copts.aliases.Len() > 0 {
n.Aliases = make([]string, copts.aliases.Len())
copy(n.Aliases, copts.aliases.GetSlice())
n.Aliases = slices.Clone(copts.aliases.GetSlice())
}
// For a user-defined network, "--link" is an endpoint option, it creates an alias. But,
// for the default bridge it defines a legacy-link.
if container.NetworkMode(n.Target).IsUserDefined() && copts.links.Len() > 0 {
n.Links = make([]string, copts.links.Len())
copy(n.Links, copts.links.GetSlice())
n.Links = slices.Clone(copts.links.GetSlice())
}
if copts.ipv4Address != nil {
if ipv4, ok := netip.AddrFromSlice(copts.ipv4Address.To4()); ok {
@@ -890,7 +854,6 @@ func parseNetworkAttachmentOpt(ep opts.NetworkAttachmentOpts) (*network.Endpoint
}
epConfig.Aliases = append(epConfig.Aliases, ep.Aliases...)
if len(ep.DriverOpts) > 0 {
epConfig.DriverOpts = make(map[string]string)
epConfig.DriverOpts = ep.DriverOpts
}
if len(ep.Links) > 0 {
@@ -934,11 +897,10 @@ func convertToStandardNotation(ports []string) ([]string, error) {
}
func parseLoggingOpts(loggingDriver string, loggingOpts []string) (map[string]string, error) {
loggingOptsMap := opts.ConvertKVStringsToMap(loggingOpts)
if loggingDriver == "none" && len(loggingOpts) > 0 {
return map[string]string{}, fmt.Errorf("invalid logging opts for driver %s", loggingDriver)
}
return loggingOptsMap, nil
return opts.ConvertKVStringsToMap(loggingOpts), nil
}
// takes a local seccomp daemon, reads the file contents for sending to the daemon
+149 -292
View File
@@ -2,7 +2,7 @@
// Copyright 2022 The nektos/act Authors. All rights reserved.
// SPDX-License-Identifier: MIT
// This file is exact copy of https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts_test.go with:
// This file is based on https://github.com/docker/cli/blob/4a63305d74332de5ceba7fcbccbc3cbb7412f5ba/cli/command/container/opts_test.go with:
// * appended with license information
// * added tests for the locally changed parseDevice, validateDevice and invalidParameter
//
@@ -69,7 +69,6 @@ func parseRun(args []string) (*container.Config, *container.HostConfig, *network
if err := flags.Parse(args); err != nil {
return nil, nil, nil, err
}
// TODO(dnephin): fix tests to accept ContainerConfig; see https://github.com/moby/moby/pull/31621
containerCfg, err := parse(flags, copts, runtime.GOOS)
if err != nil {
return nil, nil, nil, err
@@ -81,13 +80,12 @@ func setupRunFlags() (*pflag.FlagSet, *containerOptions) {
flags := pflag.NewFlagSet("run", pflag.ContinueOnError)
flags.SetOutput(io.Discard)
flags.Usage = nil
copts := addFlags(flags)
return flags, copts
return flags, addFlags(flags)
}
func mustParse(t *testing.T, args string) (*container.Config, *container.HostConfig, *networktypes.NetworkingConfig) {
t.Helper()
config, hostConfig, nwConfig, err := parseRun(append(strings.Split(args, " "), "ubuntu", "bash"))
config, hostConfig, nwConfig, err := parseRun(strings.Split(args, " "))
require.NoError(t, err)
return config, hostConfig, nwConfig
}
@@ -179,53 +177,21 @@ func TestParseRunUmask(t *testing.T) {
}
func TestParseRunAttach(t *testing.T) {
tests := []struct {
input string
expected container.Config
for _, tc := range []struct {
input string
stdin, stdout, stderr bool
}{
{
input: "",
expected: container.Config{
AttachStdout: true,
AttachStderr: true,
},
},
{
input: "-i",
expected: container.Config{
AttachStdin: true,
AttachStdout: true,
AttachStderr: true,
},
},
{
input: "-a stdin",
expected: container.Config{
AttachStdin: true,
},
},
{
input: "-a stdin -a stdout",
expected: container.Config{
AttachStdin: true,
AttachStdout: true,
},
},
{
input: "-a stdin -a stdout -a stderr",
expected: container.Config{
AttachStdin: true,
AttachStdout: true,
AttachStderr: true,
},
},
}
for _, tc := range tests {
{"", false, true, true},
{"-i", true, true, true},
{"-a stdin", true, false, false},
{"-a stdin -a stdout", true, true, false},
{"-a stdin -a stdout -a stderr", true, true, true},
} {
t.Run(tc.input, func(t *testing.T) {
config, _, _ := mustParse(t, tc.input)
require.Equal(t, tc.expected.AttachStdin, config.AttachStdin)
require.Equal(t, tc.expected.AttachStdout, config.AttachStdout)
require.Equal(t, tc.expected.AttachStderr, config.AttachStderr)
require.Equal(t, tc.stdin, config.AttachStdin)
require.Equal(t, tc.stdout, config.AttachStdout)
require.Equal(t, tc.stderr, config.AttachStderr)
})
}
}
@@ -276,104 +242,47 @@ func TestParseRunWithInvalidArgs(t *testing.T) {
}
}
func TestParseWithVolumes(t *testing.T) { //nolint:gocyclo // verbatim copy from docker/cli tests
// A single volume
arr, tryit := setupPlatformVolume([]string{`/tmp`}, []string{`c:\tmp`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds != nil {
t.Fatalf("Error parsing volume flags, %q should not mount-bind anything. Received %v", tryit, hostConfig.Binds)
} else if _, exists := config.Volumes[arr[0]]; !exists {
t.Fatalf("Error parsing volume flags, %q is missing from volumes. Received %v", tryit, config.Volumes)
}
// Two volumes
arr, tryit = setupPlatformVolume([]string{`/tmp`, `/var`}, []string{`c:\tmp`, `c:\var`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds != nil {
t.Fatalf("Error parsing volume flags, %q should not mount-bind anything. Received %v", tryit, hostConfig.Binds)
} else if _, exists := config.Volumes[arr[0]]; !exists {
t.Fatalf("Error parsing volume flags, %s is missing from volumes. Received %v", arr[0], config.Volumes)
} else if _, exists := config.Volumes[arr[1]]; !exists {
t.Fatalf("Error parsing volume flags, %s is missing from volumes. Received %v", arr[1], config.Volumes)
}
// A single bind mount
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || hostConfig.Binds[0] != arr[0] {
t.Fatalf("Error parsing volume flags, %q should mount-bind the path before the colon into the path after the colon. Received %v %v", arr[0], hostConfig.Binds, config.Volumes)
}
// Two bind mounts.
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp`, `/hostVar:/containerVar`}, []string{os.Getenv("ProgramData") + `:c:\ContainerPD`, os.Getenv("TEMP") + `:c:\containerTmp`})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
}
// Two bind mounts, first read-only, second read-write.
// TODO Windows: The Windows version uses read-write as that's the only mode it supports. Can change this post TP4
arr, tryit = setupPlatformVolume(
[]string{`/hostTmp:/containerTmp:ro`, `/hostVar:/containerVar:rw`},
[]string{os.Getenv("TEMP") + `:c:\containerTmp:rw`, os.Getenv("ProgramData") + `:c:\ContainerPD:rw`})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
}
// Similar to previous test but with alternate modes which are only supported by Linux
if runtime.GOOS != "windows" {
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp:ro,Z`, `/hostVar:/containerVar:rw,Z`}, []string{})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
func TestParseWithVolumes(t *testing.T) {
for _, tc := range []struct {
unix, windows []string
binds int
}{
{[]string{`/tmp`}, []string{`c:\tmp`}, 0},
{[]string{`/tmp`, `/var`}, []string{`c:\tmp`, `c:\var`}, 0},
{[]string{`/hostTmp:/containerTmp`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`}, 1},
{[]string{`/hostTmp:/containerTmp`, `/hostVar:/containerVar`}, []string{os.Getenv("ProgramData") + `:c:\ContainerPD`, os.Getenv("TEMP") + `:c:\containerTmp`}, 2},
{[]string{`/hostTmp:/containerTmp:ro`, `/hostVar:/containerVar:rw`}, []string{os.Getenv("TEMP") + `:c:\containerTmp:rw`, os.Getenv("ProgramData") + `:c:\ContainerPD:rw`}, 2},
{[]string{`/hostTmp:/containerTmp:ro,Z`, `/hostVar:/containerVar:rw,Z`}, nil, 2},
{[]string{`/hostTmp:/containerTmp:Z`, `/hostVar:/containerVar:z`}, nil, 2},
{[]string{`/hostTmp:/containerTmp`, `/containerVar`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`, `c:\containerTmp`}, 1},
{nil, []string{os.Getenv("SystemDrive") + `\:d:`}, 1},
} {
volumes := tc.unix
if runtime.GOOS == "windows" {
volumes = tc.windows
}
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp:Z`, `/hostVar:/containerVar:z`}, []string{})
if _, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || compareRandomizedStrings(hostConfig.Binds[0], hostConfig.Binds[1], arr[0], arr[1]) != nil {
t.Fatalf("Error parsing volume flags, `%s and %s` did not mount-bind correctly. Received %v", arr[0], arr[1], hostConfig.Binds)
if len(volumes) == 0 {
continue
}
t.Run(strings.Join(volumes, ","), func(t *testing.T) {
var args []string
for _, volume := range volumes {
args = append(args, "-v", volume)
}
config, hostConfig, _, err := parseRun(args)
require.NoError(t, err)
if tc.binds == 0 {
assert.Nil(t, hostConfig.Binds)
} else {
assert.ElementsMatch(t, volumes[:tc.binds], hostConfig.Binds)
}
expectedVolumes := map[string]struct{}{}
for _, volume := range volumes[tc.binds:] {
expectedVolumes[volume] = struct{}{}
}
assert.Equal(t, expectedVolumes, config.Volumes)
})
}
// One bind mount and one volume
arr, tryit = setupPlatformVolume([]string{`/hostTmp:/containerTmp`, `/containerVar`}, []string{os.Getenv("TEMP") + `:c:\containerTmp`, `c:\containerTmp`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || len(hostConfig.Binds) > 1 || hostConfig.Binds[0] != arr[0] {
t.Fatalf("Error parsing volume flags, %s and %s should only one and only one bind mount %s. Received %s", arr[0], arr[1], arr[0], hostConfig.Binds)
} else if _, exists := config.Volumes[arr[1]]; !exists {
t.Fatalf("Error parsing volume flags %s and %s. %s is missing from volumes. Received %v", arr[0], arr[1], arr[1], config.Volumes)
}
// Root to non-c: drive letter (Windows specific)
if runtime.GOOS == "windows" {
arr, tryit = setupPlatformVolume([]string{}, []string{os.Getenv("SystemDrive") + `\:d:`})
if config, hostConfig, _ := mustParse(t, tryit); hostConfig.Binds == nil || len(hostConfig.Binds) > 1 || hostConfig.Binds[0] != arr[0] || len(config.Volumes) != 0 {
t.Fatalf("Error parsing %s. Should have a single bind mount and no volumes", arr[0])
}
}
}
// setupPlatformVolume takes two arrays of volume specs - a Unix style
// spec and a Windows style spec. Depending on the platform being unit tested,
// it returns one of them, along with a volume string that would be passed
// on the docker CLI (e.g. -v /bar -v /foo).
func setupPlatformVolume(u, w []string) ([]string, string) {
var a []string
if runtime.GOOS == "windows" {
a = w
} else {
a = u
}
s := ""
for _, v := range a {
s = s + "-v " + v + " "
}
return a, s
}
// check if (a == c && b == d) || (a == d && b == c)
// because maps are randomized
func compareRandomizedStrings(a, b, c, d string) error {
if a == c && b == d {
return nil
}
if a == d && b == c {
return nil
}
return errors.New("strings don't match")
}
// Simple parse with MacAddress validation
@@ -414,46 +323,29 @@ func TestParseWithMemorySwap(t *testing.T) {
}
func TestParseHostname(t *testing.T) {
validHostnames := map[string]string{
"hostname": "hostname",
"host-name": "host-name",
"hostname123": "hostname123",
"123hostname": "123hostname",
"hostname-of-63-bytes-long-should-be-valid-and-without-any-error": "hostname-of-63-bytes-long-should-be-valid-and-without-any-error",
}
hostnameWithDomain := "--hostname=hostname.domainname"
hostnameWithDomainTld := "--hostname=hostname.domainname.tld"
for hostname, expectedHostname := range validHostnames {
if config, _, _ := mustParse(t, "--hostname="+hostname); config.Hostname != expectedHostname {
t.Fatalf("Expected the config to have 'hostname' as %q, got %q", expectedHostname, config.Hostname)
}
}
if config, _, _ := mustParse(t, hostnameWithDomain); config.Hostname != "hostname.domainname" || config.Domainname != "" {
t.Fatalf("Expected the config to have 'hostname' as hostname.domainname, got %q", config.Hostname)
}
if config, _, _ := mustParse(t, hostnameWithDomainTld); config.Hostname != "hostname.domainname.tld" || config.Domainname != "" {
t.Fatalf("Expected the config to have 'hostname' as hostname.domainname.tld, got %q", config.Hostname)
}
}
func TestParseHostnameDomainname(t *testing.T) {
validDomainnames := map[string]string{
"domainname": "domainname",
"domain-name": "domain-name",
"domainname123": "domainname123",
"123domainname": "123domainname",
"domainname-63-bytes-long-should-be-valid-and-without-any-errors": "domainname-63-bytes-long-should-be-valid-and-without-any-errors",
}
for domainname, expectedDomainname := range validDomainnames {
if config, _, _ := mustParse(t, "--domainname="+domainname); config.Domainname != expectedDomainname {
t.Fatalf("Expected the config to have 'domainname' as %q, got %q", expectedDomainname, config.Domainname)
}
}
if config, _, _ := mustParse(t, "--hostname=some.prefix --domainname=domainname"); config.Hostname != "some.prefix" || config.Domainname != "domainname" {
t.Fatalf("Expected the config to have 'hostname' as 'some.prefix' and 'domainname' as 'domainname', got %q and %q", config.Hostname, config.Domainname)
}
if config, _, _ := mustParse(t, "--hostname=another-prefix --domainname=domainname.tld"); config.Hostname != "another-prefix" || config.Domainname != "domainname.tld" {
t.Fatalf("Expected the config to have 'hostname' as 'another-prefix' and 'domainname' as 'domainname.tld', got %q and %q", config.Hostname, config.Domainname)
for _, tc := range []struct {
input, hostname, domainname string
}{
{"--hostname=hostname", "hostname", ""},
{"--hostname=host-name", "host-name", ""},
{"--hostname=hostname123", "hostname123", ""},
{"--hostname=123hostname", "123hostname", ""},
{"--hostname=hostname-of-63-bytes-long-should-be-valid-and-without-any-error", "hostname-of-63-bytes-long-should-be-valid-and-without-any-error", ""},
{"--hostname=hostname.domainname", "hostname.domainname", ""},
{"--hostname=hostname.domainname.tld", "hostname.domainname.tld", ""},
{"--domainname=domainname", "", "domainname"},
{"--domainname=domain-name", "", "domain-name"},
{"--domainname=domainname123", "", "domainname123"},
{"--domainname=123domainname", "", "123domainname"},
{"--domainname=domainname-63-bytes-long-should-be-valid-and-without-any-errors", "", "domainname-63-bytes-long-should-be-valid-and-without-any-errors"},
{"--hostname=some.prefix --domainname=domainname", "some.prefix", "domainname"},
{"--hostname=another-prefix --domainname=domainname.tld", "another-prefix", "domainname.tld"},
} {
t.Run(tc.input, func(t *testing.T) {
config, _, _ := mustParse(t, tc.input)
require.Equal(t, tc.hostname, config.Hostname)
require.Equal(t, tc.domainname, config.Domainname)
})
}
}
@@ -497,9 +389,15 @@ func TestParseWithExpose(t *testing.T) {
})
t.Run("merge with published", func(t *testing.T) {
// Merge with actual published port
config, _, _, err := parseRun([]string{"--publish=80", "--expose=80-81/tcp", "img", "cmd"})
config, hostConfig, _, err := parseRun([]string{"--publish=80", "--publish=127.0.0.1:8080:80", "--publish=[::1]:8081:80", "--expose=80-81/tcp"})
require.NoError(t, err)
assert.Equal(t, networktypes.PortMap{
networktypes.MustParsePort("80/tcp"): {
{},
{HostIP: netip.MustParseAddr("127.0.0.1"), HostPort: "8080"},
{HostIP: netip.MustParseAddr("::1"), HostPort: "8081"},
},
}, hostConfig.PortBindings)
assert.Len(t, config.ExposedPorts, 2)
ports := []networktypes.Port{networktypes.MustParsePort("80/tcp"), networktypes.MustParsePort("81/tcp")}
for _, port := range ports {
@@ -984,41 +882,29 @@ func TestParseRestartPolicyAutoRemove(t *testing.T) {
}
func TestParseHealth(t *testing.T) {
checkOk := func(args ...string) *container.HealthConfig {
config, _, _, err := parseRun(args)
if err != nil {
t.Fatalf("%#v: %v", args, err)
}
return config.Healthcheck
}
checkError := func(expected string, args ...string) {
config, _, _, err := parseRun(args)
if err == nil {
t.Fatalf("Expected error, but got %#v", config)
}
if err.Error() != expected {
t.Fatalf("Expected %#v, got %#v", expected, err)
}
}
health := checkOk("--no-healthcheck", "img", "cmd")
if health == nil || len(health.Test) != 1 || health.Test[0] != "NONE" {
t.Fatalf("--no-healthcheck failed: %#v", health)
}
health = checkOk("--health-cmd=/check.sh -q", "img", "cmd")
if len(health.Test) != 2 || health.Test[0] != "CMD-SHELL" || health.Test[1] != "/check.sh -q" {
t.Fatalf("--health-cmd: got %#v", health.Test)
}
if health.Timeout != 0 {
t.Fatalf("--health-cmd: timeout = %s", health.Timeout)
}
checkError("--no-healthcheck conflicts with --health-* options",
"--no-healthcheck", "--health-cmd=/check.sh -q", "img", "cmd")
health = checkOk("--health-timeout=2s", "--health-retries=3", "--health-interval=4.5s", "--health-start-period=5s", "--health-start-interval=1s", "img", "cmd")
if health.Timeout != 2*time.Second || health.Retries != 3 || health.Interval != 4500*time.Millisecond || health.StartPeriod != 5*time.Second || health.StartInterval != 1*time.Second {
t.Fatalf("--health-*: got %#v", health)
for _, tc := range []struct {
args []string
expected *container.HealthConfig
wantErr string
}{
{[]string{"--no-healthcheck", "img", "cmd"}, &container.HealthConfig{Test: []string{"NONE"}}, ""},
{[]string{"--health-cmd=/check.sh -q", "img", "cmd"}, &container.HealthConfig{Test: []string{"CMD-SHELL", "/check.sh -q"}}, ""},
{[]string{"--no-healthcheck", "--health-cmd=/check.sh -q", "img", "cmd"}, nil, "--no-healthcheck conflicts with --health-* options"},
{
[]string{"--health-timeout=2s", "--health-retries=3", "--health-interval=4.5s", "--health-start-period=5s", "--health-start-interval=1s", "img", "cmd"},
&container.HealthConfig{Timeout: 2 * time.Second, Retries: 3, Interval: 4500 * time.Millisecond, StartPeriod: 5 * time.Second, StartInterval: time.Second},
"",
},
} {
t.Run(strings.Join(tc.args, " "), func(t *testing.T) {
config, _, _, err := parseRun(tc.args)
if tc.wantErr != "" {
require.EqualError(t, err, tc.wantErr)
} else {
require.NoError(t, err)
require.Equal(t, tc.expected, config.Healthcheck)
}
})
}
}
@@ -1037,82 +923,53 @@ func TestParseLoggingOpts(t *testing.T) {
}
}
func TestParseEnvfileVariables(t *testing.T) { //nolint:dupl // verbatim copy from docker/cli tests
expErr := "--env-file: open nonexistent: no such file or directory"
if runtime.GOOS == "windows" {
expErr = "--env-file: open nonexistent: The system cannot find the file specified."
}
// env ko
if _, _, _, err := parseRun([]string{"--env-file=nonexistent", "img", "cmd"}); err == nil || err.Error() != expErr {
t.Fatalf("Expected an error with message '%s', got %v", expErr, err)
}
// env ok
config, _, _, err := parseRun([]string{"--env-file=testdata/valid.env", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Env) != 1 || config.Env[0] != "ENV1=value1" {
t.Fatalf("Expected a config with [ENV1=value1], got %v", config.Env)
}
config, _, _, err = parseRun([]string{"--env-file=testdata/valid.env", "--env=ENV2=value2", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Env) != 2 || config.Env[0] != "ENV1=value1" || config.Env[1] != "ENV2=value2" {
t.Fatalf("Expected a config with [ENV1=value1 ENV2=value2], got %v", config.Env)
func TestParseFileVariables(t *testing.T) {
for _, tc := range []struct {
flag, file, key string
}{
{"env", "valid.env", "ENV"},
{"label", "valid.label", "LABEL"},
} {
t.Run(tc.flag, func(t *testing.T) {
expectedError := "no such file or directory"
if runtime.GOOS == "windows" {
expectedError = "The system cannot find the file specified."
}
_, _, _, err := parseRun([]string{"--" + tc.flag + "-file=nonexistent"})
require.EqualError(t, err, "--"+tc.flag+"-file: open nonexistent: "+expectedError)
for _, extra := range []bool{false, true} {
args := []string{"--" + tc.flag + "-file=testdata/" + tc.file}
expected := []string{tc.key + "1=value1"}
if extra {
args = append(args, "--"+tc.flag+"="+tc.key+"2=value2")
expected = append(expected, tc.key+"2=value2")
}
config, _, _, err := parseRun(args)
require.NoError(t, err)
if tc.flag == "env" {
assert.Equal(t, expected, config.Env)
} else {
assert.Len(t, config.Labels, len(expected))
for _, label := range expected {
key, value, _ := strings.Cut(label, "=")
assert.Equal(t, value, config.Labels[key])
}
}
}
})
}
}
func TestParseEnvfileVariablesWithBOMUnicode(t *testing.T) {
// UTF8 with BOM
config, _, _, err := parseRun([]string{"--env-file=testdata/utf8.env", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
env := []string{"FOO=BAR", "HELLO=" + string([]byte{0xe6, 0x82, 0xa8, 0xe5, 0xa5, 0xbd}), "BAR=FOO"}
if len(config.Env) != len(env) {
t.Fatalf("Expected a config with %d env variables, got %v: %v", len(env), len(config.Env), config.Env)
}
for i, v := range env {
if config.Env[i] != v {
t.Fatalf("Expected a config with [%s], got %v", v, []byte(config.Env[i]))
}
}
config, _, _, err := parseRun([]string{"--env-file=testdata/utf8.env"})
require.NoError(t, err)
assert.Equal(t, []string{"FOO=BAR", "HELLO=您好", "BAR=FOO"}, config.Env)
// UTF16 with BOM
e := "invalid utf8 bytes at line"
if _, _, _, err := parseRun([]string{"--env-file=testdata/utf16.env", "img", "cmd"}); err == nil || !strings.Contains(err.Error(), e) {
t.Fatalf("Expected an error with message '%s', got %v", e, err)
}
// UTF16BE with BOM
if _, _, _, err := parseRun([]string{"--env-file=testdata/utf16be.env", "img", "cmd"}); err == nil || !strings.Contains(err.Error(), e) {
t.Fatalf("Expected an error with message '%s', got %v", e, err)
}
}
func TestParseLabelfileVariables(t *testing.T) { //nolint:dupl // verbatim copy from docker/cli tests
expErr := "--label-file: open nonexistent: no such file or directory"
if runtime.GOOS == "windows" {
expErr = "--label-file: open nonexistent: The system cannot find the file specified."
}
// label ko
if _, _, _, err := parseRun([]string{"--label-file=nonexistent", "img", "cmd"}); err == nil || err.Error() != expErr {
t.Fatalf("Expected an error with message '%s', got %v", expErr, err)
}
// label ok
config, _, _, err := parseRun([]string{"--label-file=testdata/valid.label", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Labels) != 1 || config.Labels["LABEL1"] != "value1" {
t.Fatalf("Expected a config with [LABEL1:value1], got %v", config.Labels)
}
config, _, _, err = parseRun([]string{"--label-file=testdata/valid.label", "--label=LABEL2=value2", "img", "cmd"})
if err != nil {
t.Fatal(err)
}
if len(config.Labels) != 2 || config.Labels["LABEL1"] != "value1" || config.Labels["LABEL2"] != "value2" {
t.Fatalf("Expected a config with [LABEL1:value1 LABEL2:value2], got %v", config.Labels)
for _, file := range []string{"utf16.env", "utf16be.env"} {
t.Run(file, func(t *testing.T) {
_, _, _, err := parseRun([]string{"--env-file=testdata/" + file})
require.ErrorContains(t, err, "invalid utf8 bytes at line")
})
}
}
+4 -8
View File
@@ -58,18 +58,14 @@ func RemoveImage(ctx context.Context, imageName string, force, pruneChildren boo
}
defer cli.Close()
if _, err := cli.ImageInspect(ctx, imageName); cerrdefs.IsNotFound(err) {
if _, err = cli.ImageRemove(ctx, imageName, client.ImageRemoveOptions{
Force: force,
PruneChildren: pruneChildren,
}); cerrdefs.IsNotFound(err) {
return false, nil
} else if err != nil {
return false, err
}
if _, err = cli.ImageRemove(ctx, imageName, client.ImageRemoveOptions{
Force: force,
PruneChildren: pruneChildren,
}); err != nil {
return false, err
}
return true, nil
}
+7
View File
@@ -66,4 +66,11 @@ func TestImageExistsLocally(t *testing.T) {
wrongPlatform, err := ImageExistsLocally(ctx, amd64Ref, "linux/arm64")
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.False(t, wrongPlatform)
removed, err := RemoveImage(ctx, amd64Ref, false, false)
require.NoError(t, err)
assert.True(t, removed)
removed, err = RemoveImage(ctx, amd64Ref, false, false)
require.NoError(t, err)
assert.False(t, removed)
}
+1 -10
View File
@@ -33,18 +33,9 @@ func logDockerResponse(logger *logging.Logger, dockerResponse io.ReadCloser, isE
defer dockerResponse.Close()
scanner := bufio.NewScanner(dockerResponse)
msg := dockerMessage{}
for scanner.Scan() {
line := scanner.Bytes()
msg.ID = ""
msg.Stream = ""
msg.Error = ""
msg.ErrorDetail.Message = ""
msg.Status = ""
msg.Progress = ""
var msg dockerMessage
if err := json.Unmarshal(line, &msg); err != nil {
writeLog(logger, false, "Unable to unmarshal line [%s] ==> %v", string(line), err)
continue
+12 -26
View File
@@ -6,6 +6,7 @@ package container
import (
"testing"
specs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -17,32 +18,17 @@ func TestParsePlatform(t *testing.T) {
assert.Nil(t, got)
})
t.Run("os/arch", func(t *testing.T) {
got, err := parsePlatform("linux/amd64")
require.NoError(t, err)
require.NotNil(t, got)
assert.Equal(t, "linux", got.OS)
assert.Equal(t, "amd64", got.Architecture)
assert.Empty(t, got.Variant)
})
t.Run("os/arch/variant", func(t *testing.T) {
got, err := parsePlatform("linux/arm/v7")
require.NoError(t, err)
require.NotNil(t, got)
assert.Equal(t, "linux", got.OS)
assert.Equal(t, "arm", got.Architecture)
assert.Equal(t, "v7", got.Variant)
})
t.Run("input is lowercased", func(t *testing.T) {
got, err := parsePlatform("Linux/AMD64/V8")
require.NoError(t, err)
require.NotNil(t, got)
assert.Equal(t, "linux", got.OS)
assert.Equal(t, "amd64", got.Architecture)
assert.Equal(t, "v8", got.Variant)
})
for input, want := range map[string]specs.Platform{
"linux/amd64": {OS: "linux", Architecture: "amd64"},
"linux/arm/v7": {OS: "linux", Architecture: "arm", Variant: "v7"},
"Linux/AMD64/V8": {OS: "linux", Architecture: "amd64", Variant: "v8"},
} {
t.Run(input, func(t *testing.T) {
got, err := parsePlatform(input)
require.NoError(t, err)
assert.Equal(t, &want, got)
})
}
for _, bad := range []string{
"amd64",
+24 -6
View File
@@ -48,6 +48,13 @@ var (
rawStreamPath = regexp.MustCompile(`^(/v[0-9.]+)?/(containers/[^/]+/attach|exec/[^/]+/start)$`)
)
var dockerProxyDirectories sync.Map
type dockerProxyDirectory struct {
dir, daemonDir string
socket os.FileInfo
}
func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if host := os.Getenv("DOCKER_HOST"); runtime.GOOS != "linux" || host != "" && !strings.HasPrefix(host, "unix://") {
return nil
@@ -63,15 +70,24 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
if !ok {
return nil
}
if info, err := os.Stat(daemonSocket); err != nil || info.Mode()&os.ModeSocket == 0 {
socket, err := os.Stat(daemonSocket)
if err != nil || socket.Mode()&os.ModeSocket == 0 {
return nil
}
dir, daemonDir := runnerContainerWorkdir(probeCtx, cli)
seen := daemonDir != ""
workdir, _ := os.Getwd()
key := [3]string{cli.DaemonHost(), workdir, os.TempDir()}
cached, _ := dockerProxyDirectories.Load(key)
directory, seen := cached.(dockerProxyDirectory)
seen = seen && os.SameFile(directory.socket, socket) // daemon replacement invalidates cached paths
dir, daemonDir := directory.dir, directory.daemonDir
if !seen {
if dir, err = filepath.Abs(os.TempDir()); err == nil {
daemonDir = dir
seen, err = daemonSeesDir(probeCtx, cli, dir, daemonDir)
dir, daemonDir = runnerContainerWorkdir(probeCtx, cli)
seen = daemonDir != ""
if !seen {
if dir, err = filepath.Abs(os.TempDir()); err == nil {
daemonDir = dir
seen, err = daemonSeesDir(probeCtx, cli, dir, daemonDir)
}
}
}
if err != nil {
@@ -87,9 +103,11 @@ func NewDockerProxy(ctx context.Context, job string) *DockerProxy {
}
proxy, err := StartDockerProxy(daemonSocket, dir, job)
if err != nil {
dockerProxyDirectories.Delete(key)
common.Logger(ctx).Warnf("docker proxy not started, the job gets the daemon socket directly: %v", err)
return nil
}
dockerProxyDirectories.Store(key, dockerProxyDirectory{dir, daemonDir, socket})
proxy.Socket = daemonDir + strings.TrimPrefix(proxy.Socket, dir)
return proxy
}
+43
View File
@@ -20,6 +20,7 @@ import (
"runtime"
"strconv"
"strings"
"sync/atomic"
"testing"
"time"
@@ -68,6 +69,7 @@ func TestDockerProxy(t *testing.T) {
bodies := make(chan []byte, 8)
createStarted, releaseCreate := make(chan struct{}), make(chan struct{})
streamsDone := make(chan struct{}, 2)
var probes atomic.Int32
daemonSocket := filepath.Join(shortTempDir(t), "d.sock")
listener, err := net.Listen("unix", daemonSocket)
require.NoError(t, err)
@@ -75,12 +77,28 @@ func TestDockerProxy(t *testing.T) {
switch {
case strings.HasSuffix(r.URL.Path, "/create"):
body, _ := io.ReadAll(r.Body)
if bytes.Contains(body, []byte(`"Image":"probe-image"`)) {
_, _ = w.Write([]byte(`{"Id":"probe"}`))
return
}
if r.URL.RawQuery == "wait" {
close(createStarted)
<-releaseCreate
}
bodies <- body
w.WriteHeader(http.StatusCreated)
case strings.HasSuffix(r.URL.Path, "/images/json"):
if probes.Add(1) == 1 {
http.Error(w, "probe unavailable", http.StatusServiceUnavailable)
return
}
_, _ = w.Write([]byte(`[{"Id":"probe-image"}]`))
case strings.HasSuffix(r.URL.Path, "/json"):
http.NotFound(w, r)
case strings.HasSuffix(r.URL.Path, "/archive"):
w.Header().Set("X-Docker-Container-Path-Stat", "e30=")
case r.Method == http.MethodDelete:
w.WriteHeader(http.StatusNoContent)
case r.URL.Path == "/exec/detached/start" || r.URL.Path == "/exec/error/start":
if r.URL.Path == "/exec/error/start" {
w.Header().Set("Content-Type", "application/vnd.docker.raw-stream")
@@ -211,6 +229,31 @@ func TestDockerProxy(t *testing.T) {
assert.Equal(t, "OK GET /v1.47/_ping", string(body))
})
t.Run("discovery caches only successes", func(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("automatic proxy discovery requires Linux")
}
t.Setenv("DOCKER_HOST", "unix://"+daemonSocket)
t.Setenv("TMPDIR", shortTempDir(t))
require.Nil(t, NewDockerProxy(t.Context(), "failed"))
for _, job := range []string{"first", "second"} {
proxy := NewDockerProxy(t.Context(), job)
require.NotNil(t, proxy)
require.NoError(t, proxy.Close(context.Background()))
}
assert.EqualValues(t, 2, probes.Load())
dockerProxyDirectories.Range(func(key, value any) bool {
directory, _ := value.(dockerProxyDirectory)
directory.socket, _ = os.Stat(t.TempDir())
dockerProxyDirectories.Store(key, directory)
return true
})
proxy := NewDockerProxy(t.Context(), "replaced daemon")
require.NotNil(t, proxy)
require.NoError(t, proxy.Close(context.Background()))
assert.EqualValues(t, 3, probes.Load())
})
t.Run("tunnels raw streams through stdin EOF", func(t *testing.T) {
for _, stream := range []struct{ path, upgrade string }{
{path: "/v1.47/exec/abc/start"},
+45 -64
View File
@@ -50,15 +50,12 @@ const drainGracePeriod = 2 * time.Second
// NewContainer creates a reference to a container
func NewContainer(input *NewContainerInput) ExecutionsEnvironment {
cr := new(containerReference)
cr.input = input
// Resolved up front because the image pull runs before the container is created.
cf := createFlagsFromOptions(input.allOptions())
if cf.platform != "" {
cr.input.Platform = cf.platform
input.Platform = cf.platform
}
cr.pullPolicy = cf.pull
return cr
return &containerReference{input: input, pullPolicy: cf.pull}
}
// supportsContainerImagePlatform reports whether the Docker server API version
@@ -394,17 +391,16 @@ func GetDockerClient(ctx context.Context) (cli client.APIClient, err error) {
return cli, nil
}
func GetHostInfo(ctx context.Context) (info system.Info, err error) {
var cli client.APIClient
cli, err = GetDockerClient(ctx)
func GetHostInfo(ctx context.Context) (system.Info, error) {
cli, err := GetDockerClient(ctx)
if err != nil {
return info, err
return system.Info{}, err
}
defer cli.Close()
result, err := cli.Info(ctx, client.InfoOptions{})
if err != nil {
return info, err
return system.Info{}, err
}
return result.Info, nil
@@ -476,22 +472,23 @@ func (cr *containerReference) find() common.Executor {
}
cr.id = ""
}
containers, err := cr.cli.ContainerList(ctx, client.ContainerListOptions{
All: true,
})
result, err := cr.cli.ContainerInspect(ctx, cr.input.Name, client.ContainerInspectOptions{})
if err == nil && result.Container.Name == "/"+cr.input.Name {
cr.id = result.Container.ID
return nil
} else if cerrdefs.IsNotFound(err) {
return nil
}
containers, err := cr.cli.ContainerList(ctx, client.ContainerListOptions{All: true}) // inspect also matches ids, and podman fails it with a 500 for a container removed mid-request
if err != nil {
return fmt.Errorf("failed to list containers: %w", err)
}
for _, c := range containers.Items {
for _, name := range c.Names {
if name[1:] == cr.input.Name {
cr.id = c.ID
return nil
}
for _, candidate := range containers.Items {
if slices.Contains(candidate.Names, "/"+cr.input.Name) {
cr.id = candidate.ID
break
}
}
return nil
}
}
@@ -558,32 +555,32 @@ func (input *NewContainerInput) allOptions() string {
return strings.TrimSpace(input.RunnerOptions + " " + input.WorkflowOptions)
}
func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) (*container.Config, *container.HostConfig, error) {
func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) error {
logger := common.Logger(ctx)
options := cr.input.allOptions()
if options == "" {
return config, hostConfig, nil
return nil
}
// For Gitea, checked here because the parse below is what would read those files
if err := rejectHostReadingOptions(cr.input.WorkflowOptions); err != nil {
return nil, nil, err
return err
}
// parse configuration from CLI container.options
flags, copts, cf, err := parseContainerOptions(options)
if err != nil {
return nil, nil, err
return err
}
if err := cf.validate(); err != nil {
return nil, nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
return fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
}
if len(copts.netMode.Value()) == 0 {
if err = copts.netMode.Set(cr.input.NetworkMode); err != nil {
return nil, nil, fmt.Errorf("cannot parse networkmode=%s. This is an internal error and should not happen: '%w'", cr.input.NetworkMode, err)
return fmt.Errorf("cannot parse networkmode=%s. This is an internal error and should not happen: '%w'", cr.input.NetworkMode, err)
}
}
@@ -596,7 +593,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
containerConfig, err := parse(flags, copts, runtime.GOOS)
if err != nil {
return nil, nil, fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
return fmt.Errorf("cannot process container options: '%s': '%w'", options, err)
}
// workflow aliases join the runner's own, deduped so a re-create cannot grow the input
for _, endpoint := range containerConfig.NetworkingConfig.EndpointsConfig {
@@ -611,7 +608,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
if !hostConfig.Privileged {
trusted, err := parseOptionsHostConfig(cr.input.RunnerOptions)
if err != nil {
return nil, nil, err
return err
}
sanitizeOptionsHostConfig(logger, containerConfig.HostConfig, trusted)
}
@@ -623,7 +620,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
logger.Debugf("Custom container.HostConfig from options ==> %+v", containerConfig.HostConfig)
cr.sanitizeConfig(ctx, nil, containerConfig.HostConfig) // a dropped volume must not displace the runner's own
cr.sanitizeConfig(ctx, containerConfig.HostConfig) // a dropped volume must not displace the runner's own
overlayVolumes(hostConfig, containerConfig.HostConfig)
binds := hostConfig.Binds
mounts := hostConfig.Mounts
@@ -644,7 +641,7 @@ func (cr *containerReference) mergeContainerConfigs(ctx context.Context, config
hostConfig.NetworkMode = networkMode
logger.Debugf("Merged container.HostConfig ==> %+v", hostConfig)
return config, hostConfig, nil
return nil
}
func (cr *containerReference) create(capAdd, capDrop []string) common.Executor {
@@ -710,8 +707,7 @@ func (cr *containerReference) create(capAdd, capDrop []string) common.Executor {
AutoRemove: input.AutoRemove,
}
config, hostConfig, err = cr.mergeContainerConfigs(ctx, config, hostConfig)
if err != nil {
if err := cr.mergeContainerConfigs(ctx, config, hostConfig); err != nil {
return err
}
@@ -720,18 +716,15 @@ func (cr *containerReference) create(capAdd, capDrop []string) common.Executor {
}
// For Gitea
config, hostConfig = cr.sanitizeConfig(ctx, config, hostConfig)
cr.sanitizeConfig(ctx, hostConfig)
var networkingConfig *network.NetworkingConfig
n := hostConfig.NetworkMode
// IsUserDefined and IsHost are broken on windows
if n.IsUserDefined() && n != "host" && len(input.NetworkAliases) > 0 {
endpointConfig := &network.EndpointSettings{
Aliases: input.NetworkAliases,
}
networkingConfig = &network.NetworkingConfig{
EndpointsConfig: map[string]*network.EndpointSettings{
input.NetworkMode: endpointConfig,
input.NetworkMode: {Aliases: input.NetworkAliases},
},
}
}
@@ -772,15 +765,11 @@ func (cr *containerReference) exec(cmd []string, env map[string]string, user, wo
logger.Debugf("Exec command '%s'", cmd)
isTerminal := cr.input.AllocatePTY
var wd string
if workdir != "" {
if strings.HasPrefix(workdir, "/") {
wd = workdir
} else {
wd = fmt.Sprintf("%s/%s", cr.input.WorkingDir, workdir)
}
} else {
wd = cr.input.WorkingDir
wd := cr.input.WorkingDir
if strings.HasPrefix(workdir, "/") {
wd = workdir
} else if workdir != "" {
wd = fmt.Sprintf("%s/%s", cr.input.WorkingDir, workdir)
}
logger.Debugf("Working directory '%s'", wd)
@@ -805,8 +794,7 @@ func (cr *containerReference) exec(cmd []string, env map[string]string, user, wo
}
defer resp.Close()
err = cr.waitForCommand(ctx, resp.HijackedResponse)
if err != nil {
if err := cr.waitForCommand(ctx, resp.HijackedResponse); err != nil {
return err
}
@@ -908,17 +896,14 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
return err
}
logger.Debugf("Writing tarball %s from %s", tarFile.Name(), srcPath)
defer func(tarFile *os.File) {
name := tarFile.Name()
err := tarFile.Close()
if !errors.Is(err, os.ErrClosed) {
defer func() {
if err := tarFile.Close(); !errors.Is(err, os.ErrClosed) {
logger.Error(err)
}
err = os.Remove(name)
if err != nil {
if err := os.Remove(tarFile.Name()); err != nil {
logger.Error(err)
}
}(tarFile)
}()
extractionRoot := "/"
if actPath := cr.GetActPath(); strings.HasPrefix(dstPath+"/", actPath+"/") {
extractionRoot = actPath // keeps tar entries from crossing the image's /var/run symlink
@@ -928,8 +913,7 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
}
logger.Debugf("Extracting content from '%s' to '%s'", tarFile.Name(), dstPath)
_, err = tarFile.Seek(0, 0)
if err != nil {
if _, err := tarFile.Seek(0, 0); err != nil {
return fmt.Errorf("failed to seek tar archive: %w", err)
}
return cr.upload(ctx, extractionRoot, tarFile)
@@ -1097,9 +1081,7 @@ func parseOptionsHostConfig(options string) (*container.HostConfig, error) {
return containerConfig.HostConfig, nil
}
// For Gitea
// sanitizeConfig remove the invalid configurations from `config` and `hostConfig`
func (cr *containerReference) sanitizeConfig(ctx context.Context, config *container.Config, hostConfig *container.HostConfig) (*container.Config, *container.HostConfig) {
func (cr *containerReference) sanitizeConfig(ctx context.Context, hostConfig *container.HostConfig) {
logger := common.Logger(ctx)
matcher := newValidVolumeMatcher(ctx, cr.input.ValidVolumes)
@@ -1148,13 +1130,12 @@ func (cr *containerReference) sanitizeConfig(ctx context.Context, config *contai
}
}
hostConfig.Mounts = sanitizedMounts
return config, hostConfig
}
func SanitizeBinds(ctx context.Context, validVolumes, binds []string) []string {
cr := &containerReference{input: &NewContainerInput{ValidVolumes: validVolumes}}
_, hostConfig := cr.sanitizeConfig(ctx, nil, &container.HostConfig{Binds: binds})
hostConfig := &container.HostConfig{Binds: binds}
cr.sanitizeConfig(ctx, hostConfig)
return hostConfig.Binds
}
+104 -129
View File
@@ -13,9 +13,13 @@ import (
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"gitea.com/gitea/runner/act/common"
@@ -182,11 +186,9 @@ func TestDockerExecAbort(t *testing.T) {
}, nil)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
id: "123",
cli: client,
input: &NewContainerInput{},
}
channel := make(chan error)
@@ -209,12 +211,10 @@ func TestDockerExecAbort(t *testing.T) {
func TestDockerExecFailure(t *testing.T) {
ctx := t.Context()
conn := &mockConn{}
client := &mockDockerClient{}
client.On("ExecCreate", ctx, "123", mock.AnythingOfType("client.ExecCreateOptions")).Return(mobyclient.ExecCreateResult{ID: "id"}, nil)
client.On("ExecAttach", ctx, "id", mock.AnythingOfType("client.ExecAttachOptions")).Return(mobyclient.ExecAttachResult{
Conn: conn,
Conn: &mockConn{},
Reader: bufio.NewReader(strings.NewReader("output")),
}, nil)
client.On("ExecInspect", ctx, "id", mobyclient.ExecInspectOptions{}).Return(mobyclient.ExecInspectResult{
@@ -222,11 +222,9 @@ func TestDockerExecFailure(t *testing.T) {
}, nil)
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
id: "123",
cli: client,
input: &NewContainerInput{},
}
err := cr.exec([]string{""}, map[string]string{}, "user", "workdir")(ctx)
@@ -235,7 +233,6 @@ func TestDockerExecFailure(t *testing.T) {
assert.Equal(t, ExitCodeError(1), exitErr)
assert.Equal(t, "Process completed with exit code 1.", err.Error())
conn.AssertExpectations(t)
client.AssertExpectations(t)
}
@@ -250,9 +247,6 @@ func stdcopyFrame(stream stdcopy.StdType, payload string) []byte {
return b
}
// TestDockerAttachFlushesTrailingLine verifies that wait() blocks until the
// attach() streaming goroutine has drained and flushed the container's output,
// so a final line without a trailing newline is not lost.
func TestDockerAttachFlushesTrailingLine(t *testing.T) {
ctx := t.Context()
@@ -273,18 +267,13 @@ func TestDockerAttachFlushesTrailingLine(t *testing.T) {
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 0}
errCh := make(chan error, 1)
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{
Result: (<-chan container.WaitResponse)(statusCh),
Error: (<-chan error)(errCh),
})
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
Stdout: logWriter,
Stderr: logWriter,
},
@@ -293,10 +282,6 @@ func TestDockerAttachFlushesTrailingLine(t *testing.T) {
require.NoError(t, cr.attach()(ctx))
require.NoError(t, cr.wait()(ctx))
// wait() must have blocked until the goroutine drained AND flushed; the
// trailing, non-newline-terminated line must therefore be present. Reading
// lines here is race-free because wait() synchronizes on attachDone, which
// the goroutine closes after the final append.
assert.Equal(t, []string{"line one\n", "last line without newline"}, lines)
client.AssertExpectations(t)
@@ -307,24 +292,11 @@ func TestDockerWaitFailure(t *testing.T) {
statusCh := make(chan container.WaitResponse, 1)
statusCh <- container.WaitResponse{StatusCode: 2}
errCh := make(chan error, 1)
client := &mockDockerClient{}
client.On("ContainerWait", ctx, "123", mobyclient.ContainerWaitOptions{Condition: container.WaitConditionNotRunning}).
Return(mobyclient.ContainerWaitResult{
Result: (<-chan container.WaitResponse)(statusCh),
Error: (<-chan error)(errCh),
})
Return(mobyclient.ContainerWaitResult{Result: statusCh})
cr := &containerReference{
id: "123",
cli: client,
input: &NewContainerInput{
Image: "image",
},
}
err := cr.wait()(ctx)
err := (&containerReference{id: "123", cli: client}).wait()(ctx)
var exitErr ExitCodeError
require.ErrorAs(t, err, &exitErr)
assert.Equal(t, ExitCodeError(2), exitErr)
@@ -398,62 +370,72 @@ func TestRemoveWithoutIDUsesName(t *testing.T) {
client.AssertExpectations(t)
}
// find() must drop a stale cached id so later Copy/Exec don't hit the
// daemon with a torn-down container.
func TestFindRevalidatesStaleID(t *testing.T) {
ctx := t.Context()
inspectOpts := mobyclient.ContainerInspectOptions{}
notFound := cerrdefs.ErrNotFound.WithMessage("No such container")
boom := errors.New("daemon unreachable")
newCR := func(id string) (*containerReference, *mockDockerClient) {
client := &mockDockerClient{}
return &containerReference{id: id, cli: client, input: &NewContainerInput{Name: "job-1"}}, client
for _, tc := range []struct {
name, cachedID string
cachedErr error
lookup container.InspectResponse
lookupErr error
listed []container.Summary
wantID string
retryList bool
}{
{name: "stale id cleared, name lookup empty", cachedID: "stale", cachedErr: notFound, lookupErr: notFound},
{name: "stale id cleared, name lookup repopulates", cachedID: "stale", cachedErr: notFound, lookup: container.InspectResponse{ID: "fresh", Name: "/job-1"}, wantID: "fresh"},
{name: "id match falls back to listing", lookup: container.InspectResponse{ID: "job-1abc", Name: "/somebody-else"}, listed: []container.Summary{{ID: "fresh", Names: []string{"/job-1"}}}, wantID: "fresh"},
{name: "live id kept", cachedID: "live", wantID: "live"},
{name: "transient inspect error trusts cache", cachedID: "live", cachedErr: boom, wantID: "live"},
{name: "name lookup error falls back to listing", lookupErr: boom, listed: []container.Summary{{ID: "other", Names: []string{"/somebody-else"}}, {ID: "fresh", Names: []string{"/job-1"}}}, wantID: "fresh", retryList: true},
} {
t.Run(tc.name, func(t *testing.T) {
client := &mockDockerClient{}
cr := &containerReference{id: tc.cachedID, cli: client, input: &NewContainerInput{Name: "job-1"}}
if tc.cachedID != "" {
client.On("ContainerInspect", ctx, tc.cachedID, inspectOpts).Return(mobyclient.ContainerInspectResult{}, tc.cachedErr)
}
if tc.cachedID == "" || cerrdefs.IsNotFound(tc.cachedErr) {
client.On("ContainerInspect", ctx, "job-1", inspectOpts).Return(mobyclient.ContainerInspectResult{Container: tc.lookup}, tc.lookupErr)
}
if tc.listed != nil {
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).Return(mobyclient.ContainerListResult{Items: tc.listed}, nil).Once()
}
require.NoError(t, cr.find()(ctx))
assert.Equal(t, tc.wantID, cr.id)
if tc.retryList {
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).Return(mobyclient.ContainerListResult{}, boom)
cr.id = ""
require.ErrorIs(t, cr.find()(ctx), boom)
}
client.AssertExpectations(t)
})
}
listOpts := mobyclient.ContainerListOptions{All: true}
inspectOpts := mobyclient.ContainerInspectOptions{}
}
t.Run("stale id cleared, name lookup empty", func(t *testing.T) {
cr, client := newCR("stale")
client.On("ContainerInspect", ctx, "stale", inspectOpts).Return(mobyclient.ContainerInspectResult{}, notFound)
client.On("ContainerList", ctx, listOpts).Return(mobyclient.ContainerListResult{}, nil)
require.NoError(t, cr.find()(ctx))
assert.Empty(t, cr.id)
client.AssertExpectations(t)
})
t.Run("stale id cleared, name lookup repopulates", func(t *testing.T) {
cr, client := newCR("stale")
client.On("ContainerInspect", ctx, "stale", inspectOpts).Return(mobyclient.ContainerInspectResult{}, notFound)
client.On("ContainerList", ctx, listOpts).Return(mobyclient.ContainerListResult{Items: []container.Summary{
{ID: "other", Names: []string{"/somebody-else"}},
{ID: "fresh", Names: []string{"/job-1"}},
}}, nil)
require.NoError(t, cr.find()(ctx))
assert.Equal(t, "fresh", cr.id)
client.AssertExpectations(t)
})
t.Run("live id kept", func(t *testing.T) {
cr, client := newCR("live")
client.On("ContainerInspect", ctx, "live", inspectOpts).Return(mobyclient.ContainerInspectResult{}, nil)
require.NoError(t, cr.find()(ctx))
assert.Equal(t, "live", cr.id)
client.AssertExpectations(t)
})
t.Run("transient inspect error trusts cache", func(t *testing.T) {
cr, client := newCR("live")
client.On("ContainerInspect", ctx, "live", inspectOpts).Return(mobyclient.ContainerInspectResult{}, boom)
require.NoError(t, cr.find()(ctx))
assert.Equal(t, "live", cr.id)
client.AssertExpectations(t)
})
t.Run("list error propagates", func(t *testing.T) {
cr, client := newCR("")
client.On("ContainerList", ctx, listOpts).Return(mobyclient.ContainerListResult{}, boom)
require.ErrorIs(t, cr.find()(ctx), boom)
client.AssertExpectations(t)
})
func TestRunnerArchCachedPerRun(t *testing.T) {
var infos atomic.Int32
daemon := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
writer.Header().Set("Content-Type", "application/json")
writer.Header().Set("API-Version", "1.47")
if strings.HasSuffix(request.URL.Path, "/info") {
infos.Add(1)
_, _ = io.WriteString(writer, `{"Architecture":"aarch64"}`)
}
}))
t.Cleanup(daemon.Close)
t.Setenv("DOCKER_HOST", daemon.URL)
extensions := &LinuxContainerEnvironmentExtensions{}
run := WithRunnerArch(t.Context())
var lookups sync.WaitGroup
for range 4 {
lookups.Go(func() { assert.Equal(t, "ARM64", extensions.GetRunnerContext(run)["arch"]) })
}
lookups.Wait()
assert.Equal(t, "ARM64", extensions.GetRunnerContext(t.Context())["arch"])
assert.EqualValues(t, 2, infos.Load())
}
// Every daemon entry point fails fast with a clear, container-named
@@ -461,7 +443,8 @@ func TestFindRevalidatesStaleID(t *testing.T) {
func TestRejectsMissingContainer(t *testing.T) {
ctx := t.Context()
client := &mockDockerClient{}
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).Return(mobyclient.ContainerListResult{}, nil)
client.On("ContainerInspect", ctx, "job-1", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound)
cr := &containerReference{cli: client, input: &NewContainerInput{Name: "job-1"}}
check := func(op string, err error) {
t.Helper()
@@ -498,10 +481,8 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
client := &mockDockerClient{}
client.On("ContainerInspect", ctx, "stale", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{}, cerrdefs.ErrNotFound.WithMessage("gone"))
client.On("ContainerList", ctx, mobyclient.ContainerListOptions{All: true}).
Return(mobyclient.ContainerListResult{Items: []container.Summary{
{ID: "fresh", Names: []string{"/job-1"}},
}}, nil)
client.On("ContainerInspect", ctx, "job-1", mobyclient.ContainerInspectOptions{}).
Return(mobyclient.ContainerInspectResult{Container: container.InspectResponse{ID: "fresh", Name: "/job-1"}}, nil)
client.On("CopyToContainer", ctx, "fresh", mock.Anything).
Run(func(args mock.Arguments) {
opts := args.Get(2).(mobyclient.CopyToContainerOptions)
@@ -525,9 +506,6 @@ func TestPublicCopyPipelineHandlesStaleID(t *testing.T) {
}
}
// Type assert containerReference implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &containerReference{}
func TestCheckVolumes(t *testing.T) {
testCases := []struct {
desc string
@@ -622,10 +600,11 @@ func TestCheckVolumes(t *testing.T) {
ValidVolumes: tc.validVolumes,
},
}
_, hostConf := cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{Binds: tc.binds, Mounts: tc.mounts})
hostConf := &container.HostConfig{Binds: tc.binds, Mounts: tc.mounts}
cr.sanitizeConfig(ctx, hostConf)
assert.Equal(t, tc.expectedBinds, hostConf.Binds)
assert.ElementsMatch(t, tc.expectedMounts, hostConf.Mounts)
assert.Len(t, hook.AllRecords(), len(tc.binds)+len(tc.mounts)-len(tc.expectedBinds)-len(tc.expectedMounts)) // every drop is warned about
assert.Len(t, hook.AllRecords(), len(tc.binds)+len(tc.mounts)-len(tc.expectedBinds)-len(tc.expectedMounts))
})
}
}
@@ -645,8 +624,6 @@ func TestMergeContainerConfigsDropsVolumeDriversFromWorkflows(t *testing.T) {
assert.Equal(t, "job-escape", hostConfig.Mounts[0].Source)
}
// Both of these are read here, on the runner, so a workflow could read the runner's files
// and environment with them.
func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
hostFile := filepath.Join(t.TempDir(), "host.env")
require.NoError(t, os.WriteFile(hostFile, []byte("STOLEN=from-the-host\n"), 0o600))
@@ -656,16 +633,12 @@ func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
logger := logging.New(logtest.New())
cr := &containerReference{input: &NewContainerInput{NetworkMode: "bridge", WorkflowOptions: option}}
_, _, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.ErrorContains(t, err, "not allowed in a workflow")
require.ErrorContains(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{}), "not allowed in a workflow")
// the runner reading its own files is what those options are for
cr = &containerReference{input: &NewContainerInput{NetworkMode: "bridge", RunnerOptions: option}}
_, _, err = cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{}))
}
// a bare name is no longer resolved from the runner's environment, for either source
logger := logging.New(logtest.New())
cr := &containerReference{input: &NewContainerInput{
NetworkMode: "bridge",
@@ -673,8 +646,8 @@ func TestMergeContainerConfigsKeepsTheRunnersFilesAndEnvToItself(t *testing.T) {
WorkflowOptions: "--env RUNNER_SECRET --env GIVEN=value",
}}
config, _, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
config := &container.Config{}
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), config, &container.HostConfig{}))
assert.Equal(t, []string{"RUNNER_SECRET", "RUNNER_SECRET", "GIVEN=value"}, config.Env)
}
@@ -711,7 +684,6 @@ func TestSanitizeOptionsHostConfig(t *testing.T) {
assert.Equal(t, &container.HostConfig{}, hostConfig)
}
// mergeOptions merges both option sources into a bare container, returning the result and its log.
func mergeOptions(t *testing.T, runnerOptions, workflowOptions string, privileged bool) (*container.HostConfig, *logtest.Handler) {
t.Helper()
logger, hook := logtest.NewLogger()
@@ -723,12 +695,12 @@ func mergeOptions(t *testing.T, runnerOptions, workflowOptions string, privilege
ValidVolumes: []string{"**"},
}}
_, hostConfig, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{
hostConfig := &container.HostConfig{
Privileged: privileged,
UsernsMode: container.UsernsMode("private"),
NetworkMode: container.NetworkMode("bridge"),
})
require.NoError(t, err)
}
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, hostConfig))
return hostConfig, hook
}
@@ -782,23 +754,26 @@ func TestCheckVolumesRejectsEscapingHostPaths(t *testing.T) {
}
escapingPath := allowed + string(filepath.Separator) + ".." + string(filepath.Separator) + "denied"
_, hostConf := cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{
hostConf := &container.HostConfig{
Binds: []string{escapingPath + ":/mnt"},
})
}
cr.sanitizeConfig(ctx, hostConf)
assert.Empty(t, hostConf.Binds)
linkPath := filepath.Join(allowed, "link")
if err := os.Symlink(denied, linkPath); err != nil {
t.Skipf("cannot create symlink: %v", err)
}
_, hostConf = cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{
hostConf = &container.HostConfig{
Binds: []string{linkPath + ":/mnt"},
})
}
cr.sanitizeConfig(ctx, hostConf)
assert.Empty(t, hostConf.Binds)
_, hostConf = cr.sanitizeConfig(ctx, &container.Config{}, &container.HostConfig{
hostConf = &container.HostConfig{
Binds: []string{filepath.Join(linkPath, "missing") + ":/mnt"},
})
}
cr.sanitizeConfig(ctx, hostConf)
assert.Empty(t, hostConf.Binds)
}
@@ -808,11 +783,12 @@ func TestSanitizeConfigKeepsTmpfsMounts(t *testing.T) {
for _, validVolumes := range [][]string{nil, {"data"}, {"**:ro"}} {
logger, hook := logtest.NewLogger()
_, sanitized := (&containerReference{input: &NewContainerInput{ValidVolumes: validVolumes}}).sanitizeConfig(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{Mounts: []mount.Mount{
sanitized := &container.HostConfig{Mounts: []mount.Mount{
hostConfig.Mounts[0],
{Type: mount.TypeVolume, Target: "/anonymous"},
{Type: mount.TypeTmpfs, Source: "/etc", Target: "/etc-copy"},
}})
}}
(&containerReference{input: &NewContainerInput{ValidVolumes: validVolumes}}).sanitizeConfig(common.WithLogger(t.Context(), logger), sanitized)
assert.Equal(t, hostConfig.Mounts, sanitized.Mounts)
assert.Len(t, hook.AllRecords(), 2)
}
@@ -908,11 +884,11 @@ func TestMergeContainerConfigsVolumesReplaceRunnerMounts(t *testing.T) {
},
}
_, hostConf, err := cr.mergeContainerConfigs(ctx, &container.Config{}, &container.HostConfig{
hostConf := &container.HostConfig{
Binds: []string{"/var/run/docker.sock:/var/run/docker.sock"},
Mounts: []mount.Mount{{Type: mount.TypeVolume, Source: "act-toolcache", Target: "/opt/hostedtoolcache"}},
})
require.NoError(t, err)
}
require.NoError(t, cr.mergeContainerConfigs(ctx, &container.Config{}, hostConf))
assert.Equal(t, []string{"/var/run/docker.sock:/var/run/docker.sock", "/host/tools:/opt/hostedtoolcache"}, hostConf.Binds)
assert.Empty(t, hostConf.Mounts)
}
@@ -964,8 +940,7 @@ func TestMergeContainerConfigsKeepsNetworkAliasesFromOptions(t *testing.T) {
NetworkMode: "job-network", NetworkAliases: []string{"redis"}, WorkflowOptions: "--network-alias redis-primary",
}}
_, _, err := cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{})
require.NoError(t, err)
require.NoError(t, cr.mergeContainerConfigs(common.WithLogger(t.Context(), logger), &container.Config{}, &container.HostConfig{}))
assert.Equal(t, []string{"redis", "redis-primary"}, cr.input.NetworkAliases)
}
+27 -57
View File
@@ -7,6 +7,7 @@ package container
import (
"archive/tar"
"bytes"
"cmp"
"context"
"errors"
"fmt"
@@ -64,15 +65,11 @@ func (e *HostEnvironment) processGroup(ctx context.Context) *process.Group {
}
func (e *HostEnvironment) Create(_, _ []string) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func (e *HostEnvironment) Close() common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func (e *HostEnvironment) Copy(destPath string, files ...*FileEntry) common.Executor {
@@ -168,15 +165,11 @@ func (e *HostEnvironment) GetContainerArchive(ctx context.Context, srcPath strin
}
func (e *HostEnvironment) Pull(_ bool) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func (e *HostEnvironment) Start(_ bool) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
type ptyWriter struct {
@@ -193,7 +186,6 @@ func (w *ptyWriter) Write(buf []byte) (int, error) {
}
if w.dirtyLine || len(buf) > 1 && buf[len(buf)-2] != '\n' {
_, _ = w.Out.Write([]byte("\n"))
return n, io.EOF
}
return n, io.EOF
}
@@ -242,40 +234,25 @@ func writeKeepAlive(ppty io.Writer) {
}
}
func copyPtyOutput(writer io.Writer, ppty io.Reader, finishLog context.CancelFunc) {
defer func() {
finishLog()
}()
if _, err := io.Copy(writer, ppty); err != nil {
return
}
}
func (e *HostEnvironment) UpdateFromImageEnv(_ *map[string]string) common.Executor {
return func(ctx context.Context) error {
return nil
}
return common.NewPipelineExecutor()
}
func getEnvListFromMap(env map[string]string) []string {
envList := make([]string, 0)
for k, v := range env {
envList = append(envList, fmt.Sprintf("%s=%s", k, v))
envList := make([]string, 0, len(env))
for key, value := range env {
envList = append(envList, key+"="+value)
}
return envList
}
func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline string, env map[string]string, _, workdir string) error {
envList := getEnvListFromMap(env)
var wd string
func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline string, env map[string]string, workdir string) error {
wd := e.Path
if workdir != "" {
if filepath.IsAbs(workdir) {
wd = workdir
} else {
wd = workdir
if !filepath.IsAbs(workdir) {
wd = filepath.Join(e.Path, workdir)
}
} else {
wd = e.Path
}
// Flush any buffered, not-yet-newline-terminated trailing line, as the docker backend
// does in waitForCommand, so the final line of a command's output is not lost.
@@ -288,9 +265,8 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
cmd := exec.CommandContext(ctx, f)
cmd.Path = f
cmd.Args = command
cmd.Stdin = nil
cmd.Stdout = e.StdOut
cmd.Env = envList
cmd.Env = getEnvListFromMap(env)
cmd.Stderr = e.StdOut
cmd.Dir = wd
cmd.SysProcAttr = process.SysProcAttr(cmdline, false)
@@ -322,7 +298,10 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
writer = &ptyWriter{Out: e.StdOut}
var finishLog context.CancelFunc
logctx, finishLog = context.WithCancel(context.Background())
go copyPtyOutput(writer, ppty, finishLog)
go func() {
defer finishLog()
_, _ = io.Copy(writer, ppty)
}()
go writeKeepAlive(ppty)
}
if err := cmd.Start(); err != nil {
@@ -357,13 +336,13 @@ func (e *HostEnvironment) exec(ctx context.Context, command []string, cmdline st
return err
}
func (e *HostEnvironment) Exec(command []string /*cmdline string, */, env map[string]string, user, workdir string) common.Executor {
return e.ExecWithCmdLine(command, "", env, user, workdir)
func (e *HostEnvironment) Exec(command []string, env map[string]string, _user, workdir string) common.Executor {
return e.ExecWithCmdLine(command, "", env, workdir)
}
func (e *HostEnvironment) ExecWithCmdLine(command []string, cmdline string, env map[string]string, user, workdir string) common.Executor {
func (e *HostEnvironment) ExecWithCmdLine(command []string, cmdline string, env map[string]string, workdir string) common.Executor {
return func(ctx context.Context) error {
if err := e.exec(ctx, command, cmdline, env, user, workdir); err != nil {
if err := e.exec(ctx, command, cmdline, env, workdir); err != nil {
select {
case <-ctx.Done():
return fmt.Errorf("this step has been cancelled: %w", err)
@@ -605,8 +584,7 @@ func (*HostEnvironment) GetPathVariableName() string {
}
func (e *HostEnvironment) DefaultPathVariable() string {
v, _ := os.LookupEnv(e.GetPathVariableName())
return v
return os.Getenv(e.GetPathVariableName())
}
func (*HostEnvironment) JoinPathVariable(paths ...string) string {
@@ -616,30 +594,22 @@ func (*HostEnvironment) JoinPathVariable(paths ...string) string {
// Reference for Arch values for runner.arch
// https://docs.github.com/en/actions/learn-github-actions/contexts#runner-context
func goArchToActionArch(arch string) string {
archMapper := map[string]string{
return cmp.Or(map[string]string{
"x86_64": "X64",
"amd64": "X64",
"386": "X86",
"arm": "ARM",
"aarch64": "ARM64",
"arm64": "ARM64",
}
if arch, ok := archMapper[arch]; ok {
return arch
}
return arch
}[arch], arch)
}
func goOsToActionOs(os string) string {
osMapper := map[string]string{
return cmp.Or(map[string]string{
"linux": "Linux",
"darwin": "macOS",
"windows": "Windows",
}
if os, ok := osMapper[os]; ok {
return os
}
return os
}[os], os)
}
func (e *HostEnvironment) GetRunnerContext(_ context.Context) map[string]any {
+62 -182
View File
@@ -10,7 +10,6 @@ import (
"context"
"io"
"os"
"path"
"path/filepath"
"runtime"
"strings"
@@ -24,7 +23,6 @@ import (
"github.com/stretchr/testify/require"
)
// Type assert HostEnvironment implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &HostEnvironment{}
func TestActionPlatformNames(t *testing.T) {
@@ -61,32 +59,20 @@ func TestCopyDir(t *testing.T) {
}
func TestGetContainerArchive(t *testing.T) {
dir := t.TempDir()
ctx := t.Context()
e := &HostEnvironment{
Path: filepath.Join(dir, "path"),
TmpDir: filepath.Join(dir, "tmp"),
ToolCache: filepath.Join(dir, "tool_cache"),
ActPath: filepath.Join(dir, "act_path"),
StdOut: os.Stdout,
Workdir: path.Join("testdata", "scratch"),
}
_ = os.MkdirAll(e.Path, 0o700)
_ = os.MkdirAll(e.TmpDir, 0o700)
_ = os.MkdirAll(e.ToolCache, 0o700)
_ = os.MkdirAll(e.ActPath, 0o700)
e := &HostEnvironment{Path: t.TempDir()}
expectedContent := []byte("sdde/7sh")
err := os.WriteFile(filepath.Join(e.Path, "action.yml"), expectedContent, 0o600)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
archive, err := e.GetContainerArchive(ctx, e.Path)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
defer archive.Close()
reader := tar.NewReader(archive)
h, err := reader.Next()
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(t, "action.yml", h.Name)
content, err := io.ReadAll(reader)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(t, expectedContent, content)
_, err = reader.Next()
assert.ErrorIs(t, err, io.EOF)
@@ -96,19 +82,8 @@ func TestHostEnvironmentExecExitCode(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("uses POSIX shell")
}
dir := t.TempDir()
ctx := t.Context()
e := &HostEnvironment{
Path: filepath.Join(dir, "path"),
TmpDir: filepath.Join(dir, "tmp"),
ToolCache: filepath.Join(dir, "tool_cache"),
ActPath: filepath.Join(dir, "act_path"),
StdOut: io.Discard,
Workdir: filepath.Join(dir, "path"),
}
for _, p := range []string{e.Path, e.TmpDir, e.ToolCache, e.ActPath} {
assert.NoError(t, os.MkdirAll(p, 0o700)) //nolint:testifylint // test setup
}
e := &HostEnvironment{Path: t.TempDir(), StdOut: io.Discard}
err := e.Exec([]string{"sh", "-c", "exit 3"}, map[string]string{"PATH": os.Getenv("PATH")}, "", "")(ctx)
var exitErr ExitCodeError
@@ -130,20 +105,12 @@ func TestHostEnvironmentAllocatePTY(t *testing.T) {
{name: "on", allocPTY: true, expect: "TTY"},
} {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
buf := &bytes.Buffer{}
e := &HostEnvironment{
Path: filepath.Join(dir, "path"),
TmpDir: filepath.Join(dir, "tmp"),
ToolCache: filepath.Join(dir, "tool_cache"),
ActPath: filepath.Join(dir, "act_path"),
Path: t.TempDir(),
StdOut: buf,
Workdir: filepath.Join(dir, "path"),
AllocatePTY: tc.allocPTY,
}
for _, p := range []string{e.Path, e.TmpDir, e.ToolCache, e.ActPath} {
require.NoError(t, os.MkdirAll(p, 0o700))
}
err := e.Exec(
[]string{"sh", "-c", "[ -t 1 ] && printf TTY || printf NOTTY"},
@@ -156,49 +123,26 @@ func TestHostEnvironmentAllocatePTY(t *testing.T) {
}
}
func TestHostEnvironmentRemovePreservesWorkdirByDefault(t *testing.T) {
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
miscRoot := filepath.Join(base, "misc")
path := filepath.Join(miscRoot, "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
workdir := filepath.Join(base, "workspace", "owner", "repo")
require.NoError(t, os.MkdirAll(workdir, 0o700))
e := &HostEnvironment{
Path: path,
Workdir: workdir,
CleanUp: func() {
_ = os.RemoveAll(miscRoot)
},
StdOut: os.Stdout,
func TestHostEnvironmentRemoveWorkdir(t *testing.T) {
for name, owned := range map[string]bool{"preserves caller workdir": false, "cleans owned workdir": true} {
t.Run(name, func(t *testing.T) {
miscRoot, workdir := t.TempDir(), t.TempDir()
e := &HostEnvironment{
Path: filepath.Join(miscRoot, "hostexecutor"),
Workdir: workdir,
CleanWorkdir: owned,
CleanUp: func() { _ = os.RemoveAll(miscRoot) },
}
require.NoError(t, os.MkdirAll(e.Path, 0o700))
require.NoError(t, e.Remove()(t.Context()))
_, err := os.Stat(workdir)
if owned {
assert.ErrorIs(t, err, os.ErrNotExist)
} else {
require.NoError(t, err)
}
})
}
require.NoError(t, e.Remove()(ctx))
_, err := os.Stat(workdir)
require.NoError(t, err)
}
func TestHostEnvironmentRemoveCleansWorkdirWhenOwned(t *testing.T) {
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
miscRoot := filepath.Join(base, "misc")
path := filepath.Join(miscRoot, "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
workdir := filepath.Join(base, "workspace", "123", "owner", "repo")
require.NoError(t, os.MkdirAll(workdir, 0o700))
e := &HostEnvironment{
Path: path,
Workdir: workdir,
CleanWorkdir: true,
CleanUp: func() {
_ = os.RemoveAll(miscRoot)
},
StdOut: os.Stdout,
}
require.NoError(t, e.Remove()(ctx))
_, err := os.Stat(workdir)
assert.ErrorIs(t, err, os.ErrNotExist)
}
func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) {
@@ -211,9 +155,6 @@ func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) {
<-release
return nil
}
// removeAllWithContext intentionally leaks the delete goroutine on timeout,
// and that goroutine still references removeAll. Unblock it and wait for it
// to return before restoring the var, so the restore can't race the read.
t.Cleanup(func() {
close(release)
<-stubDone
@@ -227,87 +168,42 @@ func TestRemoveAllWithContextDoesNotHangOnStuckDelete(t *testing.T) {
require.ErrorIs(t, err, context.DeadlineExceeded)
}
// TestHostEnvironmentRemoveDoesNotHangOnStuckCleanUp guards against a stalled
// CleanUp callback (e.g. an os.RemoveAll blocked by an AV/EDR filter driver or
// an unresponsive mount) wedging the runner slot forever at "Cleaning up
// container". Remove must time out the callback and complete job teardown.
func TestHostEnvironmentRemoveDoesNotHangOnStuckCleanUp(t *testing.T) {
// Keep the suite fast: shrink the per-phase teardown timeout for this test.
orig := hostCleanupTimeout
hostCleanupTimeout = 100 * time.Millisecond
t.Cleanup(func() { hostCleanupTimeout = orig })
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
path := filepath.Join(base, "misc", "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
release := make(chan struct{})
t.Cleanup(func() { close(release) }) // unblock the leaked goroutine at test end
e := &HostEnvironment{
Path: path,
CleanUp: func() {
<-release // simulate a delete syscall stuck indefinitely
},
StdOut: os.Stdout,
}
done := make(chan error, 1)
go func() { done <- e.Remove()(ctx) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatal("Remove() hung on a stuck CleanUp callback")
}
}
// TestHostEnvironmentRemoveDoesNotHangOnStuckPathRemoval guards against a
// stalled os.RemoveAll on the misc/workspace paths (same AV/EDR wedge as
// #1023) wedging job completion after the CleanUp callback has already timed
// out or finished.
func TestHostEnvironmentRemoveDoesNotHangOnStuckPathRemoval(t *testing.T) {
origTimeout := hostCleanupTimeout
hostCleanupTimeout = 100 * time.Millisecond
t.Cleanup(func() { hostCleanupTimeout = origTimeout })
release := make(chan struct{})
stubDone := make(chan struct{})
origRemoveAll := removeAll
removeAll = func(string) error {
defer close(stubDone)
<-release
return nil
}
// The stuck delete goroutine outlives the timed-out Remove and still reads
// removeAll; unblock it and wait before restoring to avoid a restore/read race.
t.Cleanup(func() {
close(release)
<-stubDone
removeAll = origRemoveAll
})
ctx := common.WithLogger(t.Context(), logging.Default())
base := t.TempDir()
path := filepath.Join(base, "misc", "hostexecutor")
require.NoError(t, os.MkdirAll(path, 0o700))
e := &HostEnvironment{
Path: path,
StdOut: os.Stdout,
}
done := make(chan error, 1)
go func() { done <- e.Remove()(ctx) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatal("Remove() hung on a stuck path removal")
func TestHostEnvironmentRemoveDoesNotHang(t *testing.T) {
for _, phase := range []string{"CleanUp callback", "path removal"} {
t.Run(phase, func(t *testing.T) {
origTimeout := hostCleanupTimeout
hostCleanupTimeout = 100 * time.Millisecond
t.Cleanup(func() { hostCleanupTimeout = origTimeout })
release, stubDone := make(chan struct{}), make(chan struct{})
block := func() {
defer close(stubDone)
<-release
}
e := &HostEnvironment{Path: filepath.Join(t.TempDir(), "misc", "hostexecutor"), StdOut: os.Stdout}
if phase == "CleanUp callback" {
e.CleanUp = block
} else {
origRemoveAll := removeAll
removeAll = func(string) error {
block()
return nil
}
t.Cleanup(func() { removeAll = origRemoveAll })
}
t.Cleanup(func() {
close(release)
<-stubDone
})
require.NoError(t, os.MkdirAll(e.Path, 0o700))
done := make(chan error, 1)
go func() { done <- e.Remove()(common.WithLogger(t.Context(), logging.Default())) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatalf("Remove() hung on a stuck %s", phase)
}
})
}
}
@@ -315,17 +211,11 @@ func TestBuildWindowsWorkspaceKillScript(t *testing.T) {
t.Run("single dir", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript([]string{`C:\workspace\job1`})
assert.Contains(t, s, `$paths = @('C:\workspace\job1')`)
// Self-PID guard is essential — without it the script could taskkill
// the PowerShell process running it.
assert.Contains(t, s, "$selfPid = $PID")
assert.Contains(t, s, "$_.ProcessId -eq $selfPid")
// Must match both ExecutablePath (binaries from the workspace) and
// CommandLine (system binaries invoked with workspace paths in args),
// both bounded by dir+separator so a name-prefix sibling is spared.
assert.Contains(t, s, `$prefix = $p + '\'`)
assert.Contains(t, s, "$_.ExecutablePath.StartsWith($prefix")
assert.Contains(t, s, "$_.CommandLine.IndexOf($prefix")
// Each matched PID must be tree-killed, not just stopped.
assert.Contains(t, s, "taskkill.exe /PID $_.ProcessId /T /F")
})
@@ -338,25 +228,16 @@ func TestBuildWindowsWorkspaceKillScript(t *testing.T) {
assert.Contains(t, s, `'C:\work\path'`)
assert.Contains(t, s, `'C:\work\workdir'`)
assert.Contains(t, s, `'C:\Users\runner\AppData\Local\Temp\job-42'`)
// Commas between entries — no trailing comma, no leading comma.
assert.Contains(t, s, `'C:\work\path','C:\work\workdir',`)
})
t.Run("path with single quote is escaped", func(t *testing.T) {
// In PowerShell single-quoted strings the only special char is the
// quote itself, escaped by doubling. A workspace path that ever
// contained `'` would inject a command into the script otherwise.
s := buildWindowsWorkspaceKillScript([]string{`C:\work\it's\path`})
assert.Contains(t, s, `'C:\work\it''s\path'`)
// And it must NOT appear unescaped — otherwise the quote would
// terminate the literal early.
assert.NotContains(t, s, `'C:\work\it's\path'`)
})
t.Run("path with wildcard metacharacters is matched literally", func(t *testing.T) {
// A path containing [ ] ? * must be embedded verbatim and matched with
// ordinal String methods, not -like, otherwise the metacharacters would
// be interpreted as wildcards and the leftover process could escape.
s := buildWindowsWorkspaceKillScript([]string{`C:\work\[job]?1`})
assert.Contains(t, s, `'C:\work\[job]?1'`)
assert.NotContains(t, s, "-like")
@@ -366,7 +247,6 @@ func TestBuildWindowsWorkspaceKillScript(t *testing.T) {
t.Run("empty dir list still produces a valid script", func(t *testing.T) {
s := buildWindowsWorkspaceKillScript(nil)
// Empty array literal — script runs, matches nothing, is a no-op.
assert.Contains(t, s, "$paths = @()")
assert.Contains(t, s, "Get-CimInstance Win32_Process")
})
@@ -16,6 +16,8 @@ import (
type LinuxContainerEnvironmentExtensions struct{}
var windowsPathRegex = regexp.MustCompile(`^([a-zA-Z]):\\(.+)$`)
// Resolves the equivalent host path inside the container
// This is required for windows and WSL 2 to translate things like C:\Users\Myproject to /mnt/users/Myproject
// For use in docker volumes and binds
@@ -31,8 +33,6 @@ func (*LinuxContainerEnvironmentExtensions) ToContainerPath(path string) string
return ""
}
// Test if the path is a windows path
windowsPathRegex := regexp.MustCompile(`^([a-zA-Z]):\\(.+)$`)
windowsPathComponents := windowsPathRegex.FindStringSubmatch(abspath)
// Return as-is if no match
@@ -43,11 +43,7 @@ func (*LinuxContainerEnvironmentExtensions) ToContainerPath(path string) string
// Convert to WSL2-compatible path if it is a windows path
// NOTE: Cannot use filepath because it will use the wrong path separators assuming we want the path to be windows
// based if running on Windows, and because we are feeding this to Docker, GoLang auto-path-translate doesn't work.
driveLetter := strings.ToLower(windowsPathComponents[1])
translatedPath := strings.ReplaceAll(windowsPathComponents[2], `\`, `/`)
// Should make something like /mnt/c/Users/person/My Folder/MyActProject
result := strings.Join([]string{"/mnt", driveLetter, translatedPath}, `/`)
return result
return "/mnt/" + strings.ToLower(windowsPathComponents[1]) + "/" + strings.ReplaceAll(windowsPathComponents[2], `\`, `/`)
}
func (*LinuxContainerEnvironmentExtensions) GetActPath() string {
@@ -69,10 +65,41 @@ func (*LinuxContainerEnvironmentExtensions) JoinPathVariable(paths ...string) st
// DefaultToolCache is where the runner mounts the tool cache inside job containers.
const DefaultToolCache = "/opt/hostedtoolcache"
type runnerArchKey struct{}
func WithRunnerArch(ctx context.Context) context.Context { // shares one architecture lookup per run
if ctx.Value(runnerArchKey{}) != nil {
return ctx
}
return context.WithValue(ctx, runnerArchKey{}, &runnerArchCache{lock: make(chan struct{}, 1)})
}
type runnerArchCache struct {
lock chan struct{} // lets cancelled jobs stop waiting for the cache
arch string
}
func cachedRunnerArch(ctx context.Context) string {
cache, ok := ctx.Value(runnerArchKey{}).(*runnerArchCache)
if !ok {
return RunnerArch(ctx)
}
select {
case cache.lock <- struct{}{}:
defer func() { <-cache.lock }()
case <-ctx.Done():
return ""
}
if cache.arch == "" {
cache.arch = RunnerArch(ctx)
}
return cache.arch
}
func (*LinuxContainerEnvironmentExtensions) GetRunnerContext(ctx context.Context) map[string]any {
return map[string]any{
"os": "Linux",
"arch": RunnerArch(ctx),
"arch": cachedRunnerArch(ctx),
"temp": "/tmp",
"tool_cache": DefaultToolCache,
}
@@ -5,15 +5,13 @@
package container
import (
"fmt"
"os"
"runtime"
"strings"
"testing"
"gitea.com/gitea/runner/act/common/logging"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestContainerPath(t *testing.T) {
@@ -23,42 +21,27 @@ func TestContainerPath(t *testing.T) {
workDir string
}
linuxcontainerext := &LinuxContainerEnvironmentExtensions{}
cwd, err := os.Getwd()
require.NoError(t, err)
tests := []containerPathJob{
{"/home/act/go/src/github.com/nektos/act", "/home/act/go/src/github.com/nektos/act", ""},
{"/home/act", `/home/act/`, ""},
{cwd, ".", ""},
}
if runtime.GOOS == "windows" {
cwd, err := os.Getwd()
if err != nil {
logging.Error(err)
}
rootDrive := os.Getenv("SystemDrive")
rootDriveLetter := strings.ReplaceAll(strings.ToLower(rootDrive), `:`, "")
for _, v := range []containerPathJob{
tests = []containerPathJob{
{"/mnt/c/Users/act/go/src/github.com/nektos/act", "C:\\Users\\act\\go\\src\\github.com\\nektos\\act\\", ""},
{"/mnt/f/work/dir", `F:\work\dir`, ""},
{"/mnt/c/windows/to/unix", "windows\\to\\unix", rootDrive + "\\"},
{fmt.Sprintf("/mnt/%v/act", rootDriveLetter), "act", rootDrive + "\\"},
} {
if v.workDir != "" {
t.Chdir(v.workDir)
}
assert.Equal(t, v.destinationPath, linuxcontainerext.ToContainerPath(v.sourcePath))
{"/mnt/" + strings.ReplaceAll(strings.ToLower(rootDrive), `:`, "") + "/act", "act", rootDrive + "\\"},
}
t.Chdir(cwd)
} else {
cwd, err := os.Getwd()
if err != nil {
logging.Error(err)
}
for _, v := range []containerPathJob{
{"/home/act/go/src/github.com/nektos/act", "/home/act/go/src/github.com/nektos/act", ""},
{"/home/act", `/home/act/`, ""},
{cwd, ".", ""},
} {
assert.Equal(t, v.destinationPath, linuxcontainerext.ToContainerPath(v.sourcePath))
}
for _, test := range tests {
if test.workDir != "" {
t.Chdir(test.workDir)
}
assert.Equal(t, test.destinationPath, (&LinuxContainerEnvironmentExtensions{}).ToContainerPath(test.sourcePath))
}
}
@@ -67,5 +50,4 @@ type typeAssertMockContainer struct {
LinuxContainerEnvironmentExtensions
}
// Type assert Container + LinuxContainerEnvironmentExtensions implements ExecutionsEnvironment
var _ ExecutionsEnvironment = &typeAssertMockContainer{}
-1
View File
@@ -75,7 +75,6 @@ func parseEnvFile(e Container, srcPath string, env *map[string]string) common.Ex
if err := s.Err(); err != nil {
return fmt.Errorf("reading env file: %w", err)
}
env = &localEnv
return nil
}
}
+30 -95
View File
@@ -12,7 +12,6 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/text/encoding"
"golang.org/x/text/encoding/unicode"
)
@@ -22,42 +21,40 @@ func newTestHostEnv(t *testing.T) (*HostEnvironment, string) {
return e, filepath.Join(e.Path, "envfile")
}
func TestParseEnvFileSingleLine(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO=bar\nBAZ=qux\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
assert.Equal(t, "qux", env["BAZ"])
}
func TestParseEnvFileMultiLine(t *testing.T) {
e, envPath := newTestHostEnv(t)
content := "FOO<<EOF\nline1\nline2\nEOF\n"
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "line1\nline2", env["FOO"])
}
func TestParseEnvFileLargeValueWithinLimit(t *testing.T) {
e, envPath := newTestHostEnv(t)
big := strings.Repeat("x", 2*1024*1024)
content := "FOO<<EOF\n" + big + "\nEOF\n"
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, big, env["FOO"])
func TestParseEnvFile(t *testing.T) {
largeValue := strings.Repeat("x", 2*1024*1024)
encodeUTF16 := func(endianness unicode.Endianness) string {
content, err := unicode.UTF16(endianness, unicode.UseBOM).NewEncoder().Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n"))
require.NoError(t, err)
return string(content)
}
for _, test := range []struct {
name, content string
want map[string]string
}{
{"single line", "FOO=bar\nBAZ=qux\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"blank lines", "\nFOO=bar\n\n \nBAZ=qux\n\n", map[string]string{"FOO": "bar", "BAZ": "qux"}},
{"multiline", "FOO<<EOF\nline1\nline2\nEOF\n", map[string]string{"FOO": "line1\nline2"}},
{"multiline blank lines", "FOO<<EOF\nline1\n\nline2\nEOF\n", map[string]string{"FOO": "line1\n\nline2"}},
{"multiline leading blank line", "FOO<<EOF\n\nline2\nEOF\n", map[string]string{"FOO": "\nline2"}},
{"large value within limit", "FOO<<EOF\n" + largeValue + "\nEOF\n", map[string]string{"FOO": largeValue}},
{"UTF8 BOM", "\xef\xbb\xbfFOO=bar\n", map[string]string{"FOO": "bar"}},
{"UTF16 little endian", encodeUTF16(unicode.LittleEndian), map[string]string{"FOO": "bar", "MULTI": "line1"}},
{"UTF16 big endian", encodeUTF16(unicode.BigEndian), map[string]string{"FOO": "bar", "MULTI": "line1"}},
} {
t.Run(test.name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte(test.content), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, test.want, env)
})
}
}
func TestParseEnvFileLineExceedsBufferReportsScannerError(t *testing.T) {
e, envPath := newTestHostEnv(t)
tooBig := strings.Repeat("x", 17*1024*1024) // over the 16 MiB cap
content := "FOO<<EOF\n" + tooBig + "\nEOF\n"
require.NoError(t, os.WriteFile(envPath, []byte(content), 0o600))
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n"+strings.Repeat("x", 17*1024*1024)+"\nEOF\n"), 0o600))
env := map[string]string{}
err := parseEnvFile(e, envPath, &env)(t.Context())
@@ -65,68 +62,6 @@ func TestParseEnvFileLineExceedsBufferReportsScannerError(t *testing.T) {
assert.Contains(t, err.Error(), "reading env file")
}
// Regression test: a blank line used to fail the job at "Complete Job", after
// every step had already been recorded as successful.
func TestParseEnvFileBlankLines(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("\nFOO=bar\n\n \nBAZ=qux\n\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
assert.Equal(t, "qux", env["BAZ"])
}
// blank lines inside a heredoc value are content, not separators
func TestParseEnvFileMultiLineKeepsBlankLines(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\nline1\n\nline2\nEOF\n"), 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "line1\n\nline2", env["FOO"])
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\n\nline2\nEOF\n"), 0o600))
env = map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "\nline2", env["FOO"])
}
func TestParseEnvFileUTF8BOM(t *testing.T) {
e, envPath := newTestHostEnv(t)
content := append([]byte{0xEF, 0xBB, 0xBF}, []byte("FOO=bar\n")...)
require.NoError(t, os.WriteFile(envPath, content, 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
}
// Windows host mode: PowerShell 5.1 redirection writes UTF-16, which used to be
// unrecognisable as KEY=VALUE, so the writes were silently ignored.
func TestParseEnvFileUTF16(t *testing.T) {
tests := []struct {
name string
encoder *encoding.Encoder
}{
{"little endian", unicode.UTF16(unicode.LittleEndian, unicode.UseBOM).NewEncoder()},
{"big endian", unicode.UTF16(unicode.BigEndian, unicode.UseBOM).NewEncoder()},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
e, envPath := newTestHostEnv(t)
content, err := tt.encoder.Bytes([]byte("FOO=bar\r\nMULTI<<EOF\r\nline1\r\nEOF\r\n"))
require.NoError(t, err)
require.NoError(t, os.WriteFile(envPath, content, 0o600))
env := map[string]string{}
require.NoError(t, parseEnvFile(e, envPath, &env)(t.Context()))
assert.Equal(t, "bar", env["FOO"])
assert.Equal(t, "line1", env["MULTI"])
})
}
}
func TestParseEnvFileMissingDelimiter(t *testing.T) {
e, envPath := newTestHostEnv(t)
require.NoError(t, os.WriteFile(envPath, []byte("FOO<<EOF\nline1\nline2\n"), 0o600))
-1
View File
@@ -1 +0,0 @@
testfile
+4 -9
View File
@@ -43,7 +43,6 @@ func (tc TarCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string,
// update the name to correctly reflect the desired destination when untaring
header.Name = path.Join(tc.DstDir, fpath)
header.Mode = int64(fi.Mode())
header.ModTime = fi.ModTime()
header.Uid = tc.UID
header.Gid = tc.GID
@@ -58,10 +57,8 @@ func (tc TarCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string,
}
// copy file data into tar writer
if _, err := io.Copy(tc.TarWriter, f); err != nil {
return err
}
return nil
_, err = io.Copy(tc.TarWriter, f)
return err
}
type CopyCollector struct {
@@ -87,10 +84,8 @@ func (cc *CopyCollector) WriteFile(fpath string, fi fs.FileInfo, linkName string
return err
}
defer df.Close()
if _, err := io.Copy(df, f); err != nil {
return err
}
return nil
_, err = io.Copy(df, f)
return err
}
type FileCollector struct {
+2 -3
View File
@@ -117,8 +117,7 @@ func SetSha(ctx context.Context, ghc *model.GithubContext, repoPath string) {
if ghc.Sha == "" {
sha, err := pinned(ctx, "sha:"+repoPath, func() (string, error) {
_, revision, err := findGitRevision(ctx, repoPath)
return revision, err
return findGitRevision(ctx, repoPath)
})
if err != nil {
logger.Warnf("unable to get git revision: %v", err)
@@ -139,5 +138,5 @@ func SetRepositoryAndOwner(ctx context.Context, ghc *model.GithubContext, repoPa
}
ghc.Repository = repo
}
ghc.RepositoryOwner = strings.Split(ghc.Repository, "/")[0]
ghc.RepositoryOwner, _, _ = strings.Cut(ghc.Repository, "/")
}
+5 -32
View File
@@ -9,28 +9,18 @@ import (
"errors"
"testing"
"gitea.com/gitea/runner/act/common/logging"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
)
func TestSetRef(t *testing.T) {
logging.SetLevel(logging.LevelDebug)
oldFindGitRef := findGitRef
oldFindGitRevision := findGitRevision
defer func() { findGitRef = oldFindGitRef }()
defer func() { findGitRevision = oldFindGitRevision }()
findGitRef = func(ctx context.Context, file string) (string, error) {
findGitRef = func(_ctx context.Context, _file string) (string, error) {
return "refs/heads/master", nil
}
findGitRevision = func(ctx context.Context, file string) (string, string, error) {
return "", "1234fakesha", nil
}
tables := []struct {
eventName string
event map[string]any
@@ -114,7 +104,7 @@ func TestSetRef(t *testing.T) {
}
t.Run("no-default-branch", func(t *testing.T) {
findGitRef = func(ctx context.Context, file string) (string, error) {
findGitRef = func(_ctx context.Context, _file string) (string, error) {
return "", errors.New("no default branch")
}
@@ -130,19 +120,13 @@ func TestSetRef(t *testing.T) {
}
func TestSetSha(t *testing.T) {
logging.SetLevel(logging.LevelDebug)
oldFindGitRef := findGitRef
oldFindGitRevision := findGitRevision
defer func() { findGitRef = oldFindGitRef }()
defer func() { findGitRevision = oldFindGitRevision }()
findGitRef = func(ctx context.Context, file string) (string, error) {
return "refs/heads/master", nil
}
findGitRevision = func(ctx context.Context, file string) (string, string, error) {
return "", "1234fakesha", nil
findGitRevision = func(_ctx context.Context, _file string) (string, error) {
return "1234fakesha", nil
}
tables := []struct {
@@ -177,11 +161,6 @@ func TestSetSha(t *testing.T) {
},
sha: "deployment-sha",
},
{
eventName: "release",
event: map[string]any{},
sha: "1234fakesha",
},
{
eventName: "push",
event: map[string]any{
@@ -195,18 +174,12 @@ func TestSetSha(t *testing.T) {
event: map[string]any{},
sha: "1234fakesha",
},
{
eventName: "no-event",
event: map[string]any{},
sha: "1234fakesha",
},
}
for _, table := range tables {
t.Run(table.eventName, func(t *testing.T) {
ghc := &model.GithubContext{
EventName: table.eventName,
BaseRef: "master",
Event: table.event,
}
@@ -219,7 +192,7 @@ func TestSetSha(t *testing.T) {
t.Run("pinned checkout keeps the first sha and ref", func(t *testing.T) {
ctx := WithPinnedCheckout(t.Context())
for _, head := range []string{"first", "moved"} {
findGitRevision = func(context.Context, string) (string, string, error) { return "", head, nil }
findGitRevision = func(context.Context, string) (string, error) { return head, nil }
findGitRef = func(context.Context, string) (string, error) { return "refs/heads/" + head, nil }
ghc := &model.GithubContext{Event: map[string]any{}}
SetSha(ctx, ghc, "/some/dir")
+11 -23
View File
@@ -13,35 +13,23 @@ import (
"testing"
)
func TestLookPath2SearchesPathAndEmptyElement(t *testing.T) {
func TestLookPath2SearchesPathAndEmptyElementOrUsesDirectPath(t *testing.T) {
dir := t.TempDir()
exe := filepath.Join(dir, "tool")
if err := os.WriteFile(exe, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
got, err := LookPath2("tool", map[string]string{"PATH": string(filepath.ListSeparator) + dir})
if err != nil {
t.Fatal(err)
}
if got != exe {
t.Fatalf("LookPath2() = %q, want %q", got, exe)
}
}
func TestLookPath2DirectPathDoesNotSearchPath(t *testing.T) {
dir := t.TempDir()
exe := filepath.Join(dir, "tool")
if err := os.WriteFile(exe, []byte("#!/bin/sh\n"), 0o755); err != nil {
t.Fatal(err)
}
got, err := LookPath2(exe, map[string]string{"PATH": ""})
if err != nil {
t.Fatal(err)
}
if got != exe {
t.Fatalf("LookPath2() = %q, want %q", got, exe)
for file, searchPath := range map[string]string{"tool": string(filepath.ListSeparator) + dir, exe: ""} {
t.Run(file, func(t *testing.T) {
got, err := LookPath2(file, map[string]string{"PATH": searchPath})
if err != nil {
t.Fatal(err)
}
if got != exe {
t.Fatalf("LookPath2() = %q, want %q", got, exe)
}
})
}
}
+3 -9
View File
@@ -28,21 +28,15 @@ func chkStat(file string) error {
}
func hasExt(file string) bool {
i := strings.LastIndex(file, ".")
if i < 0 {
return false
}
return strings.LastIndexAny(file, `:\/`) < i
return strings.LastIndex(file, ".") > strings.LastIndexAny(file, `:\/`)
}
func findExecutable(file string, exts []string) (string, error) {
if len(exts) == 0 {
return file, chkStat(file)
}
if hasExt(file) {
if chkStat(file) == nil {
return file, nil
}
if hasExt(file) && chkStat(file) == nil {
return file, nil
}
for _, e := range exts {
if f := file + e; chkStat(f) == nil {
+45 -67
View File
@@ -8,7 +8,6 @@ import (
"context"
"crypto/rand"
"crypto/sha256"
"embed"
"encoding/hex"
"errors"
"fmt"
@@ -26,6 +25,8 @@ import (
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/container"
_ "embed"
"gitea.dev/actionslib/pkg/model"
"github.com/kballard/go-shellquote"
)
@@ -47,11 +48,13 @@ type fileWriter func(filename string, data []byte, perm fs.FileMode) error
type runAction func(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor
//go:embed res/trampoline.js
var trampoline embed.FS
var trampoline string
var (
ContainerImageExistsLocally = container.ImageExistsLocally
ContainerNewDockerBuildExecutor = container.NewDockerBuildExecutor
dockerActionNamePattern = regexp.MustCompile("[^a-zA-Z0-9]")
inputEnvPattern = regexp.MustCompile("[^A-Z0-9-]")
)
func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath string, readFile actionYamlReader, writeFile fileWriter) (*model.Action, error) {
@@ -85,38 +88,31 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
logger.Debugf("Using synthetic action %v for Dockerfile", action)
return action, nil
}
if step.With != nil {
if val, ok := step.With["args"]; ok {
var b []byte
if b, err = trampoline.ReadFile("res/trampoline.js"); err != nil {
return nil, err
}
err2 := writeFile(filepath.Join(actionDir, actionPath, "trampoline.js"), b, 0o644)
if err2 != nil {
return nil, err2
}
action := &model.Action{
Name: "(Synthetic)",
Inputs: map[string]model.Input{
"cwd": {
Description: "(Actual working directory)",
Required: false,
Default: filepath.Join(actionDir, actionPath),
},
"command": {
Description: "(Actual program)",
Required: false,
Default: val,
},
},
Runs: model.ActionRuns{
Using: "node12",
Main: "trampoline.js",
},
}
logger.Debugf("Using synthetic action %v", action)
return action, nil
if val, ok := step.With["args"]; ok {
if err := writeFile(filepath.Join(actionDir, actionPath, "trampoline.js"), []byte(trampoline), 0o644); err != nil {
return nil, err
}
action := &model.Action{
Name: "(Synthetic)",
Inputs: map[string]model.Input{
"cwd": {
Description: "(Actual working directory)",
Required: false,
Default: filepath.Join(actionDir, actionPath),
},
"command": {
Description: "(Actual program)",
Required: false,
Default: val,
},
},
Runs: model.ActionRuns{
Using: "node12",
Main: "trampoline.js",
},
}
logger.Debugf("Using synthetic action %v", action)
return action, nil
}
}
}
@@ -125,8 +121,7 @@ func readActionImpl(ctx context.Context, step *model.Step, actionDir, actionPath
}
defer closer.Close()
action, err := model.ReadAction(reader)
return action, err
return model.ReadAction(reader)
}
func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actionPath, containerActionDir string) error {
@@ -153,7 +148,7 @@ func maybeCopyToActionDir(ctx context.Context, step actionStep, actionDir, actio
if !rc.Config.NoActionPatch {
// Other jobs share the directory, so patch under the copy's lock.
defer patchActions(ctx, actionScriptPaths(filepath.Join(actionDir, actionPath), step.getActionModel()))()
defer patchActions(ctx, actionScriptPaths(actionDir, filepath.Join(actionDir, actionPath), step.getActionModel()))()
}
return rc.JobContainer.CopyDir(containerActionDirCopy, actionDir+"/", false, true)(ctx) // an exported commit has no .git to apply gitignore with
@@ -166,7 +161,7 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
return func(ctx context.Context) error {
logger := common.Logger(ctx)
actionPath := ""
if remoteAction != nil && remoteAction.Path != "" {
if remoteAction != nil {
actionPath = remoteAction.Path
}
@@ -178,7 +173,6 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
return err
}
actionLocation := path.Join(actionDir, actionPath)
actionName, containerActionDir := getContainerActionPaths(step)
logger.Debugf("type=%v actionDir=%s actionPath=%s workdir=%s actionCacheDir=%s actionName=%s containerActionDir=%s", stepModel.Type(), actionDir, actionPath, rc.Config.Workdir, rc.ActionCacheDir(), actionName, containerActionDir)
@@ -196,7 +190,7 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
return rc.JobContainer.Exec(containerArgs, *step.getEnv(), "", "")(ctx)
case x.IsDocker():
location := actionLocation
location := path.Join(actionDir, actionPath)
if remoteAction == nil {
location = containerActionDir
}
@@ -215,12 +209,9 @@ func runActionImpl(step actionStep, actionDir string, remoteAction *remoteAction
rc.ApplyExtraPath(ctx, step.getEnv())
execFileName := action.Runs.Main + ".out"
buildArgs := []string{"go", "build", "-o", execFileName, action.Runs.Main}
execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor(
rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Main}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx)
default:
return fmt.Errorf("the runs.using key must be one of: %v, got %s", []string{
@@ -259,16 +250,14 @@ func dockerActionImageTag(repository, actionName string, localAction bool) strin
name = path.Join(repository, actionName)
}
// The human-readable name is sanitized by collapsing every non-alphanumeric character to "-".
sanitized := regexp.MustCompile("[^a-zA-Z0-9]").ReplaceAllString(name, "-")
sanitized := dockerActionNamePattern.ReplaceAllString(name, "-")
if localAction {
// For local actions a short hash of the raw repository and action path is appended so the tag stays unique per repository.
sum := sha256.Sum256([]byte(repository + "\x00" + actionName))
sanitized += "-" + hex.EncodeToString(sum[:])[:12]
}
// "-dockeraction" ensures that "./", "./test " won't get converted to "act-:latest", "act-test-:latest" which are invalid docker image names
image := fmt.Sprintf("%s-dockeraction:%s", sanitized, "latest")
image = "act-" + strings.TrimLeft(image, "-")
return strings.ToLower(image)
return strings.ToLower("act-" + strings.TrimLeft(sanitized+"-dockeraction:latest", "-"))
}
// One image per step and job: its stages share a build, and concurrent jobs never run each other's.
@@ -485,20 +474,15 @@ func newStepContainer(ctx context.Context, step step, image string, cmd, entrypo
}
func populateEnvsFromSavedState(env *map[string]string, step actionStep, rc *RunContext) {
state, ok := rc.IntraActionState[step.getStepModel().ID]
if ok {
for name, value := range state {
envName := "STATE_" + name
(*env)[envName] = value
}
for name, value := range rc.IntraActionState[step.getStepModel().ID] {
(*env)["STATE_"+name] = value
}
}
func populateEnvsFromInput(ctx context.Context, env *map[string]string, action *model.Action, rc *RunContext) error {
eval := rc.NewExpressionEvaluator(ctx)
for inputID, input := range action.Inputs {
envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_")
envKey = "INPUT_" + envKey
envKey := "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_")
if _, ok := (*env)[envKey]; !ok {
var err error
if (*env)[envKey], err = eval.Interpolate(ctx, input.Default); err != nil {
@@ -637,12 +621,9 @@ func runPreStep(step actionStep) common.Executor {
rc.ApplyExtraPath(ctx, step.getEnv())
execFileName := action.Runs.Pre + ".out"
buildArgs := []string{"go", "build", "-o", execFileName, action.Runs.Pre}
execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor(
rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Pre}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx)
default:
return nil
@@ -653,7 +634,7 @@ func runPreStep(step actionStep) common.Executor {
func shouldRunPostStep(step actionStep) common.Conditional {
return func(ctx context.Context) bool {
log := common.Logger(ctx)
stepResults := step.getRunContext().getStepsContext()
stepResults := step.getRunContext().StepResults
stepResult := stepResults[step.getStepModel().ID]
if stepResult == nil {
@@ -731,12 +712,9 @@ func runPostStep(step actionStep) common.Executor {
rc.ApplyExtraPath(ctx, step.getEnv())
execFileName := action.Runs.Post + ".out"
buildArgs := []string{"go", "build", "-o", execFileName, action.Runs.Post}
execArgs := []string{filepath.Join(containerActionDir, execFileName)}
return common.NewPipelineExecutor(
rc.JobContainer.Exec(buildArgs, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec(execArgs, *step.getEnv(), "", ""),
rc.JobContainer.Exec([]string{"go", "build", "-o", execFileName, action.Runs.Post}, *step.getEnv(), "", containerActionDir),
rc.JobContainer.Exec([]string{filepath.Join(containerActionDir, execFileName)}, *step.getEnv(), "", ""),
)(ctx)
default:
+9 -21
View File
@@ -8,7 +8,6 @@ import (
"context"
"errors"
"fmt"
"regexp"
"slices"
"strconv"
"strings"
@@ -32,8 +31,7 @@ func evaluateCompositeInputAndEnv(ctx context.Context, parent *RunContext, step
ee := parent.NewStepExpressionEvaluator(ctx, step)
for inputID, input := range step.getActionModel().Inputs {
envKey := regexp.MustCompile("[^A-Z0-9-]").ReplaceAllString(strings.ToUpper(inputID), "_")
envKey = "INPUT_" + strings.ToUpper(envKey)
envKey := "INPUT_" + inputEnvPattern.ReplaceAllString(strings.ToUpper(inputID), "_")
// lookup if key is defined in the step but the already
// evaluated value from the environment
@@ -211,19 +209,16 @@ func (rc *RunContext) compositeExecutor(action *model.Action) *compositeSteps {
}
stepID := step.getStepModel().ID
stepPre := rc.newCompositeCommandExecutor(step.pre())
preSteps = append(preSteps, newCompositeStepLogExecutor(stepPre, stepID))
preSteps = append(preSteps, newCompositeStepLogExecutor(rc.newCompositeCommandExecutor(step.pre()), stepID))
steps = append(steps, newCompositeStepLogExecutor(rc.newCompositeCommandExecutor(step.main()), stepID))
// run the post executor in reverse order
stepPost := rc.newCompositeCommandExecutor(step.post())
if postExecutor != nil {
stepPost := rc.newCompositeCommandExecutor(step.post())
postExecutor = newCompositeStepLogExecutor(stepPost.Finally(postExecutor), stepID)
} else {
stepPost := rc.newCompositeCommandExecutor(step.post())
postExecutor = newCompositeStepLogExecutor(stepPost, stepID)
stepPost = stepPost.Finally(postExecutor)
}
postExecutor = newCompositeStepLogExecutor(stepPost, stepID)
}
steps = append(steps, common.JobError)
@@ -241,14 +236,7 @@ func (rc *RunContext) compositeExecutor(action *model.Action) *compositeSteps {
func (rc *RunContext) newCompositeCommandExecutor(executor common.Executor) common.Executor {
return func(ctx context.Context) error {
ctx = WithCompositeLogger(ctx, &rc.Masks)
logWriter := rc.commandLogWriter(ctx)
oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter)
defer rc.JobContainer.ReplaceLogWriter(oldout, olderr)
return executor(ctx)
return rc.withCommandLogWriter(WithCompositeLogger(ctx, &rc.Masks), executor)
}
}
@@ -257,12 +245,12 @@ func newCompositeStepLogExecutor(runStep common.Executor, stepID string) common.
ctx = WithCompositeStepLogger(ctx, stepID)
logger := common.Logger(ctx)
err := runStep(ctx)
if err == nil {
err = ctx.Err()
}
if err != nil {
logger.Errorf("##[error]%s", EscapeCommandData(err.Error()))
common.SetJobError(ctx, err)
} else if ctx.Err() != nil {
logger.Errorf("##[error]%s", EscapeCommandData(ctx.Err().Error()))
common.SetJobError(ctx, ctx.Err())
}
return nil
}
+2 -28
View File
@@ -71,9 +71,9 @@ func TestAppendUniqueMasks(t *testing.T) {
want []string
}{
{
name: "appends new masks",
name: "appends each new mask once",
dst: []string{"a"},
src: []string{"b", "c"},
src: []string{"b", "c", "b", "a"},
want: []string{"a", "b", "c"},
},
{
@@ -82,12 +82,6 @@ func TestAppendUniqueMasks(t *testing.T) {
src: []string{"a", "b"},
want: []string{"a", "b"},
},
{
name: "deduplicates within src",
dst: []string{"a"},
src: []string{"b", "b", "a"},
want: []string{"a", "b"},
},
{
name: "empty src leaves dst unchanged",
dst: []string{"a"},
@@ -103,26 +97,6 @@ func TestAppendUniqueMasks(t *testing.T) {
}
}
// TestAppendUniqueMasksNoExponentialGrowth reproduces the exponential growth of
// the parent's Masks slice observed with nested/repeated composite actions. A
// composite RunContext is seeded with its parent's masks and the whole seeded
// slice was previously appended back into the parent, doubling its length on
// every composite action.
func TestAppendUniqueMasksNoExponentialGrowth(t *testing.T) {
parentMasks := []string{"secret"}
for range 20 {
// compositeRC.Masks starts as a copy of the parent's masks (it is
// seeded with parent.Masks in newCompositeRunContext).
compositeMasks := make([]string, len(parentMasks))
copy(compositeMasks, parentMasks)
parentMasks = appendUniqueMasks(parentMasks, compositeMasks)
}
assert.Equal(t, []string{"secret"}, parentMasks)
}
func TestCompositeStepIfReadsJobStatusExceptInItsOwnMainSteps(t *testing.T) {
failStep := func(rc *RunContext) {
rc.StepResults["failed"] = &model.StepResult{Conclusion: model.StepStatusFailure}
+26 -96
View File
@@ -126,7 +126,7 @@ runs:
return strings.NewReader(tt.fileContent), closerFunc(func() { closed = true }), nil
}
writeFile := func(filename string, data []byte, perm fs.FileMode) error {
writeFile := func(filename string, _data []byte, perm fs.FileMode) error {
assert.Equal(t, "actionDir/actionPath/trampoline.js", filename)
assert.Equal(t, fs.FileMode(0o644), perm)
return nil
@@ -134,7 +134,7 @@ runs:
action, err := readActionImpl(t.Context(), tt.step, "actionDir", "actionPath", readFile, writeFile)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
assert.Equal(t, tt.expected, action)
assert.Equal(t, tt.filename != "", closed)
@@ -144,94 +144,27 @@ runs:
func TestActionRunner(t *testing.T) {
t.Chdir(t.TempDir())
table := []struct {
for _, tt := range []struct {
name string
step actionStep
inputs map[string]model.Input
state map[string]string
expectedEnv map[string]string
}{
{
name: "with-input",
step: &stepActionRemote{
Step: &model.Step{
Uses: "org/repo/path@ref",
},
RunContext: &RunContext{
Config: &Config{},
Run: &model.Run{
JobID: "job",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"job": {
Name: "job",
},
},
},
},
},
action: &model.Action{
Inputs: map[string]model.Input{
"key": {
Default: "default value",
},
},
Runs: model.ActionRuns{
Using: "node16",
},
},
env: map[string]string{},
remoteAction: mustNewRemoteAction(t, "org/repo/path@ref"),
},
name: "with-input",
inputs: map[string]model.Input{"key": {Default: "default value"}},
expectedEnv: map[string]string{"INPUT_KEY": "default value"},
},
{
name: "restore-saved-state",
step: &stepActionRemote{
Step: &model.Step{
ID: "step",
Uses: "org/repo/path@ref",
},
RunContext: &RunContext{
ActionPath: "path",
Config: &Config{},
Run: &model.Run{
JobID: "job",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"job": {
Name: "job",
},
},
},
},
CurrentStep: "post-step",
StepResults: map[string]*model.StepResult{
"step": {},
},
IntraActionState: map[string]map[string]string{
"step": {
"name": "state value",
},
},
},
action: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
},
},
env: map[string]string{},
remoteAction: mustNewRemoteAction(t, "org/repo/path@ref"),
},
name: "restore-saved-state",
state: map[string]string{"name": "state value"},
expectedEnv: map[string]string{"STATE_name": "state value"},
},
}
for _, tt := range table {
} {
t.Run(tt.name, func(t *testing.T) {
ctx := t.Context()
cm := &containerMock{}
containerActionDir := "/var/run/act/actions/" + model.UsesHash("/org/repo/path@ref") + "/"
cm.On("CopyDir", containerActionDir, "dir/", false, true).Return(func(ctx context.Context) error { return nil })
cm.On("CopyDir", containerActionDir, "dir/", false, true).Return(func(_ctx context.Context) error { return nil })
envMatcher := mock.MatchedBy(func(env map[string]string) bool {
for k, v := range tt.expectedEnv {
@@ -241,14 +174,22 @@ func TestActionRunner(t *testing.T) {
}
return true
})
cm.On("Exec", []string{"node", "--preserve-symlinks-main", containerActionDir + "path"}, envMatcher, "", "").Return(func(_ctx context.Context) error { return nil })
cm.On("Exec", []string{"node", "--preserve-symlinks-main", containerActionDir + "path"}, envMatcher, "", "").Return(func(ctx context.Context) error { return nil })
tt.step.getRunContext().JobContainer = cm
err := runActionImpl(tt.step, "dir", mustNewRemoteAction(t, "org/repo/path@ref"))(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
remoteAction := mustNewRemoteAction(t, "org/repo/path@ref")
step := &stepActionRemote{
Step: &model.Step{ID: "step", Uses: "org/repo/path@ref"},
RunContext: &RunContext{
Config: &Config{},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
IntraActionState: map[string]map[string]string{"step": tt.state},
JobContainer: cm,
},
action: &model.Action{Inputs: tt.inputs, Runs: model.ActionRuns{Using: "node16"}},
env: map[string]string{},
remoteAction: remoteAction,
}
require.NoError(t, runActionImpl(step, "dir", remoteAction)(t.Context()))
cm.AssertExpectations(t)
})
}
@@ -322,7 +263,6 @@ func TestNewStepContainerDoesNotUseDockerSecrets(t *testing.T) {
newStepContainer(ctx, step, "registry.example.com/action:tag", nil, nil, "")
// DOCKER_USERNAME/DOCKER_PASSWORD should not be injected as pull credentials for docker action containers.
assert.Empty(t, captured.Username)
assert.Empty(t, captured.Password)
assert.True(t, captured.AutoRemove)
@@ -478,15 +418,11 @@ func TestExecAsDockerHoldsCloneLockForRemoteUncached(t *testing.T) {
}
func TestDockerActionImageTag(t *testing.T) {
// Remote actions already carry a unique, ref-scoped actionName (the uses
// hash), so the tag must be left untouched for backwards compatibility.
assert.Equal(t,
"act-abc123-dockeraction:latest",
dockerActionImageTag("owner/repo", "abc123", false),
)
// Local actions keep a human-readable, repository-namespaced prefix and gain a short hash suffix that makes the tag unique per (repository, actionName).
// See https://gitea.com/gitea/runner/issues/1039.
assert.Equal(t,
"act-owner-repo-baca2daaa2fe-dockeraction:latest",
dockerActionImageTag("owner/repo", "./", true),
@@ -496,16 +432,11 @@ func TestDockerActionImageTag(t *testing.T) {
dockerActionImageTag("owner/repo", "./sub", true),
)
// Sanitizing every non-alphanumeric character to "-" is lossy, so distinct inputs can collapse to the same readable prefix.
// The hash suffix must keep such cases apart, otherwise an image built for one repository is reused for another.
collisions := [][2]struct {
repoName string
actionName string
}{
// Two different repositories, both `uses: ./`: "a/b-c" and "a-b/c" both sanitize to "a-b-c".
{{"a/b-c", "./"}, {"a-b/c", "./"}},
// A repository's root action vs another repository's sub-path action:
// "owner/repo-a" + "./" and "owner/repo" + "./a" both sanitize to "owner-repo-a".
{{"owner/repo-a", "./"}, {"owner/repo", "./a"}},
}
for _, c := range collisions {
@@ -517,7 +448,6 @@ func TestDockerActionImageTag(t *testing.T) {
)
}
// Distinct local actions within the same repository keep distinct tags.
assert.NotEqual(t,
dockerActionImageTag("owner/repo", "./", true),
dockerActionImageTag("owner/repo", "./sub", true),
+79 -131
View File
@@ -66,145 +66,93 @@ func TestCancelledJobStatusEnablesAlwaysAndCancelledSteps(t *testing.T) {
assert.Equal(t, "cancelled", rc.Run.Job().Result)
}
// TestMainStepsExecutorRunsAlwaysStepsAfterCancel verifies that newMainStepsExecutor does
// not abandon the remaining steps when the run is cancelled mid-pipeline. The later step
// still runs (so a main-stage always() step is reached), it runs under a fresh,
// non-cancelled context, and the job is marked cancelled. The interrupt error is still
// propagated so callers up the chain see the cancellation.
func TestMainStepsExecutorRunsAlwaysStepsAfterCancel(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
func TestMainStepsExecutorRunsAlwaysStepsAfterInterruption(t *testing.T) {
for name, wantError := range map[string]error{"cancel": context.Canceled, "timeout": context.DeadlineExceeded} {
t.Run(name, func(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
var ctx context.Context
var interrupt func()
if wantError == context.Canceled {
ctx, interrupt = context.WithCancel(t.Context())
defer interrupt()
} else {
deadline := newControllableDeadlineContext(t.Context())
ctx, interrupt = deadline, deadline.expire
}
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
var ran []string
var laterStepCtxErr error
err := newMainStepsExecutor(rc, []common.Executor{
func(_ctx context.Context) error {
ran = append(ran, "step1")
interrupt()
return nil
},
func(ctx context.Context) error {
ran = append(ran, "always-step")
laterStepCtxErr = ctx.Err()
return nil
},
})(ctx)
var ran []string
var laterStepCtxErr error
steps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "step1")
cancel() // server cancellation lands while step1 runs
return nil
},
func(c context.Context) error {
ran = append(ran, "always-step")
laterStepCtxErr = c.Err()
return nil
},
require.ErrorIs(t, err, wantError)
assert.Equal(t, []string{"step1", "always-step"}, ran)
require.NoError(t, laterStepCtxErr)
assert.Equal(t, wantError == context.Canceled, rc.jobCancelled)
if wantError == context.DeadlineExceeded {
assert.True(t, rc.jobFailed)
assert.Equal(t, "failure", rc.getJobContext().Status)
}
})
}
err := newMainStepsExecutor(rc, steps)(ctx)
require.ErrorIs(t, err, context.Canceled, "interrupt error is propagated")
assert.Equal(t, []string{"step1", "always-step"}, ran, "the always() step still runs after cancel")
require.NoError(t, laterStepCtxErr, "remaining steps run under a fresh, non-cancelled context")
assert.True(t, rc.jobCancelled, "the job is marked cancelled")
}
// TestMainStepsExecutorMarksFailedOnTimeoutBetweenSteps guards the timeout path's symmetry with the cancel path.
// When the job deadline (timeout-minutes) lands in the gap between two steps, the job must be marked as failed (not cancelled),
// so always()/failure() cleanup steps run while default success() steps skip, and so the timed-out job is not reported as success.
func TestMainStepsExecutorMarksFailedOnTimeoutBetweenSteps(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
func TestStepsExecutorRunsMainStepsAfterPreInterruption(t *testing.T) {
for name, tt := range map[string]struct {
cancel bool
preErr error
wantError error
}{
"cancel": {cancel: true, wantError: context.Canceled},
"failure": {preErr: assert.AnError, wantError: assert.AnError},
} {
t.Run(name, func(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
ctx := t.Context()
var cancel context.CancelFunc
if tt.cancel {
ctx, cancel = context.WithCancel(ctx)
defer cancel()
}
ctx := newControllableDeadlineContext(t.Context())
var ran []string
var mainStepCtxErr error
err := newStepsExecutor(rc, []common.Executor{
func(_ctx context.Context) error {
ran = append(ran, "pre1")
if tt.cancel {
cancel()
}
return tt.preErr
},
}, []common.Executor{
func(ctx context.Context) error {
ran = append(ran, "always-step")
mainStepCtxErr = ctx.Err()
return nil
},
})(ctx)
var ran []string
var laterStepCtxErr error
steps := []common.Executor{
func(context.Context) error {
ran = append(ran, "step1")
ctx.expire()
return nil
},
func(c context.Context) error {
ran = append(ran, "always-step")
laterStepCtxErr = c.Err()
return nil
},
require.ErrorIs(t, err, tt.wantError)
assert.Equal(t, []string{"pre1", "always-step"}, ran)
require.NoError(t, mainStepCtxErr)
assert.Equal(t, tt.cancel, rc.jobCancelled)
})
}
err := newMainStepsExecutor(rc, steps)(ctx)
require.ErrorIs(t, err, context.DeadlineExceeded, "the timeout error is propagated")
assert.Equal(t, []string{"step1", "always-step"}, ran, "the always() step still runs after a timeout")
require.NoError(t, laterStepCtxErr, "remaining steps run under a fresh, non-expired context")
assert.True(t, rc.jobFailed, "a job timeout marks the job failed")
assert.False(t, rc.jobCancelled, "a timeout is not a cancellation")
// The status the real main-step `if` evaluation sees: "failure", so default success() steps skip while always()/failure() steps run.
assert.Equal(t, "failure", rc.getJobContext().Status)
}
// TestStepsExecutorRunsMainStepsAfterPreCancel verifies that a cancellation landing during the
// pre phase does not abandon the main steps: newStepsExecutor still runs the main-steps executor,
// so a main-stage always()/cancelled() step is reached (under a fresh, non-cancelled context),
// the job is marked cancelled, and the cancellation is propagated. Before the fix the `.Then(...)`
// short-circuit skipped the main steps entirely when a pre step was cancelled.
func TestStepsExecutorRunsMainStepsAfterPreCancel(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
var ran []string
var mainStepCtxErr error
preSteps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "pre1")
cancel() // server cancellation lands during the pre phase
return nil
},
}
steps := []common.Executor{
func(c context.Context) error {
ran = append(ran, "always-step")
mainStepCtxErr = c.Err()
return nil
},
}
err := newStepsExecutor(rc, preSteps, steps)(ctx)
require.ErrorIs(t, err, context.Canceled, "the cancellation is propagated")
assert.Equal(t, []string{"pre1", "always-step"}, ran, "the main always() step runs after a pre-phase cancel")
require.NoError(t, mainStepCtxErr, "the main step runs under a fresh, non-cancelled context")
assert.True(t, rc.jobCancelled, "the job is marked cancelled")
}
// TestStepsExecutorRunsMainStepsAfterPreFailure verifies that a failing pre step does not abandon
// the main steps: they still run (so a main-stage always()/failure() step is reached), and the
// pre-step error is propagated so the job is reported as failed. The main steps' own `if`
// evaluation is what skips success()-default steps, so running them here is safe.
func TestStepsExecutorRunsMainStepsAfterPreFailure(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
var ran []string
preSteps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "pre1")
return assert.AnError
},
}
steps := []common.Executor{
func(_ context.Context) error {
ran = append(ran, "always-step")
return nil
},
}
err := newStepsExecutor(rc, preSteps, steps)(t.Context())
require.ErrorIs(t, err, assert.AnError, "the pre-step error is propagated")
assert.Equal(t, []string{"pre1", "always-step"}, ran, "the main always() step runs after a pre-step failure")
assert.False(t, rc.jobCancelled, "a pre-step failure is not a cancellation")
}
// TestPreStepFailureAffectsMainStepIfStatus verifies the status path used by real
+113 -101
View File
@@ -5,61 +5,72 @@
package runner
import (
"cmp"
"context"
"fmt"
"regexp"
"strings"
"unicode"
"gitea.com/gitea/runner/act/common"
)
var commandPatternGA *regexp.Regexp
var commandPatternADO *regexp.Regexp
func init() {
commandPatternGA = regexp.MustCompile("^::([^ ]+?)( (.+?))?::([^\r\n]*)[\r\n]*$")
// excluding ']' ends the command info at the first bracket, as GitHub does
commandPatternADO = regexp.MustCompile("^##\\[([^ \\]]+)( ([^\\]]*))?]([^\r\n]*)[\r\n]*$")
}
func tryParseRawActionCommand(line string) (command string, kvPairs map[string]string, arg string, ok bool) {
command, kvPairs, arg, _, ok = tryParseActionCommand(line)
return command, kvPairs, arg, ok
}
func tryParseActionCommand(line string) (command string, kvPairs map[string]string, arg string, legacy, ok bool) {
if m := commandPatternGA.FindStringSubmatch(line); m != nil {
command = m[1]
kvPairs = parseKeyValuePairs(m[3], ",")
arg = m[4]
ok = true
} else if m := commandPatternADO.FindStringSubmatch(line); m != nil {
command = m[1]
kvPairs = parseKeyValuePairs(m[3], ";")
arg = m[4]
legacy = true
ok = true
// parseActionCommand parses like GitHub's runner, but rejects a bare \r GitHub would split at and keeps a value's leading '=' so masks match.
func parseActionCommand(line, resumeCommand string) (command string, properties map[string]string, data string, ok bool) {
line = strings.TrimRight(line, "\r\n")
if strings.ContainsAny(line, "\r\n") {
return "", nil, "", false
}
return command, kvPairs, arg, legacy, ok
trimmed := strings.TrimLeftFunc(line, unicode.IsSpace)
for _, legacy := range []bool{false, true} {
header, found := strings.CutPrefix(trimmed, "::")
separator, propertySeparator, unescapeProperty, unescapeData := "::", ",", commandPropertyUnescaper.Replace, UnescapeCommandData
if legacy {
_, header, found = strings.Cut(line, "##[")
separator, propertySeparator, unescapeProperty, unescapeData = "]", ";", UnescapeLegacyCommand, UnescapeLegacyCommand
}
if !found {
continue
}
header, data, found := strings.Cut(header, separator)
if !found {
continue
}
command, propertyList, _ := strings.Cut(header, " ")
command = strings.ToLower(command)
switch command {
case "stop-commands", "set-env", "set-output", "save-state", "add-path", "add-mask", "add-matcher",
"remove-matcher", "warning", "error", "notice", "debug", "group", "endgroup", "echo":
default:
if resumeCommand == "" || !strings.EqualFold(command, resumeCommand) {
continue
}
}
if !legacy {
propertyList = strings.TrimSpace(propertyList)
}
properties := map[string]string{}
for property := range strings.SplitSeq(propertyList, propertySeparator) {
if key, value, _ := strings.Cut(strings.TrimLeft(property, "="), "="); value != "" {
properties[strings.ToLower(key)] = unescapeProperty(value)
}
}
return command, properties, unescapeData(data), true
}
return "", nil, "", false
}
// commandHandler reports whether a line is logged, keeping per-handler command state as GitHub does.
func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
logger := common.Logger(ctx)
rawLogger := common.Logger(ctx).With(rawOutputField, true)
debug := rc.topLevelRunContext().Config.RunnerDebug()
echo := debug
resumeCommand := ""
return func(line string) bool {
command, kvPairs, arg, legacy, ok := tryParseActionCommand(line)
command, kvPairs, arg, ok := parseActionCommand(line, resumeCommand)
if !ok {
return true
}
command = strings.ToLower(command)
if resumeCommand != "" {
// There should not be any emojis in the log output for Gitea.
// Return true (not false) so the line still reaches the raw_output
// log handler; otherwise everything between ::stop-commands:: and
// its end token is silently dropped from the step log.
logger.Infof("%s", line)
// Resumed here rather than from the switch, because the end token is arbitrary
// and a token naming a real command would otherwise never resume.
if strings.EqualFold(command, resumeCommand) {
@@ -67,15 +78,9 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
}
return true
}
if legacy {
arg = UnescapeLegacyCommand(arg)
kvPairs = unescapeKvPairs(kvPairs, UnescapeLegacyCommand)
} else {
arg = UnescapeCommandData(arg)
kvPairs = unescapeKvPairs(kvPairs, unescapeCommandProperty)
}
echoed := echo
if (command == "set-env" || command == "add-path") && rc.refuseUnsecureCommand(ctx, command) {
return true
return echoed
}
switch command {
case "set-env":
@@ -83,26 +88,67 @@ func (rc *RunContext) commandHandler(ctx context.Context) common.LineHandler {
case "set-output":
rc.setOutput(ctx, kvPairs, arg)
case "add-path":
rc.addPath(ctx, arg)
rc.addPath(arg)
case "save-state":
rc.saveState(ctx, kvPairs, arg)
case "echo":
switch strings.ToLower(strings.TrimSpace(arg)) {
case "on":
echo = true
case "off":
echo = false
}
case "add-mask":
rc.AddMask(arg)
logger.Infof("%s", "***")
// The raw line is still forwarded, carrying the secret: that is how the reporter
// learns the mask, and it drops the row rather than writing it out.
rc.AddMask(ctx, arg)
if echoed {
rawLogger.Infof("::add-mask::***")
}
return false
case "stop-commands":
resumeCommand = arg
logger.Infof("%s", line)
case "save-state":
logger.Infof("%s", line)
rc.saveState(ctx, kvPairs, arg)
default:
// ::debug::, ::error::, ::warning::, ::add-matcher:: and anything unrecognised are
// passed through for the reporter and Gitea's web UI to render.
logger.Infof("%s", line)
if len(arg) > 6 { // as on GitHub, so the log does not give away the resume token
rc.AddMask(ctx, arg)
}
return true
case "debug":
return debug
case "notice", "warning", "error":
rawLogger.Infof("%s", formatAnnotation(command, kvPairs, arg))
return false
case "group", "endgroup":
return true
}
return echoed
}
}
// return true to let gitea's logger handle these special outputs also
return true
// formatAnnotation folds the location and title into the message, as Gitea's web UI drops command properties.
func formatAnnotation(level string, properties map[string]string, message string) string {
prefix := properties["file"]
if prefix != "" {
if line := properties["line"]; line != "" {
prefix += ":" + line
if col := cmp.Or(properties["col"], properties["column"]); col != "" {
prefix += ":" + col
}
}
prefix += ": "
}
if title := properties["title"]; title != "" {
prefix += title + ": "
}
return "::" + level + "::" + EscapeCommandData(prefix+message)
}
// serviceCommandHandler applies only ::add-mask::, so a service cannot set the job's env, outputs or state.
func (rc *RunContext) serviceCommandHandler(ctx context.Context) common.LineHandler {
return func(line string) bool {
command, _, arg, ok := parseActionCommand(line, "")
if !ok || command != "add-mask" {
return true
}
rc.AddMask(ctx, arg)
return false
}
}
@@ -112,7 +158,9 @@ const allowUnsecureCommandsVar = "ACTIONS_ALLOW_UNSECURE_COMMANDS"
// not run, recording the error that fails the step. GitHub disabled both because a step that
// echoes untrusted content can use them to set NODE_OPTIONS or PATH for every later step.
func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string) bool {
if rc.allowUnsecureCommandsOptIn() {
if isTruthyEnv(rc.currentStepEnv()[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.Env[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.GlobalEnv[allowUnsecureCommandsVar]) {
return false
}
@@ -130,14 +178,6 @@ func (rc *RunContext) refuseUnsecureCommand(ctx context.Context, command string)
return true
}
// allowUnsecureCommandsOptIn reports whether the workflow itself asked for the deprecated
// commands, from any env scope, as it can on GitHub.
func (rc *RunContext) allowUnsecureCommandsOptIn() bool {
return isTruthyEnv(rc.currentStepEnv()[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.Env[allowUnsecureCommandsVar]) ||
isTruthyEnv(rc.GlobalEnv[allowUnsecureCommandsVar])
}
// isTruthyEnv mirrors GitHub's bool.TryParse: only "true", in any casing.
func isTruthyEnv(v string) bool {
return strings.EqualFold(strings.TrimSpace(v), "true")
@@ -162,7 +202,6 @@ func (rc *RunContext) setEnv(ctx context.Context, kvPairs map[string]string, arg
common.Logger(ctx).With(rawOutputField, true).Errorf("##[error]%s", EscapeCommandData(message))
return
}
common.Logger(ctx).Infof("::set-env:: %s=%s", name, arg)
if rc.Env == nil {
rc.Env = make(map[string]string)
}
@@ -185,22 +224,18 @@ func (rc *RunContext) setEnvFile(ctx context.Context, kvPairs map[string]string,
}
func (rc *RunContext) setOutput(ctx context.Context, kvPairs map[string]string, arg string) {
logger := common.Logger(ctx)
stepID := rc.CurrentStep
outputName := kvPairs["name"]
result, ok := rc.StepResults[stepID]
if !ok {
logger.Infof("No outputs registered for step '%s'", stepID)
common.Logger(ctx).Infof("No outputs registered for step '%s'", stepID)
return
}
logger.Infof("::set-output:: %s=%s", outputName, arg)
result.Outputs[outputName] = arg
result.Outputs[kvPairs["name"]] = arg
}
func (rc *RunContext) addPath(ctx context.Context, arg string) {
common.Logger(ctx).Infof("::add-path:: %s", arg)
func (rc *RunContext) addPath(arg string) {
extraPath := []string{arg}
for _, v := range rc.ExtraPath {
if v != arg {
@@ -210,18 +245,6 @@ func (rc *RunContext) addPath(ctx context.Context, arg string) {
rc.ExtraPath = extraPath
}
func parseKeyValuePairs(kvPairs, separator string) map[string]string {
rtn := make(map[string]string)
kvPairList := strings.SplitSeq(kvPairs, separator)
for kvPair := range kvPairList {
kv := strings.SplitN(kvPair, "=", 2)
if len(kv) == 2 {
rtn[kv[0]] = kv[1]
}
}
return rtn
}
// A Replacer never rescans what it wrote, so "%250A" stays a literal "%0A".
var (
commandDataEscaper = strings.NewReplacer("%", "%25", "\r", "%0D", "\n", "%0A")
@@ -240,22 +263,11 @@ func UnescapeCommandData(arg string) string {
return commandDataUnescaper.Replace(arg)
}
func unescapeCommandProperty(arg string) string {
return commandPropertyUnescaper.Replace(arg)
}
// UnescapeLegacyCommand decodes a "##[cmd]" line, which also spells ";" and "]" escaped.
func UnescapeLegacyCommand(arg string) string {
return legacyCommandUnescaper.Replace(arg)
}
func unescapeKvPairs(kvPairs map[string]string, unescape func(string) string) map[string]string {
for k, v := range kvPairs {
kvPairs[k] = unescape(v)
}
return kvPairs
}
func (rc *RunContext) saveState(_ context.Context, kvPairs map[string]string, arg string) {
stepID := rc.CurrentStep
if stepID != "" {
+178 -200
View File
@@ -5,12 +5,13 @@
package runner
import (
"bytes"
"io"
"os"
"strconv"
"strings"
"testing"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.dev/actionslib/pkg/model"
@@ -18,9 +19,55 @@ import (
"github.com/stretchr/testify/require"
)
func TestParseActionCommand(t *testing.T) {
for _, test := range []struct {
name, line, resumeCommand, command string
properties map[string]string
data string
ok bool
}{
{"plain", "plain output", "", "", nil, "", false},
{"unknown v2", "::unknown::v", "", "", nil, "", false},
{"missing v2 terminator", "::set-output name=x", "", "", nil, "", false},
{"empty command", "::::", "", "", nil, "", false},
{"text before v2", "debug ::set-output name=x::v", "", "", nil, "", false},
{"missing legacy terminator", "##[debug", "", "", nil, "", false},
{"tab is not a name separator", "::set-output\tname=x::v", "", "", nil, "", false},
{"v2 leading unicode whitespace and line endings", " \t\u2003::set-output name=tab::v\r\n\r\n", "", "set-output", map[string]string{"name": "tab"}, "v", true},
{"legacy prefix and casing", "prefix ##[SET-OUTPUT NAME=mid;]v\r\n", "", "set-output", map[string]string{"name": "mid"}, "v", true},
{"only first legacy prefix", "##[unknown]##[set-output name=x]v", "", "", nil, "", false},
{"v2 delimiter boundaries", "::set-output name=a:b,other=c:d:: a::b ", "", "set-output", map[string]string{"name": "a:b", "other": "c:d"}, " a::b ", true},
{"legacy first closing bracket", "##[set-output name=x]v]tail", "", "set-output", map[string]string{"name": "x"}, "v]tail", true},
{"v2 semicolons are literal", "::set-output name=x;y=z::v", "", "set-output", map[string]string{"name": "x;y=z"}, "v", true},
{"legacy commas are literal", "##[set-output name=x,y=z]v", "", "set-output", map[string]string{"name": "x,y=z"}, "v", true},
{"values keep their leading equals", "::set-output name==x=y,=key=v,empty=,=x,spaced= =x::v", "", "set-output", map[string]string{"name": "=x=y", "key": "v", "spaced": " =x"}, "v", true},
{"v2 trims whole list only", "::set-output \tfile=x, line=2,name = value \t::v", "", "set-output", map[string]string{"file": "x", " line": "2", "name ": " value"}, "v", true},
{"legacy preserves list whitespace", "##[set-output name=q ]v", "", "set-output", map[string]string{" name": "q "}, "v", true},
{"case insensitive duplicate keys", "::SET-OUTPUT NAME=first,Name=last::v", "", "set-output", map[string]string{"name": "last"}, "v", true},
{"keys stay escaped", "::set-output na%3Ame=x::v", "", "set-output", map[string]string{"na%3ame": "x"}, "v", true},
{"v2 decode table", "::set-output name=%0D%0A%3A%2C%25%253A%252C%250A%2525%3a%2c%0d%0a%3B%5D::%0D%0A%25%3A%2C%3B%5D%250A", "", "set-output", map[string]string{"name": "\r\n:,%%3A%2C%0A%25%3a%2c%0d%0a%3B%5D"}, "\r\n%%3A%2C%3B%5D%0A", true},
{"legacy decode table", "##[set-output name=%3B%0D%0A%5D%25%253B%255D%250A%2525%3b%5d%0d%0a%3A%2C]v%5Dw%3Bx", "", "set-output", map[string]string{"name": ";\r\n]%%3B%5D%0A%25%3b%5d%0d%0a%3A%2C"}, "v]w;x", true},
{"unknown v2 falls back to legacy", "::unknown::prefix ##[set-output name=x]v", "", "set-output", map[string]string{"name": "x"}, "v", true},
{"unterminated v2 falls back to legacy", "::debug ##[set-output name=x]v", "", "set-output", map[string]string{"name": "x"}, "v", true},
{"v2 wins over legacy", "::debug::##[set-output name=x]v", "", "debug", map[string]string{}, "##[set-output name=x]v", true},
{"internal line breaks are not commands", "::debug::hidden\rplain\r\n", "", "", nil, "", false},
{"v2 resume token", "::ReSuMe-ToKeN::", "RESUME-TOKEN", "resume-token", map[string]string{}, "", true},
{"legacy resume token", "prefix ##[RESUME-TOKEN]v", "resume-token", "resume-token", map[string]string{}, "v", true},
{"different resume token", "::other-token::", "resume-token", "", nil, "", false},
} {
t.Run(test.name, func(t *testing.T) {
command, properties, data, ok := parseActionCommand(test.line, test.resumeCommand)
assert.Equal(t, test.command, command)
assert.Equal(t, test.properties, properties)
assert.Equal(t, test.data, data)
assert.Equal(t, test.ok, ok)
})
}
}
// unsecureRC opts into ::set-env:: and ::add-path::, which are refused without it.
func unsecureRC() *RunContext {
return &RunContext{Env: map[string]string{allowUnsecureCommandsVar: "true"}}
return &RunContext{Config: &Config{}, Env: map[string]string{allowUnsecureCommandsVar: "true"}}
}
func TestSetEnv(t *testing.T) {
@@ -37,212 +84,37 @@ func TestSetEnv(t *testing.T) {
a.NotContains(rc.Env, "NODE_OPTIONS")
a.NotContains(rc.Env, "node_options")
entries := hook.AllRecords()
require.Len(t, entries, 3)
a.Equal("##[error]Can't update NODE_OPTIONS environment variable using ::set-env:: command.", entries[1].Message)
a.Equal("##[error]Can't store NODE_OPTIONS output parameter using '$GITHUB_ENV' command.", entries[2].Message)
}
func TestStopCommandsKeepsSuppressedLinesInLog(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := unsecureRC()
handler := rc.commandHandler(ctx)
// Stop command processing until the matching end token is seen.
a.True(handler("::stop-commands::my-end-token\n"))
// A command-shaped line while stopped must not be executed (env unchanged),
// but must still return true so it reaches the raw_output log handler and is
// not dropped from the step log.
a.True(handler("::set-env name=x::valz\n"))
a.NotContains(rc.Env, "x")
// The matching end token resumes command processing.
a.True(handler("::my-end-token::\n"))
// Commands are processed again after resuming.
a.True(handler("::set-env name=y::valy\n"))
a.Equal("valy", rc.Env["y"])
require.Len(t, entries, 2)
a.Equal("##[error]Can't update NODE_OPTIONS environment variable using ::set-env:: command.", entries[0].Message)
a.Equal("##[error]Can't store NODE_OPTIONS output parameter using '$GITHUB_ENV' command.", entries[1].Message)
}
func TestSetOutput(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := new(RunContext)
rc.StepResults = make(map[string]*model.StepResult)
handler := rc.commandHandler(ctx)
rc := &RunContext{Config: &Config{}, CurrentStep: "my-step", StepResults: map[string]*model.StepResult{"my-step": {Outputs: map[string]string{}}}}
handler := rc.commandHandler(t.Context())
rc.CurrentStep = "my-step"
rc.StepResults[rc.CurrentStep] = &model.StepResult{
Outputs: make(map[string]string),
}
handler("::set-output name=x::valz\n")
a.Equal("valz", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x::percent2%25\n")
a.Equal("percent2%", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x::percent2%25%0Atest\n")
a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x::percent2%25%0Atest another3%25test\n")
a.Equal("percent2%\ntest another3%test", rc.StepResults["my-step"].Outputs["x"])
handler("::set-output name=x%3A::percent2%25%0Atest\n")
a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x:"])
handler("::set-output name=x%3A%2C%0A%25%0D%3A::percent2%25%0Atest\n")
a.Equal("percent2%\ntest", rc.StepResults["my-step"].Outputs["x:,\n%\r:"])
handler("::set-output name=symbol::std::vector")
a.Equal("std::vector", rc.StepResults["my-step"].Outputs["symbol"])
handler("::set-output name=a=b::value\n")
a.Equal("value", rc.StepResults["my-step"].Outputs["a=b"])
handler("##[set-output name=legacy%3B%5D]value%3B%5D")
a.Equal("value;]", rc.StepResults["my-step"].Outputs["legacy;]"])
handler("##[set-output name=bracket]value]tail")
a.Equal("value]tail", rc.StepResults["my-step"].Outputs["bracket"])
handler("::set-output name=modern%3B%5D::value%3B%5D\n")
a.Equal("value%3B%5D", rc.StepResults["my-step"].Outputs["modern%3B%5D"])
handler("::set-output NAME=modern%3A::v%0D%0A%25%3B%5D%250A\n")
handler("##[set-output name=legacy%3B%5D]v%0D%0A%25%3B%5D%250A\n")
assert.Equal(t, map[string]string{"modern:": "v\r\n%%3B%5D%0A", "legacy;]": "v\r\n%;]%0A"}, rc.StepResults["my-step"].Outputs)
}
func TestAddpath(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := unsecureRC()
handler := rc.commandHandler(ctx)
handler("::add-path::/zoo\n")
a.Equal("/zoo", rc.ExtraPath[0])
handler("::add-path::/boo\n")
a.Equal("/boo", rc.ExtraPath[0])
}
func TestStopCommands(t *testing.T) {
logger, hook := logtest.NewLogger()
a := assert.New(t)
ctx := common.WithLogger(t.Context(), logger)
rc := unsecureRC()
handler := rc.commandHandler(ctx)
handler("::set-env name=x::valz\n")
a.Equal("valz", rc.Env["x"])
handler("::stop-commands::MY-END-TOKEN\n")
handler("::set-env name=x::abcd\n")
a.Equal("valz", rc.Env["x"])
handler("::my-end-token::\n")
handler("::set-env name=x::abcd\n")
a.Equal("abcd", rc.Env["x"])
messages := make([]string, 0)
for _, entry := range hook.AllRecords() {
messages = append(messages, entry.Message)
for _, prefix := range []string{"::add-path::", "##[add-path]"} {
t.Run(prefix, func(t *testing.T) {
rc := unsecureRC()
handler := rc.commandHandler(t.Context())
handler(prefix + "/zoo\n")
handler(prefix + "/boo\n")
assert.Equal(t, []string{"/boo", "/zoo"}, rc.ExtraPath)
handler(prefix + "/zoo\n")
assert.Equal(t, []string{"/zoo", "/boo"}, rc.ExtraPath)
})
}
a.Contains(messages, "::set-env name=x::abcd\n")
}
// The end token is arbitrary, so one that happens to name a real command must still resume
// rather than being swallowed by that command's case.
func TestStopCommandsResumesOnCommandNamedToken(t *testing.T) {
a := assert.New(t)
rc := unsecureRC()
handler := rc.commandHandler(t.Context())
handler("::stop-commands::add-mask\n")
handler("::set-env name=x::suppressed\n")
a.NotContains(rc.Env, "x")
handler("::add-mask::\n")
handler("::set-env name=x::resumed\n")
a.Equal("resumed", rc.Env["x"])
}
func TestAddpathADO(t *testing.T) {
a := assert.New(t)
ctx := t.Context()
rc := unsecureRC()
handler := rc.commandHandler(ctx)
handler("##[add-path]/zoo\n")
a.Equal("/zoo", rc.ExtraPath[0])
handler("##[add-path]/boo\n")
a.Equal("/boo", rc.ExtraPath[0])
}
func TestAddmask(t *testing.T) {
logger, hook := logtest.NewLogger()
a := assert.New(t)
ctx := t.Context()
loggerCtx := common.WithLogger(ctx, logger)
rc := new(RunContext)
handler := rc.commandHandler(loggerCtx)
handler("::ADD-MASK::my::secret")
a.Equal("***", hook.LastRecord().Message)
a.Equal([]string{"my::secret"}, rc.Masks)
}
// based on https://stackoverflow.com/a/10476304
func captureOutput(t *testing.T, f func()) string {
old := os.Stdout
r, w, _ := os.Pipe()
os.Stdout = w
f()
outC := make(chan string)
go func() {
var buf bytes.Buffer
_, err := io.Copy(&buf, r)
if err != nil {
a := assert.New(t)
a.Fail("io.Copy failed")
}
outC <- buf.String()
}()
w.Close()
os.Stdout = old
out := <-outC
return out
}
func TestAddmaskUsemask(t *testing.T) {
rc := new(RunContext)
rc.StepResults = make(map[string]*model.StepResult)
rc.CurrentStep = "my-step"
rc.StepResults[rc.CurrentStep] = &model.StepResult{
Outputs: make(map[string]string),
}
a := assert.New(t)
config := &Config{
Secrets: map[string]string{},
InsecureSecrets: false,
}
re := captureOutput(t, func() {
ctx := t.Context()
ctx = WithJobLogger(ctx, "0", "testjob", config, &rc.Masks, map[string]any{})
handler := rc.commandHandler(ctx)
handler("::add-mask::secret\n")
handler("::set-output:: token=secret\n")
})
a.Equal("[testjob] ***\n[testjob] ::set-output:: = token=***\n", re)
}
func TestSaveState(t *testing.T) {
rc := &RunContext{
Config: &Config{},
CurrentStep: "step",
StepResults: map[string]*model.StepResult{},
}
@@ -278,12 +150,12 @@ func TestUnsecureCommands(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
a := assert.New(t)
rc := &RunContext{Env: tt.jobEnv}
rc := &RunContext{Config: &Config{}, Env: tt.jobEnv}
rc.setCurrentStepEnv(tt.stepEnv)
handler := rc.commandHandler(t.Context())
handler("::set-env name=x::valz\n")
handler("::add-path::/opt/bin\n")
a.False(handler("::set-env name=x::valz\n"))
a.False(handler("::add-path::/opt/bin\n"))
if !tt.optedIn {
a.Empty(rc.Env["x"])
@@ -299,3 +171,109 @@ func TestUnsecureCommands(t *testing.T) {
})
}
}
type maskRecorder struct {
logging.Handler
masks []string
}
func (m *maskRecorder) AddMask(mask string) { m.masks = append(m.masks, mask) }
func TestCommandLogWriter(t *testing.T) {
for _, tt := range []struct {
name string
debug bool
steps [][]string
log []string
masks []string
}{
{
name: "applied commands are hidden unless echoed",
steps: [][]string{{
"::set-env name=x::y", "::add-path::/bin", "::set-output name=a::b", "##[save-state name=a]b", "::add-matcher::m.json", "::remove-matcher owner=m::",
"::echo::On%0A",
"::set-env name=x::y", "::add-path::/bin", "::set-output name=a::b", "##[save-state name=a]b", "::add-matcher::m.json", "::remove-matcher owner=m::",
"::ADD-MASK::my::secret", "::echo::bogus", "::echo::off", "::add-mask ::empty-properties", "::unknown::v", "plain",
}},
log: []string{
"::set-env name=x::y", "::add-path::/bin", "::set-output name=a::b", "##[save-state name=a]b", "::add-matcher::m.json", "::remove-matcher owner=m::",
"::add-mask::***", "::echo::bogus", "::echo::off", "::unknown::v", "plain",
},
masks: []string{"my::secret", "empty-properties"},
},
{
name: "step debug shows debug lines and echoes commands until echo is off",
debug: true,
steps: [][]string{{"::debug::first%0Asecond", "::set-output name=a::b", "::echo::off", "::debug::again", "::set-output name=a::b"}},
log: []string{"::debug::first%0Asecond", "::set-output name=a::b", "::echo::off", "::debug::again"},
},
{
name: "debug lines are hidden without step debug and an invalid echo changes nothing",
steps: [][]string{{"::echo::bogus", "::set-output name=a::b", "::echo::on", "::debug::d"}},
},
{
name: "annotations fold their location once whether echoed or not",
steps: [][]string{{
"::echo::on",
"::error file=main.go,line=12,col=5,title=vet::undefined: x",
"::warning File=a%3Ab.go,Line=1,column=2,title=100%252C::50%25%0Anext",
"::warning file=x,line=1,col=2,column=3::m",
"::notice line=9,col=4,title=title::m",
"::error file=x,col=2::m",
"::error file=x,line=1::m",
"::notice::std::vector",
"##[error file=x%3By]m%5D",
}},
log: []string{
"::error::main.go:12:5: vet: undefined: x", "::warning::a:b.go:1:2: 100%252C: 50%25%0Anext", "::warning::x:1:2: m",
"::notice::title: m", "::error::x: m", "::error::x:1: m", "::notice::std::vector", "::error::x;y: m]",
},
},
{
name: "stop-commands keeps lines verbatim until resumed",
steps: [][]string{{"::stop-commands::resume-token", "::add-mask::kept", "::set-output name=a::b", "::RESUME-TOKEN::", "::add-mask::masked"}},
log: []string{"::stop-commands::resume-token", "::add-mask::kept", "::set-output name=a::b", "::RESUME-TOKEN::"},
masks: []string{"resume-token", "masked"},
},
{
name: "a short stop token naming a command stays unmasked and resumes in any case",
steps: [][]string{{"::stop-commands::NOTICE", "::notice::", "::add-mask::masked"}},
log: []string{"::stop-commands::NOTICE", "::notice::"},
masks: []string{"masked"},
},
{
name: "state does not carry over to the next step",
steps: [][]string{{"::echo::on", "::stop-commands::token77"}, {"::set-output name=a::b", "::add-mask::masked"}},
log: []string{"::stop-commands::token77"},
masks: []string{"token77", "masked"},
},
} {
t.Run(tt.name, func(t *testing.T) {
logger, hook := logtest.NewLogger()
recorder := &maskRecorder{}
ctx := common.WithLoggerHandler(common.WithLogger(t.Context(), logger), recorder)
rc := &RunContext{
Config: &Config{},
Parent: &RunContext{Config: &Config{Secrets: map[string]string{"ACTIONS_STEP_DEBUG": strconv.FormatBool(tt.debug)}}},
Env: map[string]string{allowUnsecureCommandsVar: "true"},
CurrentStep: "step",
StepResults: map[string]*model.StepResult{"step": {Outputs: map[string]string{}}},
}
for _, lines := range tt.steps {
writer := rc.commandLogWriter(ctx)
for _, line := range lines {
_, err := io.WriteString(writer, line+"\n")
require.NoError(t, err)
}
}
var log []string
for _, record := range hook.AllRecords() {
log = append(log, strings.TrimSuffix(record.Message, "\n"))
}
assert.Equal(t, tt.log, log)
assert.Equal(t, tt.masks, recorder.masks)
assert.Equal(t, tt.masks, rc.Masks)
})
}
}
+10 -22
View File
@@ -23,56 +23,44 @@ type containerMock struct {
}
func (cm *containerMock) Create(capAdd, capDrop []string) common.Executor {
args := cm.Called(capAdd, capDrop)
return args.Get(0).(func(context.Context) error)
return cm.Called(capAdd, capDrop).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Pull(forcePull bool) common.Executor {
args := cm.Called(forcePull)
return args.Get(0).(func(context.Context) error)
return cm.Called(forcePull).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Start(attach bool) common.Executor {
args := cm.Called(attach)
return args.Get(0).(func(context.Context) error)
return cm.Called(attach).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Remove() common.Executor {
args := cm.Called()
return args.Get(0).(func(context.Context) error)
return cm.Called().Get(0).(func(context.Context) error)
}
func (cm *containerMock) Close() common.Executor {
args := cm.Called()
return args.Get(0).(func(context.Context) error)
return cm.Called().Get(0).(func(context.Context) error)
}
func (cm *containerMock) UpdateFromEnv(srcPath string, env *map[string]string) common.Executor {
args := cm.Called(srcPath, env)
return args.Get(0).(func(context.Context) error)
return cm.Called(srcPath, env).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Copy(destPath string, files ...*container.FileEntry) common.Executor {
args := cm.Called(destPath, files)
return args.Get(0).(func(context.Context) error)
return cm.Called(destPath, files).Get(0).(func(context.Context) error)
}
func (cm *containerMock) CopyDir(destPath, srcPath string, useGitIgnore, skipGitDir bool) common.Executor {
args := cm.Called(destPath, srcPath, useGitIgnore, skipGitDir)
return args.Get(0).(func(context.Context) error)
return cm.Called(destPath, srcPath, useGitIgnore, skipGitDir).Get(0).(func(context.Context) error)
}
func (cm *containerMock) Exec(command []string, env map[string]string, user, workdir string) common.Executor {
args := cm.Called(command, env, user, workdir)
return args.Get(0).(func(context.Context) error)
return cm.Called(command, env, user, workdir).Get(0).(func(context.Context) error)
}
func (cm *containerMock) GetContainerArchive(ctx context.Context, srcPath string) (io.ReadCloser, error) {
args := cm.Called(ctx, srcPath)
err, hasErr := args.Get(1).(error)
if !hasErr {
err = nil
}
err, _ := args.Get(1).(error)
return args.Get(0).(io.ReadCloser), err
}
+21 -34
View File
@@ -49,15 +49,15 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
result := model.WorkflowCallResult{
Outputs: map[string]string{},
}
unlock := lockJob(job)
maps.Copy(result.Outputs, job.Outputs)
unlock()
workflowCallResult[jobName] = &result
}
}
}
ghc := rc.getGithubContext(ctx)
inputs := getEvaluatorInputs(rc, rc.actionInputs, ghc)
ee := &exprparser.EvaluationEnvironment{
Github: ghc,
Env: env,
@@ -65,13 +65,13 @@ func (rc *RunContext) NewExpressionEvaluatorWithEnv(ctx context.Context, env map
Jobs: &workflowCallResult,
// todo: should be unavailable
// but required to interpolate/evaluate the step outputs on the job
Steps: rc.getStepsContext(),
Steps: rc.StepResults,
Secrets: getWorkflowSecrets(rc),
Vars: getWorkflowVars(ctx, rc),
Vars: rc.Config.Vars,
Strategy: strategy,
Matrix: rc.Matrix,
Needs: exprparser.NeedsContext(rc.Run),
Inputs: inputs,
Inputs: getEvaluatorInputs(rc, rc.actionInputs, ghc),
HashFiles: getHashFilesFunction(ctx, rc),
}
ee.Runner = rc.getRunnerContext(ctx)
@@ -103,9 +103,9 @@ func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step,
Github: step.getGithubContext(ctx),
Env: *step.getEnv(),
Job: jobContext,
Steps: rc.getStepsContext(),
Steps: rc.StepResults,
Secrets: getWorkflowSecrets(rc),
Vars: getWorkflowVars(ctx, rc),
Vars: rc.Config.Vars,
Strategy: exprparser.StrategyContext(rc.Run.Job().Strategy, rc.jobIndex, rc.jobTotal),
Matrix: rc.Matrix,
Needs: exprparser.NeedsContext(rc.Run),
@@ -125,7 +125,7 @@ func (rc *RunContext) newStepExpressionEvaluator(ctx context.Context, step step,
}
func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.Value) (any, error) {
hashFiles := func(v []reflect.Value) (any, error) {
return func(v []reflect.Value) (any, error) {
if rc.JobContainer != nil {
timeed, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
@@ -137,14 +137,12 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
for i, p := range v {
s := p.String()
if i == 0 {
if strings.HasPrefix(s, "--") {
if strings.EqualFold(s, "--follow-symbolic-links") {
followSymlink = true
continue
}
return "", fmt.Errorf("invalid glob option %s, available option: '--follow-symbolic-links'", s)
if i == 0 && strings.HasPrefix(s, "--") {
if strings.EqualFold(s, "--follow-symbolic-links") {
followSymlink = true
continue
}
return "", fmt.Errorf("invalid glob option %s, available option: '--follow-symbolic-links'", s)
}
patterns = append(patterns, s)
}
@@ -167,20 +165,14 @@ func getHashFilesFunction(ctx context.Context, rc *RunContext) func(v []reflect.
rc.JobContainer.ReplaceLogWriter(stdout, stderr)
return nil
})(timeed)
output := hout.String() + "\n" + herr.String()
guard := "__OUTPUT__"
outstart := strings.Index(output, guard)
if outstart != -1 {
outstart += len(guard)
outend := strings.Index(output[outstart:], guard)
if outend != -1 {
return output[outstart : outstart+outend], nil
if _, output, found := strings.Cut(hout.String()+"\n"+herr.String(), "__OUTPUT__"); found {
if hash, _, found := strings.Cut(output, "__OUTPUT__"); found {
return hash, nil
}
}
}
return "", nil
}
return hashFiles
}
type expressionEvaluator struct {
@@ -189,14 +181,15 @@ type expressionEvaluator struct {
type ExpressionEvaluator = expressionEvaluator
var addMaskExpression = regexp.MustCompile(`::add-mask::.*`)
func (ee expressionEvaluator) evaluate(ctx context.Context, in string, defaultStatusCheck exprparser.DefaultStatusCheck) (any, error) {
logger := common.Logger(ctx)
logger.Debugf("evaluating expression '%s'", in)
evaluated, err := ee.interpreter.Evaluate(in, defaultStatusCheck)
// evaluated is an any: %t renders everything but a bool as "%!t(string=...)"
printable := regexp.MustCompile(`::add-mask::.*`).ReplaceAllString(fmt.Sprintf("%v", evaluated), "::add-mask::***)")
logger.Debugf("expression '%s' evaluated to '%s'", in, printable)
logger.Debugf("expression '%s' evaluated to '%s'", in, addMaskExpression.ReplaceAllString(fmt.Sprintf("%v", evaluated), "::add-mask::***)"))
return evaluated, err
}
@@ -249,15 +242,13 @@ func getEvaluatorInputs(rc *RunContext, stepInputs map[string]any, ghc *model.Gi
if stepInputs != nil { // an action's own inputs, which workflow inputs must not shadow
return maps.Clone(stepInputs)
}
inputs := map[string]any{}
// only a called workflow's job itself takes the inputs Gitea resolved,
// composite actions copy its Config but have their own `inputs`
if rc.Config.WorkflowCallInputs != nil && rc.Parent == nil {
maps.Copy(inputs, rc.Config.WorkflowCallInputs)
return inputs
return maps.Clone(rc.Config.WorkflowCallInputs)
}
inputs := map[string]any{}
maps.Copy(inputs, rc.workflowCallInputs)
switch ghc.EventName {
@@ -327,7 +318,3 @@ func getWorkflowSecrets(rc *RunContext) map[string]string {
}
return rc.Config.Secrets
}
func getWorkflowVars(_ context.Context, rc *RunContext) map[string]string {
return rc.Config.Vars
}
+91 -105
View File
@@ -6,6 +6,7 @@ package runner
import (
"strings"
"sync"
"testing"
"gitea.dev/actionslib/pkg/exprparser"
@@ -84,71 +85,61 @@ func TestEvaluateRunContext(t *testing.T) {
ee := rc.NewExpressionEvaluator(t.Context())
tables := []struct {
in string
out any
errMesg string
in string
out any
}{
{" 1 ", 1.0, ""},
// {"1 + 3", "4", ""},
// {"(1 + 3) * -2", "-8", ""},
{"'my text'", "my text", ""},
{"contains('my text', 'te')", true, ""},
{"contains('my TEXT', 'te')", true, ""},
{"contains(fromJSON('[\"my text\"]'), 'te')", false, ""},
{"contains(fromJSON('[\"foo\",\"bar\"]'), 'bar')", true, ""},
{"startsWith('hello world', 'He')", true, ""},
{"endsWith('hello world', 'ld')", true, ""},
{"format('0:{0} 2:{2} 1:{1}', 'zero', 'one', 'two')", "0:zero 2:two 1:one", ""},
{"join(fromJSON('[\"hello\"]'),'octocat')", "hello", ""},
{"join(fromJSON('[\"hello\",\"mona\",\"the\"]'),'octocat')", "hellooctocatmonaoctocatthe", ""},
{"join('hello','mona')", "hello", ""},
{"toJSON(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}", ""},
{"toJson(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}", ""},
{"(fromJSON('{\"foo\":\"bar\"}')).foo", "bar", ""},
{"(fromJson('{\"foo\":\"bar\"}')).foo", "bar", ""},
{"(fromJson('[\"foo\",\"bar\"]'))[1]", "bar", ""},
// github does return an empty string for non-existent files
{"hashFiles('**/non-extant-files')", "", ""},
{"hashFiles('**/non-extant-files', '**/more-non-extant-files')", "", ""},
{"hashFiles('**/non.extant.files')", "", ""},
{"hashFiles('**/non''extant''files')", "", ""},
{"success()", true, ""},
{"failure()", false, ""},
{"always()", true, ""},
{"cancelled()", false, ""},
{"github.workflow", "test-workflow", ""},
{"github.actor", "nektos/act", ""},
{"github.run_id", "1", ""},
{"github.run_number", "1", ""},
{"job.status", "success", ""},
{"matrix.os", "Linux", ""},
{"matrix.foo", "bar", ""},
{"env.key", "value", ""},
{"secrets.CASE_INSENSITIVE_SECRET", "value", ""},
{"secrets.case_insensitive_secret", "value", ""},
{"vars.CASE_INSENSITIVE_VAR", "value", ""},
{"vars.case_insensitive_var", "value", ""},
{"format('{{0}}', 'test')", "{0}", ""},
{"format('{{{0}}}', 'test')", "{test}", ""},
{"format('}}')", "}", ""},
{"format('echo Hello {0} ${{Test}}', 'World')", "echo Hello World ${Test}", ""},
{"format('echo Hello {0} ${{Test}}', github.undefined_property)", "echo Hello ${Test}", ""},
{"format('echo Hello {0}{1} ${{Te{0}st}}', github.undefined_property, 'World')", "echo Hello World ${Test}", ""},
{"format('{0}', '{1}', 'World')", "{1}", ""},
{"format('{{{0}', '{1}', 'World')", "{{1}", ""},
{" 1 ", 1.0},
{"'my text'", "my text"},
{"contains('my text', 'te')", true},
{"contains('my TEXT', 'te')", true},
{"contains(fromJSON('[\"my text\"]'), 'te')", false},
{"contains(fromJSON('[\"foo\",\"bar\"]'), 'bar')", true},
{"startsWith('hello world', 'He')", true},
{"endsWith('hello world', 'ld')", true},
{"format('0:{0} 2:{2} 1:{1}', 'zero', 'one', 'two')", "0:zero 2:two 1:one"},
{"join(fromJSON('[\"hello\"]'),'octocat')", "hello"},
{"join(fromJSON('[\"hello\",\"mona\",\"the\"]'),'octocat')", "hellooctocatmonaoctocatthe"},
{"join('hello','mona')", "hello"},
{"toJSON(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}"},
{"toJson(env)", "{\n \"ACT\": \"true\",\n \"ACT_SKIP_CHECKOUT\": \"true\",\n \"key\": \"value\"\n}"},
{"(fromJSON('{\"foo\":\"bar\"}')).foo", "bar"},
{"(fromJson('{\"foo\":\"bar\"}')).foo", "bar"},
{"(fromJson('[\"foo\",\"bar\"]'))[1]", "bar"},
{"hashFiles('**/non-extant-files')", ""},
{"hashFiles('**/non-extant-files', '**/more-non-extant-files')", ""},
{"hashFiles('**/non.extant.files')", ""},
{"hashFiles('**/non''extant''files')", ""},
{"success()", true},
{"failure()", false},
{"always()", true},
{"cancelled()", false},
{"github.workflow", "test-workflow"},
{"github.actor", "nektos/act"},
{"github.run_id", "1"},
{"github.run_number", "1"},
{"job.status", "success"},
{"matrix.os", "Linux"},
{"matrix.foo", "bar"},
{"env.key", "value"},
{"secrets.CASE_INSENSITIVE_SECRET", "value"},
{"secrets.case_insensitive_secret", "value"},
{"vars.CASE_INSENSITIVE_VAR", "value"},
{"vars.case_insensitive_var", "value"},
{"format('{{0}}', 'test')", "{0}"},
{"format('{{{0}}}', 'test')", "{test}"},
{"format('}}')", "}"},
{"format('echo Hello {0} ${{Test}}', 'World')", "echo Hello World ${Test}"},
{"format('echo Hello {0} ${{Test}}', github.undefined_property)", "echo Hello ${Test}"},
{"format('echo Hello {0}{1} ${{Te{0}st}}', github.undefined_property, 'World')", "echo Hello World ${Test}"},
{"format('{0}', '{1}', 'World')", "{1}"},
{"format('{{{0}', '{1}', 'World')", "{{1}"},
}
for _, table := range tables {
t.Run(table.in, func(t *testing.T) {
assertObject := assert.New(t)
out, err := ee.evaluate(t.Context(), table.in, exprparser.DefaultStatusCheckNone)
if table.errMesg == "" {
assertObject.NoError(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.out, out, table.in)
} else {
assertObject.Error(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.errMesg, err.Error(), table.in)
}
require.NoError(t, err)
assert.Equal(t, table.out, out)
})
}
}
@@ -164,33 +155,26 @@ func TestEvaluateStep(t *testing.T) {
ee := rc.NewStepExpressionEvaluator(t.Context(), step)
tables := []struct {
in string
out any
errMesg string
in string
out any
}{
{"steps.idwithnothing.conclusion", model.StepStatusSuccess.String(), ""},
{"steps.idwithnothing.outcome", model.StepStatusFailure.String(), ""},
{"steps.idwithnothing.outputs.foowithnothing", "barwithnothing", ""},
{"steps.id-with-hyphens.conclusion", model.StepStatusSuccess.String(), ""},
{"steps.id-with-hyphens.outcome", model.StepStatusFailure.String(), ""},
{"steps.id-with-hyphens.outputs.foo-with-hyphens", "bar-with-hyphens", ""},
{"steps.id_with_underscores.conclusion", model.StepStatusSuccess.String(), ""},
{"steps.id_with_underscores.outcome", model.StepStatusFailure.String(), ""},
{"steps.id_with_underscores.outputs.foo_with_underscores", "bar_with_underscores", ""},
{"inputs.forged", nil, ""}, // INPUT_* env is not an input
{"steps.idwithnothing.conclusion", model.StepStatusSuccess.String()},
{"steps.idwithnothing.outcome", model.StepStatusFailure.String()},
{"steps.idwithnothing.outputs.foowithnothing", "barwithnothing"},
{"steps.id-with-hyphens.conclusion", model.StepStatusSuccess.String()},
{"steps.id-with-hyphens.outcome", model.StepStatusFailure.String()},
{"steps.id-with-hyphens.outputs.foo-with-hyphens", "bar-with-hyphens"},
{"steps.id_with_underscores.conclusion", model.StepStatusSuccess.String()},
{"steps.id_with_underscores.outcome", model.StepStatusFailure.String()},
{"steps.id_with_underscores.outputs.foo_with_underscores", "bar_with_underscores"},
{"inputs.forged", nil},
}
for _, table := range tables {
t.Run(table.in, func(t *testing.T) {
assertObject := assert.New(t)
out, err := ee.evaluate(t.Context(), table.in, exprparser.DefaultStatusCheckNone)
if table.errMesg == "" {
assertObject.NoError(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.out, out, table.in)
} else {
assertObject.Error(err, table.in) //nolint:testifylint // pre-existing issue from nektos/act
assertObject.Equal(table.errMesg, err.Error(), table.in)
}
require.NoError(t, err)
assert.Equal(t, table.out, out)
})
}
}
@@ -282,10 +266,8 @@ func TestInterpolate(t *testing.T) {
}
func TestGetEvaluatorInputsBoolean(t *testing.T) {
workflows := map[string]string{
"workflow_call": `
on:
workflow_call:
workflow := func(event string) string {
return "on:\n " + event + `:
inputs:
flag:
type: boolean
@@ -295,20 +277,7 @@ on:
default: gitea
unset:
type: boolean
`,
"workflow_dispatch": `
on:
workflow_dispatch:
inputs:
flag:
type: boolean
default: true
name:
type: string
default: gitea
unset:
type: boolean
`,
`
}
tables := []struct {
@@ -317,7 +286,6 @@ on:
flag any
}{
{
// Gitea >= 1.27 resolves the inputs server-side and sends native JSON types
name: "native bool true",
event: map[string]any{"inputs": map[string]any{"flag": true}},
flag: true,
@@ -344,10 +312,10 @@ on:
},
}
for eventName, workflow := range workflows {
for _, eventName := range []string{"workflow_call", "workflow_dispatch"} {
for _, table := range tables {
t.Run(eventName+"/"+table.name, func(t *testing.T) {
wf, err := model.ReadWorkflow(strings.NewReader(workflow))
wf, err := model.ReadWorkflow(strings.NewReader(workflow(eventName)))
require.NoError(t, err)
rc := &RunContext{
@@ -365,7 +333,7 @@ on:
}
t.Run("deferred workflow call inputs", func(t *testing.T) {
workflow, err := model.ReadWorkflow(strings.NewReader(workflows["workflow_call"] + `
workflow, err := model.ReadWorkflow(strings.NewReader(workflow("workflow_call") + `
jobs:
test: {}
call:
@@ -405,7 +373,6 @@ on:
config := &Config{Workdir: ".", WorkflowCallInputs: map[string]any{"name": "caller", "target": "dispatch"}}
ghc := &model.GithubContext{EventName: "workflow_dispatch", Event: map[string]any{"inputs": map[string]any{"name": "dispatch"}}}
// the event's own inputs must not override those Gitea resolved
job := &RunContext{Config: config, Run: &model.Run{JobID: "job1", Workflow: wf}}
assert.Equal(t, map[string]any{"name": "caller", "target": "dispatch"}, getEvaluatorInputs(job, nil, ghc))
@@ -497,6 +464,25 @@ jobs:
}
a, b := containerName("a"), containerName("b")
assert.NotContains(t, a, "s3cr3t") // it reaches the container name, which no log masker covers
assert.NotEqual(t, a, b) // masking the name must not collapse two jobs onto one container
assert.NotContains(t, a, "s3cr3t")
assert.NotEqual(t, a, b)
}
func TestNewExpressionEvaluatorReadsJobOutputsWhileCombinationsWriteThem(t *testing.T) {
rc := createRunContext(t)
rc.caller = &caller{runContext: createRunContext(t)}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
job := rc.Run.Job()
var writes sync.WaitGroup
writes.Go(func() {
for range 100 {
unlock := lockJob(job)
job.Outputs = map[string]string{"out": "value"}
unlock()
}
})
for range 100 {
rc.NewExpressionEvaluator(t.Context())
}
writes.Wait()
}
+43 -32
View File
@@ -7,6 +7,7 @@ package runner
import (
"archive/tar"
"bytes"
"cmp"
"context"
"encoding/base64"
"encoding/json/v2"
@@ -19,10 +20,12 @@ import (
"slices"
"strconv"
"strings"
"sync"
"time"
"unicode"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/container"
"gitea.dev/actionslib/pkg/exprparser"
@@ -95,6 +98,39 @@ func printPrepareActions(rc *RunContext, preparers []actionPreparer) common.Exec
rawLogger := common.Logger(ctx).With(rawOutputField, true)
rawLogger.Infof("Prepare all required actions")
fetchCtx, cancelFetches := context.WithCancel(ctx)
var fetches sync.WaitGroup
defer func() { cancelFetches(); fetches.Wait() }()
slots := make(chan struct{}, 4)
job := rc.topLevelRunContext()
if job.actionDownloads == nil {
job.actionDownloads = map[string]*actionDownload{}
}
for _, preparer := range preparers {
action, ok := preparer.(*stepActionRemote)
if !ok || strings.Contains(action.Step.Uses, "${{") || action.action != nil {
continue
}
github := action.getGithubContext(common.WithLogger(ctx, logging.New(logging.DiscardHandler)))
remote, err := newRemoteAction(action.Step.Uses, github, action.RunContext.Config.GitHubInstance)
if err != nil || action.skipsLocalCheckout(remote, github) {
continue
}
key := remote.downloadKey(action.RunContext.Config.DefaultActionURL())
if job.actionDownloads[key] != nil {
continue
}
download := &actionDownload{fetched: make(chan struct{})}
job.actionDownloads[key] = download
fetches.Go(func() {
defer close(download.fetched)
defer common.RecoverPanic(&download.err)
slots <- struct{}{}
defer func() { <-slots }()
download.dir, download.sha, download.err = action.fetchAction(fetchCtx, remote, github.Token)
})
}
for _, preparer := range preparers {
if err := preparer.prepareActionExecutor()(ctx); err != nil {
// No step has run yet, so the failure belongs to the job.
@@ -120,10 +156,7 @@ func (e setupError) Unwrap() error { return e.error }
func printCompleteJobName(rc *RunContext) common.Executor {
return func(ctx context.Context) error {
// Name holds a matrix combination; JobName is the shared name GitHub reports.
name := rc.JobName
if name == "" {
name = rc.Name
}
name := cmp.Or(rc.JobName, rc.Name)
if name == "" && rc.Run != nil {
name = rc.Run.JobID
}
@@ -323,10 +356,7 @@ func newStepsExecutor(rc *RunContext, preSteps, steps []common.Executor) common.
return preErr // as on GitHub, no step runs after the job setup failed
}
mainErr := mainExecutor(ctx)
if preErr != nil {
return preErr
}
return mainErr
return cmp.Or(preErr, mainErr)
}
}
@@ -512,8 +542,7 @@ func tryUploadJobSummary(ctx context.Context, rc *RunContext) {
ctx, cancel := context.WithTimeout(ctx, jobSummaryUploadPhaseTimeout)
defer cancel()
env := rc.GetEnv()
caps := strings.TrimSpace(env["GITEA_ACTIONS_CAPABILITIES"])
if !hasJobSummaryCapability(caps) {
if !hasJobSummaryCapability(env["GITEA_ACTIONS_CAPABILITIES"]) {
// Server did not advertise support. Do not attempt upload.
return
}
@@ -646,19 +675,18 @@ func readSingleFileFromContainerArchive(ctx context.Context, env container.Execu
}
defer rc.Close()
requestedPath := path.Clean(strings.TrimPrefix(p, "/"))
tr := tar.NewReader(rc)
for {
header, err := tr.Next()
if err == io.EOF {
return nil, false
}
if err != nil {
return nil, false
}
if header.Typeflag != tar.TypeReg {
continue
}
if !archiveEntryMatchesPath(header.Name, p) {
entryName := path.Clean(strings.TrimPrefix(header.Name, "/"))
if entryName != requestedPath && entryName != path.Base(requestedPath) {
continue
}
// Summaries larger than the limit are truncated rather than dropped, so the
@@ -681,12 +709,6 @@ func readSingleFileFromContainerArchive(ctx context.Context, env container.Execu
}
}
func archiveEntryMatchesPath(entryName, requestedPath string) bool {
entryName = path.Clean(strings.TrimPrefix(entryName, "/"))
requestedPath = path.Clean(strings.TrimPrefix(requestedPath, "/"))
return entryName == requestedPath || entryName == path.Base(requestedPath)
}
// useJobStageLogger tags the log entries of a job level executor with the stage they belong to.
func useJobStageLogger(stage string, executor common.Executor) common.Executor {
return func(ctx context.Context) error {
@@ -697,17 +719,6 @@ func useJobStageLogger(stage string, executor common.Executor) common.Executor {
func useStepLogger(rc *RunContext, stepModel *model.Step, stage stepStage, executor common.Executor) common.Executor {
return func(ctx context.Context) error {
ctx = withStepLogger(ctx, stepModel.Number, stepModel.ID, rc.ExprEval.InterpolateName(ctx, stepModel.String()), stage.String())
logWriter := rc.commandLogWriter(ctx)
oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter)
defer rc.JobContainer.ReplaceLogWriter(oldout, olderr)
// Flush any buffered, not-yet-newline-terminated trailing line once the
// step has finished, so the final line of the step's output is not lost
// when it is not newline-terminated.
defer common.FlushWriter(logWriter)
return executor(ctx)
return rc.withCommandLogWriter(ctx, executor)
}
}
+204 -263
View File
@@ -14,13 +14,17 @@ import (
"io"
"net/http"
"net/http/httptest"
"path/filepath"
"slices"
"strconv"
"strings"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/git"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
@@ -118,14 +122,13 @@ func (sfm *stepFactoryMock) newStep(model *model.Step, rc *RunContext) (step, er
type actionPreparerMock struct {
reference string
sha string
err error
prepared int
}
func (apm *actionPreparerMock) prepareActionExecutor() common.Executor {
return func(context.Context) error {
apm.prepared++
return apm.err
return nil
}
}
@@ -133,24 +136,85 @@ func (apm *actionPreparerMock) actionDownloadInfo() (string, string) {
return apm.reference, apm.sha
}
func TestPrintPrepareActionsGolden(t *testing.T) {
buf := &bytes.Buffer{}
logger := logging.New(logging.NewLevelHandler(newJobLogHandler(buf, cyan), logging.LevelInfo))
ctx := common.WithLogger(t.Context(), logger.With("job", "j1"))
func TestPrintPrepareActionsFetchesConcurrently(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
buf := &bytes.Buffer{}
ctx := common.WithLogger(t.Context(), logging.New(logging.NewLevelHandler(newJobLogHandler(buf, cyan), logging.LevelInfo)).With("job", "j1"))
rc := newPrefetchRunContext(t)
started, release := make(chan string), make(chan struct{})
var fetched atomic.Int32
setFetch(t, func(_ctx context.Context, input git.NewGitCloneExecutorInput) (string, string, error) {
fetched.Add(1)
started <- input.Ref
<-release
return filepath.Join(input.Dir, "commit"), "sha-" + input.Ref, nil
})
preparers := append(newRemotePreparers(rc, "org/repo@v1", "org/repo/sub@v1", "org/repo@v2", "org/repo@v3", "org/repo@v4", "org/repo@v5"), &actionPreparerMock{})
finished := make(chan error)
go func() { finished <- printPrepareActions(rc, preparers)(ctx) }()
for range 4 {
<-started
}
synctest.Wait()
assert.EqualValues(t, 4, fetched.Load())
close(release)
<-started
require.NoError(t, <-finished)
assert.EqualValues(t, 5, fetched.Load())
lines := []string{"[j1] | Prepare all required actions"}
for _, ref := range []string{"v1", "v2", "v3", "v4", "v5"} {
lines = append(lines, fmt.Sprintf("[j1] | Download action repository 'org/repo@%s' (SHA:sha-%s)", ref, ref))
}
assert.Equal(t, strings.Join(append(lines, ""), "\n"), buf.String())
})
}
preparers := []actionPreparer{
&actionPreparerMock{reference: "actions/checkout@v7", sha: "9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0"},
// A step that downloads nothing, such as the checkout of the workflow's own repository.
&actionPreparerMock{},
func TestPrintPrepareActionsFailsJobAndCancelsLaterDownloads(t *testing.T) {
for name, panicFetch := range map[string]bool{"error": false, "panic": true} {
t.Run(name, func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
rc := newPrefetchRunContext(t)
setFetch(t, func(ctx context.Context, input git.NewGitCloneExecutorInput) (string, string, error) {
if input.Ref == "v1" {
if panicFetch {
panic("broken archive")
}
return "", "", assert.AnError
}
<-ctx.Done()
return "", "", ctx.Err()
})
ctx := common.WithJobErrorContainer(common.WithLogger(t.Context(), logging.New(logging.DiscardHandler)))
remaining := &actionPreparerMock{}
err := printPrepareActions(rc, append(newRemotePreparers(rc, "org/repo@v1", "org/repo@v2"), remaining))(ctx)
if panicFetch {
require.ErrorContains(t, err, "panic: broken archive")
} else {
require.ErrorIs(t, err, assert.AnError)
}
require.ErrorIs(t, err, common.JobError(ctx))
assert.True(t, rc.jobFailed)
assert.Zero(t, remaining.prepared)
})
})
}
require.NoError(t, printPrepareActions(&RunContext{}, preparers)(ctx))
}
want := strings.Join([]string{
"[j1] | Prepare all required actions",
"[j1] | Download action repository 'actions/checkout@v7' (SHA:9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0)",
"",
}, "\n")
assert.Equal(t, want, buf.String())
func newPrefetchRunContext(t *testing.T) *RunContext {
return &RunContext{
Config: &Config{ActionCacheDir: t.TempDir(), PresetGitHubContext: &model.GithubContext{}},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
}
}
func newRemotePreparers(rc *RunContext, uses ...string) []actionPreparer {
preparers := make([]actionPreparer, 0, len(uses))
for _, use := range uses {
preparers = append(preparers, &stepActionRemote{Step: &model.Step{Uses: use}, RunContext: rc, readAction: func(_ctx context.Context, _step *model.Step, _actionDir, _actionPath string, _readFile actionYamlReader, _writeFile fileWriter) (*model.Action, error) {
return &model.Action{}, nil
}})
}
return preparers
}
func TestPrintPrepareActionsSkipsWithoutActions(t *testing.T) {
@@ -163,26 +227,6 @@ func TestPrintPrepareActionsSkipsWithoutActions(t *testing.T) {
assert.Empty(t, buf.String())
}
func TestPrintPrepareActionsFailsJobOnDownloadError(t *testing.T) {
logger := logging.New(logtest.New())
ctx := common.WithJobErrorContainer(common.WithLogger(t.Context(), logger.With("job", "j1")))
downloadErr := errors.New("failed to fetch \"actions/checkout\"")
rc := &RunContext{}
remaining := &actionPreparerMock{reference: "actions/setup-go@v6", sha: "sha"}
err := printPrepareActions(rc, []actionPreparer{
&actionPreparerMock{err: downloadErr},
remaining,
})(ctx)
require.ErrorIs(t, err, downloadErr)
// No step has run yet, so the failure has to be recorded against the job itself.
assert.Equal(t, downloadErr, common.JobError(ctx))
assert.True(t, rc.jobFailed)
assert.Zero(t, remaining.prepared)
}
func TestNewStepsExecutorSkipsStepsAfterSetupFailure(t *testing.T) {
for name, tt := range map[string]struct {
preErr error
@@ -320,6 +364,7 @@ func TestNewJobExecutor(t *testing.T) {
output string
startError error
cancelOnStart bool
timeout bool
}{
{
name: "zeroSteps",
@@ -470,17 +515,28 @@ func TestNewJobExecutor(t *testing.T) {
result: "cancelled",
cancelOnStart: true,
},
}
contains := func(needle string, haystack []string) bool {
return slices.Contains(haystack, needle)
{
name: "post steps run after timeout",
steps: []*model.Step{{ID: "1"}},
preSteps: []bool{false},
postSteps: []bool{true},
executedSteps: []string{"startContainer", "step1", "post1", "interpolateOutputs", "stopContainer", "closeContainer"},
result: "failure",
timeout: true,
},
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
fmt.Printf("::group::%s\n", tt.name) //nolint:forbidigo // pre-existing issue from nektos/act
ctx, cancel := context.WithCancel(common.WithJobErrorContainer(t.Context()))
ctx := common.WithJobErrorContainer(t.Context())
cancel := func() {}
var expire func()
if tt.timeout {
deadline := newControllableDeadlineContext(ctx)
ctx, expire = deadline, deadline.expire
} else {
ctx, cancel = context.WithCancel(ctx)
}
defer cancel()
jim := &jobInfoMock{}
sfm := &stepFactoryMock{}
@@ -519,7 +575,7 @@ func TestNewJobExecutor(t *testing.T) {
sfm.On("newStep", stepModel, rc).Return(sm, nil)
sm.On("pre").Return(func(ctx context.Context) error {
sm.On("pre").Return(func(_ctx context.Context) error {
if tt.preSteps[i] {
executorOrder = append(executorOrder, "pre"+stepModel.ID)
}
@@ -528,6 +584,10 @@ func TestNewJobExecutor(t *testing.T) {
sm.On("main").Return(func(ctx context.Context) error {
executorOrder = append(executorOrder, "step"+stepModel.ID)
if tt.timeout {
expire()
return ctx.Err()
}
if tt.hasError {
return errors.New("error")
}
@@ -538,6 +598,9 @@ func TestNewJobExecutor(t *testing.T) {
if tt.postSteps[i] {
executorOrder = append(executorOrder, "post"+stepModel.ID)
}
if tt.timeout {
require.NoError(t, ctx.Err(), "post step must run against a fresh, non-expired context")
}
return nil
})
@@ -555,7 +618,7 @@ func TestNewJobExecutor(t *testing.T) {
return nil
})
if contains("stopContainer", tt.executedSteps) {
if slices.Contains(tt.executedSteps, "stopContainer") {
jim.On("stopContainer").Return(func(ctx context.Context) error {
executorOrder = append(executorOrder, "stopContainer")
require.NoError(t, ctx.Err())
@@ -569,7 +632,7 @@ func TestNewJobExecutor(t *testing.T) {
}
if len(tt.steps) > 0 {
jim.On("closeContainer").Return(func(ctx context.Context) error {
jim.On("closeContainer").Return(func(_ctx context.Context) error {
executorOrder = append(executorOrder, "closeContainer")
return nil
})
@@ -590,8 +653,6 @@ func TestNewJobExecutor(t *testing.T) {
jim.AssertExpectations(t)
sfm.AssertExpectations(t)
fmt.Println("::endgroup::") //nolint:forbidigo // pre-existing issue from nektos/act
})
}
}
@@ -622,123 +683,31 @@ func (ctx *controllableDeadlineContext) expire() {
close(ctx.done)
}
// TestNewJobExecutorRunsPostStepsAfterTimeout guards the timeout-minutes cleanup
// path: when a job exceeds its timeout the job context is DeadlineExceeded, but
// the post steps (cleanup hooks like actions/checkout post and cache save) must
// still run against a fresh, non-expired context, and the job must still be
// reported as failed.
func TestNewJobExecutorRunsPostStepsAfterTimeout(t *testing.T) {
ctx := newControllableDeadlineContext(common.WithJobErrorContainer(t.Context()))
jim := &jobInfoMock{}
sfm := &stepFactoryMock{}
rc := &RunContext{
JobContainer: &jobContainerMock{},
Run: &model.Run{
JobID: "test",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"test": {},
},
},
},
Config: &Config{},
}
rc.ExprEval = rc.NewExpressionEvaluator(ctx)
stepModel := &model.Step{ID: "1"}
jim.On("steps").Return([]*model.Step{stepModel})
jim.On("matrix").Return(map[string]any{})
jim.On("startContainer").Return(func(ctx context.Context) error { return nil })
jim.On("interpolateOutputs").Return(func(ctx context.Context) error { return nil })
jim.On("closeContainer").Return(func(ctx context.Context) error { return nil })
// The job timed out, so it must be reported as failed and still cleaned up.
jim.On("stopContainer").Return(func(context.Context) error { return nil })
jim.On("result", "failure")
sm := &stepMock{}
sfm.On("newStep", stepModel, rc).Return(sm, nil)
sm.On("pre").Return(func(ctx context.Context) error { return nil })
sm.On("main").Return(func(stepCtx context.Context) error {
ctx.expire()
return stepCtx.Err()
})
var postRan bool
var postCtxErr error
sm.On("post").Return(func(ctx context.Context) error {
postRan = true
postCtxErr = ctx.Err()
return nil
})
executor := newJobExecutor(jim, sfm, rc)
// The executor itself returns nil on timeout: the failure is surfaced through
// the job result ("failure", asserted via the result mock below), not the
// return value.
require.NoError(t, executor(ctx))
assert.True(t, postRan, "post step must run after a job timeout")
require.NoError(t, postCtxErr, "post step must run against a fresh, non-expired context")
jim.AssertExpectations(t)
sfm.AssertExpectations(t)
sm.AssertExpectations(t)
}
// TestSetJobResultMatrixContinueOnError exercises the parallel-matrix path
// end-to-end: two combinations share one *model.Job and continue-on-error is
// keyed on matrix.experimental, so one combination tolerates its failure and the
// other does not. The job is reported as continue-on-error only when EVERY failing
// combination was tolerated; a single firm failure makes the whole job firm, and
// handleFailure then fails the run.
func TestSetJobResultMatrixContinueOnError(t *testing.T) {
const jobYAML = "continue-on-error: ${{ matrix.experimental }}\nruns-on: ubuntu-latest"
newSharedJob := func(t *testing.T) (*model.Job, *model.Workflow) {
t.Helper()
var job *model.Job
require.NoError(t, yaml.Unmarshal([]byte(jobYAML), &job))
return job, &model.Workflow{
Name: "workflow1",
Jobs: map[string]*model.Job{"job1": job},
}
}
planFor := func(wf *model.Workflow) *model.Plan {
return &model.Plan{Stages: []*model.Stage{{Runs: []*model.Run{{Workflow: wf, JobID: "job1"}}}}}
}
ctx := t.Context()
for name, tolerated := range map[string]bool{
"one tolerated and one firm failure fails the run": false,
"all tolerated failures do not fail the run": true,
} {
t.Run(name, func(t *testing.T) {
var job *model.Job
require.NoError(t, yaml.Unmarshal([]byte("continue-on-error: ${{ matrix.experimental }}\nruns-on: ubuntu-latest"), &job))
workflow := &model.Workflow{Name: "workflow1", Jobs: map[string]*model.Job{"job1": job}}
for _, experimental := range []bool{true, tolerated} {
rc := newTestRC(workflow, map[string]any{"experimental": experimental})
setJobResult(ctx, rc, rc, false)
}
// fail drives a single matrix combination through the failure path; each
// RunContext is its own jobInfo (rc implements jobInfo) and shares the job.
fail := func(wf *model.Workflow, experimental bool) {
rc := newTestRC(wf, map[string]any{"experimental": experimental})
setJobResult(ctx, rc, rc, false)
assert.Equal(t, "failure", job.Result)
assert.Equal(t, tolerated, job.ContinueOnError)
err := handleFailure(&model.Plan{Stages: []*model.Stage{{Runs: []*model.Run{{Workflow: workflow, JobID: "job1"}}}}})(ctx)
if tolerated {
require.NoError(t, err)
} else {
require.Error(t, err)
}
})
}
t.Run("one tolerated and one firm failure fails the run", func(t *testing.T) {
job, wf := newSharedJob(t)
// Order is intentional: the tolerated combination finishes first, then the
// firm one. The firm-failure latch must still win regardless of order.
fail(wf, true)
fail(wf, false)
assert.Equal(t, "failure", job.Result)
assert.False(t, job.ContinueOnError, "a single firm failure must make the whole job firm")
assert.Error(t, handleFailure(planFor(wf))(ctx))
})
t.Run("all tolerated failures do not fail the run", func(t *testing.T) {
job, wf := newSharedJob(t)
fail(wf, true)
fail(wf, true)
assert.Equal(t, "failure", job.Result)
assert.True(t, job.ContinueOnError, "every failing combination was tolerated")
assert.NoError(t, handleFailure(planFor(wf))(ctx))
})
}
func TestHasJobSummaryCapability(t *testing.T) {
@@ -925,44 +894,34 @@ func TestTryUploadJobSummaryUploadsEachStepIndependentlyWithInstanceTransport(t
cm.AssertExpectations(t)
}
func TestTryUploadJobSummaryRequiresExactCapability(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
func TestTryUploadJobSummarySkipsUnsupportedUploads(t *testing.T) {
for name, tt := range map[string]struct {
capabilities string
runtimeToken string
}{
"requires exact capability": {"not-job-summary,job-summary-v2", fakeRuntimeToken(34)},
"requires job ID": {"job-summary", "not-a-jwt"},
} {
t.Run(name, func(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _r *http.Request) {
requests++
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": "not-job-summary,job-summary-v2",
"ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": fakeRuntimeToken(34),
"GITEA_RUN_ID": "12",
}, &containerMock{}, 1)
rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": tt.capabilities,
"ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": tt.runtimeToken,
"GITEA_RUN_ID": "12",
}, &containerMock{}, 1)
tryUploadJobSummary(t.Context(), rc)
tryUploadJobSummary(t.Context(), rc)
assert.Equal(t, 0, requests)
}
func TestTryUploadJobSummarySkipsWhenJobIDMissingFromToken(t *testing.T) {
requests := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
rc := newJobSummaryRC(map[string]string{
"GITEA_ACTIONS_CAPABILITIES": "job-summary",
"ACTIONS_RUNTIME_URL": server.URL,
"ACTIONS_RUNTIME_TOKEN": "not-a-jwt",
"GITEA_RUN_ID": "12",
}, &containerMock{}, 1)
tryUploadJobSummary(t.Context(), rc)
assert.Equal(t, 0, requests)
assert.Zero(t, requests)
})
}
}
func TestExtractJobIDFromRuntimeToken(t *testing.T) {
@@ -991,47 +950,39 @@ func TestReadSingleFileFromContainerArchiveFindsMatchingRegularFile(t *testing.T
cm.AssertExpectations(t)
}
func TestReadSingleFileFromContainerArchiveTruncatesWhenTooLarge(t *testing.T) {
logger, hook := logtest.NewLogger()
ctx := common.WithLogger(t.Context(), logger)
cm := &containerMock{}
content := strings.Repeat("a", 300)
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "SUMMARY.md", body: content}))),
nil,
).Once()
func TestReadSingleFileFromContainerArchiveSizeLimit(t *testing.T) {
for name, tt := range map[string]struct {
content string
maxBytes int64
want string
}{
"truncates oversized summary": {strings.Repeat("a", 300), 200, strings.Repeat("a", 200-len(jobSummaryTruncationMarker)) + jobSummaryTruncationMarker},
"keeps exact limit": {"abc", 3, "abc"},
} {
t.Run(name, func(t *testing.T) {
logger, hook := logtest.NewLogger()
ctx := common.WithLogger(t.Context(), logger)
cm := &containerMock{}
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "SUMMARY.md", body: tt.content}))),
nil,
).Once()
const maxBytes = 200
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", maxBytes)
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", tt.maxBytes)
// Oversized summaries are truncated to the limit (reserving room for the marker)
// rather than dropped entirely, and the truncation marker is appended.
assert.True(t, ok)
assert.LessOrEqual(t, len(body), maxBytes)
keep := maxBytes - len(jobSummaryTruncationMarker)
assert.Equal(t, []byte(content[:keep]+jobSummaryTruncationMarker), body)
if assert.Len(t, hook.AllRecords(), 1) {
assert.Contains(t, hook.AllRecords()[0].Message, "job summary truncated")
assert.True(t, ok)
assert.Equal(t, []byte(tt.want), body)
assert.LessOrEqual(t, int64(len(body)), tt.maxBytes)
if int64(len(tt.content)) > tt.maxBytes {
if assert.Len(t, hook.AllRecords(), 1) {
assert.Contains(t, hook.LastRecord().Message, "job summary truncated")
}
} else {
assert.Empty(t, hook.AllRecords())
}
cm.AssertExpectations(t)
})
}
cm.AssertExpectations(t)
}
func TestReadSingleFileFromContainerArchiveKeepsExactLimitWithoutWarning(t *testing.T) {
logger, hook := logtest.NewLogger()
ctx := common.WithLogger(t.Context(), logger)
cm := &containerMock{}
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/SUMMARY.md").Return(
io.NopCloser(bytes.NewReader(tarArchive(t, tarEntry{name: "SUMMARY.md", body: "abc"}))),
nil,
).Once()
body, ok := readSingleFileFromContainerArchive(ctx, cm, "/var/run/act/workflow/SUMMARY.md", 3)
// A summary that is exactly at the limit is kept whole and not flagged as truncated.
assert.True(t, ok)
assert.Equal(t, []byte("abc"), body)
assert.Empty(t, hook.AllRecords())
cm.AssertExpectations(t)
}
type tarEntry struct {
@@ -1137,34 +1088,24 @@ func TestEvaluateJobContinueOnError(t *testing.T) {
}
func TestJobSetContinueOnError(t *testing.T) {
t.Run("first call true", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(true)
assert.True(t, j.ContinueOnError)
})
t.Run("first call false", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(false)
assert.False(t, j.ContinueOnError)
})
t.Run("true then false locks to false", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(true)
j.SetContinueOnError(false)
assert.False(t, j.ContinueOnError)
})
t.Run("false then true stays false", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(false)
j.SetContinueOnError(true)
assert.False(t, j.ContinueOnError)
})
t.Run("true then true stays true", func(t *testing.T) {
j := &model.Job{}
j.SetContinueOnError(true)
j.SetContinueOnError(true)
assert.True(t, j.ContinueOnError)
})
for name, tt := range map[string]struct {
values []bool
want bool
}{
"first call true": {[]bool{true}, true},
"first call false": {[]bool{false}, false},
"true then false locks to false": {[]bool{true, false}, false},
"false then true stays false": {[]bool{false, true}, false},
"true then true stays true": {[]bool{true, true}, true},
} {
t.Run(name, func(t *testing.T) {
job := &model.Job{}
for _, value := range tt.values {
job.SetContinueOnError(value)
}
assert.Equal(t, tt.want, job.ContinueOnError)
})
}
}
func TestTryUploadJobSummaryMasksSecrets(t *testing.T) {
+1 -1
View File
@@ -56,7 +56,7 @@ func (rc *RunContext) runJobHook(ctx context.Context, hookPath, name string) err
err := rc.setupHookFileCommands(ctx, env)
if err == nil {
err = rc.JobContainer.Exec(cmd, env, "", "")(ctx)
err = rc.withCommandLogWriter(ctx, rc.JobContainer.Exec(cmd, env, "", ""))
}
// Processed even on failure, so a hook that exports what it managed to set up before
// failing still hands it to the job.
+26
View File
@@ -29,6 +29,8 @@ type hookContainer struct {
err error
envFile map[string]string
pathTar []byte
output string
stdout io.Writer
}
func (c *hookContainer) ToContainerPath(path string) string { return path }
@@ -41,10 +43,21 @@ func (c *hookContainer) GetRunnerContext(context.Context) map[string]any {
func (c *hookContainer) Exec(command []string, env map[string]string, _, _ string) common.Executor {
return func(context.Context) error {
c.cmd, c.env = command, env
if c.output != "" {
if _, err := io.WriteString(c.stdout, c.output); err != nil {
return err
}
}
return c.err
}
}
func (c *hookContainer) ReplaceLogWriter(stdout, _ io.Writer) (io.Writer, io.Writer) {
old := c.stdout
c.stdout = stdout
return old, nil
}
func (c *hookContainer) UpdateFromEnv(_ string, env *map[string]string) common.Executor {
return func(context.Context) error {
maps.Copy(*env, c.envFile)
@@ -90,6 +103,19 @@ func TestRunJobHook(t *testing.T) {
assert.Equal(t, "/var/run/act/workflow/hook-path.txt", jobContainer.env["GITHUB_PATH"])
})
t.Run("each hook has its own command state", func(t *testing.T) {
jobContainer := &hookContainer{output: "::stop-commands::token77"}
rc, ctx := newHookRunContext(jobContainer, &Config{JobStartedHook: "/started.sh", JobCompletedHook: "/completed.sh"})
jobWriter := rc.commandLogWriter(ctx)
jobContainer.stdout = jobWriter
require.NoError(t, rc.runJobStartedHook(ctx))
jobContainer.output = "::add-mask::s3cret"
require.NoError(t, rc.runJobCompletedHook(ctx))
assert.Equal(t, []string{"token77", "s3cret"}, rc.Masks)
assert.Same(t, jobWriter, jobContainer.stdout)
})
// Each hook reads its own variable, so a swapped constant cannot pass.
t.Run("falls back to the GitHub environment variables", func(t *testing.T) {
for name, hook := range map[string]struct {
+7 -29
View File
@@ -51,29 +51,19 @@ const (
)
var (
colors []int
colors = []int{blue, yellow, green, magenta, red, gray, cyan}
nextColor int
mux sync.Mutex
)
func init() {
nextColor = 0
colors = []int{
blue, yellow, green, magenta, red, gray, cyan,
}
}
type masksContextKey string
const masksContextKeyVal = masksContextKey("logrus.FieldLogger")
// Logger returns the appropriate logger for current context
func Masks(ctx context.Context) *[]string {
val := ctx.Value(masksContextKeyVal)
if val != nil {
if masks, ok := val.(*[]string); ok {
return masks
}
if masks, ok := ctx.Value(masksContextKeyVal).(*[]string); ok {
return masks
}
return &[]string{}
}
@@ -178,7 +168,7 @@ var secretValueEncoders = []func(string) string{
uriDataEscape,
url.QueryEscape, // the form-encoded twin of uriDataEscape, which spells a space "+"
url.PathEscape,
xmlDataEscape,
xmlDataEscaper.Replace,
trimDoubleQuotes,
}
@@ -225,10 +215,6 @@ var xmlDataEscaper = strings.NewReplacer(
"'", "&apos;",
)
func xmlDataEscape(v string) string {
return xmlDataEscaper.Replace(v)
}
func trimDoubleQuotes(v string) string {
if len(v) > 8 && strings.HasPrefix(v, `"`) && strings.HasSuffix(v, `"`) {
return v[1 : len(v)-1]
@@ -407,7 +393,7 @@ func (f textFormatter) format(r logging.Record) ([]byte, error) {
b := &bytes.Buffer{}
// the web renderer decodes command data, so this local view has to as well
if _, _, _, ok := tryParseRawActionCommand(r.Message + "\n"); ok {
if _, _, _, ok := parseActionCommand(r.Message, ""); ok && (strings.HasPrefix(r.Message, "::") || strings.HasPrefix(r.Message, "##[")) {
r.Message = UnescapeCommandData(r.Message)
}
r.Message = strings.TrimSuffix(r.Message, "\n")
@@ -468,7 +454,8 @@ func (f textFormatter) print(b *bytes.Buffer, r logging.Record) {
}
func (f textFormatter) isColored() bool {
isColored := checkIfTerminal(f.w)
file, ok := f.w.(*os.File)
isColored := ok && term.IsTerminal(int(file.Fd()))
if force, ok := os.LookupEnv("CLICOLOR_FORCE"); ok && force != "0" {
isColored = true
@@ -481,15 +468,6 @@ func (f textFormatter) isColored() bool {
return isColored
}
func checkIfTerminal(w io.Writer) bool {
switch v := w.(type) {
case *os.File:
return term.IsTerminal(int(v.Fd()))
default:
return false
}
}
// maskSecrets hides this job's secrets in a value that reaches somewhere the log maskers cannot,
// such as a container name or a job summary. Masks added at runtime count, so a summary written
// after ::add-mask:: is covered too.
+11 -45
View File
@@ -14,59 +14,25 @@ import (
)
func TestMaxParallelStrategy(t *testing.T) {
tests := []struct {
for _, tt := range []struct {
name string
maxParallelString string
expectedMaxParallel int
}{
{
name: "max-parallel-1",
maxParallelString: "1",
expectedMaxParallel: 1,
},
{
name: "max-parallel-2",
maxParallelString: "2",
expectedMaxParallel: 2,
},
{
name: "max-parallel-default",
maxParallelString: "",
expectedMaxParallel: 4,
},
{
name: "max-parallel-10-clamped-to-combinations",
maxParallelString: "10",
expectedMaxParallel: 5,
},
{
name: "max-parallel-invalid-falls-back-to-default",
maxParallelString: "tow",
expectedMaxParallel: 4,
},
}
for _, tt := range tests {
{"max-parallel-1", "1", 1},
{"max-parallel-2", "2", 2},
{"max-parallel-default", "", 4},
{"max-parallel-10-clamped-to-combinations", "10", 5},
{"max-parallel-invalid-falls-back-to-default", "tow", 4},
} {
t.Run(tt.name, func(t *testing.T) {
matrix := map[string][]any{
"version": {1, 2, 3, 4, 5},
}
var rawMatrix yaml.Node
err := rawMatrix.Encode(matrix)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
job := &model.Job{
Strategy: &model.Strategy{
MaxParallelString: tt.maxParallelString,
RawMatrix: rawMatrix,
},
}
require.NoError(t, rawMatrix.Encode(map[string][]any{"version": {1, 2, 3, 4, 5}}))
job := &model.Job{Strategy: &model.Strategy{MaxParallelString: tt.maxParallelString, RawMatrix: rawMatrix}}
matrixes, err := job.GetMatrixes()
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
assert.NotNil(t, matrixes)
assert.Len(t, matrixes, 5)
require.NoError(t, err)
require.Len(t, matrixes, 5)
assert.Equal(t, tt.expectedMaxParallel, maxParallelFor(job.Strategy, len(matrixes)))
})
}
+31 -12
View File
@@ -9,7 +9,7 @@ import (
"os"
"path/filepath"
"regexp"
"strings"
"slices"
"gitea.com/gitea/runner/act/common"
@@ -40,8 +40,9 @@ const (
cacheURLEnv = "ACTIONS_CACHE_URL"
resultsURLEnv = "ACTIONS_RESULTS_URL"
// localhostHost is the suffix isGhes accepts.
// localhostHost is the suffix isGhes accepts, and gheHost the one it tests just before.
localhostHost = ".LOCALHOST"
gheHost = ".GHE.COM"
// artifactRefusal is the only thing the gate guards in @actions/artifact, which is what makes
// such a bundle safe to open. A bundle carrying neither toolkit uses isGhes for something this
@@ -57,8 +58,10 @@ var (
// testing it, while undici, bundled into all of these actions, tests a lowercase ".localhost"
// in isURLPotentiallyTrustworthy. Opening that one would tell its HTTP client that every URL
// is trustworthy. Uppercase, the literal occurs nowhere but this test, across 118 bundles
// covering every major version of sixteen actions.
localhostTest = regexp.MustCompile(`endsWith\s*\(\s*` + quoted(regexp.QuoteMeta(localhostHost)) + `\s*\)`)
// covering every major version of sixteen actions. Anchoring on the .GHE.COM test, 5 to 35
// bytes before it, keeps any other library's uppercase test out.
localhostTest = regexp.MustCompile(`endsWith\s*\(\s*` + quoted(regexp.QuoteMeta(gheHost)) + `\s*\)(?s:.{0,128}?)` +
`endsWith\s*\(\s*` + quoted(regexp.QuoteMeta(localhostHost)) + `\s*\)`)
// serviceURLBranches matches both branches of getCacheServiceURL at once: the v1 branch reads
// the cache URL and falls back to the results URL, the v2 branch just below reads the results
@@ -83,26 +86,41 @@ func quoted(pattern string) string {
return "(?:'" + pattern + "'|\"" + pattern + "\"|`" + pattern + "`)"
}
// actionScriptPaths returns the entrypoints of a node action, the only kind with a bundle. Only
// actionScriptPaths returns a node action's entrypoints and the chunks bundled beside them. Only
// remote actions get here: a local one lives in the user's checkout, which the runner does not
// rewrite.
func actionScriptPaths(dir string, action *model.Action) []string {
// rewrite. Scripts may leave dir for the rest of the checkout, as setup-gradle's ../dist does.
func actionScriptPaths(checkout, dir string, action *model.Action) []string {
if action == nil || !action.Runs.Using.IsNode() {
return nil
}
root, err := filepath.EvalSymlinks(checkout)
if err != nil {
return nil
}
var paths []string
for _, script := range []string{action.Runs.Pre, action.Runs.Main, action.Runs.Post} {
if script == "" {
continue
}
path := filepath.Join(dir, script)
// `runs` is the action's own yaml, and a key pointing outside its directory is not ours.
if rel, err := filepath.Rel(dir, path); err != nil || strings.HasPrefix(rel, "..") {
path, err := filepath.EvalSymlinks(filepath.Join(dir, script))
if err != nil {
continue
}
// `runs` is the action's own yaml, and a key resolving outside its checkout is not ours.
bundleDir := filepath.Dir(path)
if rel, err := filepath.Rel(root, bundleDir); err != nil || !filepath.IsLocal(rel) {
continue
}
paths = append(paths, path)
entries, _ := os.ReadDir(bundleDir)
for _, entry := range entries {
if entry.Type().IsRegular() && slices.Contains([]string{".js", ".cjs", ".mjs"}, filepath.Ext(entry.Name())) {
paths = append(paths, filepath.Join(bundleDir, entry.Name()))
}
}
}
return paths
slices.Sort(paths)
return slices.Compact(paths)
}
// patchActions returns a restore: the tree is shared, so a job with patching off gets no edits.
@@ -164,7 +182,8 @@ func patchedBundle(data []byte) ([]byte, bool) {
opened := localhostTest.ReplaceAllFunc(data, func(test []byte) []byte {
// Drop the hostname from the test rather than rewriting the call, so the bundle's own
// quoting survives and the result stays valid even inside a string literal.
return bytes.Replace(test, []byte(localhostHost), nil, 1)
before, after, _ := bytes.CutLast(test, []byte(localhostHost))
return slices.Concat(before, after)
})
if cache {
opened = serviceURLBranches.ReplaceAll(opened, cacheURLFirst)
+61 -91
View File
@@ -20,16 +20,13 @@ import (
"gitea.com/gitea/runner/act/artifactcache"
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// actionsCacheRef pins the actions/cache release this is verified against. Bump it
// deliberately: a new release is exactly what can stop the patch matching.
const actionsCacheRef = "v6.1.0"
// bundleFromGitHub downloads one entrypoint, keeping it in the user cache dir so repeated runs
// cost nothing. The bundles are megabytes, too large to vendor.
func bundleFromGitHub(t *testing.T, repo, ref, path string) string {
t.Helper()
@@ -52,6 +49,7 @@ func bundleFromGitHub(t *testing.T, repo, ref, path string) string {
t.Skipf("cannot reach %s: %v", url, err)
}
defer resp.Body.Close()
require.NotEqual(t, http.StatusNotFound, resp.StatusCode, "%s is not in this release", url)
if resp.StatusCode != http.StatusOK {
t.Skipf("GET %s: %s", url, resp.Status)
}
@@ -63,17 +61,28 @@ func bundleFromGitHub(t *testing.T, repo, ref, path string) string {
return bundle
}
// jobEnv is the environment a job gets from this runner, which is what decides where an action's
// toolkit looks for the cache and artifact services.
type jobEnv struct {
workspace, runnerTemp string
cacheURL, resultsURL string
token string
}
// runActionEntrypoint runs one action entrypoint the way a job would. Adding another action to these tests
// means downloading its entrypoint with bundleFromGitHub and calling this with its inputs, whose
// names are the ones the action's own action.yml uses.
func newActionJobEnv(t *testing.T, token string, credential artifactcache.JobCredential) jobEnv {
t.Helper()
handler, err := artifactcache.StartHandler(artifactcache.Options{Dir: filepath.Join(t.TempDir(), "cache"), OutboundIP: "127.0.0.1"})
require.NoError(t, err)
t.Cleanup(func() { _ = handler.Close() })
t.Cleanup(handler.RegisterJob(token, credential))
return jobEnv{
workspace: tempDirPath(t),
runnerTemp: tempDirPath(t),
cacheURL: handler.ExternalURL(),
resultsURL: handler.ExternalURL(),
token: token,
}
}
func runActionEntrypoint(t *testing.T, script string, env jobEnv, inputs map[string]string) string {
t.Helper()
state := filepath.Join(env.runnerTemp, "state")
@@ -108,7 +117,6 @@ func runActionEntrypoint(t *testing.T, script string, env jobEnv, inputs map[str
return string(out)
}
// patchedAction downloads one entrypoint and patches it exactly as a downloaded action would be.
func patchedAction(t *testing.T, repo, ref, entrypoint string) string {
t.Helper()
@@ -120,8 +128,6 @@ func patchedAction(t *testing.T, repo, ref, entrypoint string) string {
return script
}
// tempDirPath is TempDir with symlinks resolved, because macOS hands out /var paths that resolve
// to /private/var and the client derives archive paths relative to the workspace.
func tempDirPath(t *testing.T) string {
t.Helper()
@@ -130,32 +136,13 @@ func tempDirPath(t *testing.T) string {
return dir
}
// The whole chain against the pinned release, whose bundles ship unminified: patch them, run the
// real client with an ordinary Gitea server URL and a results URL that goes nowhere, and have it
// save and restore through this runner's cache server. The unreachable results URL is the point,
// it is what proves the cache reaches the runner without the runner fronting Gitea. If a release
// stops matching the patch the client falls back to v1 and this fails on the version line, which
// is the signal to look at the new bundle.
func TestCacheServiceV2EndToEnd(t *testing.T) {
requireHostTools(t, "node")
restore := patchedAction(t, "actions/cache", actionsCacheRef, "dist/restore/index.js")
save := patchedAction(t, "actions/cache", actionsCacheRef, "dist/save/index.js")
handler, err := artifactcache.StartHandler(artifactcache.Options{Dir: filepath.Join(t.TempDir(), "cache"), OutboundIP: "127.0.0.1"})
require.NoError(t, err)
t.Cleanup(func() { _ = handler.Close() })
const token, repo = "e2e-runtime-token", "testuser/testrepo"
handler.RegisterJob(token, artifactcache.JobCredential{Repo: repo})
env := jobEnv{
workspace: tempDirPath(t),
runnerTemp: tempDirPath(t),
cacheURL: handler.ExternalURL(),
// The results service is the cache server's too, which is what the runner advertises.
resultsURL: handler.ExternalURL(),
token: token,
}
env := newActionJobEnv(t, "e2e-runtime-token", artifactcache.JobCredential{Repo: "testuser/testrepo"})
require.NoError(t, os.MkdirAll(filepath.Join(env.workspace, "to-cache"), 0o755))
content := []byte("cached through the patched gate")
require.NoError(t, os.WriteFile(filepath.Join(env.workspace, "to-cache", "data.txt"), content, 0o600))
@@ -177,46 +164,33 @@ func TestCacheServiceV2EndToEnd(t *testing.T) {
got, err := os.ReadFile(filepath.Join(env.workspace, "to-cache", "data.txt"))
require.NoError(t, err)
assert.Equal(t, content, got)
// Untouched, the same client takes a Gitea host for GHES and stays on v1, which reaches the
// cache server on its own address. That is what a runner without a results service of its own
// leaves its jobs with, so it has to round trip too.
env.workspace = tempDirPath(t)
v1 := runActionEntrypoint(t, bundleFromGitHub(t, "actions/cache", actionsCacheRef, "dist/restore/index.js"), env, inputs)
require.Contains(t, v1, "Cache service version: v1")
require.Contains(t, v1, "Cache restored from key: "+key)
}
// The gate and the URL getter are separate functions, and a bundler may put either first: the gap
// between them runs from 159 to 1179 bytes across these actions, which is why neither edit is
// anchored on that distance. One entrypoint from each of the families that bundle the cache
// toolkit, patched but not run, is what keeps a future release from quietly matching only one of
// the two shapes and leaving every cache on v1.
func TestPatchedBundleAcrossActions(t *testing.T) {
for _, tc := range []struct {
repo, ref, path string
wantPatched bool
}{
// The cache toolkit, in each bundler shape and from a spread of ecosystems, including the
// actions that drive a Go or a Rust cache client of their own.
{"actions/cache", actionsCacheRef, "dist/restore/index.js", true},
{"actions/setup-node", "v7.1.0", "dist/cache-save/index.js", true},
{"actions/setup-python", "v7.0.0", "dist/setup/index.js", true},
{"actions/setup-go", "v7.0.0", "dist/setup/index.js", true},
{"actions/setup-java", "v5.7.0", "dist/setup/index.js", true},
{"actions/setup-java", "v6.1.0", "dist/setup/295.index.js", true},
{"ruby/setup-ruby", "v1.331.0", "dist/index.js", true},
{"pnpm/action-setup", "v6.1.0", "dist/index.js", true},
{"oven-sh/setup-bun", "v2.2.0", "dist/setup/index.js", true},
{"Swatinem/rust-cache", "v2.9.1", "dist/restore/index.js", true},
{"Swatinem/rust-cache", "v2.9.2", "dist/cache-CTfQoCTL.js", true},
{"docker/build-push-action", "v7.4.0", "dist/index.cjs", true},
// The artifact toolkit, where the gate is a refusal and there is nothing to redirect.
// v4.4.0 is the first release whose gate carries the localhost test this matches; the
// releases before it refuse in a shape the runner leaves alone.
{"gradle/actions", "v6.4.0", "dist/setup-gradle/main/index.js", true},
{"actions/upload-artifact", "v4.4.0", "dist/upload/index.js", true},
{"actions/upload-artifact", "v7.0.2", "dist/upload/index.js", true},
{"actions/download-artifact", "v8.0.2", "dist/index.js", true},
// Neither toolkit's gate, so these have to come back byte for byte. sccache-action is the
// one that exports ACTIONS_CACHE_SERVICE_V2 itself, for the Rust client it installs.
{"actions/checkout", "v7.0.1", "dist/index.js", false},
{"mozilla-actions/sccache-action", "v0.0.11", "dist/setup/index.js", false},
} {
@@ -235,10 +209,8 @@ func TestPatchedBundleAcrossActions(t *testing.T) {
assert.NotContains(t, string(out), ".LOCALHOST", "a copy of the gate was missed")
if !strings.Contains(string(data), CacheServiceV2Env) {
return // the artifact toolkit: a refusal to open, and no URL to move
return
}
// Only the reads inside getCacheServiceURL are rewritten. The others, such as the
// feature-availability check, must be left as they are.
assert.NotZero(t, strings.Count(string(out), "(process.env."+cacheURLEnv+"||process.env"),
"the cache service URL was not redirected")
assert.Equal(t, strings.Count(string(data), resultsURLEnv), strings.Count(string(out), resultsURLEnv),
@@ -247,10 +219,6 @@ func TestPatchedBundleAcrossActions(t *testing.T) {
}
}
// The stock artifact actions refuse on a Gitea host until the gate is opened, and then they talk
// to the results service, which is this runner's cache server forwarding the artifact half on to
// Gitea. Running the real upload-artifact against a stand-in Gitea covers both halves at once:
// the patch, and the forwarding the job's registration set up.
func TestUploadArtifactThroughTheResultsService(t *testing.T) {
requireHostTools(t, "node")
@@ -275,12 +243,12 @@ func TestUploadArtifactThroughTheResultsService(t *testing.T) {
case "download":
w.Header().Set("Content-Type", "application/zip")
_, _ = w.Write(zipped)
default: // the zip on its way up, in the blocks the Azure protocol puts it in
default:
body, _ := io.ReadAll(r.Body)
switch r.URL.Query().Get("comp") {
case "block":
zipped = append(zipped, body...)
case "blocklist": // the ordering document, not content
case "blocklist":
default:
zipped = body
}
@@ -288,24 +256,11 @@ func TestUploadArtifactThroughTheResultsService(t *testing.T) {
}
}))
defer gitea.Close()
handler, err := artifactcache.StartHandler(artifactcache.Options{Dir: filepath.Join(t.TempDir(), "cache"), OutboundIP: "127.0.0.1"})
require.NoError(t, err)
t.Cleanup(func() { _ = handler.Close() })
// The artifact client decodes the runtime token for the run ids it puts in its requests, where
// the cache client only presents it, so this one has to be shaped like Gitea's.
token := "e30." + base64.RawURLEncoding.EncodeToString([]byte(`{"scp":"Actions.Results:11:22"}`)) + ".sig"
defer handler.RegisterJob(token, artifactcache.JobCredential{Repo: "testuser/testrepo", Results: gitea.URL})()
env := newActionJobEnv(t, "e30."+base64.RawURLEncoding.EncodeToString([]byte(`{"scp":"Actions.Results:11:22"}`))+".sig",
artifactcache.JobCredential{Repo: "testuser/testrepo", Results: gitea.URL})
upload := patchedAction(t, "actions/upload-artifact", "v7.0.2", "dist/upload/index.js")
env := jobEnv{
workspace: tempDirPath(t),
runnerTemp: tempDirPath(t),
cacheURL: handler.ExternalURL(),
resultsURL: handler.ExternalURL(),
token: token,
}
uploaded := []byte("through the results service")
require.NoError(t, os.WriteFile(filepath.Join(env.workspace, "artifact.txt"), uploaded, 0o600))
@@ -316,9 +271,6 @@ func TestUploadArtifactThroughTheResultsService(t *testing.T) {
})
require.Contains(t, out, "has been successfully uploaded")
// And back down again: listing and downloading go the same way, and the signed URL the
// artifact service hands out is fetched straight from it.
download := patchedAction(t, "actions/download-artifact", "v8.0.2", "dist/index.js")
env.workspace = tempDirPath(t)
out = runActionEntrypoint(t, download, env, map[string]string{
@@ -335,27 +287,12 @@ func TestUploadArtifactThroughTheResultsService(t *testing.T) {
assert.Equal(t, uploaded, got)
}
// The setup actions carry the same toolkit and reach the same service, from a key of their own
// making. setup-node is the cheapest of them to run: given a lockfile and no version to install,
// it does the cache lookup and nothing else.
func TestSetupActionFindsTheCacheService(t *testing.T) {
requireHostTools(t, "node", "npm")
setup := patchedAction(t, "actions/setup-node", "v7.1.0", "dist/setup/index.js")
handler, err := artifactcache.StartHandler(artifactcache.Options{Dir: filepath.Join(t.TempDir(), "cache"), OutboundIP: "127.0.0.1"})
require.NoError(t, err)
t.Cleanup(func() { _ = handler.Close() })
const token = "setup-runtime-token"
defer handler.RegisterJob(token, artifactcache.JobCredential{Repo: "testuser/testrepo"})()
env := jobEnv{
workspace: tempDirPath(t),
runnerTemp: tempDirPath(t),
cacheURL: handler.ExternalURL(),
resultsURL: handler.ExternalURL(),
token: token,
}
env := newActionJobEnv(t, "setup-runtime-token", artifactcache.JobCredential{Repo: "testuser/testrepo"})
require.NoError(t, os.WriteFile(filepath.Join(env.workspace, "package-lock.json"),
[]byte(`{"lockfileVersion":3}`), 0o600))
@@ -364,3 +301,36 @@ func TestSetupActionFindsTheCacheService(t *testing.T) {
require.Contains(t, out, "Cache service version: v2")
require.Contains(t, out, "npm cache is not found", "the lookup did not reach the cache server")
}
func TestCodeSplitActionFindsTheCacheService(t *testing.T) {
requireHostTools(t, "node", "rustc")
dir := tempDirPath(t)
for _, name := range []string{
"action.yml", "dist/restore.js", "dist/save.js", "dist/cleanup-BWEbZ6YT.js",
"dist/cache-CTfQoCTL.js", "dist/cache-DC26rUaF.js", "dist/state-cjs-CkvFUaNy.js",
} {
body, err := os.ReadFile(bundleFromGitHub(t, "Swatinem/rust-cache", "v2.9.2", name))
require.NoError(t, err)
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), body, 0o600))
}
body, err := os.ReadFile(filepath.Join(dir, "action.yml"))
require.NoError(t, err)
action, err := model.ReadAction(strings.NewReader(string(body)))
require.NoError(t, err)
inputs := map[string]string{"lookup-only": "true", "add-rust-environment-hash-key": "false", "cache-bin": "false"}
for name, input := range action.Inputs {
if _, ok := inputs[name]; !ok {
inputs[name] = input.Default
}
}
t.Setenv("CARGO_HOME", tempDirPath(t))
defer patchActions(t.Context(), actionScriptPaths(dir, dir, action))()
out := runActionEntrypoint(t, filepath.Join(dir, action.Runs.Main),
newActionJobEnv(t, "split-runtime-token", artifactcache.JobCredential{Repo: "testuser/testrepo"}), inputs)
require.Contains(t, out, "Cache service version: v2")
require.Contains(t, out, "No cache found.", "the lookup did not reach the cache server")
}
+52 -9
View File
@@ -8,6 +8,7 @@ import (
"os"
"os/exec"
"path/filepath"
"slices"
"strings"
"testing"
@@ -84,6 +85,21 @@ func TestPatchedBundleLeavesTrustworthyURLCheckAlone(t *testing.T) {
assert.True(t, gateOpened(string(out)))
}
func TestPatchedBundleOpensOnlyTheIsGhesTest(t *testing.T) {
const unrelated = `function isLocalName(h){return h.toUpperCase().endsWith('.LOCALHOST')}`
for _, bundle := range []string{unrelated + "\n" + gateTSC, gateTSC + "\n" + unrelated} {
out, patched := patchedBundle([]byte(bundle))
require.True(t, patched)
assert.Contains(t, string(out), unrelated)
assert.Contains(t, string(out), "!h.endsWith('')")
}
apart := strings.Replace(gateTSC, "&&!h.endsWith('.LOCALHOST')", "&&"+strings.Repeat(" ", 200)+"!h.endsWith('.LOCALHOST')", 1)
_, patched := patchedBundle([]byte(apart))
assert.False(t, patched)
}
// The artifact toolkit puts the same gate in front of a plain refusal, with no URL to move, so
// opening it is what lets the stock upload-artifact work against Gitea instead of aborting.
func TestPatchedBundleOpensTheArtifactRefusal(t *testing.T) {
@@ -211,16 +227,42 @@ func bundleFile(t *testing.T, body string) string {
}
func TestActionScriptPaths(t *testing.T) {
node := &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "dist/restore/index.js", Post: "dist/save/index.js"}}
assert.Equal(t, []string{"/a/dist/restore/index.js", "/a/dist/save/index.js"}, actionScriptPaths("/a", node))
dir, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err)
dist := filepath.Join(dir, "dist")
require.NoError(t, os.MkdirAll(filepath.Join(dist, "sub"), 0o755))
require.NoError(t, os.Mkdir(filepath.Join(dist, "directory.js"), 0o755))
for _, name := range []string{"dist/restore.js", "dist/save.js", "dist/cache-CTfQoCTL.js", "dist/295.index.js", "dist/chunk.cjs", "dist/chunk.mjs", "dist/LICENSE.txt", "dist/sub/deep.js", "pre.js"} {
require.NoError(t, os.WriteFile(filepath.Join(dir, name), nil, 0o644))
}
outside, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err)
require.NoError(t, os.WriteFile(filepath.Join(outside, "index.js"), nil, 0o644))
require.NoError(t, os.Symlink(filepath.Join(outside, "index.js"), filepath.Join(dist, "linked.js")))
var chunks []string
for _, name := range []string{"295.index.js", "cache-CTfQoCTL.js", "chunk.cjs", "chunk.mjs", "restore.js", "save.js"} {
chunks = append(chunks, filepath.Join(dist, name))
}
node := &model.Action{Runs: model.ActionRuns{Using: "node20", Pre: "pre.js", Main: "dist/restore.js", Post: "dist/save.js"}}
assert.Equal(t, append(slices.Clone(chunks), filepath.Join(dir, "pre.js")), actionScriptPaths(dir, dir, node))
// Only a node action has a bundle to patch.
assert.Nil(t, actionScriptPaths("/a", &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "alpine"}}))
assert.Nil(t, actionScriptPaths("/a", nil))
assert.Nil(t, actionScriptPaths(dir, dir, &model.Action{Runs: model.ActionRuns{Using: "docker", Image: "alpine"}}))
assert.Nil(t, actionScriptPaths(dir, dir, nil))
// An action naming a file outside its own directory does not get it rewritten.
escaping := &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "../../elsewhere/index.js"}}
assert.Nil(t, actionScriptPaths("/a", escaping))
// An action naming a file outside its checkout does not get it rewritten.
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "linked")))
escaping, err := filepath.Rel(dir, filepath.Join(outside, "index.js"))
require.NoError(t, err)
for _, main := range []string{"../../elsewhere/index.js", escaping, "dist/linked.js", "linked/index.js"} {
assert.Nil(t, actionScriptPaths(dir, dir, &model.Action{Runs: model.ActionRuns{Using: "node20", Main: main}}), main)
}
require.NoError(t, os.Symlink("dist", filepath.Join(dir, "build")))
assert.Equal(t, chunks, actionScriptPaths(dir, dir, &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "build/restore.js"}}))
assert.Equal(t, chunks, actionScriptPaths(dir, filepath.Join(dir, "setup"), &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "../dist/restore.js"}}))
}
func TestPatchActionsAtTheContainerCopy(t *testing.T) {
@@ -231,15 +273,16 @@ func TestPatchActionsAtTheContainerCopy(t *testing.T) {
sar := &stepActionRemote{
Step: &model.Step{Uses: "owner/repo/sub@v1"},
remoteAction: &remoteAction{Org: "owner", Repo: "repo", Path: "sub", Ref: "v1"},
action: &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "index.js"}},
action: &model.Action{Runs: model.ActionRuns{Using: "node20", Main: "../dist/index.js"}},
actionDir: t.TempDir(),
RunContext: &RunContext{
Config: &Config{ActionCacheDir: t.TempDir(), NoActionPatch: noPatch},
JobContainer: cm,
},
}
script := filepath.Join(sar.actionDir, "sub", "index.js")
script := filepath.Join(sar.actionDir, "dist", "cache.js")
require.NoError(t, os.MkdirAll(filepath.Dir(script), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(filepath.Dir(script), "index.js"), []byte("require('./cache.js')"), 0o600))
require.NoError(t, os.WriteFile(script, []byte(gateTSC), 0o600))
var copied string
+24 -38
View File
@@ -107,7 +107,13 @@ func newReusableWorkflowExecutor(rc *RunContext, directory, workflow string, cac
return err
}
runner, err := newReusableWorkflowRunner(rc)
runner, err := (&runnerImpl{
config: rc.Config,
caller: &caller{
runContext: rc,
reusedWorkflowJobResults: map[string]string{},
},
}).configure()
if err != nil {
return err
}
@@ -119,20 +125,6 @@ func newReusableWorkflowExecutor(rc *RunContext, directory, workflow string, cac
}
}
func newReusableWorkflowRunner(rc *RunContext) (*runnerImpl, error) {
runner := &runnerImpl{
config: rc.Config,
eventJSON: rc.EventJSON,
caller: &caller{
runContext: rc,
reusedWorkflowJobResults: map[string]string{}, // For Gitea
},
}
return runner.configure()
}
type remoteReusableWorkflow struct {
*model.ReusableWorkflowUses
URL string
@@ -163,38 +155,32 @@ func newRemoteReusableWorkflow(baseURL, uses string) (*remoteReusableWorkflow, e
// For Gitea
func setReusedWorkflowCallerResult(rc *RunContext, runner *runnerImpl) common.Executor {
return func(ctx context.Context) error {
caller := runner.caller
allJobDone := true
hasFailure := false
for _, result := range caller.reusedWorkflowJobResults {
for _, result := range runner.caller.reusedWorkflowJobResults {
if result == "pending" {
allJobDone = false
break
return nil
}
if result == "failure" {
hasFailure = true
}
}
if allJobDone {
reusedWorkflowJobResult := "success"
reusedWorkflowJobResultMessage := "succeeded"
if hasFailure {
reusedWorkflowJobResult = "failure"
reusedWorkflowJobResultMessage = "failed"
}
reusedWorkflowJobResult := "success"
reusedWorkflowJobResultMessage := "succeeded"
if hasFailure {
reusedWorkflowJobResult = "failure"
reusedWorkflowJobResultMessage = "failed"
}
if rc.caller != nil {
rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, reusedWorkflowJobResult)
} else {
// Serialize this shared Job.Result write against the other matrix combos
// and setJobResult (same lockJob key).
unlock := lockJob(rc.Run.Job())
rc.result(reusedWorkflowJobResult)
unlock()
common.Logger(ctx).With("jobResult", reusedWorkflowJobResult).Infof("Job %s", reusedWorkflowJobResultMessage)
}
if rc.caller != nil {
rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, reusedWorkflowJobResult)
} else {
// Serialize this shared Job.Result write against the other matrix combos
// and setJobResult (same lockJob key).
unlock := lockJob(rc.Run.Job())
rc.result(reusedWorkflowJobResult)
unlock()
common.Logger(ctx).With("jobResult", reusedWorkflowJobResult).Infof("Job %s", reusedWorkflowJobResultMessage)
}
return nil
+49 -111
View File
@@ -82,9 +82,9 @@ type RunContext struct {
platformImage string // container.image or the runs-on pick, resolved once by isEnabled
jobIndex, jobTotal int // strategy.job-index and job-total, total 0 when unknown
jobEnv map[string]string // workflow and job env as evaluated for the container and service specs
dockerActionImages map[*model.Step]string // Dockerfile action images built for this job, removed when it ends
actionDownloads map[string]actionDownload // remote action repositories by clone URL and ref, fetched once per job
jobEnv map[string]string // workflow and job env as evaluated for the container and service specs
dockerActionImages map[*model.Step]string // Dockerfile action images built for this job, removed when it ends
actionDownloads map[string]*actionDownload // remote action repositories by clone URL and ref, fetched once per job
// summaryFileInitialized tracks which per-step summary files (workflow/step-summary-N.md)
// have already been created on the JobContainer. The runner sets up file-command files
@@ -164,8 +164,12 @@ func (rc *RunContext) markInterrupted(err error) {
}
}
func (rc *RunContext) AddMask(mask string) {
// AddMask also hands the mask to the job logger's extra handler, which receives records unmasked.
func (rc *RunContext) AddMask(ctx context.Context, mask string) {
rc.Masks = append(rc.Masks, mask)
if masker, ok := common.LoggerHandler(ctx).(interface{ AddMask(string) }); ok {
masker.AddMask(mask)
}
}
func (rc *RunContext) String() string {
@@ -225,8 +229,7 @@ func (rc *RunContext) networkNameForGitea() (string, bool) {
}
func getDockerDaemonSocketMountPath(daemonPath string) string {
if before, after, ok := strings.Cut(daemonPath, "://"); ok {
scheme := before
if scheme, after, ok := strings.Cut(daemonPath, "://"); ok {
switch {
case strings.EqualFold(scheme, "npipe"):
// linux container mount on windows, use the default socket path of the VM / wsl2
@@ -247,10 +250,7 @@ func getDockerDaemonSocketMountPath(daemonPath string) string {
// without mutating the shared Config. Parallel jobs in a plan share one *Config, so a job
// must never write to it.
func (rc *RunContext) containerDaemonSocket() string {
if rc.Config.ContainerDaemonSocket == "" {
return "/var/run/docker.sock"
}
return rc.Config.ContainerDaemonSocket
return cmp.Or(rc.Config.ContainerDaemonSocket, "/var/run/docker.sock")
}
const sharedToolCacheVolume = "act-toolcache" // mounted only when the tool cache is shared
@@ -300,10 +300,7 @@ func (rc *RunContext) startDockerProxy(ctx context.Context) {
// toolCache returns the tool cache path the job sees, relocatable through RUNNER_TOOL_CACHE.
func (rc *RunContext) toolCache(fallback string) string {
if path := rc.GetEnv()["RUNNER_TOOL_CACHE"]; path != "" {
return path
}
return fallback
return cmp.Or(rc.GetEnv()["RUNNER_TOOL_CACHE"], fallback)
}
// runnerEnv returns a container's RUNNER_* variables, derived from the values runner.tool_cache
@@ -547,10 +544,7 @@ func (rc *RunContext) startJobContainer() common.Executor {
rc.Env["JOB_CONTAINER_NAME"] = name
delete(rc.jobEnv, "JOB_CONTAINER_NAME") // the runtime name wins over a workflow env of the same name
envList := make([]string, 0)
envList = append(envList, rc.runnerEnv(ctx)...)
envList = append(envList, fmt.Sprintf("%s=%s", "LANG", "C.UTF-8")) // Use same locale as GitHub Actions
envList := append(rc.runnerEnv(ctx), "LANG=C.UTF-8") // Use same locale as GitHub Actions
// specify the network to which the container will connect when `docker create` stage. (like execute command line: docker create --network <networkName> <image>)
// if using service containers, will create a new network for the containers.
@@ -567,12 +561,8 @@ func (rc *RunContext) startJobContainer() common.Executor {
rc.serviceContainers = append(rc.serviceContainers, &serviceContainer{name: serviceID, image: input.Image, container: newContainer(input)})
}
// For Gitea, `jobContainerNetwork` should be the same as `networkName`
jobContainerNetwork := networkName
ext := container.LinuxContainerEnvironmentExtensions{}
containerInput := &container.NewContainerInput{
Cmd: nil,
Entrypoint: []string{"sleep", strconv.FormatInt(int64(rc.Config.ContainerMaxLifetime.Round(time.Second).Seconds()), 10)},
WorkingDir: ext.ToContainerPath(rc.Config.Workdir),
Image: image,
@@ -580,7 +570,7 @@ func (rc *RunContext) startJobContainer() common.Executor {
Password: password,
Name: name,
Env: envList,
NetworkMode: jobContainerNetwork,
NetworkMode: networkName,
NetworkAliases: []string{rc.Name},
Stdout: logWriter,
Stderr: logWriter,
@@ -672,7 +662,7 @@ func (rc *RunContext) jobServices(ctx context.Context, networkName string) (map[
return nil, fmt.Errorf("failed to parse service %s ports: %w", serviceID, err)
}
serviceLog := common.NewLineWriter(rawLogLine(ctx)) // raw, so a service's output cannot set the job's env, outputs or state
serviceLog := common.NewLineWriter(rc.serviceCommandHandler(ctx), rawLogLine(ctx))
inputs[serviceID] = &container.NewContainerInput{
Name: createContainerName(rc.jobContainerName(), serviceID),
Image: spec.Image,
@@ -705,6 +695,15 @@ func (rc *RunContext) commandLogWriter(ctx context.Context) io.Writer {
return common.NewLineWriter(rc.commandHandler(ctx), rawLogLine(ctx))
}
// withCommandLogWriter runs executor with a command handler of its own on the job environment's output.
func (rc *RunContext) withCommandLogWriter(ctx context.Context, executor common.Executor) error {
logWriter := rc.commandLogWriter(ctx)
oldout, olderr := rc.JobContainer.ReplaceLogWriter(logWriter, logWriter)
defer rc.JobContainer.ReplaceLogWriter(oldout, olderr)
defer common.FlushWriter(logWriter) // the last line of output may lack its newline
return executor(ctx)
}
func rawLogLine(ctx context.Context) common.LineHandler {
rawLogger := common.Logger(ctx).With(rawOutputField, true)
return func(line string) bool {
@@ -819,9 +818,7 @@ func (rc *RunContext) ApplyExtraPath(ctx context.Context, env *map[string]string
cenv := map[string]string{}
var cpath string
if err := rc.JobContainer.UpdateFromImageEnv(&cenv)(ctx); err == nil {
if p, ok := cenv[path]; ok {
cpath = p
}
cpath = cenv[path]
}
if len(cpath) == 0 {
cpath = rc.JobContainer.DefaultPathVariable()
@@ -853,7 +850,7 @@ func (rc *RunContext) UpdateExtraPath(ctx context.Context, githubEnvPath string)
for s.Scan() {
line := s.Text()
if len(line) > 0 {
rc.addPath(ctx, line)
rc.addPath(line)
}
}
if err := s.Err(); err != nil {
@@ -862,7 +859,7 @@ func (rc *RunContext) UpdateExtraPath(ctx context.Context, githubEnvPath string)
return nil
}
func (rc *RunContext) stopJobContainer() common.Executor {
func (rc *RunContext) stopContainer() common.Executor {
return func(ctx context.Context) error {
if rc.cleanUpJobContainer != nil {
return rc.cleanUpJobContainer(ctx)
@@ -922,7 +919,7 @@ func (rc *RunContext) stopServiceContainers() common.Executor {
const defaultServiceReadyTimeout = 5 * time.Minute
const serviceReadyPollInterval = time.Second
const serviceReadyPollInterval = 250 * time.Millisecond
// reportUnstartedServices logs a service that did not start. The steps that need it
// report it better than the runner can, so the job carries on.
@@ -1105,9 +1102,8 @@ func (rc *RunContext) ActionCacheDir() string {
if rc.Config.ActionCacheDir != "" {
return rc.Config.ActionCacheDir
}
var xdgCache string
var ok bool
if xdgCache, ok = os.LookupEnv("XDG_CACHE_HOME"); !ok || xdgCache == "" {
xdgCache := os.Getenv("XDG_CACHE_HOME")
if xdgCache == "" {
if home, err := os.UserHomeDir(); err == nil {
xdgCache = filepath.Join(home, ".cache")
} else if xdgCache, err = filepath.Abs("."); err != nil {
@@ -1179,10 +1175,6 @@ func (rc *RunContext) IsHostEnv() bool {
return rc.containerSpec.Image == "" && strings.EqualFold(rc.platformImage, "-self-hosted")
}
func (rc *RunContext) stopContainer() common.Executor {
return rc.stopJobContainer()
}
func (rc *RunContext) closeContainer() common.Executor {
return func(ctx context.Context) error {
if rc.JobContainer != nil {
@@ -1404,29 +1396,20 @@ func mergeMaps(maps ...map[string]string) map[string]string {
return rtnMap
}
var containerNamePattern = regexp.MustCompile("[^a-zA-Z0-9]")
func createContainerName(parts ...string) string {
name := strings.Join(parts, "-")
pattern := regexp.MustCompile("[^a-zA-Z0-9]")
name = pattern.ReplaceAllString(name, "-")
name = containerNamePattern.ReplaceAllString(name, "-")
name = strings.ReplaceAll(name, "--", "-")
hash := sha256.Sum256([]byte(name))
// SHA256 is 64 hex characters. So trim name to 63 characters to make room for the hash and separator
trimmedName := strings.Trim(trimToLen(name, 63), "-")
trimmedName := strings.Trim(name[:min(len(name), 63)], "-")
return fmt.Sprintf("%s-%x", trimmedName, hash)
}
func trimToLen(s string, l int) string {
if l < 0 {
l = 0
}
if len(s) > l {
return s[:l]
}
return s
}
// ownStatus is the result of this context's own steps, which for a composite action is its action status
func (rc *RunContext) ownStatus() string {
if rc.jobFailed {
@@ -1471,10 +1454,6 @@ func (rc *RunContext) getJobContext() *model.JobContext {
return jobContext
}
func (rc *RunContext) getStepsContext() map[string]*model.StepResult {
return rc.StepResults
}
// getRunnerContext returns the `runner` context: what the execution environment knows
// (os, arch, temp, tool_cache) plus what only the runner process knows.
func (rc *RunContext) getRunnerContext(ctx context.Context) map[string]any {
@@ -1508,9 +1487,9 @@ func (rc *RunContext) getGithubContext(ctx context.Context) *model.GithubContext
ghc := &model.GithubContext{
Event: make(map[string]any),
Workflow: rc.Run.Workflow.Name,
RunID: rc.Config.Env["GITHUB_RUN_ID"],
RunNumber: rc.Config.Env["GITHUB_RUN_NUMBER"],
Actor: rc.Config.Actor,
RunID: cmp.Or(rc.Config.Env["GITHUB_RUN_ID"], "1"),
RunNumber: cmp.Or(rc.Config.Env["GITHUB_RUN_NUMBER"], "1"),
Actor: cmp.Or(rc.Config.Actor, "nektos/act"), // for configs that require a default rather than being run as a cmd
EventName: rc.Config.EventName,
Action: rc.CurrentStep,
Token: rc.Config.Token,
@@ -1519,8 +1498,8 @@ func (rc *RunContext) getGithubContext(ctx context.Context) *model.GithubContext
ActionRepository: rc.Env["GITHUB_ACTION_REPOSITORY"],
ActionRef: rc.Env["GITHUB_ACTION_REF"],
RepositoryOwner: rc.Config.Env["GITHUB_REPOSITORY_OWNER"],
RetentionDays: rc.Config.Env["GITHUB_RETENTION_DAYS"],
RunnerPerflog: rc.Config.Env["RUNNER_PERFLOG"],
RetentionDays: cmp.Or(rc.Config.Env["GITHUB_RETENTION_DAYS"], "0"),
RunnerPerflog: cmp.Or(rc.Config.Env["RUNNER_PERFLOG"], "/dev/null"),
RunnerTrackingID: rc.Config.Env["RUNNER_TRACKING_ID"],
Repository: rc.Config.Env["GITHUB_REPOSITORY"],
Ref: rc.Config.Env["GITHUB_REF"],
@@ -1535,40 +1514,16 @@ func (rc *RunContext) getGithubContext(ctx context.Context) *model.GithubContext
ghc.EventPath = rc.JobContainer.GetActPath() + "/workflow/event.json"
}
if ghc.RunID == "" {
ghc.RunID = "1"
}
if preset := rc.Config.PresetGitHubContext; preset != nil {
merged := *preset
merged.Workflow, merged.Job, merged.Workspace, merged.EventPath = ghc.Workflow, ghc.Job, ghc.Workspace, ghc.EventPath
merged.Action, merged.ActionPath, merged.ActionRepository, merged.ActionRef = ghc.Action, ghc.ActionPath, ghc.ActionRepository, ghc.ActionRef
merged.RunnerPerflog, merged.RunnerTrackingID = ghc.RunnerPerflog, ghc.RunnerTrackingID
*ghc = merged
if ghc.RunNumber == "" {
ghc.RunNumber = "1"
}
if ghc.RetentionDays == "" {
ghc.RetentionDays = "0"
}
if ghc.RunnerPerflog == "" {
ghc.RunnerPerflog = "/dev/null"
}
// Backwards compatibility for configs that require
// a default rather than being run as a cmd
if ghc.Actor == "" {
ghc.Actor = "nektos/act"
}
{ // Adapt to Gitea
if preset := rc.Config.PresetGitHubContext; preset != nil {
merged := *preset
merged.Workflow, merged.Job, merged.Workspace, merged.EventPath = ghc.Workflow, ghc.Job, ghc.Workspace, ghc.EventPath
merged.Action, merged.ActionPath, merged.ActionRepository, merged.ActionRef = ghc.Action, ghc.ActionPath, ghc.ActionRepository, ghc.ActionRef
merged.RunnerPerflog, merged.RunnerTrackingID = ghc.RunnerPerflog, ghc.RunnerTrackingID
*ghc = merged
rc.setInstanceURLs(ghc)
ghc.GraphQLURL = "" // Gitea doesn't support graphql
return ghc
}
rc.setInstanceURLs(ghc)
ghc.GraphQLURL = "" // Gitea doesn't support graphql
return ghc
}
if rc.EventJSON != "" {
@@ -1621,10 +1576,6 @@ func withHTTPS(u string) string {
}
func isLocalCheckout(ghc *model.GithubContext, step *model.Step) bool {
if step.Type() == model.StepTypeInvalid {
// This will be errored out by the executor later, we need this here to avoid a null panic though
return false
}
if step.Type() != model.StepTypeUsesActionRemote {
return false
}
@@ -1689,7 +1640,8 @@ func (rc *RunContext) withGithubEnv(ctx context.Context, github *model.GithubCon
}
if rc.Config.ArtifactServerPath != "" {
setActionRuntimeVars(rc, env)
env["ACTIONS_RUNTIME_URL"] = cmp.Or(os.Getenv("ACTIONS_RUNTIME_URL"), "http://"+net.JoinHostPort(rc.Config.ArtifactServerAddr, rc.Config.ArtifactServerPort)+"/")
env["ACTIONS_RUNTIME_TOKEN"] = cmp.Or(os.Getenv("ACTIONS_RUNTIME_TOKEN"), "token")
}
if imageOS := rc.imageOS(ctx); imageOS != "" {
@@ -1744,20 +1696,6 @@ func imageOSFromImage(image string) string {
return ""
}
func setActionRuntimeVars(rc *RunContext, env map[string]string) {
actionsRuntimeURL := os.Getenv("ACTIONS_RUNTIME_URL")
if actionsRuntimeURL == "" {
actionsRuntimeURL = "http://" + net.JoinHostPort(rc.Config.ArtifactServerAddr, rc.Config.ArtifactServerPort) + "/"
}
env["ACTIONS_RUNTIME_URL"] = actionsRuntimeURL
actionsRuntimeToken := os.Getenv("ACTIONS_RUNTIME_TOKEN")
if actionsRuntimeToken == "" {
actionsRuntimeToken = "token"
}
env["ACTIONS_RUNTIME_TOKEN"] = actionsRuntimeToken
}
func registryCredentials(credentials map[string]string, prefix string) (string, string, error) {
if credentials == nil {
return "", "", nil
+167 -445
View File
@@ -40,13 +40,6 @@ import (
)
func TestRunContext_EvalBool(t *testing.T) {
var yml yaml.Node
err := yml.Encode(map[string][]any{
"os": {"Linux", "Windows"},
"foo": {"bar", "baz"},
})
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
rc := &RunContext{
Config: &Config{
Workdir: ".",
@@ -61,26 +54,12 @@ func TestRunContext_EvalBool(t *testing.T) {
Workflow: &model.Workflow{
Name: "test-workflow",
Jobs: map[string]*model.Job{
"job1": {
Strategy: &model.Strategy{
RawMatrix: yml,
},
},
"job1": {},
},
},
},
Matrix: map[string]any{
"os": "Linux",
"foo": "bar",
},
StepResults: map[string]*model.StepResult{
"id1": {
Conclusion: model.StepStatusSuccess,
Outcome: model.StepStatusFailure,
Outputs: map[string]string{
"foo": "bar",
},
},
"id1": {Conclusion: model.StepStatusSuccess, Outcome: model.StepStatusFailure},
},
}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
@@ -90,21 +69,12 @@ func TestRunContext_EvalBool(t *testing.T) {
out bool
wantErr bool
}{
// The basic ones
{in: "failure()", out: false},
{in: "success()", out: true},
{in: "cancelled()", out: false},
{in: "always()", out: true},
// TODO: move to sc.NewExpressionEvaluator(), because "steps" context is not available here
// {in: "steps.id1.conclusion == 'success'", out: true},
// {in: "steps.id1.conclusion != 'success'", out: false},
// {in: "steps.id1.outcome == 'failure'", out: true},
// {in: "steps.id1.outcome != 'failure'", out: false},
{in: "true", out: true},
{in: "false", out: false},
// TODO: This does not throw an error, because the evaluator does not know if the expression is inside ${{ }} or not
// {in: "!true", wantErr: true},
// {in: "!false", wantErr: true},
{in: "1 != 0", out: true},
{in: "1 != 1", out: false},
{in: "${{ 1 != 0 }}", out: true},
@@ -113,32 +83,24 @@ func TestRunContext_EvalBool(t *testing.T) {
{in: "1 == 1", out: true},
{in: "1 > 2", out: false},
{in: "1 < 2", out: true},
// And or
{in: "true && false", out: false},
{in: "true && 1 < 2", out: true},
{in: "false || 1 < 2", out: true},
{in: "false || false", out: false},
// None boolable
{in: "env.UNKNOWN == 'true'", out: false},
{in: "env.UNKNOWN", out: false},
// Inline expressions
{in: "env.SOME_TEXT", out: true},
{in: "env.SOME_TEXT == 'text'", out: true},
{in: "env.SOMETHING_TRUE == 'true'", out: true},
{in: "env.SOMETHING_FALSE == 'true'", out: false},
{in: "env.SOMETHING_TRUE", out: true},
{in: "env.SOMETHING_FALSE", out: true},
// TODO: This does not throw an error, because the evaluator does not know if the expression is inside ${{ }} or not
// {in: "!env.SOMETHING_TRUE", wantErr: true},
// {in: "!env.SOMETHING_FALSE", wantErr: true},
{in: "${{ !env.SOMETHING_TRUE }}", out: false},
{in: "${{ !env.SOMETHING_FALSE }}", out: false},
{in: "${{ ! env.SOMETHING_TRUE }}", out: false},
{in: "${{ ! env.SOMETHING_FALSE }}", out: false},
{in: "${{ env.SOMETHING_TRUE }}", out: true},
{in: "${{ env.SOMETHING_FALSE }}", out: true},
{in: "${{ !env.SOMETHING_TRUE }}", out: false},
{in: "${{ !env.SOMETHING_FALSE }}", out: false},
{in: "${{ !env.SOMETHING_TRUE && true }}", out: false},
{in: "${{ !env.SOMETHING_FALSE && true }}", out: false},
{in: "${{ !env.SOMETHING_TRUE || true }}", out: true},
@@ -146,14 +108,11 @@ func TestRunContext_EvalBool(t *testing.T) {
{in: "${{ env.SOMETHING_TRUE && true }}", out: true},
{in: "${{ env.SOMETHING_FALSE || true }}", out: true},
{in: "${{ env.SOMETHING_FALSE || false }}", out: true},
// TODO: This does not throw an error, because the evaluator does not know if the expression is inside ${{ }} or not
// {in: "!env.SOMETHING_TRUE || true", wantErr: true},
{in: "${{ env.SOMETHING_TRUE == 'true'}}", out: true},
{in: "${{ env.SOMETHING_FALSE == 'true'}}", out: false},
{in: "${{ env.SOMETHING_FALSE == 'false'}}", out: true},
{in: "${{ env.SOMETHING_FALSE }} && ${{ env.SOMETHING_TRUE }}", out: true},
// All together now
{in: "false || env.SOMETHING_TRUE == 'true'", out: true},
{in: "true || env.SOMETHING_FALSE == 'true'", out: true},
{in: "true && env.SOMETHING_TRUE == 'true'", out: true},
@@ -162,33 +121,26 @@ func TestRunContext_EvalBool(t *testing.T) {
{in: "env.SOMETHING_FALSE == 'true' && true", out: false},
{in: "${{ env.SOMETHING_FALSE == 'true' }} && true", out: true},
{in: "true && ${{ env.SOMETHING_FALSE == 'true' }}", out: true},
// Check github context
{in: "github.actor == 'nektos/act'", out: true},
{in: "github.actor == 'unknown'", out: false},
{in: "github.job == 'job1'", out: true},
// The special ACT flag
{in: "${{ env.ACT }}", out: true},
{in: "${{ !env.ACT }}", out: false},
// Invalid expressions should be reported
{in: "INVALID_EXPRESSION", wantErr: true},
}
for _, table := range tables {
t.Run(table.in, func(t *testing.T) {
assertObject := assert.New(t)
b, err := EvalBool(t.Context(), rc.ExprEval, table.in, exprparser.DefaultStatusCheckSuccess)
if table.wantErr {
assertObject.Error(err) //nolint:testifylint // pre-existing issue from nektos/act
require.Error(t, err)
}
assertObject.Equal(table.out, b, fmt.Sprintf("Expected %s to be %v, was %v", table.in, table.out, b)) //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, table.out, b)
})
}
}
// fakeContainer turns every container operation into a no-op, so startJobContainer
// runs without a Docker daemon. The embedded interface is nil, so any method the
// test does not exercise panics rather than silently doing the wrong thing.
type fakeContainer struct {
container.ExecutionsEnvironment
}
@@ -244,7 +196,7 @@ func startJobContainerInputs(t *testing.T, workflowYAML string, cfg *Config) []*
t.Cleanup(func() { newContainer = origNewContainer })
cfg.Workdir = "/tmp"
cfg.ContainerNetworkMode = "host" // an explicit network mode creates no network
cfg.ContainerNetworkMode = "host"
cfg.Env = map[string]string{}
cfg.Secrets = map[string]string{}
@@ -273,6 +225,8 @@ func startJobContainerInputs(t *testing.T, workflowYAML string, cfg *Config) []*
w.WriteHeader(http.StatusNoContent)
case strings.HasSuffix(r.URL.Path, "/info"):
_, _ = io.WriteString(w, `{"Architecture":"amd64","OSType":"linux"}`)
case strings.Contains(r.URL.Path, "/containers/") && strings.HasSuffix(r.URL.Path, "/json"):
http.NotFound(w, r)
default:
t.Errorf("unexpected Docker request: %s %s", r.Method, r.URL)
http.NotFound(w, r)
@@ -285,8 +239,6 @@ func startJobContainerInputs(t *testing.T, workflowYAML string, cfg *Config) []*
return inputs
}
// Regression test: a service without a `credentials:` block resolves to empty
// credentials, which used to overwrite the job container's own credentials.
func TestStartJobContainerKeepsJobCredentialsWithServices(t *testing.T) {
inputs := startJobContainerInputs(t, `
name: test
@@ -315,9 +267,7 @@ jobs:
credentials[in.Image] = [2]string{in.Username, in.Password}
}
// the job container keeps its own credentials, whichever services exist
require.Equal(t, [2]string{"job-user", "job-password"}, credentials["registry.example/private:latest"])
// each service keeps its own, and a service without credentials gets none
require.Equal(t, [2]string{"db-user", "db-password"}, credentials["postgres:latest"])
require.Equal(t, [2]string{"", ""}, credentials["redis:latest"])
}
@@ -333,10 +283,10 @@ jobs:
- data:/data
`, &Config{ValidVolumes: []string{"data"}})[0]
require.Equal(t, "redis:latest", redis.Image) // services are built before the job container
require.Equal(t, "redis:latest", redis.Image)
require.Equal(t, []string{"data"}, redis.ValidVolumes)
require.Equal(t, map[string]string{"data": "/data"}, redis.Mounts)
require.Empty(t, redis.Binds) // the docker socket is the job container's alone
require.Empty(t, redis.Binds)
require.Empty(t, redis.WorkingDir)
require.Nil(t, redis.Entrypoint)
require.Nil(t, redis.Cmd)
@@ -386,8 +336,6 @@ jobs:
require.Equal(t, "--volumes-from "+job.Name, redis.WorkflowOptions)
}
// Only the workflow's options may be stripped later, so the two sources have to reach the
// container apart from each other.
func TestStartJobContainerKeepsRunnerOptionsApartFromWorkflowOptions(t *testing.T) {
inputs := startJobContainerInputs(t, `
name: test
@@ -411,12 +359,9 @@ jobs:
}
require.Equal(t, [2]string{"--device /dev/fuse", "--cap-add SYS_PTRACE"}, options["registry.example/job:latest"])
// a service container gets no options from the runner's config today
require.Equal(t, [2]string{"", "--shm-size 1g"}, options["redis:latest"])
}
// A service container reaches the internet the same way the job does, so it inherits the
// job's proxy; a service that sets the variable itself keeps its own value.
func TestStartJobContainerGivesServicesTheJobProxy(t *testing.T) {
inputs := startJobContainerInputs(t, `
name: test
@@ -440,14 +385,11 @@ jobs:
require.Contains(t, redis.Env, "http_proxy=http://proxy:3128")
require.Contains(t, redis.Env, "no_proxy=internal.example")
// the service's own env wins over what the runner injected, without dropping the rest
require.Contains(t, db.Env, "no_proxy=db-only.example")
require.NotContains(t, db.Env, "no_proxy=internal.example")
require.Contains(t, db.Env, "http_proxy=http://proxy:3128")
}
// act builds Dockerfile actions through the API, which does not pre-populate the proxy
// build args the docker CLI would, so the RUN steps would have no network behind a proxy.
func TestProxyBuildArgs(t *testing.T) {
rc := &RunContext{Config: &Config{ProxyEnv: map[string]string{"http_proxy": "http://proxy:3128"}}}
@@ -456,69 +398,56 @@ func TestProxyBuildArgs(t *testing.T) {
require.Len(t, args, 1)
require.Equal(t, "http://proxy:3128", *args["http_proxy"])
// a job without a proxy builds exactly as it does today
require.Nil(t, (&RunContext{Config: &Config{}}).proxyBuildArgs())
}
func TestRunContext_GetBindsAndMounts(t *testing.T) {
rctemplate := &RunContext{
Name: "TestRCName",
Run: &model.Run{
Workflow: &model.Workflow{
Name: "TestWorkflowName",
newRunContext := func(config *Config) *RunContext {
return &RunContext{
Name: "TestRCName",
Run: &model.Run{
Workflow: &model.Workflow{Name: "TestWorkflowName"},
},
},
Config: &Config{
BindWorkdir: false,
},
Config: config,
}
}
rctemplate := newRunContext(&Config{})
tests := []struct {
windowsPath bool
name string
rc *RunContext
wantbind string
wantmount string
name, want string
}{
{false, "/mnt/linux", rctemplate, "/mnt/linux", "/mnt/linux"},
{false, "/mnt/path with spaces/linux", rctemplate, "/mnt/path with spaces/linux", "/mnt/path with spaces/linux"},
{true, "C:\\Users\\TestPath\\MyTestPath", rctemplate, "/mnt/c/Users/TestPath/MyTestPath", "/mnt/c/Users/TestPath/MyTestPath"},
{true, "C:\\Users\\Test Path with Spaces\\MyTestPath", rctemplate, "/mnt/c/Users/Test Path with Spaces/MyTestPath", "/mnt/c/Users/Test Path with Spaces/MyTestPath"},
{true, "/LinuxPathOnWindowsShouldFail", rctemplate, "", ""},
{false, "/mnt/linux", "/mnt/linux"},
{false, "/mnt/path with spaces/linux", "/mnt/path with spaces/linux"},
{true, `C:\Users\TestPath\MyTestPath`, "/mnt/c/Users/TestPath/MyTestPath"},
{true, `C:\Users\Test Path with Spaces\MyTestPath`, "/mnt/c/Users/Test Path with Spaces/MyTestPath"},
{true, "/LinuxPathOnWindowsShouldFail", ""},
}
isWindows := runtime.GOOS == "windows"
for _, testcase := range tests {
// pin for scopelint
if (runtime.GOOS == "windows") != testcase.windowsPath {
continue
}
for _, bindWorkDir := range []bool{true, false} {
// pin for scopelint
testBindSuffix := ""
if bindWorkDir {
testBindSuffix = "Bind"
}
t.Run(testcase.name+testBindSuffix, func(t *testing.T) {
rctemplate.Config.Workdir = testcase.name
rctemplate.Config.BindWorkdir = bindWorkDir
gotbind, gotmount := rctemplate.GetBindsAndMounts(t.Context())
// Only run windows path tests on windows and non-windows on non-windows
if (isWindows && testcase.windowsPath) || (!isWindows && !testcase.windowsPath) {
t.Run((testcase.name + testBindSuffix), func(t *testing.T) {
config := testcase.rc.Config
config.Workdir = testcase.name
config.BindWorkdir = bindWorkDir
gotbind, gotmount := rctemplate.GetBindsAndMounts(t.Context())
// Name binds/mounts are either/or
if config.BindWorkdir {
fullBind := testcase.name + ":" + testcase.wantbind
if runtime.GOOS == "darwin" {
fullBind += ":delegated"
}
assert.Contains(t, gotbind, fullBind)
} else {
mountkey := testcase.rc.jobContainerName()
assert.EqualValues(t, testcase.wantmount, gotmount[mountkey]) //nolint:testifylint // pre-existing issue from nektos/act
if bindWorkDir {
fullBind := testcase.name + ":" + testcase.want
if runtime.GOOS == "darwin" {
fullBind += ":delegated"
}
})
}
assert.Contains(t, gotbind, fullBind)
} else {
assert.Equal(t, testcase.want, gotmount[rctemplate.jobContainerName()])
}
})
}
}
@@ -546,21 +475,12 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
})
require.NoError(t, err)
rc := &RunContext{
Name: "TestRCName",
Run: &model.Run{
Workflow: &model.Workflow{
Name: "TestWorkflowName",
},
rc := newRunContext(&Config{
ValidVolumes: []string{"**"},
Secrets: map[string]string{
"MAME": "/host/mame/roms",
},
Config: &Config{
BindWorkdir: false,
ValidVolumes: []string{"**"},
Secrets: map[string]string{
"MAME": "/host/mame/roms",
},
},
}
})
rc.Run.JobID = "job1"
rc.Run.Workflow.Jobs = map[string]*model.Job{"job1": job}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
@@ -574,20 +494,11 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
for _, testcase := range tests {
t.Run(testcase.name, func(t *testing.T) {
rc := &RunContext{
Name: "TestRCName",
Run: &model.Run{
Workflow: &model.Workflow{
Name: "TestWorkflowName",
},
},
Config: &Config{
BindWorkdir: false,
SharedToolCache: true, // so OverridesToolCache has a mount to displace
ValidVolumes: []string{"**"},
},
containerSpec: model.ContainerSpec{Volumes: testcase.volumes},
}
rc := newRunContext(&Config{
SharedToolCache: true,
ValidVolumes: []string{"**"},
})
rc.containerSpec.Volumes = testcase.volumes
rc.Run.JobID = "job1"
gotbind, gotmount := rc.GetBindsAndMounts(t.Context())
@@ -601,8 +512,6 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
assert.Equal(t, gotmount[k], v)
}
// Docker rejects a container with two mounts on one target, so the job's own
// volumes must displace the runner's rather than pile up next to them.
targets := map[string]bool{}
for _, bind := range gotbind {
parsed, err := container.ParseVolume(bind)
@@ -619,11 +528,7 @@ func TestRunContext_GetBindsAndMounts(t *testing.T) {
})
t.Run("ToolCacheMount", func(t *testing.T) {
rc := &RunContext{
Name: "TestRCName",
Run: &model.Run{Workflow: &model.Workflow{Name: "TestWorkflowName"}},
Config: &Config{},
}
rc := newRunContext(&Config{})
_, gotmount := rc.GetBindsAndMounts(t.Context())
assert.NotContains(t, gotmount, sharedToolCacheVolume)
@@ -666,15 +571,11 @@ func TestRunContextValidVolumes(t *testing.T) {
got := rc.validVolumes()
// the configured volumes plus the ones the runner mounts automatically
assert.Subset(t, got, []string{"my-vol", "/host/path", sharedToolCacheVolume, name, name + "-env", "/var/run/docker.sock"})
// deriving the list must never mutate or grow the shared Config slice: parallel matrix
// combinations share one *Config, and the previous in-place append was a data race.
assert.Equal(t, []string{"my-vol", "/host/path"}, rc.Config.ValidVolumes)
assert.Len(t, rc.validVolumes(), len(got), "repeated calls must be stable, not accumulate")
// a job may mount it only while the runner does
rc.Config.SharedToolCache = false
assert.NotContains(t, rc.validVolumes(), sharedToolCacheVolume)
@@ -892,7 +793,6 @@ func TestInterpolateOutputsIsPerMatrixComboKeepingNeedsOutputs(t *testing.T) {
require.Equal(t, map[string]string{"o": "from-gitea"}, needed.Outputs)
}
// A whole-value `outputs:` expression only reaches the typed field through DecodeRaw.
func TestInterpolateOutputsFromExpression(t *testing.T) {
var rawOutputs yaml.Node
require.NoError(t, rawOutputs.Encode(`${{ fromJSON('{"o":"resolved"}') }}`))
@@ -910,7 +810,7 @@ func TestGetGitHubContext(t *testing.T) {
logging.SetLevel(logging.LevelDebug)
cwd, err := os.Getwd()
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
rc := &RunContext{
Config: &Config{
@@ -937,8 +837,6 @@ func TestGetGitHubContext(t *testing.T) {
ghc := rc.getGithubContext(t.Context())
logging.Debugf("%v", ghc)
actor := "nektos/act"
if a := os.Getenv("ACT_ACTOR"); a != "" {
actor = a
@@ -954,15 +852,15 @@ func TestGetGitHubContext(t *testing.T) {
owner = o
}
assert.Equal(t, ghc.RunID, "1") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, ghc.RunNumber, "1") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, ghc.RetentionDays, "0") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, ghc.Actor, actor)
assert.Equal(t, ghc.Repository, repo)
assert.Equal(t, ghc.RepositoryOwner, owner)
assert.Equal(t, ghc.RunnerPerflog, "/dev/null") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, ghc.Token, rc.Config.Secrets["GITHUB_TOKEN"])
assert.Equal(t, ghc.Job, "job1") //nolint:testifylint // pre-existing issue from nektos/act
assert.Equal(t, "1", ghc.RunID)
assert.Equal(t, "1", ghc.RunNumber)
assert.Equal(t, "0", ghc.RetentionDays)
assert.Equal(t, actor, ghc.Actor)
assert.Equal(t, repo, ghc.Repository)
assert.Equal(t, owner, ghc.RepositoryOwner)
assert.Equal(t, "/dev/null", ghc.RunnerPerflog)
assert.Equal(t, rc.Config.Secrets["GITHUB_TOKEN"], ghc.Token)
assert.Equal(t, "job1", ghc.Job)
rc.Config.PresetGitHubContext = &model.GithubContext{Actor: "preset-actor", TriggeringActor: "triggerer", Job: "preset-job"}
ghc = rc.getGithubContext(t.Context())
@@ -1068,229 +966,88 @@ func createJob(t *testing.T, input, result string) *model.Job {
}
func TestRunContextRunsOnPlatformNames(t *testing.T) {
logging.SetLevel(logging.LevelDebug)
assertObject := assert.New(t)
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
})
assertObject.Equal([]string{"ubuntu-latest"}, rc.runsOnPlatformNames(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ${{ 'ubuntu-latest' }}`, ""),
})
assertObject.Equal([]string{"ubuntu-latest"}, rc.runsOnPlatformNames(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: [self-hosted, my-runner]`, ""),
})
assertObject.Equal([]string{"self-hosted", "my-runner"}, rc.runsOnPlatformNames(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: [self-hosted, "${{ 'my-runner' }}"]`, ""),
})
assertObject.Equal([]string{"self-hosted", "my-runner"}, rc.runsOnPlatformNames(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ${{ fromJSON('["ubuntu-latest"]') }}`, ""),
})
assertObject.Equal([]string{"ubuntu-latest"}, rc.runsOnPlatformNames(t.Context()))
// test missing / invalid runs-on
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `name: something`, ""),
})
assertObject.Equal([]string{}, rc.runsOnPlatformNames(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on:
mapping: value`, ""),
})
assertObject.Equal([]string{}, rc.runsOnPlatformNames(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ${{ invalid expression }}`, ""),
})
assertObject.Equal([]string{}, rc.runsOnPlatformNames(t.Context()))
}
func TestRunContextIsEnabled(t *testing.T) {
logging.SetLevel(logging.LevelDebug)
assertObject := assert.New(t)
// success()
rc := createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest
env:
BROKEN: ${{ fromJSON('invalid') }}
if: success()`, ""),
})
assertObject.True(rc.isEnabled(t.Context()))
require.Error(t, evaluateJobEnvAndDefaults(t.Context(), rc))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "failure"),
"job2": createJob(t, `runs-on: ubuntu-latest
needs: [job1]
if: success()`, ""),
})
rc.Run.JobID = "job2"
assertObject.False(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "success"),
"job2": createJob(t, `runs-on: ubuntu-latest
needs: [job1]
if: success()`, ""),
})
rc.Run.JobID = "job2"
assertObject.True(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "failure"),
"job2": createJob(t, `runs-on: ubuntu-latest
if: success()`, ""),
})
rc.Run.JobID = "job2"
assertObject.True(rc.isEnabled(t.Context()))
// failure()
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest
if: failure()`, ""),
})
assertObject.False(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "failure"),
"job2": createJob(t, `runs-on: ubuntu-latest
needs: [job1]
if: failure()`, ""),
})
rc.Run.JobID = "job2"
assertObject.True(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "success"),
"job2": createJob(t, `runs-on: ubuntu-latest
needs: [job1]
if: failure()`, ""),
})
rc.Run.JobID = "job2"
assertObject.False(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "failure"),
"job2": createJob(t, `runs-on: ubuntu-latest
if: failure()`, ""),
})
rc.Run.JobID = "job2"
assertObject.False(rc.isEnabled(t.Context()))
// always()
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest
if: always()`, ""),
})
assertObject.True(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "failure"),
"job2": createJob(t, `runs-on: ubuntu-latest
needs: [job1]
if: always()`, ""),
})
rc.Run.JobID = "job2"
assertObject.True(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "success"),
"job2": createJob(t, `runs-on: ubuntu-latest
needs: [job1]
if: always()`, ""),
})
rc.Run.JobID = "job2"
assertObject.True(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, "success"),
"job2": createJob(t, `runs-on: ubuntu-latest
if: always()`, ""),
})
rc.Run.JobID = "job2"
assertObject.True(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `uses: ./.github/workflows/reusable.yml`, ""),
})
assertObject.True(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `uses: ./.github/workflows/reusable.yml
if: false`, ""),
})
assertObject.False(rc.isEnabled(t.Context()))
rc = createIfTestRunContext(map[string]*model.Job{
"job1": createJob(t, `if: ${{ fromJSON('not-json') }}`, ""),
})
enabled, err := rc.isEnabled(t.Context())
require.NoError(t, err)
assertObject.False(enabled)
}
func TestRunContextGetEnv(t *testing.T) {
tests := []struct {
description string
rc *RunContext
targetEnv string
want string
for _, testcase := range []struct {
job string
want []string
}{
{
description: "Env from Config should overwrite",
rc: &RunContext{
Config: &Config{
Env: map[string]string{"OVERWRITTEN": "true"},
},
{"runs-on: ubuntu-latest", []string{"ubuntu-latest"}},
{"runs-on: ${{ 'ubuntu-latest' }}", []string{"ubuntu-latest"}},
{"runs-on: [self-hosted, my-runner]", []string{"self-hosted", "my-runner"}},
{`runs-on: [self-hosted, "${{ 'my-runner' }}"]`, []string{"self-hosted", "my-runner"}},
{`runs-on: ${{ fromJSON('["ubuntu-latest"]') }}`, []string{"ubuntu-latest"}},
{"name: something", []string{}},
{"runs-on: {mapping: value}", []string{}},
{"runs-on: ${{ invalid expression }}", []string{}},
} {
t.Run(testcase.job, func(t *testing.T) {
rc := createIfTestRunContext(map[string]*model.Job{"job1": createJob(t, testcase.job, "")})
assert.Equal(t, testcase.want, rc.runsOnPlatformNames(t.Context()))
})
}
}
func TestRunContextIsEnabled(t *testing.T) {
for _, testcase := range []struct {
name, job, neededResult string
want, envError bool
}{
{name: "broken env fails at setup", job: "runs-on: ubuntu-latest\nenv:\n BROKEN: ${{ fromJSON('invalid') }}\nif: success()", want: true, envError: true},
{name: "success with failed dependency", job: "runs-on: ubuntu-latest\nneeds: [needed]\nif: success()", neededResult: "failure"},
{name: "success with successful dependency", job: "runs-on: ubuntu-latest\nneeds: [needed]\nif: success()", neededResult: "success", want: true},
{name: "success with unrelated failed job", job: "runs-on: ubuntu-latest\nif: success()", neededResult: "failure", want: true},
{name: "failure without dependencies", job: "runs-on: ubuntu-latest\nif: failure()"},
{name: "failure with failed dependency", job: "runs-on: ubuntu-latest\nneeds: [needed]\nif: failure()", neededResult: "failure", want: true},
{name: "failure with successful dependency", job: "runs-on: ubuntu-latest\nneeds: [needed]\nif: failure()", neededResult: "success"},
{name: "failure with unrelated failed job", job: "runs-on: ubuntu-latest\nif: failure()", neededResult: "failure"},
{name: "always without dependencies", job: "runs-on: ubuntu-latest\nif: always()", want: true},
{name: "always with failed dependency", job: "runs-on: ubuntu-latest\nneeds: [needed]\nif: always()", neededResult: "failure", want: true},
{name: "always with successful dependency", job: "runs-on: ubuntu-latest\nneeds: [needed]\nif: always()", neededResult: "success", want: true},
{name: "always with unrelated successful job", job: "runs-on: ubuntu-latest\nif: always()", neededResult: "success", want: true},
{name: "reusable workflow", job: "uses: ./.github/workflows/reusable.yml", want: true},
{name: "disabled reusable workflow", job: "uses: ./.github/workflows/reusable.yml\nif: false"},
{name: "invalid condition", job: "if: ${{ fromJSON('not-json') }}"},
} {
t.Run(testcase.name, func(t *testing.T) {
jobs := map[string]*model.Job{"job1": createJob(t, testcase.job, "")}
if testcase.neededResult != "" {
jobs["needed"] = createJob(t, "runs-on: ubuntu-latest", testcase.neededResult)
}
rc := createIfTestRunContext(jobs)
enabled, err := rc.isEnabled(t.Context())
require.NoError(t, err)
assert.Equal(t, testcase.want, enabled)
if testcase.envError {
require.Error(t, evaluateJobEnvAndDefaults(t.Context(), rc))
}
})
}
}
func TestRunContextGetEnv(t *testing.T) {
for _, testcase := range []struct {
name string
env map[string]string
want string
}{
{"Env from Config should overwrite", map[string]string{"OVERWRITTEN": "true"}, "true"},
{"No overwrite occurs", map[string]string{"SOME_OTHER_VAR": "true"}, "false"},
} {
t.Run(testcase.name, func(t *testing.T) {
rc := &RunContext{
Config: &Config{Env: testcase.env},
Run: &model.Run{
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{"test": {Name: "test"}},
Env: map[string]string{"OVERWRITTEN": "false"},
},
JobID: "test",
},
},
targetEnv: "OVERWRITTEN",
want: "true",
},
{
description: "No overwrite occurs",
rc: &RunContext{
Config: &Config{
Env: map[string]string{"SOME_OTHER_VAR": "true"},
},
Run: &model.Run{
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{"test": {Name: "test"}},
Jobs: map[string]*model.Job{"test": {}},
Env: map[string]string{"OVERWRITTEN": "false"},
},
JobID: "test",
},
},
targetEnv: "OVERWRITTEN",
want: "false",
},
}
for _, test := range tests {
t.Run(test.description, func(t *testing.T) {
envMap := test.rc.GetEnv()
assert.EqualValues(t, test.want, envMap[test.targetEnv]) //nolint:testifylint // pre-existing issue from nektos/act
}
assert.Equal(t, testcase.want, rc.GetEnv()["OVERWRITTEN"])
})
}
}
// a remote composite action re-evaluates its env per stage, so inputs must follow it
func TestSetActionEnvRefreshesInputs(t *testing.T) {
rc := &RunContext{}
rc.setActionEnv(map[string]string{"INPUT_MSG": "pre"})
@@ -1432,7 +1189,7 @@ func TestRunContext_cleanupFailedStart(t *testing.T) {
slices.Sort(operations[:3])
slices.Sort(operations[len(operations)-4 : len(operations)-1])
assert.Equal(t, strings.Fields(want+" job.Remove service.Close service.Remove job.Close"), operations)
assert.Empty(t, rc.Masks)
assert.Equal(t, strings.Contains(want, "service.DumpLogs"), slices.Contains(rc.Masks, "leak"))
assert.Equal(t, testcase.pullError == nil && testcase.health != container.HealthUnhealthy, slices.ContainsFunc(hook.AllRecords(), func(entry logging.Record) bool {
return strings.HasPrefix(entry.Message, "::group::Print service container logs")
}))
@@ -1450,15 +1207,40 @@ func TestWaitForServiceContainers(t *testing.T) {
}
}
t.Run("returns as soon as a service without a healthcheck runs", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return(&container.Info{ID: "id", State: "running", Health: container.HealthNone}, nil).Once()
rc := newRunContext(0, &serviceContainer{name: "redis", container: service})
require.NoError(t, rc.waitForServiceContainers()(t.Context()))
service.AssertExpectations(t)
})
for _, testcase := range []struct {
name, serviceName string
timeout time.Duration
info *container.Info
inspectError error
wantError, wantID string
}{
{name: "returns as soon as a service without a healthcheck runs", serviceName: "redis", info: &container.Info{ID: "id", State: "running", Health: container.HealthNone}},
{name: "fails with the probe output when a service is unhealthy", serviceName: "postgres", info: &container.Info{State: "running", Health: container.HealthUnhealthy, HealthOutput: "connection refused"}, wantError: "the service 'postgres' is unhealthy: connection refused"},
{name: "lets the steps run when a service exits without a healthcheck", serviceName: "postgres", info: &container.Info{State: "exited", ExitCode: 2, Health: container.HealthNone}},
{name: "proceeds when the container is gone", serviceName: "postgres", inspectError: container.ErrContainerNotFound},
{name: "does not wait when the timeout is negative", serviceName: "postgres", timeout: -1, info: &container.Info{ID: "id", State: "running", Health: container.HealthStarting}, wantID: "id"},
{name: "fails on an inspect error even when the timeout is negative", serviceName: "postgres", timeout: -1, inspectError: errors.New("daemon is gone"), wantError: "failed to inspect service 'postgres'"},
} {
t.Run(testcase.name, func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).Return(testcase.info, testcase.inspectError).Once()
if testcase.info != nil && testcase.info.Health == container.HealthUnhealthy {
service.On("DumpLogs", mock.Anything).Return(nil).Once()
}
svc := &serviceContainer{name: testcase.serviceName, container: service}
err := newRunContext(testcase.timeout, svc).waitForServiceContainers()(t.Context())
if testcase.wantError != "" {
require.Error(t, err)
assert.Contains(t, err.Error(), testcase.wantError)
} else {
require.NoError(t, err)
}
if testcase.wantID != "" {
assert.Equal(t, testcase.wantID, svc.info.ID)
}
service.AssertExpectations(t)
})
}
t.Run("polls at a fixed interval and logs starting only once", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
@@ -1475,46 +1257,12 @@ func TestWaitForServiceContainers(t *testing.T) {
var output bytes.Buffer
logger := logging.New(logging.NewLevelHandler(newJobLogHandler(&output, cyan), logging.LevelInfo))
require.NoError(t, newRunContext(0, &serviceContainer{name: "postgres", container: service}).waitForServiceContainers()(common.WithLogger(t.Context(), logger)))
assert.Equal(t, []time.Duration{0, time.Second, 2 * time.Second}, pollTimes)
assert.Equal(t, []time.Duration{0, serviceReadyPollInterval, 2 * serviceReadyPollInterval}, pollTimes)
assert.Equal(t, 1, strings.Count(output.String(), "postgres service is starting."))
assert.Contains(t, output.String(), "postgres service is healthy.")
})
})
t.Run("fails with the probe output when a service is unhealthy", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).Return(&container.Info{
State: "running",
Health: container.HealthUnhealthy,
HealthOutput: "connection refused",
}, nil).Once()
service.On("DumpLogs", mock.Anything).Return(nil).Once()
rc := newRunContext(0, &serviceContainer{name: "postgres", container: service})
err := rc.waitForServiceContainers()(t.Context())
require.Error(t, err)
assert.Contains(t, err.Error(), "the service 'postgres' is unhealthy: connection refused")
service.AssertExpectations(t)
})
t.Run("lets the steps run when a service exits without a healthcheck", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return(&container.Info{State: "exited", ExitCode: 2, Health: container.HealthNone}, nil).Once()
rc := newRunContext(0, &serviceContainer{name: "postgres", container: service})
require.NoError(t, rc.waitForServiceContainers()(t.Context()))
})
t.Run("proceeds when the container is gone", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).
Return((*container.Info)(nil), container.ErrContainerNotFound).Once()
rc := newRunContext(0, &serviceContainer{name: "postgres", container: service})
require.NoError(t, rc.waitForServiceContainers()(t.Context()))
})
t.Run("fails right away when one service fails while another is still starting", func(t *testing.T) {
failing := &containerMock{}
failing.On("Inspect", mock.Anything).
@@ -1563,29 +1311,6 @@ func TestWaitForServiceContainers(t *testing.T) {
assert.Contains(t, err.Error(), "did not become healthy within")
})
t.Run("does not wait when the timeout is negative", func(t *testing.T) {
service := &containerMock{}
// Still described once, so the `job.services` context is filled either way.
service.On("Inspect", mock.Anything).
Return(&container.Info{ID: "id", State: "running", Health: container.HealthStarting}, nil).Once()
svc := &serviceContainer{name: "postgres", container: service}
rc := newRunContext(-1, svc)
require.NoError(t, rc.waitForServiceContainers()(t.Context()))
service.AssertExpectations(t)
assert.Equal(t, "id", svc.info.ID)
})
t.Run("fails on an inspect error even when the timeout is negative", func(t *testing.T) {
service := &containerMock{}
service.On("Inspect", mock.Anything).Return((*container.Info)(nil), errors.New("daemon is gone")).Once()
rc := newRunContext(-1, &serviceContainer{name: "postgres", container: service})
err := rc.waitForServiceContainers()(t.Context())
require.Error(t, err)
assert.Contains(t, err.Error(), "failed to inspect service 'postgres'")
})
t.Run("is a no-op without services", func(t *testing.T) {
require.NoError(t, newRunContext(0).waitForServiceContainers()(t.Context()))
})
@@ -1614,9 +1339,7 @@ func TestGetJobContextReportsContainers(t *testing.T) {
jobContainerID: "job-container-id",
serviceContainers: []*serviceContainer{
{name: "postgres", info: &container.Info{ID: "svc-id", Ports: map[string]string{"5432": "49153"}}},
// A service that publishes no port reports an empty map, as GitHub does.
{name: "redis", info: &container.Info{ID: "redis-id", Ports: map[string]string{}}},
// A service that never reported is left out rather than reported as empty.
{name: "mailhog"},
},
}
@@ -1654,7 +1377,6 @@ func TestCaptureJobContainerInfoExportsDockerWorkspace(t *testing.T) {
assert.NotContains(t, rc.Env, "GITEA_DOCKER_WORKSPACE")
}
// A job that never started a container reports an empty context, not a placeholder.
func TestGetJobContextWithoutContainer(t *testing.T) {
jobContext := (&RunContext{}).getJobContext()
+8 -18
View File
@@ -94,20 +94,14 @@ type Config struct {
JobCompletedHook string // script run inside the job environment after the job's last step; ACTIONS_RUNNER_HOOK_JOB_COMPLETED is read from Env when empty
}
// RunnerDebug reports whether debug logging is on, exposed as `runner.debug` and
// RUNNER_DEBUG. Only the secret also makes the reporter keep ::debug:: output, the env
// is accepted for `exec` and for runners configured with it.
// RunnerDebug reports whether debug logging is on, exposed as `runner.debug` and RUNNER_DEBUG.
func (c Config) RunnerDebug() bool {
return c.Secrets["ACTIONS_STEP_DEBUG"] == "true" || c.Env["ACTIONS_STEP_DEBUG"] == "true"
}
// GetToken: Adapt to Gitea
func (c Config) GetToken() string {
token := c.Secrets["GITHUB_TOKEN"]
if c.Secrets["GITEA_TOKEN"] != "" {
token = c.Secrets["GITEA_TOKEN"]
}
return token
return cmp.Or(c.Secrets["GITEA_TOKEN"], c.Secrets["GITHUB_TOKEN"])
}
// DefaultActionURL returns the host used for implicit remote actions.
@@ -187,8 +181,7 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
stagePipeline := make([]common.Executor, 0)
logging.Debugf("Plan Stages: %v", plan.Stages)
for i := range plan.Stages {
stage := plan.Stages[i]
for _, stage := range plan.Stages {
stagePipeline = append(stagePipeline, func(ctx context.Context) error {
pipeline := make([]common.Executor, 0)
for _, run := range stage.Runs {
@@ -270,9 +263,7 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
if len(matrixes) > 1 {
rc.Name = fmt.Sprintf("%s-%d", rc.Name, i+1)
}
if len(rc.String()) > maxJobNameLen {
maxJobNameLen = len(rc.String())
}
maxJobNameLen = max(maxJobNameLen, len(rc.String()))
if rc.caller != nil { // For Gitea
rc.caller.setReusedWorkflowJobResult(rc.Run.JobID, "pending")
}
@@ -324,10 +315,7 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
logging.Debugf("Using CPU count for parallelism: %d", parallelism)
}
// Don't exceed the number of pipeline elements
if parallelism > len(pipeline) {
parallelism = len(pipeline)
}
parallelism = min(parallelism, len(pipeline))
logging.Infof("Executing %d pipeline elements with parallelism %d", len(pipeline), parallelism)
return common.NewParallelExecutor(parallelism, pipeline...)(ctx)
@@ -335,7 +323,9 @@ func (runner *runnerImpl) NewPlanExecutor(plan *model.Plan) common.Executor {
}
executor := common.NewPipelineExecutor(stagePipeline...).Then(handleFailure(plan))
return func(ctx context.Context) error { return executor(ghcontext.WithPinnedCheckout(ctx)) }
return func(ctx context.Context) error {
return executor(container.WithRunnerArch(ghcontext.WithPinnedCheckout(ctx)))
}
}
func handleFailure(plan *model.Plan) common.Executor {
+14 -19
View File
@@ -165,11 +165,14 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
if topRC.summaryFileInitialized == nil {
topRC.summaryFileInitialized = map[int]bool{}
}
_ = rc.JobContainer.Copy(actPath, files...)(ctx)
initialFiles := files
if !topRC.summaryFileInitialized[stepSummaryIndex] {
_ = rc.JobContainer.Copy(actPath, &container.FileEntry{Name: summaryFileCommand, Mode: 0o666})(ctx)
initialFiles = append([]*container.FileEntry{{Name: summaryFileCommand, Mode: 0o666}}, files...)
topRC.summaryFileInitialized[stepSummaryIndex] = true
}
if err := rc.JobContainer.Copy(actPath, initialFiles...)(ctx); err != nil && len(initialFiles) > len(files) {
_ = rc.JobContainer.Copy(actPath, files...)(ctx) // the summary goes first and the resets retry alone, so neither failure blocks the other
}
// The command handler needs the step's env to judge ACTIONS_ALLOW_UNSECURE_COMMANDS.
// Cloned: the step executor keeps writing to its own env map after this point, on a
@@ -212,7 +215,7 @@ func runStepExecutor(step step, stage stepStage, executor common.Executor) commo
} else {
stepResult.Outcome = model.StepStatusFailure
continueOnError, parseErr := isContinueOnError(ctx, stepModel.RawContinueOnError, step, stage)
continueOnError, parseErr := isContinueOnError(ctx, stepModel.RawContinueOnError, step)
if parseErr != nil {
stepResult.Conclusion = model.StepStatusFailure
logger.With("stepResult", stepResult.Conclusion).Infof("Failure - %s %s", stage, stepString)
@@ -251,7 +254,10 @@ func evaluateStepTimeout(ctx context.Context, exprEval *expressionEvaluator, ste
func setupEnv(ctx context.Context, step step) error {
rc := step.getRunContext()
mergeEnv(ctx, step)
env := step.getEnv()
// container env is the image's baseline, which job env and $GITHUB_ENV override
mergeIntoMap(step, env, rc.containerSpec.Env, rc.GetEnv())
rc.withGithubEnv(ctx, step.getGithubContext(ctx), *env)
var err error
exprEval := rc.NewExpressionEvaluator(ctx)
@@ -279,16 +285,6 @@ func setupInputs(ctx context.Context, step step) error {
return nil
}
func mergeEnv(ctx context.Context, step step) {
env := step.getEnv()
rc := step.getRunContext()
// container env is the image's baseline, which job env and $GITHUB_ENV override
mergeIntoMap(step, env, rc.containerSpec.Env, rc.GetEnv())
rc.withGithubEnv(ctx, step.getGithubContext(ctx), *env)
}
func isStepEnabled(ctx context.Context, expr string, step step, stage stepStage) (bool, error) {
rc := step.getRunContext()
@@ -312,9 +308,9 @@ func isStepEnabled(ctx context.Context, expr string, step step, stage stepStage)
return runStep, nil
}
func isContinueOnError(ctx context.Context, expr string, step step, _ stepStage) (bool, error) {
func isContinueOnError(ctx context.Context, expr string, step step) (bool, error) {
// https://github.com/github/docs/blob/3ae84420bd10997bb5f35f629ebb7160fe776eae/content/actions/reference/workflow-syntax-for-github-actions.md?plain=true#L962
if len(strings.TrimSpace(expr)) == 0 {
if strings.TrimSpace(expr) == "" {
return false, nil
}
@@ -352,7 +348,7 @@ func mergeIntoMapCaseInsensitive(target map[string]string, maps ...map[string]st
if k, ok := foldKeys[foldKey]; ok {
return k
}
foldKeys[strings.ToLower(foldKey)] = s
foldKeys[foldKey] = s
return s
}
for _, m := range maps {
@@ -363,8 +359,7 @@ func mergeIntoMapCaseInsensitive(target map[string]string, maps ...map[string]st
}
func symlinkJoin(filename, sym, parent string) (string, error) {
dir := path.Dir(filename)
dest := path.Join(dir, sym)
dest := path.Join(path.Dir(filename), sym)
prefix := path.Clean(parent) + "/"
if strings.HasPrefix(dest, prefix) || prefix == "./" {
return dest, nil
+81 -199
View File
@@ -17,28 +17,37 @@ import (
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"go.yaml.in/yaml/v4"
"github.com/stretchr/testify/require"
)
type stepActionLocalMocks struct {
type stepActionMocks struct {
mock.Mock
}
func (salm *stepActionLocalMocks) runAction(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor {
func (salm *stepActionMocks) runAction(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor {
args := salm.Called(step, actionDir, remoteAction)
return args.Get(0).(func(context.Context) error)
}
func (salm *stepActionLocalMocks) readAction(_ context.Context, step *model.Step, actionDir, actionPath string, readFile actionYamlReader, writeFile fileWriter) (*model.Action, error) {
func (salm *stepActionMocks) readAction(_ctx context.Context, step *model.Step, actionDir, actionPath string, readFile actionYamlReader, writeFile fileWriter) (*model.Action, error) {
args := salm.Called(step, actionDir, actionPath, readFile, writeFile)
return args.Get(0).(*model.Action), args.Error(1)
}
func expectStepFileCommands(t *testing.T, cm *containerMock) {
t.Helper()
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
for _, filename := range []string{"envs.txt", "statecmd.txt", "outputcmd.txt"} {
cm.On("UpdateFromEnv", "/var/run/act/workflow/"+filename, mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
}
cm.On("GetContainerArchive", t.Context(), "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
}
func TestStepActionLocalTest(t *testing.T) {
ctx := t.Context()
cm := &containerMock{}
salm := &stepActionLocalMocks{}
salm := &stepActionMocks{}
sal := &stepActionLocal{
readAction: salm.readAction,
@@ -53,13 +62,7 @@ func TestStepActionLocalTest(t *testing.T) {
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"1": {
Defaults: model.Defaults{
Run: model.RunDefaults{
Shell: "bash",
},
},
},
"1": {},
},
},
},
@@ -74,208 +77,87 @@ func TestStepActionLocalTest(t *testing.T) {
salm.On("readAction", sal.Step, filepath.Clean("/tmp/path/to/action"), "", mock.Anything, mock.Anything).
Return(&model.Action{}, nil)
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
expectStepFileCommands(t, cm)
salm.On("runAction", sal, filepath.Clean("/tmp/path/to/action"), (*remoteAction)(nil)).Return(noopExecutor)
err := sal.pre()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
err = sal.main()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
require.NoError(t, err)
cm.AssertExpectations(t)
salm.AssertExpectations(t)
}
func TestStepActionLocalPost(t *testing.T) {
table := []struct {
name string
stepModel *model.Step
actionModel *model.Action
initialStepResults map[string]*model.StepResult
err error
mocks struct {
env bool
exec bool
}
}{
{
name: "main-success",
stepModel: &model.Step{
ID: "step",
Uses: "./local/action",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "always()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusSuccess,
Outcome: model.StepStatusSuccess,
Outputs: map[string]string{},
},
},
mocks: struct {
env bool
exec bool
func TestStepActionPost(t *testing.T) {
for _, kind := range []string{"local", "remote"} {
t.Run(kind, func(t *testing.T) {
for _, test := range []struct {
name string
status model.StepStatus
postIf string
run bool
}{
env: true,
exec: true,
},
},
{
name: "main-failed",
stepModel: &model.Step{
ID: "step",
Uses: "./local/action",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "always()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusFailure,
Outcome: model.StepStatusFailure,
Outputs: map[string]string{},
},
},
mocks: struct {
env bool
exec bool
}{
env: true,
exec: true,
},
},
{
name: "skip-if-failed",
stepModel: &model.Step{
ID: "step",
Uses: "./local/action",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "success()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusFailure,
Outcome: model.StepStatusFailure,
Outputs: map[string]string{},
},
},
mocks: struct {
env bool
exec bool
}{
env: false,
exec: false,
},
},
{
name: "skip-if-main-skipped",
stepModel: &model.Step{
ID: "step",
If: yaml.Node{Value: "failure()"},
Uses: "./local/action",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "always()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusSkipped,
Outcome: model.StepStatusSkipped,
Outputs: map[string]string{},
},
},
mocks: struct {
env bool
exec bool
}{
env: false,
exec: false,
},
},
}
{"main-success", model.StepStatusSuccess, "always()", true},
{"main-failed", model.StepStatusFailure, "always()", true},
{"skip-if-failed", model.StepStatusFailure, "success()", false},
{"skip-if-main-skipped", model.StepStatusSkipped, "always()", false},
} {
t.Run(test.name, func(t *testing.T) {
ctx := t.Context()
cm := &containerMock{}
rc := &RunContext{
Config: &Config{GitHubInstance: "https://github.com", ActionCacheDir: t.TempDir()},
JobContainer: cm,
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
StepResults: map[string]*model.StepResult{"step": {Conclusion: test.status}},
}
rc.ExprEval = rc.NewExpressionEvaluator(ctx)
action := &model.Action{Runs: model.ActionRuns{Using: "node16", Post: "post.js", PostIf: test.postIf}}
var currentStep step
if kind == "remote" {
currentStep = &stepActionRemote{
RunContext: rc,
Step: &model.Step{ID: "step", Uses: "remote/action@v1"},
action: action,
env: map[string]string{},
remoteAction: mustNewRemoteAction(t, "remote/action@v1"),
}
if test.status == model.StepStatusSuccess {
rc.IntraActionState = map[string]map[string]string{"step": {"key": "value"}}
}
} else {
currentStep = &stepActionLocal{
RunContext: rc,
Step: &model.Step{ID: "step", Uses: "./local/action"},
action: action,
env: map[string]string{},
}
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
ctx := t.Context()
if test.run {
cm.On("Exec", mock.MatchedBy(func(args []string) bool {
if len(args) != 3 || args[0] != "node" || args[1] != "--preserve-symlinks-main" {
return false
}
if kind == "local" {
return strings.HasSuffix(args[2], "runner/local/action/post.js")
}
return strings.Contains(args[2], "post.js")
}), *currentStep.getEnv(), "", "").Return(noopExecutor)
expectStepFileCommands(t, cm)
}
cm := &containerMock{}
sal := &stepActionLocal{
env: map[string]string{},
RunContext: &RunContext{
Config: &Config{
GitHubInstance: "https://github.com",
},
JobContainer: cm,
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"1": {},
},
},
},
StepResults: tt.initialStepResults,
},
Step: tt.stepModel,
action: tt.actionModel,
require.NoError(t, currentStep.post()(ctx))
if rc.IntraActionState != nil {
assert.Equal(t, "value", (*currentStep.getEnv())["STATE_key"])
}
assert.NotContains(t, rc.StepResults, "post-step")
cm.AssertExpectations(t)
})
}
sal.RunContext.ExprEval = sal.RunContext.NewExpressionEvaluator(ctx)
if tt.mocks.exec {
suffixMatcher := func(suffix string) any {
return mock.MatchedBy(func(array []string) bool {
return len(array) == 3 && array[0] == "node" && array[1] == "--preserve-symlinks-main" &&
strings.HasSuffix(array[2], suffix)
})
}
cm.On("Exec", suffixMatcher("runner/local/action/post.js"), sal.env, "", "").Return(func(ctx context.Context) error { return tt.err })
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
}
err := sal.post()(ctx)
assert.Equal(t, tt.err, err)
assert.Equal(t, sal.RunContext.StepResults["post-step"], (*model.StepResult)(nil)) //nolint:testifylint // pre-existing issue from nektos/act
cm.AssertExpectations(t)
})
}
}
+40 -22
View File
@@ -31,16 +31,32 @@ type stepActionRemote struct {
env map[string]string
remoteAction *remoteAction
actionDir string // the repository at the resolved commit, shared by the job's actions in it
resolvedSha string // set on the step that made the job's download of the repository
resolvedSha string // set on the first step preparing the repository
}
type actionDownload struct {
dir string
err error
dir, sha string
err error
fetched chan struct{}
}
var stepActionRemoteFetch = git.FetchAction
func (sar *stepActionRemote) fetchAction(ctx context.Context, remote *remoteAction, token string) (string, string, error) {
cloneURL := remote.CloneURL(sar.RunContext.Config.DefaultActionURL())
return stepActionRemoteFetch(ctx, sar.RunContext.Config.withInstanceAuth(git.NewGitCloneExecutorInput{
URL: cloneURL,
Ref: remote.Ref,
Dir: filepath.Join(sar.RunContext.ActionCacheDir(), model.UsesHash(remote.downloadKey(sar.RunContext.Config.DefaultActionURL()))),
OfflineMode: sar.RunContext.Config.ActionOfflineMode,
Depth: sar.RunContext.Config.ActionCloneDepth,
}, token))
}
func (sar *stepActionRemote) skipsLocalCheckout(remote *remoteAction, github *model.GithubContext) bool {
return remote.IsCheckout() && isLocalCheckout(github, sar.Step) && !sar.RunContext.Config.NoSkipCheckout
}
func (sar *stepActionRemote) prepareActionExecutor() common.Executor {
return func(ctx context.Context) error {
if sar.remoteAction != nil && sar.action != nil {
@@ -63,29 +79,30 @@ func (sar *stepActionRemote) prepareActionExecutor() common.Executor {
return err
}
if sar.remoteAction.IsCheckout() && isLocalCheckout(github, sar.Step) && !sar.RunContext.Config.NoSkipCheckout {
if sar.skipsLocalCheckout(sar.remoteAction, github) {
common.Logger(ctx).Debugf("Skipping local actions/checkout because workdir was already copied")
return nil
}
cloneURL := sar.remoteAction.CloneURL(sar.RunContext.Config.DefaultActionURL())
job := sar.RunContext.topLevelRunContext()
key := cloneURL + "@" + sar.remoteAction.Ref
download, ok := job.actionDownloads[key]
if !ok {
key := sar.remoteAction.downloadKey(sar.RunContext.Config.DefaultActionURL())
download := job.actionDownloads[key]
if download == nil {
download = &actionDownload{}
// For Gitea
// github.Token, unlike Config.GetToken, survives the composite config copy, which nils Config.Secrets.
download.dir, sar.resolvedSha, download.err = stepActionRemoteFetch(ctx, sar.RunContext.Config.withInstanceAuth(git.NewGitCloneExecutorInput{
URL: cloneURL,
Ref: sar.remoteAction.Ref,
Dir: filepath.Join(sar.RunContext.ActionCacheDir(), model.UsesHash(key)),
OfflineMode: sar.RunContext.Config.ActionOfflineMode,
Depth: sar.RunContext.Config.ActionCloneDepth,
}, github.Token))
download.dir, download.sha, download.err = sar.fetchAction(ctx, sar.remoteAction, github.Token)
if job.actionDownloads == nil {
job.actionDownloads = map[string]actionDownload{}
job.actionDownloads = map[string]*actionDownload{}
}
job.actionDownloads[key] = download
} else if download.fetched != nil {
select {
case <-download.fetched:
case <-ctx.Done():
return ctx.Err()
}
}
sar.resolvedSha, download.sha = download.sha, ""
var refErr *git.Error
switch {
case errors.As(download.err, &refErr) && errors.Is(download.err, git.ErrShortRef):
@@ -137,7 +154,7 @@ func (sar *stepActionRemote) main() common.Executor {
defer rawLogger.Infof("::endgroup::")
github := sar.getGithubContext(ctx)
if sar.remoteAction.IsCheckout() && isLocalCheckout(github, sar.Step) && !sar.RunContext.Config.NoSkipCheckout {
if sar.skipsLocalCheckout(sar.remoteAction, github) {
if sar.RunContext.Config.BindWorkdir {
common.Logger(ctx).Debugf("Skipping local actions/checkout because you bound your workspace")
return nil
@@ -188,7 +205,7 @@ func (sar *stepActionRemote) getIfExpression(ctx context.Context, stage stepStag
switch stage {
case stepStagePre:
github := sar.getGithubContext(ctx)
if sar.remoteAction.IsCheckout() && isLocalCheckout(github, sar.Step) && !sar.RunContext.Config.NoSkipCheckout {
if sar.skipsLocalCheckout(sar.remoteAction, github) {
// skip local checkout pre step
return "false"
}
@@ -244,6 +261,10 @@ type remoteAction struct {
Ref string
}
func (ra *remoteAction) downloadKey(defaultURL string) string {
return ra.CloneURL(defaultURL) + "@" + ra.Ref
}
func (ra *remoteAction) CloneURL(u string) string {
if ra.URL == "" {
// keep an absolute local path as-is (used by tests to resolve actions from a local
@@ -269,10 +290,7 @@ func (ra *remoteAction) Reference() string {
}
func (ra *remoteAction) IsCheckout() bool {
if ra.Org == "actions" && ra.Repo == "checkout" {
return true
}
return false
return ra.Org == "actions" && ra.Repo == "checkout"
}
// newRemoteAction resolves `self:` on instanceURL and `$/` in the enclosing composite action, else the workflow's repo and commit.
+44 -470
View File
@@ -5,11 +5,9 @@
package runner
import (
"bytes"
"cmp"
"context"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path"
@@ -28,10 +26,6 @@ import (
"go.yaml.in/yaml/v4"
)
type stepActionRemoteMocks struct {
mock.Mock
}
func setFetch(t *testing.T, fetch func(context.Context, git.NewGitCloneExecutorInput) (string, string, error)) {
original := stepActionRemoteFetch
stepActionRemoteFetch = fetch
@@ -74,108 +68,21 @@ func TestShortSHAActionRejected(t *testing.T) {
require.ErrorContains(t, err, "shortened version of a commit SHA")
}
func (sarm *stepActionRemoteMocks) readAction(_ context.Context, step *model.Step, actionDir, actionPath string, readFile actionYamlReader, writeFile fileWriter) (*model.Action, error) {
args := sarm.Called(step, actionDir, actionPath, readFile, writeFile)
return args.Get(0).(*model.Action), args.Error(1)
}
func (sarm *stepActionRemoteMocks) runAction(step actionStep, actionDir string, remoteAction *remoteAction) common.Executor {
args := sarm.Called(step, actionDir, remoteAction)
return args.Get(0).(func(context.Context) error)
}
func TestStepActionRemote(t *testing.T) {
table := []struct {
name string
stepModel *model.Step
result *model.StepResult
mocks struct {
env bool
cloned bool
read bool
run bool
}
runError error
for _, tt := range []struct {
name, condition string
status model.StepStatus
runError error
}{
{
name: "run-successful",
stepModel: &model.Step{
ID: "step",
Uses: "remote/action@v1",
},
result: &model.StepResult{
Conclusion: model.StepStatusSuccess,
Outcome: model.StepStatusSuccess,
Outputs: map[string]string{},
},
mocks: struct {
env bool
cloned bool
read bool
run bool
}{
env: true,
cloned: true,
read: true,
run: true,
},
},
{
name: "run-skipped",
stepModel: &model.Step{
ID: "step",
Uses: "remote/action@v1",
If: yaml.Node{Value: "false"},
},
result: &model.StepResult{
Conclusion: model.StepStatusSkipped,
Outcome: model.StepStatusSkipped,
Outputs: map[string]string{},
},
mocks: struct {
env bool
cloned bool
read bool
run bool
}{
env: true,
cloned: true,
read: true,
run: false,
},
},
{
name: "run-error",
stepModel: &model.Step{
ID: "step",
Uses: "remote/action@v1",
},
result: &model.StepResult{
Conclusion: model.StepStatusFailure,
Outcome: model.StepStatusFailure,
Outputs: map[string]string{},
},
mocks: struct {
env bool
cloned bool
read bool
run bool
}{
env: true,
cloned: true,
read: true,
run: true,
},
runError: errors.New("error"),
},
}
for _, tt := range table {
{name: "run-successful", status: model.StepStatusSuccess},
{name: "run-skipped", condition: "false", status: model.StepStatusSkipped},
{name: "run-error", status: model.StepStatusFailure, runError: assert.AnError},
} {
t.Run(tt.name, func(t *testing.T) {
ctx := t.Context()
cm := &containerMock{}
sarm := &stepActionRemoteMocks{}
sarm := &stepActionMocks{}
clonedAction := false
actionDir := t.TempDir()
@@ -201,37 +108,23 @@ func TestStepActionRemote(t *testing.T) {
StepResults: map[string]*model.StepResult{},
JobContainer: cm,
},
Step: tt.stepModel,
Step: &model.Step{ID: "step", Uses: "remote/action@v1", If: yaml.Node{Value: tt.condition}},
readAction: sarm.readAction,
runAction: sarm.runAction,
}
sar.RunContext.ExprEval = sar.RunContext.NewExpressionEvaluator(ctx)
if tt.mocks.read {
sarm.On("readAction", sar.Step, actionDir, "", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
}
if tt.mocks.run {
sarm.On("runAction", sar, actionDir, mustNewRemoteAction(t, sar.Step.Uses)).Return(func(ctx context.Context) error { return tt.runError })
sarm.On("readAction", sar.Step, actionDir, "", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
if tt.status != model.StepStatusSkipped {
sarm.On("runAction", sar, actionDir, mustNewRemoteAction(t, sar.Step.Uses)).Return(func(_ctx context.Context) error { return tt.runError })
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
expectStepFileCommands(t, cm)
}
err := sar.pre()(ctx)
if err == nil {
err = sar.main()(ctx)
}
assert.Equal(t, tt.runError, err)
assert.Equal(t, tt.mocks.cloned, clonedAction)
assert.Equal(t, tt.result, sar.RunContext.StepResults["step"])
require.NoError(t, sar.pre()(ctx))
assert.Equal(t, tt.runError, sar.main()(ctx))
assert.True(t, clonedAction)
assert.Equal(t, &model.StepResult{Conclusion: tt.status, Outcome: tt.status, Outputs: map[string]string{}}, sar.RunContext.StepResults["step"])
sarm.AssertExpectations(t)
cm.AssertExpectations(t)
@@ -314,341 +207,33 @@ func TestStepActionRemotePrepare(t *testing.T) {
assert.Equal(t, "restore", path.Base(restoreContainerDir))
}
func TestStepActionRemotePost(t *testing.T) {
table := []struct {
name string
stepModel *model.Step
actionModel *model.Action
initialStepResults map[string]*model.StepResult
IntraActionState map[string]map[string]string
expectedEnv map[string]string
err error
mocks struct {
env bool
exec bool
}
}{
{
name: "main-success",
stepModel: &model.Step{
ID: "step",
Uses: "remote/action@v1",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "always()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusSuccess,
Outcome: model.StepStatusSuccess,
Outputs: map[string]string{},
},
},
IntraActionState: map[string]map[string]string{
"step": {
"key": "value",
},
},
expectedEnv: map[string]string{
"STATE_key": "value",
},
mocks: struct {
env bool
exec bool
}{
env: true,
exec: true,
},
},
{
name: "main-failed",
stepModel: &model.Step{
ID: "step",
Uses: "remote/action@v1",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "always()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusFailure,
Outcome: model.StepStatusFailure,
Outputs: map[string]string{},
},
},
mocks: struct {
env bool
exec bool
}{
env: true,
exec: true,
},
},
{
name: "skip-if-failed",
stepModel: &model.Step{
ID: "step",
Uses: "remote/action@v1",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "success()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusFailure,
Outcome: model.StepStatusFailure,
Outputs: map[string]string{},
},
},
mocks: struct {
env bool
exec bool
}{
env: true,
exec: false,
},
},
{
name: "skip-if-main-skipped",
stepModel: &model.Step{
ID: "step",
If: yaml.Node{Value: "failure()"},
Uses: "remote/action@v1",
},
actionModel: &model.Action{
Runs: model.ActionRuns{
Using: "node16",
Post: "post.js",
PostIf: "always()",
},
},
initialStepResults: map[string]*model.StepResult{
"step": {
Conclusion: model.StepStatusSkipped,
Outcome: model.StepStatusSkipped,
Outputs: map[string]string{},
},
},
mocks: struct {
env bool
exec bool
}{
env: false,
exec: false,
},
},
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
ctx := t.Context()
cm := &containerMock{}
sar := &stepActionRemote{
env: map[string]string{},
RunContext: &RunContext{
Config: &Config{
GitHubInstance: "https://github.com",
ActionCacheDir: t.TempDir(),
},
JobContainer: cm,
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"1": {},
},
},
},
StepResults: tt.initialStepResults,
IntraActionState: tt.IntraActionState,
},
Step: tt.stepModel,
action: tt.actionModel,
// post only ever runs after prepareActionExecutor resolved the action
remoteAction: mustNewRemoteAction(t, tt.stepModel.Uses),
}
sar.RunContext.ExprEval = sar.RunContext.NewExpressionEvaluator(ctx)
if tt.mocks.exec {
// Use mock.MatchedBy to match the exec command with hash-based path
execMatcher := mock.MatchedBy(func(args []string) bool {
if len(args) != 3 {
return false
}
return args[0] == "node" && args[1] == "--preserve-symlinks-main" && strings.Contains(args[2], "post.js")
})
cm.On("Exec", execMatcher, sar.env, "", "").Return(func(ctx context.Context) error { return tt.err })
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
}
err := sar.post()(ctx)
assert.Equal(t, tt.err, err)
if tt.expectedEnv != nil {
for key, value := range tt.expectedEnv {
assert.Equal(t, value, sar.env[key])
}
}
// Enshure that StepResults is nil in this test
assert.Equal(t, sar.RunContext.StepResults["post-step"], (*model.StepResult)(nil)) //nolint:testifylint // pre-existing issue from nektos/act
cm.AssertExpectations(t)
})
}
}
func Test_newRemoteAction(t *testing.T) {
tests := []struct {
action string
want *remoteAction
wantCloneURL string
}{
{
action: "actions/heroku@main",
want: &remoteAction{
URL: "",
Org: "actions",
Repo: "heroku",
Path: "",
Ref: "main",
},
wantCloneURL: "https://github.com/actions/heroku",
},
{
action: "actions/aws/ec2@main",
want: &remoteAction{
URL: "",
Org: "actions",
Repo: "aws",
Path: "ec2",
Ref: "main",
},
wantCloneURL: "https://github.com/actions/aws",
},
{
action: "./.github/actions/my-action", // it's valid for GitHub, but act don't support it
want: nil,
},
{
action: "docker://alpine:3.8", // it's valid for GitHub, but act don't support it
want: nil,
},
{
action: "https://gitea.com/actions/heroku@main", // it's invalid for GitHub, but gitea supports it
want: &remoteAction{
URL: "https://gitea.com",
Org: "actions",
Repo: "heroku",
Path: "",
Ref: "main",
},
wantCloneURL: "https://gitea.com/actions/heroku",
},
{
action: "https://gitea.com/actions/aws/ec2@main", // it's invalid for GitHub, but gitea supports it
want: &remoteAction{
URL: "https://gitea.com",
Org: "actions",
Repo: "aws",
Path: "ec2",
Ref: "main",
},
wantCloneURL: "https://gitea.com/actions/aws",
},
{
action: "http://gitea.com/actions/heroku@main", // it's invalid for GitHub, but gitea supports it
want: &remoteAction{
URL: "http://gitea.com",
Org: "actions",
Repo: "heroku",
Path: "",
Ref: "main",
},
wantCloneURL: "http://gitea.com/actions/heroku",
},
{
action: "http://gitea.com/actions/aws/ec2@main", // it's invalid for GitHub, but gitea supports it
want: &remoteAction{
URL: "http://gitea.com",
Org: "actions",
Repo: "aws",
Path: "ec2",
Ref: "main",
},
wantCloneURL: "http://gitea.com/actions/aws",
},
{
action: "ssh://[email protected]/actions/heroku@main", // it's invalid for GitHub, but gitea supports it
want: &remoteAction{
URL: "ssh://[email protected]",
Org: "actions",
Repo: "heroku",
Path: "",
Ref: "main",
},
wantCloneURL: "ssh://[email protected]/actions/heroku",
},
{
action: "ssh://[email protected]/actions/aws/ec2@main", // the ssh user is kept as part of the host segment
want: &remoteAction{
URL: "ssh://[email protected]",
Org: "actions",
Repo: "aws",
Path: "ec2",
Ref: "main",
},
wantCloneURL: "ssh://[email protected]/actions/aws",
},
{
action: "ssh://gitea.com/onlyonesegment@main", // missing org/repo after the host
want: nil,
},
{
action: "self:owner/repo/sub@v1",
want: &remoteAction{
URL: "https://gitea.example.com",
Org: "owner",
Repo: "repo",
Path: "sub",
Ref: "v1",
},
wantCloneURL: "https://gitea.example.com/owner/repo",
},
}
for _, tt := range tests {
t.Run(tt.action, func(t *testing.T) {
got, _ := newRemoteAction(tt.action, nil, "gitea.example.com")
assert.Equalf(t, tt.want, got, "newRemoteAction(%v)", tt.action)
check := func(action string, want *remoteAction, wantCloneURL string) {
t.Run(action, func(t *testing.T) {
got, _ := newRemoteAction(action, nil, "gitea.example.com")
assert.Equalf(t, want, got, "newRemoteAction(%v)", action)
cloneURL := ""
if got != nil {
cloneURL = got.CloneURL("github.com")
}
assert.Equalf(t, tt.wantCloneURL, cloneURL, "newRemoteAction(%v).CloneURL()", tt.action)
assert.Equalf(t, wantCloneURL, cloneURL, "newRemoteAction(%v).CloneURL()", action)
})
}
for _, instance := range []string{"", "https://gitea.com", "http://gitea.com", "ssh://[email protected]"} {
for _, repository := range []struct{ name, path string }{{"heroku", ""}, {"aws", "ec2"}} {
action := path.Join("actions", repository.name, repository.path) + "@main"
if instance != "" {
action = instance + "/" + action
}
check(action, &remoteAction{URL: instance, Org: "actions", Repo: repository.name, Path: repository.path, Ref: "main"},
cmp.Or(instance, "https://github.com")+"/actions/"+repository.name)
}
}
for _, action := range []string{"./.github/actions/my-action", "docker://alpine:3.8", "ssh://gitea.com/onlyonesegment@main"} {
check(action, nil, "")
}
check("self:owner/repo/sub@v1", &remoteAction{URL: "https://gitea.example.com", Org: "owner", Repo: "repo", Path: "sub", Ref: "v1"},
"https://gitea.example.com/owner/repo")
_, err := newRemoteAction("self:owner/repo@v1", nil, "")
require.ErrorContains(t, err, "without a Gitea instance")
}
@@ -734,7 +319,6 @@ func Test_remoteActionReference(t *testing.T) {
}{
{uses: "actions/checkout@v7", want: "actions/checkout@v7"},
{uses: "actions/aws/ec2@main", want: "actions/aws/ec2@main"},
// The download source can be interpolated from a secret and must stay out of the log.
{uses: "https://gitea.example.com/actions/checkout@v7", want: "actions/checkout@v7"},
}
for _, tt := range tests {
@@ -744,9 +328,6 @@ func Test_remoteActionReference(t *testing.T) {
}
}
// TestStepActionRemotePreResolvesDownloadedCommit runs the real download path against a local
// git repository standing in for the actions instance, so the reported commit is the one the
// clone actually checked out.
func TestStepActionRemotePreResolvesDownloadedCommit(t *testing.T) {
instance := t.TempDir()
actionDir := filepath.Join(instance, "actions", "setup-go")
@@ -754,9 +335,6 @@ func TestStepActionRemotePreResolvesDownloadedCommit(t *testing.T) {
require.NoError(t, os.WriteFile(filepath.Join(actionDir, "action.yml"),
[]byte("name: setup-go\nruns:\n using: node20\n main: index.js\n"), 0o600))
// Supply an identity on the commit so the test does not depend on a
// git identity being configured in the environment; a CI runner without
// user.name/user.email would otherwise fail "commit" with exit code 128.
for _, args := range [][]string{
{"init", "--initial-branch=main", actionDir},
{"-C", actionDir, "add", "action.yml"},
@@ -785,6 +363,7 @@ func TestStepActionRemotePreResolvesDownloadedCommit(t *testing.T) {
readAction: readActionImpl,
}
require.NoError(t, printPrepareActions(sar.RunContext, []actionPreparer{sar})(t.Context()))
require.NoError(t, sar.prepareActionExecutor()(t.Context()))
reference, sha := sar.actionDownloadInfo()
@@ -813,9 +392,6 @@ func Test_safeFilename(t *testing.T) {
}
}
// Regression: a nested action in a composite cloned anonymously (401) because the
// composite RunContext nils Config.Secrets. The token must come from github.Token,
// which survives the config copy; the host gate must still withhold it cross-host.
func TestStepActionRemoteCloneTokenSurvivesNilSecrets(t *testing.T) {
const wantToken = "job-token"
@@ -849,7 +425,7 @@ func TestStepActionRemoteCloneTokenSurvivesNilSecrets(t *testing.T) {
return actionDir, "", nil
})
sarm := &stepActionRemoteMocks{}
sarm := &stepActionMocks{}
sar := &stepActionRemote{
Step: &model.Step{Uses: "org/repo@v1"},
RunContext: &RunContext{
@@ -857,10 +433,8 @@ func TestStepActionRemoteCloneTokenSurvivesNilSecrets(t *testing.T) {
GitHubInstance: tt.gitHubInstance,
DefaultActionInstance: tt.defaultActionInstance,
ActionCacheDir: t.TempDir(),
// Mirrors the state of a composite RunContext: job secrets are
// stripped, but the job token is still reachable via Config.Token.
Secrets: nil,
Token: wantToken,
Secrets: nil,
Token: wantToken,
},
Run: &model.Run{
JobID: "1",
@@ -876,7 +450,7 @@ func TestStepActionRemoteCloneTokenSurvivesNilSecrets(t *testing.T) {
sarm.On("readAction", sar.Step, actionDir, "", mock.Anything, mock.Anything).Return(&model.Action{}, nil)
err := sar.prepareActionExecutor()(ctx)
err := printPrepareActions(sar.RunContext, []actionPreparer{sar})(ctx)
require.NoError(t, err)
assert.Equal(t, tt.wantCloneToken, capturedToken)
+34 -129
View File
@@ -5,8 +5,6 @@
package runner
import (
"bytes"
"io"
"slices"
"strings"
"testing"
@@ -15,7 +13,6 @@ import (
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
)
@@ -24,7 +21,6 @@ func TestStepDockerMain(t *testing.T) {
var input *container.NewContainerInput
// mock the new container call
origContainerNewContainer := ContainerNewContainer
ContainerNewContainer = func(containerInput *container.NewContainerInput) container.ExecutionsEnvironment {
input = containerInput
@@ -79,22 +75,11 @@ func TestStepDockerMain(t *testing.T) {
cm.On("Close").Return(noopExecutor)
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
err := sd.main()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
expectStepFileCommands(t, cm)
require.NoError(t, sd.main()(ctx))
assert.Equal(t, "node:14", input.Image)
// DOCKER_USERNAME/DOCKER_PASSWORD secrets should not be used as implicit pull credentials for docker:// action containers.
assert.Empty(t, input.Username)
assert.Empty(t, input.Password)
assert.True(t, input.AutoRemove)
@@ -102,90 +87,20 @@ func TestStepDockerMain(t *testing.T) {
cm.AssertExpectations(t)
}
func TestStepDockerNewStepContainerAllocatePTY(t *testing.T) {
for _, tc := range []struct {
name string
allocPTY bool
}{
{name: "off", allocPTY: false},
{name: "on", allocPTY: true},
} {
t.Run(tc.name, func(t *testing.T) {
cm := &containerMock{}
var captured *container.NewContainerInput
origContainerNewContainer := ContainerNewContainer
ContainerNewContainer = func(input *container.NewContainerInput) container.ExecutionsEnvironment {
captured = input
return cm
}
defer func() {
ContainerNewContainer = origContainerNewContainer
}()
ctx := t.Context()
sd := &stepDocker{
RunContext: &RunContext{
StepResults: map[string]*model.StepResult{},
Config: &Config{
AllocatePTY: tc.allocPTY,
PlatformPicker: func(_ []string) string {
return "node:14"
},
},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{"1": {}},
},
},
JobContainer: cm,
},
Step: &model.Step{ID: "1", Uses: "docker://node:14"},
}
sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx)
newStepContainer(ctx, sd, "node:14", []string{"echo", "hi"}, nil, "")
assert.Equal(t, tc.allocPTY, captured.AllocatePTY)
})
}
}
func TestStepDockerNewStepContainerNetworkModeAndPath(t *testing.T) {
func TestStepDockerNewStepContainer(t *testing.T) {
inheritedPath := (&container.LinuxContainerEnvironmentExtensions{}).DefaultPathVariable()
cases := []struct {
name string
platform string
path string
expectDefault bool
keepsPath bool
for _, test := range []struct {
name, platform, image, path, greeting string
allocPTY, expectDefault, keepsPath bool
}{
{
name: "docker mode attaches to job container network",
platform: "node:14",
path: inheritedPath,
expectDefault: false,
keepsPath: true,
},
{
name: "host mode uses default network and the image's PATH",
platform: "-self-hosted",
path: inheritedPath,
expectDefault: true,
},
{
name: "host mode keeps a PATH that is not the runner's",
platform: "-self-hosted",
path: "/custom/bin",
expectDefault: true,
keepsPath: true,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
{name: "allocate PTY/off", platform: "node:14", image: "node:14", greeting: "hi"},
{name: "allocate PTY/on", platform: "node:14", image: "node:14", greeting: "hi", allocPTY: true},
{name: "docker mode attaches to job container network", platform: "node:14", image: "alpine:3.20", path: inheritedPath, greeting: "hello", keepsPath: true},
{name: "host mode uses default network and the image's PATH", platform: "-self-hosted", image: "alpine:3.20", path: inheritedPath, greeting: "hello", expectDefault: true},
{name: "host mode keeps a PATH that is not the runner's", platform: "-self-hosted", image: "alpine:3.20", path: "/custom/bin", greeting: "hello", expectDefault: true, keepsPath: true},
} {
t.Run(test.name, func(t *testing.T) {
cm := &containerMock{}
var captured *container.NewContainerInput
origContainerNewContainer := ContainerNewContainer
ContainerNewContainer = func(input *container.NewContainerInput) container.ExecutionsEnvironment {
@@ -197,48 +112,38 @@ func TestStepDockerNewStepContainerNetworkModeAndPath(t *testing.T) {
}()
ctx := t.Context()
platform := tc.platform
sd := &stepDocker{
RunContext: &RunContext{
StepResults: map[string]*model.StepResult{},
Config: &Config{
PlatformPicker: func(_ []string) string {
return platform
},
},
Run: &model.Run{
JobID: "1",
Workflow: &model.Workflow{
Jobs: map[string]*model.Job{
"1": {},
},
AllocatePTY: test.allocPTY,
PlatformPicker: func(_platforms []string) string {
return test.platform
},
},
Run: &model.Run{JobID: "1", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"1": {}}}},
JobContainer: cm,
},
Step: &model.Step{
ID: "1",
Uses: "docker://alpine:3.20",
},
env: map[string]string{"PATH": tc.path},
Step: &model.Step{ID: "1", Uses: "docker://" + test.image},
}
sd.RunContext.ExprEval = sd.RunContext.NewExpressionEvaluator(ctx)
if test.path != "" {
sd.env = map[string]string{"PATH": test.path}
require.NoError(t, sd.RunContext.resolvePlatformImage(ctx))
assert.Equal(t, test.expectDefault, sd.RunContext.IsHostEnv(), "IsHostEnv mismatch for platform %q", test.platform)
}
require.NoError(t, sd.RunContext.resolvePlatformImage(ctx))
assert.Equal(t, tc.expectDefault, sd.RunContext.IsHostEnv(), "IsHostEnv mismatch for platform %q", tc.platform)
newStepContainer(ctx, sd, "alpine:3.20", []string{"echo", "hello"}, nil, "")
assert.Equal(t, tc.keepsPath, slices.Contains(captured.Env, "PATH="+tc.path))
if tc.expectDefault {
assert.Equal(t, "default", captured.NetworkMode,
"host-mode step container must use 'default' network, got %q",
captured.NetworkMode)
} else {
assert.True(t, strings.HasPrefix(captured.NetworkMode, "container:"),
"docker-mode step container must attach to job container network, got %q",
captured.NetworkMode)
newStepContainer(ctx, sd, test.image, []string{"echo", test.greeting}, nil, "")
assert.Equal(t, test.allocPTY, captured.AllocatePTY)
if test.path != "" {
assert.Equal(t, test.keepsPath, slices.Contains(captured.Env, "PATH="+test.path))
if test.expectDefault {
assert.Equal(t, "default", captured.NetworkMode,
"host-mode step container must use 'default' network, got %q", captured.NetworkMode)
} else {
assert.True(t, strings.HasPrefix(captured.NetworkMode, "container:"),
"docker-mode step container must attach to job container network, got %q", captured.NetworkMode)
}
}
})
}
+13 -62
View File
@@ -9,74 +9,25 @@ import (
"gitea.dev/actionslib/pkg/model"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestStepFactoryNewStep(t *testing.T) {
table := []struct {
for _, test := range []struct {
name string
model *model.Step
check func(s step) bool
want step
}{
{
name: "StepRemoteAction",
model: &model.Step{
Uses: "remote/action@v1",
},
check: func(s step) bool {
_, ok := s.(*stepActionRemote)
return ok
},
},
{
name: "StepLocalAction",
model: &model.Step{
Uses: "./action@v1",
},
check: func(s step) bool {
_, ok := s.(*stepActionLocal)
return ok
},
},
{
name: "StepDocker",
model: &model.Step{
Uses: "docker://image:tag",
},
check: func(s step) bool {
_, ok := s.(*stepDocker)
return ok
},
},
{
name: "StepRun",
model: &model.Step{
Run: "cmd",
},
check: func(s step) bool {
_, ok := s.(*stepRun)
return ok
},
},
{
name: "StepBuiltinAction",
model: &model.Step{
Uses: "builtin:checkout",
},
check: func(s step) bool {
_, ok := s.(*stepActionBuiltin)
return ok
},
},
}
for _, tt := range table {
t.Run(tt.name, func(t *testing.T) {
sf := &stepFactoryImpl{}
step, err := sf.newStep(tt.model, &RunContext{})
assert.True(t, tt.check(step))
assert.NoError(t, err)
{"StepRemoteAction", &model.Step{Uses: "remote/action@v1"}, &stepActionRemote{}},
{"StepLocalAction", &model.Step{Uses: "./action@v1"}, &stepActionLocal{}},
{"StepDocker", &model.Step{Uses: "docker://image:tag"}, &stepDocker{}},
{"StepRun", &model.Step{Run: "cmd"}, &stepRun{}},
{"StepBuiltinAction", &model.Step{Uses: "builtin:checkout"}, &stepActionBuiltin{}},
} {
t.Run(test.name, func(t *testing.T) {
step, err := (&stepFactoryImpl{}).newStep(test.model, &RunContext{})
require.NoError(t, err)
assert.IsType(t, test.want, step)
})
}
}
+24 -50
View File
@@ -5,6 +5,7 @@
package runner
import (
"cmp"
"context"
"fmt"
"io"
@@ -23,8 +24,6 @@ import (
yaml "go.yaml.in/yaml/v4"
)
var builtinShells = []string{"bash", "sh", "pwsh", "powershell", "cmd", "python"}
type stepRun struct {
Step *model.Step
RunContext *RunContext
@@ -41,14 +40,24 @@ func (sr *stepRun) pre() common.Executor { return common.NewPipelineExecutor() }
func (sr *stepRun) main() common.Executor {
sr.env = map[string]string{}
return runStepExecutor(sr, stepStageMain, common.NewPipelineExecutor(
sr.setupShellCommandExecutor(),
func(ctx context.Context) error {
scriptName, script, err := sr.setupShellCommand(ctx)
if err != nil {
return err
}
return sr.RunContext.JobContainer.Copy(sr.RunContext.JobContainer.GetActPath(), &container.FileEntry{
Name: scriptName,
Mode: 0o755,
Body: script,
})(ctx)
},
func(ctx context.Context) error {
rc := sr.getRunContext()
// Apply ::add-path:: effects before printing so PATH is accurate in the env: block.
rc.ApplyExtraPath(ctx, &sr.env)
sr.printRunScriptActionDetails(ctx)
if he, ok := rc.JobContainer.(*container.HostEnvironment); ok && he != nil {
return he.ExecWithCmdLine(sr.cmd, sr.cmdline, sr.env, "", sr.WorkingDirectory)(ctx)
return he.ExecWithCmdLine(sr.cmd, sr.cmdline, sr.env, sr.WorkingDirectory)(ctx)
}
return rc.JobContainer.Exec(sr.cmd, sr.env, "", sr.WorkingDirectory)(ctx)
},
@@ -86,11 +95,7 @@ func printRunActionHeader(ctx context.Context, step *model.Step, env map[string]
}
rawLogger := common.Logger(ctx).With(rawOutputField, true)
title := step.Uses
if step.Name != "" {
title = step.Name
}
rawLogger.Infof("::group::Run %s", EscapeCommandData(title))
rawLogger.Infof("::group::Run %s", EscapeCommandData(cmp.Or(step.Name, step.Uses)))
if len(step.With) > 0 {
rawLogger.Infof("with:")
@@ -223,30 +228,6 @@ func (sr *stepRun) getIfExpression(_ context.Context, _ stepStage) string {
return sr.Step.If.Value
}
func (sr *stepRun) setupShellCommandExecutor() common.Executor {
return func(ctx context.Context) error {
scriptName, script, err := sr.setupShellCommand(ctx)
if err != nil {
return err
}
rc := sr.getRunContext()
return rc.JobContainer.Copy(rc.JobContainer.GetActPath(), &container.FileEntry{
Name: scriptName,
Mode: 0o755,
Body: script,
})(ctx)
}
}
func getScriptName(rc *RunContext, step *model.Step) string {
scriptName := step.ID
for rcs := rc; rcs.Parent != nil; rcs = rcs.Parent {
scriptName = fmt.Sprintf("%s-composite-%s", rcs.Parent.CurrentStep, scriptName)
}
return "workflow/" + scriptName
}
// TODO: Currently we just ignore top level keys, BUT we should return proper error on them
// BUTx2 I leave this for when we rewrite act to use actionlint for workflow validation
// so we return proper errors before any execution or spawning containers
@@ -272,8 +253,9 @@ func (sr *stepRun) setupShellCommand(ctx context.Context) (name, script string,
sr.interpolatedScript = script
// GitHub matches the built-in names case-insensitively, so `shell: PWSH` is valid
if slices.Contains(builtinShells, strings.ToLower(step.Shell)) {
step.Shell = strings.ToLower(step.Shell)
switch shell := strings.ToLower(step.Shell); shell {
case "bash", "sh", "pwsh", "powershell", "cmd", "python":
step.Shell = shell
}
scCmd := step.ShellCommand()
@@ -285,7 +267,11 @@ func (sr *stepRun) setupShellCommand(ctx context.Context) (name, script string,
}
sr.shellCommand = scCmd
name = getScriptName(sr.RunContext, step)
name = step.ID
for rc := sr.RunContext; rc.Parent != nil; rc = rc.Parent {
name = fmt.Sprintf("%s-composite-%s", rc.Parent.CurrentStep, name)
}
name = "workflow/" + name
// Reference: https://github.com/actions/runner/blob/8109c962f09d9acc473d92c595ff43afceddb347/src/Runner.Worker/Handlers/ScriptHandlerHelpers.cs#L47-L64
// Reference: https://github.com/actions/runner/blob/8109c962f09d9acc473d92c595ff43afceddb347/src/Runner.Worker/Handlers/ScriptHandlerHelpers.cs#L19-L27
@@ -330,13 +316,7 @@ func (sr *stepRun) setupShell(ctx context.Context, eval *expressionEvaluator) (b
if err != nil {
return false, fmt.Errorf("unable to interpolate the shell: %w", err)
}
if shell == "" {
shell = rc.jobRunDefaults.Shell
}
if shell == "" {
shell = rc.Run.Workflow.Defaults.Run.Shell
}
step.Shell = shell
step.Shell = cmp.Or(shell, rc.jobRunDefaults.Shell, rc.Run.Workflow.Defaults.Run.Shell)
implicitShell := step.Shell == ""
if implicitShell {
@@ -381,13 +361,7 @@ func (sr *stepRun) setupWorkingDirectory(ctx context.Context, eval *expressionEv
if err != nil {
return fmt.Errorf("unable to interpolate the working directory: %w", err)
}
if workingdirectory == "" {
workingdirectory = rc.jobRunDefaults.WorkingDirectory
}
// top level keys in workflow file like `defaults` or `env` can't hold expressions
if workingdirectory == "" {
workingdirectory = rc.Run.Workflow.Defaults.Run.WorkingDirectory
}
sr.WorkingDirectory = workingdirectory
sr.WorkingDirectory = cmp.Or(workingdirectory, rc.jobRunDefaults.WorkingDirectory, rc.Run.Workflow.Defaults.Run.WorkingDirectory)
return nil
}
+2 -15
View File
@@ -5,7 +5,6 @@
package runner
import (
"bytes"
"cmp"
"context"
"io"
@@ -60,20 +59,8 @@ func TestStepRun(t *testing.T) {
cm.On("Copy", "/var/run/act", []*container.FileEntry{fileEntry}).Return(noopExecutor)
cm.On("Exec", []string{"bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "/var/run/act/workflow/1.sh"}, mock.AnythingOfType("map[string]string"), "", "workdir").Return(noopExecutor)
cm.On("Copy", "/var/run/act", mock.AnythingOfType("[]*container.FileEntry")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/envs.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/statecmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
cm.On("UpdateFromEnv", "/var/run/act/workflow/outputcmd.txt", mock.AnythingOfType("*map[string]string")).Return(noopExecutor)
ctx := t.Context()
cm.On("GetContainerArchive", ctx, "/var/run/act/workflow/pathcmd.txt").Return(io.NopCloser(&bytes.Buffer{}), nil)
err := sr.main()(ctx)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
expectStepFileCommands(t, cm)
require.NoError(t, sr.main()(t.Context()))
cm.AssertExpectations(t)
}
+96 -183
View File
@@ -9,12 +9,12 @@ import (
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"testing/synctest"
"gitea.com/gitea/runner/act/common"
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/act/common/logging/logtest"
"gitea.com/gitea/runner/act/container"
"gitea.com/gitea/runner/internal/pkg/telemetry"
@@ -115,11 +115,6 @@ func (sm *stepMock) getRunContext() *RunContext {
return args.Get(0).(*RunContext)
}
func (sm *stepMock) getGithubContext(ctx context.Context) *model.GithubContext {
args := sm.Called()
return args.Get(0).(*RunContext).getGithubContext(ctx)
}
func (sm *stepMock) getStepModel() *model.Step {
args := sm.Called()
return args.Get(0).(*model.Step)
@@ -131,9 +126,6 @@ func (sm *stepMock) getEnv() *map[string]string {
}
func TestSetupEnv(t *testing.T) {
cm := &containerMock{}
sm := &stepMock{}
rc := &RunContext{
Config: &Config{
Env: map[string]string{
@@ -155,21 +147,16 @@ func TestSetupEnv(t *testing.T) {
Env: map[string]string{
"RC_KEY": "rcvalue",
},
JobContainer: cm,
JobContainer: &containerMock{},
}
step := &model.Step{}
require.NoError(t, step.RawWith.Encode(map[string]string{"STEP_WITH": "with-value", "ID": "${{ fromJSON('1234567') }}"}))
env := map[string]string{}
sm.On("getRunContext").Return(rc)
sm.On("getGithubContext").Return(rc)
sm.On("getStepModel").Return(step)
sm.On("getEnv").Return(&env)
testStep := &stepRun{RunContext: rc, Step: step, env: env}
require.NoError(t, setupEnv(t.Context(), testStep))
require.NoError(t, setupInputs(t.Context(), testStep))
require.NoError(t, setupEnv(t.Context(), sm))
require.NoError(t, setupInputs(t.Context(), sm))
// These are commit or system specific
delete(env, "GITHUB_REF")
delete(env, "GITHUB_REF_NAME")
delete(env, "GITHUB_REF_TYPE")
@@ -178,7 +165,6 @@ func TestSetupEnv(t *testing.T) {
delete(env, "GITHUB_REPOSITORY")
delete(env, "GITHUB_REPOSITORY_OWNER")
delete(env, "GITHUB_ACTOR")
// Host-dependent, asserted in TestRunContextWithGithubEnvRunnerValues instead.
delete(env, "RUNNER_NAME")
delete(env, "RUNNER_WORKSPACE")
@@ -210,8 +196,6 @@ func TestSetupEnv(t *testing.T) {
"RUNNER_TRACKING_ID": "",
}, env)
cm.AssertExpectations(t)
for _, expression := range []string{"inputs.args", "env.ARGS"} {
t.Run("deferred inputs from "+expression, func(t *testing.T) {
workflow, err := model.ReadWorkflow(strings.NewReader(`
@@ -275,169 +259,96 @@ jobs:
}
func TestIsStepEnabled(t *testing.T) {
createTestStep := func(t *testing.T, input string) step {
var step *model.Step
err := yaml.Unmarshal([]byte(input), &step)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
return &stepRun{
RunContext: &RunContext{
Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }},
StepResults: map[string]*model.StepResult{},
Env: map[string]string{},
Run: &model.Run{
JobID: "job1",
Workflow: &model.Workflow{
Name: "workflow1",
Jobs: map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
for _, test := range []struct {
name string
result *model.StepResult
success bool
}{
{"no prior steps", nil, true},
{"prior step succeeded", &model.StepResult{Conclusion: model.StepStatusSuccess}, true},
{"prior step failed", &model.StepResult{Conclusion: model.StepStatusFailure}, false},
} {
t.Run(test.name, func(t *testing.T) {
for _, condition := range []struct {
expression string
want bool
}{{"success()", test.success}, {"failure()", !test.success}, {"always()", true}} {
t.Run(condition.expression, func(t *testing.T) {
step := &stepRun{
RunContext: &RunContext{
Config: &Config{},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
StepResults: map[string]*model.StepResult{},
},
},
},
Step: &model.Step{If: yaml.Node{Value: condition.expression}},
}
if test.result != nil {
step.RunContext.StepResults["prior"] = test.result
}
enabled, err := isStepEnabled(t.Context(), step.getIfExpression(t.Context(), stepStageMain), step, stepStageMain)
require.NoError(t, err)
assert.Equal(t, condition.want, enabled)
})
}
})
}
t.Run("env and step inputs are unavailable at every stage", func(t *testing.T) {
step := &stepRun{
RunContext: &RunContext{
Config: &Config{},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
Env: map[string]string{"INPUT_FORGED": "leaked"},
},
Step: step,
Step: &model.Step{},
env: map[string]string{"INPUT_FORGED": "leaked"},
}
}
logging.SetLevel(logging.LevelDebug)
assertObject := assert.New(t)
// success()
step := createTestStep(t, "if: success()")
assertObject.True(isStepEnabled(t.Context(), step.getIfExpression(t.Context(), stepStageMain), step, stepStageMain))
step = createTestStep(t, "if: success()")
step.getRunContext().StepResults["a"] = &model.StepResult{
Conclusion: model.StepStatusSuccess,
}
assertObject.True(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
step = createTestStep(t, "if: success()")
step.getRunContext().StepResults["a"] = &model.StepResult{
Conclusion: model.StepStatusFailure,
}
assertObject.False(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
// failure()
step = createTestStep(t, "if: failure()")
assertObject.False(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
step = createTestStep(t, "if: failure()")
step.getRunContext().StepResults["a"] = &model.StepResult{
Conclusion: model.StepStatusSuccess,
}
assertObject.False(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
step = createTestStep(t, "if: failure()")
step.getRunContext().StepResults["a"] = &model.StepResult{
Conclusion: model.StepStatusFailure,
}
assertObject.True(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
// always()
step = createTestStep(t, "if: always()")
assertObject.True(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
step = createTestStep(t, "if: always()")
step.getRunContext().StepResults["a"] = &model.StepResult{
Conclusion: model.StepStatusSuccess,
}
assertObject.True(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
step = createTestStep(t, "if: always()")
step.getRunContext().StepResults["a"] = &model.StepResult{
Conclusion: model.StepStatusFailure,
}
assertObject.True(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
// neither env nor the step's own with: values are inputs, at any stage
step = createTestStep(t, "if: inputs.forged")
step.getRunContext().Env["INPUT_FORGED"] = "leaked"
*step.getEnv() = map[string]string{"INPUT_FORGED": "leaked"}
assertObject.False(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStageMain))
assertObject.False(isStepEnabled(t.Context(), step.getStepModel().If.Value, step, stepStagePost))
for _, stage := range []stepStage{stepStageMain, stepStagePost} {
enabled, err := isStepEnabled(t.Context(), "inputs.forged", step, stage)
require.NoError(t, err)
assert.False(t, enabled)
}
})
}
func TestIsContinueOnError(t *testing.T) {
createTestStep := func(t *testing.T, input string) step {
var step *model.Step
err := yaml.Unmarshal([]byte(input), &step)
assert.NoError(t, err) //nolint:testifylint // pre-existing issue from nektos/act
return &stepRun{
RunContext: &RunContext{
Config: &Config{Workdir: ".", PlatformPicker: func([]string) string { return "ubuntu-latest" }},
StepResults: map[string]*model.StepResult{},
Env: map[string]string{},
Run: &model.Run{
JobID: "job1",
Workflow: &model.Workflow{
Name: "workflow1",
Jobs: map[string]*model.Job{
"job1": createJob(t, `runs-on: ubuntu-latest`, ""),
},
},
for _, test := range []struct {
name, expression string
env map[string]string
want, wantError bool
}{
{name: "absent"},
{name: "explicit true", expression: "true", want: true},
{name: "explicit false", expression: "false"},
{name: "expression true", expression: "${{ 'test' == 'test' }}", want: true},
{name: "expression false", expression: "${{ 'test' != 'test' }}"},
{name: "step inputs are unavailable", expression: "${{ inputs.forged }}", env: map[string]string{"INPUT_FORGED": "true"}},
{name: "invalid expression", expression: "${{ 'test' != test }}", wantError: true},
} {
t.Run(test.name, func(t *testing.T) {
step := &stepRun{
RunContext: &RunContext{
Config: &Config{},
Run: &model.Run{JobID: "job", Workflow: &model.Workflow{Jobs: map[string]*model.Job{"job": {}}}},
},
},
Step: step,
}
Step: &model.Step{},
env: test.env,
}
continueOnError, err := isContinueOnError(t.Context(), test.expression, step)
if test.wantError {
require.Error(t, err)
} else {
require.NoError(t, err)
}
assert.Equal(t, test.want, continueOnError)
})
}
logging.SetLevel(logging.LevelDebug)
assertObject := assert.New(t)
// absent
step := createTestStep(t, "name: test")
continueOnError, err := isContinueOnError(t.Context(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.False(continueOnError)
assertObject.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
// explcit true
step = createTestStep(t, "continue-on-error: true")
continueOnError, err = isContinueOnError(t.Context(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.True(continueOnError)
assertObject.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
// explicit false
step = createTestStep(t, "continue-on-error: false")
continueOnError, err = isContinueOnError(t.Context(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.False(continueOnError)
assertObject.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
// expression true
step = createTestStep(t, "continue-on-error: ${{ 'test' == 'test' }}")
continueOnError, err = isContinueOnError(t.Context(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.True(continueOnError)
assertObject.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
// expression false
step = createTestStep(t, "continue-on-error: ${{ 'test' != 'test' }}")
continueOnError, err = isContinueOnError(t.Context(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.False(continueOnError)
assertObject.NoError(err) //nolint:testifylint // pre-existing issue from nektos/act
// the step's own with: values are not inputs
step = createTestStep(t, "continue-on-error: ${{ inputs.forged }}")
*step.getEnv() = map[string]string{"INPUT_FORGED": "true"}
continueOnError, err = isContinueOnError(t.Context(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.False(continueOnError)
require.NoError(t, err)
// expression parse error
step = createTestStep(t, "continue-on-error: ${{ 'test' != test }}")
continueOnError, err = isContinueOnError(t.Context(), step.getStepModel().RawContinueOnError, step, stepStageMain)
assertObject.False(continueOnError)
assertObject.Error(err)
}
// A refused ::set-env::/::add-path:: records a job-scoped error. When the step that
// produced it also fails on its own, the refusal must be cleared at the step boundary, so
// it fails only that step and never leaks onto a later step that runs anyway (if: always()).
func TestRunStepExecutorDoesNotLeakRefusalToNextStep(t *testing.T) {
cm := &containerMock{}
noop := func(context.Context) error { return nil }
cm.On("Copy", mock.Anything, mock.Anything).Return(noop)
cm.On("UpdateFromEnv", mock.Anything, mock.Anything).Return(noop)
cm.On("Copy", mock.Anything, mock.Anything).Return(noopExecutor)
cm.On("UpdateFromEnv", mock.Anything, mock.Anything).Return(noopExecutor)
rc := &RunContext{
Config: &Config{Env: map[string]string{}},
@@ -450,29 +361,21 @@ func TestRunStepExecutorDoesNotLeakRefusalToNextStep(t *testing.T) {
JobContainer: cm,
}
rc.ExprEval = rc.NewExpressionEvaluator(t.Context())
// Dryrun skips reading the path file back from the (mocked) container.
ctx := common.WithDryrun(t.Context(), true)
// A refusal parsed out of the job container's own output belongs to no step, so the
// first step must not be failed by it.
rc.commandHandler(ctx)("::set-env name=setup::y\n")
stepSetup := &stepRun{RunContext: rc, Step: &model.Step{ID: "setup"}, env: map[string]string{}}
require.NoError(t, runStepExecutor(stepSetup, stepStageMain, func(context.Context) error { return nil })(ctx))
require.NoError(t, runStepExecutor(stepSetup, stepStageMain, noopExecutor)(ctx))
// Step A refuses a ::set-env:: and then fails on its own.
stepA := &stepRun{RunContext: rc, Step: &model.Step{ID: "a"}, env: map[string]string{}}
errA := runStepExecutor(stepA, stepStageMain, func(context.Context) error {
rc.commandHandler(ctx)("::set-env name=x::y\n")
return errors.New("boom")
})(ctx)
// The step fails with its own error, not the refusal.
require.ErrorContains(t, errA, "boom")
// Step B runs despite step A's failure (if: always()) and issues no unsecure command;
// it must not inherit step A's refusal.
stepB := &stepRun{RunContext: rc, Step: &model.Step{ID: "b", If: yaml.Node{Value: "always()"}}, env: map[string]string{}}
errB := runStepExecutor(stepB, stepStageMain, func(context.Context) error { return nil })(ctx)
require.NoError(t, errB)
require.NoError(t, runStepExecutor(stepB, stepStageMain, noopExecutor)(ctx))
}
type fileCommandTestContainer struct {
@@ -540,10 +443,11 @@ func TestRunStepExecutorParity(t *testing.T) {
}
})
t.Run("file command error honors continue-on-error", func(t *testing.T) {
t.Run("file command error honors continue-on-error and keeps the step summary", func(t *testing.T) {
step := newStep(t, &model.Step{ID: "commands", RawContinueOnError: "true"})
logger, hook := logtest.NewLogger()
err := runStepExecutor(step, stepStageMain, func(context.Context) error {
require.NoError(t, os.WriteFile(step.env["GITHUB_STEP_SUMMARY"], []byte("report"), 0o600))
require.NoError(t, os.WriteFile(step.env["GITHUB_ENV"], []byte("GOOD=1\nmalformed\n"), 0o600))
require.NoError(t, os.WriteFile(step.env["GITHUB_OUTPUT"], []byte("kept=value\n"), 0o600))
require.NoError(t, os.WriteFile(step.env["GITHUB_STATE"], []byte("saved=value\n"), 0o600))
@@ -563,6 +467,15 @@ func TestRunStepExecutorParity(t *testing.T) {
require.NoError(t, readErr)
assert.Empty(t, contents)
}
summary, err := os.ReadFile(step.env["GITHUB_STEP_SUMMARY"])
require.NoError(t, err)
assert.Equal(t, "report", string(summary))
})
t.Run("a broken summary path still resets the command files", func(t *testing.T) {
step := newStep(t, &model.Step{ID: "noop"})
require.NoError(t, os.MkdirAll(filepath.Join(step.RunContext.JobContainer.GetActPath(), "workflow", "step-summary-0.md"), 0o700))
require.NoError(t, runStepExecutor(step, stepStageMain, noopExecutor)(t.Context()))
})
t.Run("file commands survive a job timeout", func(t *testing.T) {
+9 -30
View File
@@ -5,44 +5,23 @@
package e2e
import (
"testing"
)
import "testing"
func testActionsCacheRoundTrip(t *testing.T) {
func testCacheAndArtifactRoundTrips(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
v2 bool
name, workflow, label string
options runnerOptions
}{
{name: "cache_v2", v2: true},
{name: "cache_v1", v2: false},
{"cache_v2", "cache.yml", "e2e-cache", runnerOptions{cacheV2: new(true)}},
{"cache_v1", "cache.yml", "e2e-cache", runnerOptions{cacheV2: new(false)}},
{"artifact_through_the_cache_server", "artifact.yml", "e2e-artifact", runnerOptions{cacheV2: new(true)}},
{"artifact_direct_with_no_reachable_cache", "artifact.yml", "e2e-artifact", runnerOptions{cacheHost: "cache.invalid", cacheV2: new(false)}},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
api, repo, _ := startIsolatedScenario(t, "cache.yml", "e2e-cache", runnerOptions{cacheV2: &tc.v2})
wfRun := waitForRun(t, api, repo)
requireSuccess(t, api, repo, wfRun.ID)
})
}
}
func testArtifactRoundTrip(t *testing.T) {
t.Parallel()
v2 := true
for _, tc := range []struct {
name string
options runnerOptions
}{
{"through_the_cache_server", runnerOptions{cacheV2: &v2}},
{"direct_with_no_reachable_cache", runnerOptions{cacheHost: "cache.invalid", cacheV2: new(bool)}},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
api, repo, _ := startIsolatedScenario(t, "artifact.yml", "e2e-artifact", tc.options)
api, repo, _ := startIsolatedScenario(t, tc.workflow, tc.label, tc.options)
wfRun := waitForRun(t, api, repo)
requireSuccess(t, api, repo, wfRun.ID)
+1 -2
View File
@@ -26,8 +26,7 @@ func TestCompatibility(t *testing.T) {
})
t.Run("payloads", testPayloads)
t.Run("cache", testActionsCacheRoundTrip)
t.Run("artifact", testArtifactRoundTrip)
t.Run("cache_and_artifacts", testCacheAndArtifactRoundTrips)
t.Run("cancellation_and_log_streaming", testRunCancellation)
t.Run("dispatch", testWorkflowDispatch)
t.Run("builtin_checkout", testBuiltinCheckout)
+1 -5
View File
@@ -21,10 +21,6 @@ const runTimeout = 3 * time.Minute
var nonRepoChars = regexp.MustCompile(`[^a-zA-Z0-9._-]+`)
func repoName(t *testing.T) string {
return strings.ToLower(nonRepoChars.ReplaceAllString(t.Name(), "-"))
}
var fixture *GiteaFixture
var skipReason string
@@ -59,7 +55,7 @@ func runSuite(m *testing.M) int {
func newScenario(t *testing.T) (*GiteaAPI, string) {
t.Helper()
repo := repoName(t)
repo := strings.ToLower(nonRepoChars.ReplaceAllString(t.Name(), "-"))
api := &GiteaAPI{baseURL: fixture.baseURL, token: fixture.adminToken}
if err := api.CreateRepo(t.Context(), repo); err != nil {
t.Fatalf("create repo: %v", err)
+11 -17
View File
@@ -19,7 +19,6 @@ import (
"gitea.com/gitea/runner/internal/pkg/config"
"gitea.com/gitea/runner/internal/pkg/labels"
"connectrpc.com/connect"
pingv1 "gitea.dev/actionslib/ping/v1"
runnerv1 "gitea.dev/actionslib/runner/v1"
)
@@ -48,10 +47,7 @@ func startRunner(t *testing.T, repo, labelName string, options runnerOptions) *p
if options.cacheV2 != nil {
cfg.Cache.V2 = options.cacheV2
}
cfg.Container.DockerHost = os.Getenv("DOCKER_HOST")
if cfg.Container.DockerHost == "" {
cfg.Container.DockerHost = "unix:///var/run/docker.sock"
}
cfg.Container.DockerHost = cmp.Or(os.Getenv("DOCKER_HOST"), "unix:///var/run/docker.sock")
cfg.Cache.Dir = t.TempDir() + "/cache"
cfg.Cache.Host = options.cacheHost
cfg.Runner.Insecure = true
@@ -76,44 +72,42 @@ func startRunner(t *testing.T, repo, labelName string, options runnerOptions) *p
labelNames := []string{label.Name}
pingCli := client.New(fixture.baseURL, cfg.Runner.TLSConfig(), "", "", config.RequestTimeout, cfg.Runner.ExtraHeaders)
if _, err := pingCli.Ping(ctx, connect.NewRequest(&pingv1.PingRequest{Data: t.Name()})); err != nil {
if _, err := pingCli.Ping(ctx, &pingv1.PingRequest{Data: t.Name()}); err != nil {
t.Fatalf("ping %s: %v", fixture.baseURL, err)
}
regResp, err := pingCli.Register(ctx, connect.NewRequest(&runnerv1.RegisterRequest{
regResp, err := pingCli.Register(ctx, &runnerv1.RegisterRequest{
Name: t.Name(),
Token: token,
Version: "e2e",
Labels: labelNames,
Ephemeral: options.ephemeral,
Capabilities: run.RunnerCapabilities(),
}))
})
if err != nil {
t.Fatalf("register runner: %v", err)
}
if options.ephemeral && !regResp.Msg.Runner.Ephemeral {
if options.ephemeral && !regResp.Runner.Ephemeral {
t.Fatal("gitea did not grant ephemeral registration")
}
reg := &config.Registration{
ID: regResp.Msg.Runner.Id,
UUID: regResp.Msg.Runner.Uuid,
Name: regResp.Msg.Runner.Name,
Token: regResp.Msg.Runner.Token,
ID: regResp.Runner.Id,
UUID: regResp.Runner.Uuid,
Name: regResp.Runner.Name,
Token: regResp.Runner.Token,
Address: fixture.baseURL,
Labels: []string{rawLabel},
Ephemeral: regResp.Msg.Runner.Ephemeral,
Ephemeral: regResp.Runner.Ephemeral,
}
cli := client.New(fixture.baseURL, cfg.Runner.TLSConfig(), reg.UUID, reg.Token, config.RequestTimeout, cfg.Runner.ExtraHeaders)
runner := run.NewRunner(cfg, reg, cli)
declResp, err := runner.Declare(ctx, labelNames)
if err != nil {
if _, err := runner.Declare(ctx, labelNames); err != nil {
_ = runner.Close()
t.Fatalf("declare runner: %v", err)
}
runner.SetCapabilitiesFromDeclare(declResp)
poller := poll.New(cfg, cli, runner)
go poller.Poll()
+3 -8
View File
@@ -5,9 +5,9 @@ go 1.27
toolchain go1.27.3
require (
connectrpc.com/connect v1.21.0
gitea.dev/actionslib v1.4.1
github.com/bmatcuk/doublestar/v4 v4.10.2
connectrpc.com/connect/v2 v2.0.1
gitea.dev/actionslib v1.5.0
github.com/bmatcuk/doublestar/v4 v4.10.3
github.com/coder/websocket v1.8.15
github.com/containerd/errdefs v1.0.0
github.com/creack/pty v1.1.24
@@ -15,7 +15,6 @@ require (
github.com/docker/cli v29.8.2+incompatible
github.com/docker/go-connections v0.8.1
github.com/docker/go-units v0.5.0
github.com/johannesboyne/gofakes3 v1.2.0
github.com/joho/godotenv v1.5.1
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
github.com/moby/moby/api v1.56.1
@@ -27,7 +26,6 @@ require (
github.com/spf13/cobra v1.10.2
github.com/spf13/pflag v1.0.10
github.com/stretchr/testify v1.12.1
github.com/timshannon/bolthold v0.0.0-20240314194003-30aac6950928
go.etcd.io/bbolt v1.5.0
go.opentelemetry.io/otel v1.47.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0
@@ -59,13 +57,10 @@ require (
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46 // indirect
github.com/stretchr/objx v0.5.3 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 // indirect
go.opentelemetry.io/otel/log v1.47.0 // indirect
go.opentelemetry.io/otel/metric v1.47.0 // indirect
go.shabbyrobe.org/gocovmerge v0.0.0-20230507111327-fa4f82cfbf4d // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/tools v0.49.0 // indirect
)
+6 -63
View File
@@ -1,43 +1,15 @@
connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
gitea.dev/actionslib v1.4.1 h1:DWlUnzDfuuMArQWS6yZ+vwJu9av7sZVMOox7li/kyu4=
gitea.dev/actionslib v1.4.1/go.mod h1:RbXaZudZ04YndjBBxZJF9rZAqU6SKGkaSD1sob5lQbE=
connectrpc.com/connect/v2 v2.0.1 h1:WTAycxe87UqxSM5oxS30d1dalZ+C5xz8Ec8O6zVYkHQ=
connectrpc.com/connect/v2 v2.0.1/go.mod h1:N32OLrstcPUzkykFynnefFro3+xzSdOrZlK2jHmS/xo=
gitea.dev/actionslib v1.5.0 h1:dC3v2HTWbbvyNBSdg2W3kd4WNC/p+FnZaKpl1MdzM00=
gitea.dev/actionslib v1.5.0/go.mod h1:C9wmOFgDBCXyRvg4A3KTX8Vszs8v9+elH4QLGVDolqU=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/aws/aws-sdk-go-v2 v1.41.5 h1:dj5kopbwUsVUVFgO4Fi5BIT3t4WyqIDjGKCangnV/yY=
github.com/aws/aws-sdk-go-v2 v1.41.5/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 h1:eBMB84YGghSocM7PsjmmPffTa+1FBUeNvGvFou6V/4o=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI=
github.com/aws/aws-sdk-go-v2/credentials v1.17.67 h1:9KxtdcIA/5xPNQyZRgUSpYOE6j9Bc4+D7nZua0KGYOM=
github.com/aws/aws-sdk-go-v2/credentials v1.17.67/go.mod h1:p3C44m+cfnbv763s52gCqrjaqyPikj9Sg47kUVaNZQQ=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.75 h1:S61/E3N01oral6B3y9hZ2E1iFDqCZPPOBoBQretCnBI=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.75/go.mod h1:bDMQbkI1vJbNjnvJYpPTSNYBkI/VIv18ngWb/K84tkk=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 h1:Rgg6wvjjtX8bNHcvi9OnXWwcE0a2vGpbwmtICOsvcf4=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21/go.mod h1:A/kJFst/nm//cyqonihbdpQZwiUhhzpqTsdbhDdRF9c=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 h1:PEgGVtPoB6NTpPrBgqSE5hE/o47Ij9qk/SEZFbUOe9A=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21/go.mod h1:p+hz+PRAYlY3zcpJhPwXlLC4C+kqn70WIHwnzAfs6ps=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 h1:rWyie/PxDRIdhNf4DzRk0lvjVOqFJuNnO8WwaIRVxzQ=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22/go.mod h1:zd/JsJ4P7oGfUhXn1VyLqaRZwPmZwg44Jf2dS84Dm3Y=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 h1:5EniKhLZe4xzL7a+fU3C2tfUN4nWIqlLesfrjkuPFTY=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 h1:JRaIgADQS/U6uXDqlPiefP32yXTda7Kqfx+LgspooZM=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13/go.mod h1:CEuVn5WqOMilYl+tbccq8+N2ieCy0gVn3OtRb0vBNNM=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 h1:c31//R3xgIJMSC8S6hEVq+38DcvUlgFY0FM6mSI5oto=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21/go.mod h1:r6+pf23ouCB718FUxaqzZdbpYFyDtehyZcmP5KL9FkA=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 h1:ZlvrNcHSFFWURB8avufQq9gFsheUgjVD9536obIknfM=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21/go.mod h1:cv3TNhVrssKR0O/xxLJVRfd2oazSnZnkUeTf6ctUwfQ=
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3 h1:HwxWTbTrIHm5qY+CAEur0s/figc3qwvLWsNkF4RPToo=
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3/go.mod h1:uoA43SdFwacedBfSgfFSjjCvYe8aYBS7EnU5GZ/YKMM=
github.com/aws/smithy-go v1.24.2 h1:FzA3bu/nt/vDvmnkg+R8Xl46gmzEDam6mZ1hzmwXFng=
github.com/aws/smithy-go v1.24.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bmatcuk/doublestar/v4 v4.10.2 h1:eF7W7HWKg3z9NrWV9pTLnNeoXaqq3Tq9DNKXVMfoCnw=
github.com/bmatcuk/doublestar/v4 v4.10.2/go.mod h1:xBQ8jztBU6kakFMg+8WGxn0c6z1fTSPVIjEY1Wr7jzc=
github.com/bmatcuk/doublestar/v4 v4.10.3 h1:1UIv1+fqH3++ViVkpnaPTB+Pi1UHycnPFAq14zgK+gI=
github.com/bmatcuk/doublestar/v4 v4.10.3/go.mod h1:xBQ8jztBU6kakFMg+8WGxn0c6z1fTSPVIjEY1Wr7jzc=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cevatbarisyilmaz/ara v0.0.4 h1:SGH10hXpBJhhTlObuZzTuFn1rrdmjQImITXnZVPSodc=
github.com/cevatbarisyilmaz/ara v0.0.4/go.mod h1:BfFOxnUd6Mj6xmcvRxHN3Sr21Z1T3U2MYkYOmoQe4Ts=
github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNUA=
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
@@ -47,8 +19,6 @@ github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmC
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/cli v29.8.2+incompatible h1:2zgdFuoFst2T80oS42vhKGxkd0JRo305eIEKTsxR7pQ=
@@ -70,8 +40,6 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/johannesboyne/gofakes3 v1.2.0 h1:I9VEzPWvvAUAGzDlhYFoZjF0AXMlkcEyZlmBwiI6Oms=
github.com/johannesboyne/gofakes3 v1.2.0/go.mod h1:UHhRZRod9rENGFrUWTYnQHZqlNgSmjOq8DaD/ATQYRM=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 h1:Z9n2FFNUXsshfwJMBgNA0RU6/i7WVaAegv3PtuIHPMs=
@@ -94,7 +62,6 @@ github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo=
@@ -104,31 +71,17 @@ github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46 h1:GHRpF1pTW19a8tTFrMLUcfWwyC0pnifVo2ClaLq+hP8=
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46/go.mod h1:uAQ5PCi+MFsC7HjREoAz1BU+Mq60+05gifQSsHSDG/8=
github.com/spf13/afero v1.2.1 h1:qgMbHoJbPbw579P+1zVY+6n4nIFuIchaIjzZ/I/Yq8M=
github.com/spf13/afero v1.2.1/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4=
github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/timshannon/bolthold v0.0.0-20240314194003-30aac6950928 h1:zjNCuOOhh1TKRU0Ru3PPPJt80z7eReswCao91gBLk00=
github.com/timshannon/bolthold v0.0.0-20240314194003-30aac6950928/go.mod h1:PCFYfAEfKT+Nd6zWvUpsXduMR1bXFLf0uGSlEF05MCI=
go.etcd.io/bbolt v1.3.8/go.mod h1:N9Mkw9X8x5fupy0IKsmuqVtoGDyxsaDlbk4Rd05IAQw=
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
go.etcd.io/gofail v0.1.0/go.mod h1:VZBCXYGZhHAinaBiiqYvuDynvahNsAyLFwB3kEHKz1M=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI=
@@ -149,8 +102,6 @@ go.opentelemetry.io/otel/trace v1.47.0 h1:JOjX/Oci8K94QHddo+bbfya/Ai/nf6/dt9ZfrF
go.opentelemetry.io/otel/trace v1.47.0/go.mod h1:jNaSLa2PZEYFG6fRjJABAu+bw4FS08uDmPg28lTghu0=
go.opentelemetry.io/proto/otlp v1.11.1 h1:SCrPqH9NE5CmBrOqpjNCiT0X5mGwMyGERXdogfR7Yjw=
go.opentelemetry.io/proto/otlp v1.11.1/go.mod h1:wq8W4aSf6bOy8RZyESOeQFr9Pu7wydWX/TY4sgvrTPg=
go.shabbyrobe.org/gocovmerge v0.0.0-20230507111327-fa4f82cfbf4d h1:Ns9kd1Rwzw7t0BR8XMphenji4SmIoNZPn8zhYmaVKP8=
go.shabbyrobe.org/gocovmerge v0.0.0-20230507111327-fa4f82cfbf4d/go.mod h1:92Uoe3l++MlthCm+koNi0tcUCX3anayogF0Pa/sp24k=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
@@ -164,23 +115,15 @@ golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce h1:xcEWjVhvbDy+nHP67nPDDpbYrY+ILlfndk4bRioVHaU=
gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q=
gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA=
pgregory.net/rapid v1.2.0 h1:keKAYRcjm+e1F0oAuU5F5+YPAWcyxNNRK2wud503Gnk=
+6 -23
View File
@@ -4,6 +4,7 @@
package cmd
import (
"cmp"
"errors"
"fmt"
@@ -33,32 +34,14 @@ func runCacheServer(configFile *string, cacheArgs *cacheServerArgs) func(cmd *co
return errors.New("cache-server does not support cache.s3; set cache.s3 on the runners instead")
}
var (
dir = cfg.Cache.Dir
host = cfg.Cache.Host
port = cfg.Cache.Port
)
// cacheArgs has higher priority
if cacheArgs.Dir != "" {
dir = cacheArgs.Dir
}
if cacheArgs.Host != "" {
host = cacheArgs.Host
}
if cacheArgs.Port != 0 {
port = cacheArgs.Port
}
secret := cfg.Cache.ExternalSecret
if secret == "" {
if cfg.Cache.ExternalSecret == "" {
return errors.New("cache.external_secret (or cache.external_secret_file) must be set for cache-server; configure the same value on each runner that points at this server via cache.external_server")
}
cacheHandler, err := artifactcache.StartHandler(artifactcache.Options{
Dir: dir,
OutboundIP: host,
Port: port,
InternalSecret: secret,
Dir: cmp.Or(cacheArgs.Dir, cfg.Cache.Dir),
OutboundIP: cmp.Or(cacheArgs.Host, cfg.Cache.Host),
Port: cmp.Or(cacheArgs.Port, cfg.Cache.Port),
InternalSecret: cfg.Cache.ExternalSecret,
Policy: run.CachePolicy(cfg),
Logger: logging.Default().With("module", "cache_request"),
})
+19 -36
View File
@@ -31,7 +31,7 @@ import (
"gitea.com/gitea/runner/internal/pkg/telemetry"
"gitea.com/gitea/runner/internal/pkg/ver"
"connectrpc.com/connect"
"connectrpc.com/connect/v2"
runnerv1 "gitea.dev/actionslib/runner/v1"
"github.com/spf13/cobra"
"golang.org/x/term"
@@ -163,8 +163,8 @@ func runDaemon(ctx context.Context, daemArgs *daemonArgs, configFile *string) fu
}
}
if !slices.Equal(reg.Labels, ls.ToStrings()) {
reg.Labels = ls.ToStrings()
if labelStrings := ls.ToStrings(); !slices.Equal(reg.Labels, labelStrings) {
reg.Labels = labelStrings
if err := config.SaveRegistration(cfg.Runner.File, reg); err != nil {
return fmt.Errorf("failed to save runner config: %w", err)
}
@@ -188,7 +188,7 @@ func runDaemon(ctx context.Context, daemArgs *daemonArgs, configFile *string) fu
}()
// declare the labels of the runner before fetching tasks
var resp *connect.Response[runnerv1.DeclareResponse]
var resp *runnerv1.DeclareResponse
for attempt := 0; ; attempt++ {
resp, err = runner.Declare(ctx, ls.Names())
if err == nil || !shouldRetryDeclare(err) || ctx.Err() != nil {
@@ -218,14 +218,13 @@ func runDaemon(ctx context.Context, daemArgs *daemonArgs, configFile *string) fu
return err
default:
logging.Infof("runner: %s, with version: %s, with labels: %v, declare successfully",
resp.Msg.Runner.Name, resp.Msg.Runner.Version, resp.Msg.Runner.Labels)
resp.Runner.Name, resp.Runner.Version, resp.Runner.Labels)
}
runner.SetCapabilitiesFromDeclare(resp)
poller := poll.New(cfg, cli, runner)
metrics.Init() // OTLP exports the registry even without the server
metrics.RunnerInfo.WithLabelValues(ver.Version(), resp.Msg.Runner.Name).Set(1)
metrics.RunnerInfo.WithLabelValues(ver.Version(), resp.Runner.Name).Set(1)
metrics.RunnerCapacity.Set(float64(cfg.Runner.Capacity))
metrics.RegisterUptimeFunc(time.Now())
metrics.RegisterRunningJobsFunc(runner.RunningCount, cfg.Runner.Capacity)
@@ -245,36 +244,24 @@ func runDaemon(ctx context.Context, daemArgs *daemonArgs, configFile *string) fu
}
if daemArgs.Once || reg.Ephemeral {
done := make(chan struct{})
go func() {
defer close(done)
poller.PollOnce(func() { removeEphemeralRegistration(reg, cfg.Runner.File) })
}()
// shutdown when we complete a job or cancel is requested
select {
case <-ctx.Done():
case <-done:
}
go poller.PollOnce(func() { removeEphemeralRegistration(reg, cfg.Runner.File) })
} else {
go poller.Poll()
// Stop either on an external cancellation or when the poller shuts
// itself down (e.g. after the runner has been unregistered).
select {
case <-ctx.Done():
case <-poller.Done():
}
}
logging.Infof("runner: %s shutdown initiated, waiting %s for running jobs to complete before shutting down", resp.Msg.Runner.Name, cfg.Runner.ShutdownTimeout)
select {
case <-ctx.Done():
case <-poller.Done():
}
logging.Infof("runner: %s shutdown initiated, waiting %s for running jobs to complete before shutting down", resp.Runner.Name, cfg.Runner.ShutdownTimeout)
ctx, cancel := context.WithTimeout(context.Background(), cfg.Runner.ShutdownTimeout)
defer cancel()
err = poller.Shutdown(ctx)
if err != nil {
logging.Warnf("runner: %s cancelled in progress jobs during shutdown", resp.Msg.Runner.Name)
logging.Warnf("runner: %s cancelled in progress jobs during shutdown", resp.Runner.Name)
}
if poller.RegistrationRejected() {
@@ -330,13 +317,8 @@ func splitLabels(s string) []string {
// initLogging sets up the process-wide logger.
func initLogging(cfg *config.Config) {
callPrettyfier := func(f *runtime.Frame) (string, string) {
// get function name
s := strings.Split(f.Function, ".")
funcname := "[" + s[len(s)-1] + "]"
// get file name and line number
_, filename := path.Split(f.File)
filename = "[" + filename + ":" + strconv.Itoa(f.Line) + "]"
return funcname, filename
return "[" + f.Function[strings.LastIndexByte(f.Function, '.')+1:] + "]", "[" + filename + ":" + strconv.Itoa(f.Line) + "]"
}
isTerm := term.IsTerminal(int(os.Stdout.Fd()))
@@ -408,11 +390,12 @@ func getDockerSocketPath(configDockerHost string) (string, error) {
}
for _, p := range commonSocketPaths {
if _, err := os.Lstat(os.ExpandEnv(p)); err == nil {
socket := os.ExpandEnv(p)
if _, err := os.Lstat(socket); err == nil {
if strings.HasPrefix(p, `\\.\`) {
return "npipe://" + filepath.ToSlash(os.ExpandEnv(p)), nil
return "npipe://" + filepath.ToSlash(socket), nil
}
return "unix://" + filepath.ToSlash(os.ExpandEnv(p)), nil
return "unix://" + filepath.ToSlash(socket), nil
}
}
+27 -24
View File
@@ -6,7 +6,6 @@ package cmd
import (
"context"
"crypto/tls"
"errors"
"net/http"
"net/http/httptest"
"os"
@@ -19,7 +18,8 @@ import (
"gitea.com/gitea/runner/act/common/logging"
"gitea.com/gitea/runner/internal/pkg/config"
"connectrpc.com/connect"
"connectrpc.com/connect/v2"
"connectrpc.com/connect/v2/connecthttp"
runnerv1 "gitea.dev/actionslib/runner/v1"
"gitea.dev/actionslib/runner/v1/runnerv1connect"
"github.com/stretchr/testify/require"
@@ -83,7 +83,7 @@ func TestInitLoggingSetsLevelAndCaller(t *testing.T) {
require.Contains(t, buf.String(), "[daemon_test.go:")
}
var errUnregisteredRunner = errors.New("rpc error: code = Unauthenticated desc = unregistered runner")
var errUnregisteredRunner = connect.NewError(connect.CodeUnknown, "rpc error: code = Unauthenticated desc = unregistered runner")
type daemonTestService struct {
runnerv1connect.UnimplementedRunnerServiceHandler
@@ -93,9 +93,9 @@ type daemonTestService struct {
cancel context.CancelFunc
}
func (s *daemonTestService) Declare(context.Context, *connect.Request[runnerv1.DeclareRequest]) (*connect.Response[runnerv1.DeclareResponse], error) {
func (s *daemonTestService) Declare(context.Context, *runnerv1.DeclareRequest) (*runnerv1.DeclareResponse, error) {
if s.declareCalls.Add(1) == 1 {
return nil, connect.NewError(connect.CodeUnavailable, errors.New("bad gateway"))
return nil, connect.NewError(connect.CodeUnavailable, "bad gateway")
}
if s.declareErr != nil {
if s.beforeTask {
@@ -103,10 +103,10 @@ func (s *daemonTestService) Declare(context.Context, *connect.Request[runnerv1.D
}
return nil, s.declareErr
}
return connect.NewResponse(&runnerv1.DeclareResponse{Runner: &runnerv1.Runner{Name: "test"}}), nil
return &runnerv1.DeclareResponse{Runner: &runnerv1.Runner{Name: "test"}}, nil
}
func (s *daemonTestService) FetchTask(context.Context, *connect.Request[runnerv1.FetchTaskRequest]) (*connect.Response[runnerv1.FetchTaskResponse], error) {
func (s *daemonTestService) FetchTask(context.Context, *runnerv1.FetchTaskRequest) (*runnerv1.FetchTaskResponse, error) {
if s.fetchErr != nil {
return nil, s.fetchErr
}
@@ -114,15 +114,15 @@ func (s *daemonTestService) FetchTask(context.Context, *connect.Request[runnerv1
s.cancel()
return nil, context.Canceled
}
return connect.NewResponse(&runnerv1.FetchTaskResponse{Task: &runnerv1.Task{Id: 1, Context: &structpb.Struct{}, WorkflowPayload: []byte("invalid: [")}}), nil
return &runnerv1.FetchTaskResponse{Task: &runnerv1.Task{Id: 1, Context: &structpb.Struct{}, WorkflowPayload: []byte("invalid: [")}}, nil
}
func (*daemonTestService) UpdateTask(_ context.Context, req *connect.Request[runnerv1.UpdateTaskRequest]) (*connect.Response[runnerv1.UpdateTaskResponse], error) {
return connect.NewResponse(&runnerv1.UpdateTaskResponse{State: req.Msg.State}), nil
func (*daemonTestService) UpdateTask(_ context.Context, req *runnerv1.UpdateTaskRequest) (*runnerv1.UpdateTaskResponse, error) {
return &runnerv1.UpdateTaskResponse{State: req.State}, nil
}
func (*daemonTestService) UpdateLog(_ context.Context, req *connect.Request[runnerv1.UpdateLogRequest]) (*connect.Response[runnerv1.UpdateLogResponse], error) {
return connect.NewResponse(&runnerv1.UpdateLogResponse{AckIndex: req.Msg.Index + int64(len(req.Msg.Rows))}), nil
func (*daemonTestService) UpdateLog(_ context.Context, req *runnerv1.UpdateLogRequest) (*runnerv1.UpdateLogResponse, error) {
return &runnerv1.UpdateLogResponse{AckIndex: req.Index + int64(len(req.Rows))}, nil
}
func TestDaemonRetriesDeclareOutagesAndRemovesOnlyConsumedEphemeralRegistration(t *testing.T) {
@@ -138,18 +138,21 @@ func TestDaemonRetriesDeclareOutagesAndRemovesOnlyConsumedEphemeralRegistration(
{"ephemeral interrupted before task", true, true, nil, nil, false},
{"persistent once", false, false, nil, nil, false},
{"ephemeral rejected on fetch", true, false, nil, errUnregisteredRunner, true},
{"ephemeral rejected on fetch with 401", true, false, nil, connect.NewError(connect.CodeUnauthenticated, errors.New("unregistered runner")), true},
{"ephemeral proxy 401 on fetch", true, false, nil, connect.NewError(connect.CodeUnauthenticated, errors.New("HTTP status 401 Unauthorized")), false},
{"ephemeral rejected on fetch with 401", true, false, nil, connect.NewError(connect.CodeUnauthenticated, "unregistered runner"), true},
{"ephemeral proxy 401 on fetch", true, false, nil, connect.NewError(connect.CodeUnauthenticated, "HTTP status 401 Unauthorized"), false},
{"ephemeral rejected on declare", true, false, errUnregisteredRunner, nil, true},
{"persistent rejected on declare", false, false, errUnregisteredRunner, nil, false},
{"ephemeral stopped during declare outage", true, true, connect.NewError(connect.CodeUnavailable, errors.New("bad gateway")), nil, false},
{"ephemeral stopped during declare outage", true, true, connect.NewError(connect.CodeUnavailable, "bad gateway"), nil, false},
} {
t.Run(tc.name, func(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
defer cancel()
service := &daemonTestService{declareErr: tc.declareErr, fetchErr: tc.fetchErr, beforeTask: tc.beforeTask, cancel: cancel}
_, handler := runnerv1connect.NewRunnerServiceHandler(service)
server := httptest.NewServer(http.StripPrefix("/api/actions", handler))
rpcServer := connect.NewServer()
runnerv1connect.RegisterRunnerServiceHandler(rpcServer, service)
mux := http.NewServeMux()
connecthttp.Mount(mux, rpcServer, connecthttp.WithReadMaxBytes(0))
server := httptest.NewServer(http.StripPrefix("/api/actions", mux))
defer server.Close()
dir := t.TempDir()
regFile := filepath.Join(dir, "registration.json")
@@ -177,13 +180,13 @@ func TestShouldRetryDeclare(t *testing.T) {
err error
retry bool
}{
{connect.NewError(connect.CodeUnavailable, errors.New("bad gateway")), true},
{connect.NewWireError(connect.CodeInternal, errors.New("update runner")), true},
{connect.NewError(connect.CodeUnavailable, &tls.CertificateVerificationError{Err: errors.New("unknown authority")}), false},
{connect.NewError(connect.CodeUnavailable, http.ErrSchemeMismatch), false},
{connect.NewError(connect.CodeUnavailable, tls.AlertError(42)), false},
{connect.NewError(connect.CodePermissionDenied, errors.New("forbidden")), false},
{connect.NewWireError(connect.CodeUnknown, errors.New("unregistered runner")), false},
{connect.NewError(connect.CodeUnavailable, "bad gateway"), true},
{connect.NewError(connect.CodeInternal, "update runner").WithRemote(), true},
{connect.NewError(connect.CodeUnavailable, "certificate verification").WithCause(&tls.CertificateVerificationError{}), false},
{connect.NewError(connect.CodeUnavailable, "scheme mismatch").WithCause(http.ErrSchemeMismatch), false},
{connect.NewError(connect.CodeUnavailable, "TLS alert").WithCause(tls.AlertError(42)), false},
{connect.NewError(connect.CodePermissionDenied, "forbidden"), false},
{connect.NewError(connect.CodeUnknown, "unregistered runner").WithRemote(), false},
} {
require.Equal(t, tc.retry, shouldRetryDeclare(tc.err), tc.err.Error())
}
+43 -75
View File
@@ -203,12 +203,8 @@ func parseKVAndFile(rawKVs []string, filePath string) map[string]string {
_ = readEnvs(filePath, result)
for _, raw := range rawKVs {
parts := strings.SplitN(raw, "=", 2)
if len(parts) == 2 {
result[parts[0]] = parts[1]
} else {
result[parts[0]] = ""
}
key, value, _ := strings.Cut(raw, "=")
result[key] = value
}
return result
}
@@ -265,39 +261,25 @@ func printList(plan *model.Plan) {
for i, stage := range plan.Stages {
for _, r := range stage.Runs {
jobID := r.JobID
line := lineInfoDef{
jobID: jobID,
jobID: r.JobID,
jobName: r.String(),
stage: strconv.Itoa(i),
wfName: r.Workflow.Name,
wfFile: r.Workflow.File,
events: strings.Join(r.Workflow.On(), `,`),
}
if _, ok := jobs[jobID]; ok {
if jobs[r.JobID] {
duplicateJobIDs = true
} else {
jobs[jobID] = true
}
jobs[r.JobID] = true
lineInfos = append(lineInfos, line)
if jobIDMaxWidth < len(line.jobID) {
jobIDMaxWidth = len(line.jobID)
}
if jobNameMaxWidth < len(line.jobName) {
jobNameMaxWidth = len(line.jobName)
}
if stageMaxWidth < len(line.stage) {
stageMaxWidth = len(line.stage)
}
if wfNameMaxWidth < len(line.wfName) {
wfNameMaxWidth = len(line.wfName)
}
if wfFileMaxWidth < len(line.wfFile) {
wfFileMaxWidth = len(line.wfFile)
}
if eventsMaxWidth < len(line.events) {
eventsMaxWidth = len(line.events)
}
jobIDMaxWidth = max(jobIDMaxWidth, len(line.jobID))
jobNameMaxWidth = max(jobNameMaxWidth, len(line.jobName))
stageMaxWidth = max(stageMaxWidth, len(line.stage))
wfNameMaxWidth = max(wfNameMaxWidth, len(line.wfName))
wfFileMaxWidth = max(wfFileMaxWidth, len(line.wfFile))
eventsMaxWidth = max(eventsMaxWidth, len(line.events))
}
}
@@ -356,21 +338,15 @@ func runExecList(planner model.WorkflowPlanner, execArgs *executeArgs) error {
case execArgs.job != "":
logging.Infof("Preparing plan with a job: %s", execArgs.job)
filterPlan, err = planner.PlanJob(execArgs.job)
if err != nil {
return err
}
case filterEventName != "":
logging.Infof("Preparing plan for a event: %s", filterEventName)
filterPlan, err = planner.PlanEvent(filterEventName)
if err != nil {
return err
}
default:
logging.Infof("Preparing plan with all jobs")
filterPlan, err = planner.PlanAll()
if err != nil {
return err
}
}
if err != nil {
return err
}
printList(filterPlan)
@@ -385,32 +361,31 @@ func (i *executeArgs) runnerConfig(eventName string, env, proxyEnv map[string]st
}
return &runner.Config{
Workdir: i.Workdir(),
BindWorkdir: false,
ForcePull: i.forcePull,
ForceRebuild: i.forceRebuild,
JSONLogger: i.jsonLogger,
Env: env,
ProxyEnv: proxyEnv,
Vars: i.LoadVars(),
Secrets: i.LoadSecrets(),
InsecureSecrets: i.insecureSecrets,
Privileged: i.privileged,
UsernsMode: i.usernsMode,
ContainerArchitecture: i.containerArchitecture,
ContainerDaemonSocket: i.containerDaemonSocket,
UseGitIgnore: i.useGitIgnore,
GitHubInstance: i.githubInstance,
ContainerCapAdd: i.containerCapAdd,
ContainerCapDrop: i.containerCapDrop,
ContainerOptions: i.containerOptions,
ArtifactServerPath: i.artifactServerPath,
ArtifactServerPort: i.artifactServerPort,
ArtifactServerAddr: i.artifactServerAddr,
NoSkipCheckout: i.noSkipCheckout,
EventName: eventName,
EventJSON: eventJSON,
// PresetGitHubContext: preset,
Workdir: i.Workdir(),
BindWorkdir: false,
ForcePull: i.forcePull,
ForceRebuild: i.forceRebuild,
JSONLogger: i.jsonLogger,
Env: env,
ProxyEnv: proxyEnv,
Vars: i.LoadVars(),
Secrets: i.LoadSecrets(),
InsecureSecrets: i.insecureSecrets,
Privileged: i.privileged,
UsernsMode: i.usernsMode,
ContainerArchitecture: i.containerArchitecture,
ContainerDaemonSocket: i.containerDaemonSocket,
UseGitIgnore: i.useGitIgnore,
GitHubInstance: i.githubInstance,
ContainerCapAdd: i.containerCapAdd,
ContainerCapDrop: i.containerCapDrop,
ContainerOptions: i.containerOptions,
ArtifactServerPath: i.artifactServerPath,
ArtifactServerPort: i.artifactServerPort,
ArtifactServerAddr: i.artifactServerAddr,
NoSkipCheckout: i.noSkipCheckout,
EventName: eventName,
EventJSON: eventJSON,
ContainerNamePrefix: "GITEA-ACTIONS-TASK-" + eventName,
ContainerMaxLifetime: maxLifetime,
ContainerNetworkMode: container.NetworkMode(i.network),
@@ -465,15 +440,12 @@ func runExec(ctx context.Context, execArgs *executeArgs) func(cmd *cobra.Command
if execArgs.job != "" {
logging.Infof("Planning job: %s", execArgs.job)
plan, err = planner.PlanJob(execArgs.job)
if err != nil {
return err
}
} else {
logging.Infof("Planning jobs for event: %s", eventName)
plan, err = planner.PlanEvent(eventName)
if err != nil {
return err
}
}
if err != nil {
return err
}
maxLifetime := 3 * time.Hour
@@ -548,11 +520,7 @@ func runExec(ctx context.Context, execArgs *executeArgs) func(cmd *cobra.Command
config.Env["ACT_EXEC"] = "true"
config.Secrets[actionsRuntimeTokenEnvName] = actionsRuntimeToken
if t := config.Secrets["GITEA_TOKEN"]; t != "" {
config.Token = t
} else if t := config.Secrets["GITHUB_TOKEN"]; t != "" {
config.Token = t
}
config.Token = cmp.Or(config.Secrets["GITEA_TOKEN"], config.Secrets["GITHUB_TOKEN"])
if !execArgs.debug {
logLevel := logging.LevelInfo
+15 -26
View File
@@ -21,7 +21,6 @@ import (
"gitea.com/gitea/runner/internal/pkg/lock"
"gitea.com/gitea/runner/internal/pkg/ver"
"connectrpc.com/connect"
pingv1 "gitea.dev/actionslib/ping/v1"
runnerv1 "gitea.dev/actionslib/runner/v1"
"github.com/spf13/cobra"
@@ -118,10 +117,7 @@ func (r *registerInputs) validate() error {
if r.Token == "" {
return errors.New("token is empty")
}
if len(r.Labels) > 0 {
return validateLabels(r.Labels)
}
return nil
return validateLabels(r.Labels)
}
func validateLabels(ls []string) error {
@@ -142,20 +138,14 @@ func (r *registerInputs) stageValue(stage registerStage) string {
case StageInputRunnerName:
return r.RunnerName
case StageInputLabels:
if len(r.Labels) > 0 {
return strings.Join(r.Labels, ",")
}
return strings.Join(r.Labels, ",")
}
return ""
}
func (r *registerInputs) assignToNext(stage registerStage, value string, cfg *config.Config) registerStage {
// must set instance address and token.
// if empty, keep current stage.
if stage == StageInputInstance || stage == StageInputToken {
if value == "" {
return stage
}
if (stage == StageInputInstance || stage == StageInputToken) && value == "" {
return stage
}
// set hostname for runner name if empty
@@ -223,11 +213,10 @@ func initInputs(regArgs *registerArgs) (*registerInputs, error) {
case regArgs.Token != "":
token = regArgs.Token
default:
envToken, ok := os.LookupEnv(registerTokenEnvVar)
if !ok || envToken == "" {
token = os.Getenv(registerTokenEnvVar)
if token == "" {
return nil, fmt.Errorf("missing token, token-file argument, or %s environment variable", registerTokenEnvVar)
}
token = envToken
}
inputs := &registerInputs{
InstanceAddr: strings.TrimSpace(regArgs.InstanceAddr),
@@ -371,9 +360,9 @@ func doRegister(ctx context.Context, cfg *config.Config, inputs *registerInputs)
cfg.Runner.ExtraHeaders,
)
if _, err := cli.Ping(ctx, connect.NewRequest(&pingv1.PingRequest{
if _, err := cli.Ping(ctx, &pingv1.PingRequest{
Data: inputs.RunnerName,
})); err != nil {
}); err != nil {
return fmt.Errorf("cannot ping the Gitea instance server: %w", err)
}
logging.Debug("Successfully pinged the Gitea instance server")
@@ -395,25 +384,25 @@ func doRegister(ctx context.Context, cfg *config.Config, inputs *registerInputs)
ls[i] = l.Name
}
// register new runner.
resp, err := cli.Register(ctx, connect.NewRequest(&runnerv1.RegisterRequest{
resp, err := cli.Register(ctx, &runnerv1.RegisterRequest{
Name: reg.Name,
Token: reg.Token,
Version: ver.Version(),
Labels: ls,
Ephemeral: reg.Ephemeral,
Capabilities: run.RunnerCapabilities(),
}))
})
if err != nil {
logging.WithError(err).Error("poller: cannot register new runner")
return err
}
reg.ID = resp.Msg.Runner.Id
reg.UUID = resp.Msg.Runner.Uuid
reg.Name = resp.Msg.Runner.Name
reg.Token = resp.Msg.Runner.Token
reg.ID = resp.Runner.Id
reg.UUID = resp.Runner.Uuid
reg.Name = resp.Runner.Name
reg.Token = resp.Runner.Token
if inputs.Ephemeral != resp.Msg.Runner.Ephemeral {
if inputs.Ephemeral != resp.Runner.Ephemeral {
// TODO we cannot remove the configuration via runner api, if we return an error here we just fill the database
logging.Error("poller: cannot register new runner as ephemeral upgrade Gitea to gain security, run-once will be used automatically")
}
+54 -89
View File
@@ -65,7 +65,6 @@ func TestRegisterInputsValidate(t *testing.T) {
func TestValidateLabels(t *testing.T) {
require.NoError(t, validateLabels([]string{"ubuntu:host", "ubuntu:docker://node:18"}))
// a colon that is not a supported schema is part of the label name
require.NoError(t, validateLabels([]string{"pool:e57e18d4-10d4-406f-93bf-60f127221bdd"}))
require.Error(t, validateLabels([]string{"ubuntu:host", ""}))
}
@@ -149,97 +148,63 @@ func TestRegisterInputsAssignToNext(t *testing.T) {
}
func TestInitInputs(t *testing.T) {
t.Run("missing token", func(t *testing.T) {
_, err := initInputs(&registerArgs{
InstanceAddr: "http://localhost:3000",
RunnerName: "runner",
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
for _, tc := range []struct {
name, tokenFile, wantErr string
env *string
}{
{name: "missing token", wantErr: "missing token, token-file argument, or GITEA_RUNNER_REGISTRATION_TOKEN environment variable"},
{name: "empty token", env: new(""), wantErr: "missing token, token-file argument, or GITEA_RUNNER_REGISTRATION_TOKEN environment variable"},
{
name: "invalid token file", tokenFile: "/tmp/nonexistent", env: new("from-env"),
wantErr: "cannot read the token file: /tmp/nonexistent, open /tmp/nonexistent: no such file or directory",
},
} {
t.Run(tc.name, func(t *testing.T) {
if tc.env != nil {
t.Setenv(registerTokenEnvVar, *tc.env)
}
_, err := initInputs(&registerArgs{
InstanceAddr: "http://localhost:3000",
TokenFile: tc.tokenFile,
RunnerName: "runner",
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
})
require.EqualError(t, err, tc.wantErr)
})
require.EqualError(t, err, "missing token, token-file argument, or GITEA_RUNNER_REGISTRATION_TOKEN environment variable")
})
}
t.Run("empty token", func(t *testing.T) {
t.Setenv(registerTokenEnvVar, "")
_, err := initInputs(&registerArgs{
InstanceAddr: "http://localhost:3000",
Token: "",
TokenFile: "",
RunnerName: "runner",
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
for _, tc := range []struct {
name, address, token, runnerName, wantToken string
file bool
}{
{name: "valid token", address: " http://localhost:3000 ", token: "from-plain-arg\n", runnerName: "runner\n", wantToken: "from-plain-arg"},
{name: "valid token file", address: "http://localhost:3000", runnerName: "runner", file: true, wantToken: "from-file"},
{name: "token from environment variable", address: "http://localhost:3000", runnerName: "runner", wantToken: "from-env"},
} {
t.Run(tc.name, func(t *testing.T) {
t.Setenv(registerTokenEnvVar, "from-env")
var tokenFile string
if tc.file {
tokenFile = filepath.Join(t.TempDir(), "from-file")
require.NoError(t, os.WriteFile(tokenFile, []byte("from-file\n"), 0o600))
}
inputs, err := initInputs(&registerArgs{
InstanceAddr: tc.address,
Token: tc.token,
TokenFile: tokenFile,
RunnerName: tc.runnerName,
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
})
require.NoError(t, err)
require.Equal(t, "http://localhost:3000", inputs.InstanceAddr)
require.Equal(t, tc.wantToken, inputs.Token)
require.Equal(t, "runner", inputs.RunnerName)
require.True(t, inputs.Ephemeral)
require.Equal(t, []string{"ubuntu:host ", " ubuntu:docker://node:18"}, inputs.Labels)
})
require.EqualError(t, err, "missing token, token-file argument, or GITEA_RUNNER_REGISTRATION_TOKEN environment variable")
})
t.Run("invalid token file", func(t *testing.T) {
t.Setenv(registerTokenEnvVar, "from-env")
_, err := initInputs(&registerArgs{
InstanceAddr: "http://localhost:3000",
TokenFile: "/tmp/nonexistent",
RunnerName: "runner",
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
})
require.EqualError(t, err, "cannot read the token file: /tmp/nonexistent, open /tmp/nonexistent: no such file or directory")
})
t.Run("valid token", func(t *testing.T) {
t.Setenv(registerTokenEnvVar, "from-env")
inputs, err := initInputs(&registerArgs{
InstanceAddr: " http://localhost:3000 ",
Token: "from-plain-arg\n",
RunnerName: "runner\n",
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
})
require.NoError(t, err)
require.Equal(t, "http://localhost:3000", inputs.InstanceAddr)
require.Equal(t, "from-plain-arg", inputs.Token)
require.Equal(t, "runner", inputs.RunnerName)
require.True(t, inputs.Ephemeral)
require.Equal(t, []string{"ubuntu:host ", " ubuntu:docker://node:18"}, inputs.Labels)
})
t.Run("valid token file", func(t *testing.T) {
t.Setenv(registerTokenEnvVar, "from-env")
tokenFile, createErr := os.CreateTemp(t.TempDir(), "from-file")
require.NoError(t, createErr)
defer tokenFile.Close()
_, writeErr := tokenFile.WriteString("from-file\n")
require.NoError(t, writeErr)
_ = tokenFile.Sync()
inputs, err := initInputs(&registerArgs{
InstanceAddr: "http://localhost:3000",
TokenFile: tokenFile.Name(),
RunnerName: "runner",
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
})
require.NoError(t, err)
require.Equal(t, "http://localhost:3000", inputs.InstanceAddr)
require.Equal(t, "from-file", inputs.Token)
require.Equal(t, "runner", inputs.RunnerName)
require.True(t, inputs.Ephemeral)
require.Equal(t, []string{"ubuntu:host ", " ubuntu:docker://node:18"}, inputs.Labels)
})
t.Run("token from environment variable", func(t *testing.T) {
t.Setenv(registerTokenEnvVar, "from-env")
inputs, err := initInputs(&registerArgs{
InstanceAddr: "http://localhost:3000",
RunnerName: "runner",
Ephemeral: true,
Labels: " ubuntu:host , ubuntu:docker://node:18 ",
})
require.NoError(t, err)
require.Equal(t, "http://localhost:3000", inputs.InstanceAddr)
require.Equal(t, "from-env", inputs.Token)
require.Equal(t, "runner", inputs.RunnerName)
require.True(t, inputs.Ephemeral)
require.Equal(t, []string{"ubuntu:host ", " ubuntu:docker://node:18"}, inputs.Labels)
})
}
t.Run("empty labels", func(t *testing.T) {
inputs, _ := initInputs(&registerArgs{
+25 -37
View File
@@ -17,7 +17,7 @@ import (
"gitea.com/gitea/runner/internal/pkg/config"
"gitea.com/gitea/runner/internal/pkg/metrics"
"connectrpc.com/connect"
"connectrpc.com/connect/v2"
runnerv1 "gitea.dev/actionslib/runner/v1"
)
@@ -83,8 +83,6 @@ func New(cfg *config.Config, client client.Client, runner TaskRunner) *Poller {
jobsCtx, shutdownJobs := context.WithCancel(context.Background())
done := make(chan struct{})
p := &Poller{
client: client,
runner: runner,
@@ -96,11 +94,11 @@ func New(cfg *config.Config, client client.Client, runner TaskRunner) *Poller {
jobsCtx: jobsCtx,
shutdownJobs: shutdownJobs,
done: done,
done: make(chan struct{}),
availabilityReady: true,
availabilityReason: "ok",
}
p.lastHealthyPoll.Store(time.Now().UnixNano())
p.availabilityReady = true
p.availabilityReason = "ok"
return p
}
@@ -132,8 +130,8 @@ func (p *Poller) Poll() {
continue
}
task, ok := p.fetchTask(p.pollingCtx, s)
if !ok {
task := p.fetchTask(p.pollingCtx, s)
if task == nil {
p.runIdleMaintenance()
<-sem
if !p.waitBackoff(s) {
@@ -166,8 +164,8 @@ func (p *Poller) PollOnce(onTaskFetched func()) {
}
continue
}
task, ok := p.fetchTask(p.pollingCtx, s)
if !ok {
task := p.fetchTask(p.pollingCtx, s)
if task == nil {
p.runIdleMaintenance()
if !p.waitBackoff(s) {
return
@@ -348,16 +346,16 @@ func (p *Poller) runTaskWithRecover(ctx context.Context, task *runnerv1.Task) {
}
}
func (p *Poller) fetchTask(ctx context.Context, s *workerState) (*runnerv1.Task, bool) {
func (p *Poller) fetchTask(ctx context.Context, s *workerState) *runnerv1.Task {
reqCtx, cancel := context.WithTimeout(ctx, p.cfg.Runner.FetchTimeout)
defer cancel()
// Load the version value that was in the cache when the request was sent.
v := p.tasksVersion.Load()
start := time.Now()
resp, err := p.client.FetchTask(reqCtx, connect.NewRequest(&runnerv1.FetchTaskRequest{
resp, err := p.client.FetchTask(reqCtx, &runnerv1.FetchTaskRequest{
TasksVersion: v,
}))
})
// Our own deadline proves nothing either way: today Gitea answers immediately,
// so it means a slow server, and once it holds the request open it is an idle
@@ -370,7 +368,7 @@ func (p *Poller) fetchTask(ctx context.Context, s *workerState) (*runnerv1.Task,
}
s.consecutiveEmpty++
metrics.PollFetchTotal.WithLabelValues(metrics.LabelResultEmpty).Inc()
return nil, false
return nil
}
s.fetchTimedOut = false
metrics.PollFetchDuration.Observe(time.Since(start).Seconds())
@@ -385,50 +383,40 @@ func (p *Poller) fetchTask(ctx context.Context, s *workerState) (*runnerv1.Task,
p.unregistered.Store(true)
p.registrationRejected.Store(rejected)
p.shutdownPolling()
return nil, false
}
// Not a long poll, so a deadline can mean the server assigned a task
// this runner never received.
if errors.Is(err, context.DeadlineExceeded) {
logging.WithError(err).Errorf("fetching a task timed out after %s", p.cfg.Runner.FetchTimeout)
} else {
logging.WithError(err).Error("failed to fetch task")
return nil
}
logging.WithError(err).Error("failed to fetch task")
p.lastPollFailed.Store(true)
s.consecutiveErrors++
metrics.PollFetchTotal.WithLabelValues(metrics.LabelResultError).Inc()
metrics.ClientErrors.WithLabelValues(metrics.LabelMethodFetchTask, connect.CodeOf(err).String()).Inc()
return nil, false
return nil
}
p.markHealthyPoll()
p.lastHealthyPoll.Store(time.Now().UnixNano())
p.lastPollFailed.Store(false)
// Successful response — reset error counter.
s.consecutiveErrors = 0
if resp == nil || resp.Msg == nil {
if resp == nil {
s.consecutiveEmpty++
metrics.PollFetchTotal.WithLabelValues(metrics.LabelResultEmpty).Inc()
return nil, false
return nil
}
if resp.Msg.TasksVersion > v {
p.tasksVersion.CompareAndSwap(v, resp.Msg.TasksVersion)
if resp.TasksVersion > v {
p.tasksVersion.CompareAndSwap(v, resp.TasksVersion)
}
if resp.Msg.Task == nil {
if resp.Task == nil {
s.consecutiveEmpty++
metrics.PollFetchTotal.WithLabelValues(metrics.LabelResultEmpty).Inc()
return nil, false
return nil
}
// got a task, set `tasksVersion` to zero to force query db in next request.
p.tasksVersion.CompareAndSwap(resp.Msg.TasksVersion, 0)
p.tasksVersion.CompareAndSwap(resp.TasksVersion, 0)
metrics.PollFetchTotal.WithLabelValues(metrics.LabelResultTask).Inc()
return resp.Msg.Task, true
}
func (p *Poller) markHealthyPoll() {
p.lastHealthyPoll.Store(time.Now().UnixNano())
p.lastPollFailed.Store(false)
return resp.Task
}
+57 -115
View File
@@ -15,22 +15,18 @@ import (
"gitea.com/gitea/runner/internal/pkg/client/mocks"
"gitea.com/gitea/runner/internal/pkg/config"
connect_go "connectrpc.com/connect"
"connectrpc.com/connect/v2"
runnerv1 "gitea.dev/actionslib/runner/v1"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
)
// TestPoller_WorkerStateCounters verifies that workerState correctly tracks
// consecutive empty responses independently per state instance, and that
// fetchTask increments only the relevant counter.
func TestPoller_WorkerStateCounters(t *testing.T) {
client := mocks.NewClient(t)
client.On("FetchTask", mock.Anything, mock.Anything).Return(
func(_ context.Context, _ *connect_go.Request[runnerv1.FetchTaskRequest]) (*connect_go.Response[runnerv1.FetchTaskResponse], error) {
// Always return an empty response.
return connect_go.NewResponse(&runnerv1.FetchTaskResponse{}), nil
func(_ context.Context, _ *runnerv1.FetchTaskRequest) (*runnerv1.FetchTaskResponse, error) {
return &runnerv1.FetchTaskResponse{}, nil
},
)
@@ -42,24 +38,17 @@ func TestPoller_WorkerStateCounters(t *testing.T) {
s1 := &workerState{}
s2 := &workerState{}
// Each worker independently observes one empty response.
_, ok := p.fetchTask(ctx, s1)
require.False(t, ok)
_, ok = p.fetchTask(ctx, s2)
require.False(t, ok)
require.Nil(t, p.fetchTask(ctx, s1))
require.Nil(t, p.fetchTask(ctx, s2))
assert.Equal(t, int64(1), s1.consecutiveEmpty, "worker 1 should only count its own empty response")
assert.Equal(t, int64(1), s2.consecutiveEmpty, "worker 2 should only count its own empty response")
// Worker 1 sees a second empty; worker 2 stays at 1.
_, ok = p.fetchTask(ctx, s1)
require.False(t, ok)
require.Nil(t, p.fetchTask(ctx, s1))
assert.Equal(t, int64(2), s1.consecutiveEmpty)
assert.Equal(t, int64(1), s2.consecutiveEmpty, "worker 2's counter must not be affected by worker 1's empty fetches")
}
// TestPoller_FetchErrorIncrementsErrorsOnly verifies that a fetch error
// increments only the per-worker error counter, not the empty counter.
func TestPoller_FetchErrorIncrementsErrorsOnly(t *testing.T) {
client := mocks.NewClient(t)
client.On("FetchTask", mock.Anything, mock.Anything).Return(nil, errors.New("network unreachable"))
@@ -69,15 +58,11 @@ func TestPoller_FetchErrorIncrementsErrorsOnly(t *testing.T) {
p := &Poller{client: client, cfg: cfg}
s := &workerState{}
_, ok := p.fetchTask(t.Context(), s)
require.False(t, ok)
require.Nil(t, p.fetchTask(t.Context(), s))
assert.Equal(t, int64(1), s.consecutiveErrors)
assert.Equal(t, int64(0), s.consecutiveEmpty)
}
// A deadline is the idle path once the server holds the request open, and a
// symptom before then, so it must neither reset the error count nor assert that
// the server is reachable.
func TestPoller_FetchTimeoutIsNoSignal(t *testing.T) {
client := mocks.NewClient(t)
client.On("FetchTask", mock.Anything, mock.Anything).Return(nil, context.DeadlineExceeded)
@@ -90,27 +75,22 @@ func TestPoller_FetchTimeoutIsNoSignal(t *testing.T) {
hook := logtest.SetDefault(t)
s := &workerState{consecutiveErrors: 2}
_, ok := p.fetchTask(t.Context(), s)
require.False(t, ok)
require.Nil(t, p.fetchTask(t.Context(), s))
assert.Equal(t, int64(2), s.consecutiveErrors)
assert.Equal(t, int64(1), s.consecutiveEmpty)
assert.True(t, p.lastPollFailed.Load(), "a timeout must not clear a known failure")
// An idle runner against a server holding the request open must not flood.
_, _ = p.fetchTask(t.Context(), s)
p.fetchTask(t.Context(), s)
assert.Len(t, hook.AllRecords(), 1)
}
// TestPoller_FetchUnauthenticatedStopsPolling verifies that an Unauthenticated
// response marks the runner as unregistered and cancels the polling context so
// the daemon can exit instead of retrying forever.
func TestPoller_FetchUnauthenticatedStopsPolling(t *testing.T) {
for name, tc := range map[string]struct {
fetchErr error
rejected bool
}{
"proxy": {connect_go.NewError(connect_go.CodeUnauthenticated, errors.New("401 Unauthorized")), false},
"gitea": {connect_go.NewWireError(connect_go.CodeUnknown, errors.New("rpc error: code = Unauthenticated desc = unregistered runner")), true},
"proxy": {connect.NewError(connect.CodeUnauthenticated, "401 Unauthorized"), false},
"gitea": {connect.NewError(connect.CodeUnknown, "rpc error: code = Unauthenticated desc = unregistered runner").WithRemote(), true},
} {
t.Run(name, func(t *testing.T) {
client := mocks.NewClient(t)
@@ -120,8 +100,7 @@ func TestPoller_FetchUnauthenticatedStopsPolling(t *testing.T) {
require.NoError(t, err)
p := New(cfg, client, nil)
s := &workerState{}
_, ok := p.fetchTask(t.Context(), s)
require.False(t, ok)
require.Nil(t, p.fetchTask(t.Context(), s))
assert.True(t, p.Unregistered(), "runner should be marked unregistered")
assert.Equal(t, tc.rejected, p.RegistrationRejected())
@@ -135,8 +114,6 @@ func TestPoller_FetchUnauthenticatedStopsPolling(t *testing.T) {
}
}
// TestPoller_CalculateInterval verifies the exponential backoff math is
// correctly driven by the workerState counters.
func TestPoller_CalculateInterval(t *testing.T) {
cfg, err := config.LoadDefault("")
require.NoError(t, err)
@@ -165,7 +142,6 @@ func TestPoller_CalculateInterval(t *testing.T) {
}
}
// atomicMax atomically updates target to max(target, val).
func atomicMax(target *atomic.Int64, val int64) {
for {
old := target.Load()
@@ -205,7 +181,6 @@ func TestPollerReady(t *testing.T) {
require.NoError(t, err)
poller := New(cfg, nil, &availabilityRunner{})
// /readyz reuses the availability the poll loop last recorded.
ready, reason := poller.Ready(time.Second)
assert.True(t, ready)
assert.Equal(t, "ok", reason)
@@ -242,7 +217,7 @@ func TestPollerPausesAndResumesForLocalAvailability(t *testing.T) {
cli := mocks.NewClient(t)
cli.On("FetchTask", mock.Anything, mock.Anything).Maybe().Run(func(mock.Arguments) {
fetches.Add(1)
}).Return(connect_go.NewResponse(&runnerv1.FetchTaskResponse{}), nil)
}).Return(&runnerv1.FetchTaskResponse{}, nil)
cfg, err := config.LoadDefault("")
require.NoError(t, err)
@@ -291,68 +266,50 @@ func (m *idleAwareRunner) OnIdle(_ context.Context) {
m.idleCalls.Add(1)
}
func TestPollerPollCallsOnIdle(t *testing.T) {
cli := mocks.NewClient(t)
cli.On("FetchTask", mock.Anything, mock.Anything).Return(
func(_ context.Context, _ *connect_go.Request[runnerv1.FetchTaskRequest]) (*connect_go.Response[runnerv1.FetchTaskResponse], error) {
return connect_go.NewResponse(&runnerv1.FetchTaskResponse{}), nil
},
)
func TestPollerCallsOnIdle(t *testing.T) {
for _, pollOnce := range []bool{false, true} {
name := "Poll"
if pollOnce {
name = "PollOnce"
}
t.Run(name, func(t *testing.T) {
cli := mocks.NewClient(t)
cli.On("FetchTask", mock.Anything, mock.Anything).Return(&runnerv1.FetchTaskResponse{}, nil)
cfg, err := config.LoadDefault("")
require.NoError(t, err)
cfg.Runner.Capacity = 1
cfg.Runner.FetchInterval = 10 * time.Millisecond
cfg.Runner.FetchIntervalMax = 10 * time.Millisecond
cfg, err := config.LoadDefault("")
require.NoError(t, err)
if !pollOnce {
cfg.Runner.Capacity = 1
}
cfg.Runner.FetchInterval = 10 * time.Millisecond
cfg.Runner.FetchIntervalMax = 10 * time.Millisecond
runner := &idleAwareRunner{}
poller := New(cfg, cli, runner)
runner := &idleAwareRunner{}
poller := New(cfg, cli, runner)
var wg sync.WaitGroup
wg.Go(func() {
if pollOnce {
poller.PollOnce(func() {})
} else {
poller.Poll()
}
})
var wg sync.WaitGroup
wg.Go(poller.Poll)
require.Eventually(t, func() bool {
return runner.idleCalls.Load() > 0
}, time.Second, 10*time.Millisecond)
require.Eventually(t, func() bool {
return runner.idleCalls.Load() > 0
}, time.Second, 10*time.Millisecond)
ctx, cancel := context.WithTimeout(t.Context(), time.Second)
defer cancel()
require.NoError(t, poller.Shutdown(ctx))
wg.Wait()
}
func TestPollerPollOnceCallsOnIdle(t *testing.T) {
cli := mocks.NewClient(t)
cli.On("FetchTask", mock.Anything, mock.Anything).Return(
func(_ context.Context, _ *connect_go.Request[runnerv1.FetchTaskRequest]) (*connect_go.Response[runnerv1.FetchTaskResponse], error) {
return connect_go.NewResponse(&runnerv1.FetchTaskResponse{}), nil
},
)
cfg, err := config.LoadDefault("")
require.NoError(t, err)
cfg.Runner.FetchInterval = 10 * time.Millisecond
cfg.Runner.FetchIntervalMax = 10 * time.Millisecond
runner := &idleAwareRunner{}
poller := New(cfg, cli, runner)
var wg sync.WaitGroup
wg.Go(func() { poller.PollOnce(func() {}) })
require.Eventually(t, func() bool {
return runner.idleCalls.Load() > 0
}, time.Second, 10*time.Millisecond)
ctx, cancel := context.WithTimeout(t.Context(), time.Second)
defer cancel()
require.NoError(t, poller.Shutdown(ctx))
wg.Wait()
ctx, cancel := context.WithTimeout(t.Context(), time.Second)
defer cancel()
require.NoError(t, poller.Shutdown(ctx))
wg.Wait()
})
}
}
func TestPollerPollOnceCallsOnTaskFetchedBeforeRunningIt(t *testing.T) {
cli := mocks.NewClient(t)
cli.On("FetchTask", mock.Anything, mock.Anything).Return(connect_go.NewResponse(&runnerv1.FetchTaskResponse{Task: &runnerv1.Task{Id: 1}}), nil)
cli.On("FetchTask", mock.Anything, mock.Anything).Return(&runnerv1.FetchTaskResponse{Task: &runnerv1.Task{Id: 1}}, nil)
cfg, err := config.LoadDefault("")
require.NoError(t, err)
runner := &mockRunner{}
@@ -362,9 +319,6 @@ func TestPollerPollOnceCallsOnTaskFetchedBeforeRunningIt(t *testing.T) {
assert.Equal(t, int64(1), runner.totalCompleted.Load())
}
// TestPoller_ConcurrencyLimitedByCapacity verifies that with capacity=3 and
// 6 available tasks, at most 3 tasks run concurrently, and FetchTask is
// never called concurrently (single poller).
func TestPoller_ConcurrencyLimitedByCapacity(t *testing.T) {
const (
capacity = 3
@@ -380,17 +334,17 @@ func TestPoller_ConcurrencyLimitedByCapacity(t *testing.T) {
cli := mocks.NewClient(t)
cli.On("FetchTask", mock.Anything, mock.Anything).Return(
func(_ context.Context, _ *connect_go.Request[runnerv1.FetchTaskRequest]) (*connect_go.Response[runnerv1.FetchTaskResponse], error) {
func(_ context.Context, _ *runnerv1.FetchTaskRequest) (*runnerv1.FetchTaskResponse, error) {
atomicMax(&maxFetchConcur, fetchConcur.Add(1))
defer fetchConcur.Add(-1)
n := tasksReturned.Add(1)
if n <= totalTasks {
return connect_go.NewResponse(&runnerv1.FetchTaskResponse{
return &runnerv1.FetchTaskResponse{
Task: &runnerv1.Task{Id: n},
}), nil
}, nil
}
return connect_go.NewResponse(&runnerv1.FetchTaskResponse{}), nil
return &runnerv1.FetchTaskResponse{}, nil
},
)
@@ -427,29 +381,20 @@ func TestPoller_ConcurrencyLimitedByCapacity(t *testing.T) {
"all tasks should have been executed")
}
// TestPoller_ShutdownForcesJobsOnTimeout locks in the fix for a
// pre-existing bug where Shutdown's timeout branch used a blocking
// `<-p.done` receive, leaving p.shutdownJobs() unreachable. With a
// task parked on jobsCtx and a Shutdown deadline shorter than the
// task's natural completion, Shutdown must force-cancel via
// shutdownJobs() and return ctx.Err() promptly — not block until the
// task would have finished on its own.
func TestPoller_ShutdownForcesJobsOnTimeout(t *testing.T) {
var served atomic.Bool
cli := mocks.NewClient(t)
cli.On("FetchTask", mock.Anything, mock.Anything).Return(
func(_ context.Context, _ *connect_go.Request[runnerv1.FetchTaskRequest]) (*connect_go.Response[runnerv1.FetchTaskResponse], error) {
func(_ context.Context, _ *runnerv1.FetchTaskRequest) (*runnerv1.FetchTaskResponse, error) {
if served.CompareAndSwap(false, true) {
return connect_go.NewResponse(&runnerv1.FetchTaskResponse{
return &runnerv1.FetchTaskResponse{
Task: &runnerv1.Task{Id: 1},
}), nil
}, nil
}
return connect_go.NewResponse(&runnerv1.FetchTaskResponse{}), nil
return &runnerv1.FetchTaskResponse{}, nil
},
)
// delay >> Shutdown timeout: Run only returns when jobsCtx is
// cancelled by shutdownJobs().
runner := &mockRunner{delay: 30 * time.Second}
cfg, err := config.LoadDefault("")
@@ -474,9 +419,6 @@ func TestPoller_ShutdownForcesJobsOnTimeout(t *testing.T) {
elapsed := time.Since(start)
require.ErrorIs(t, err, context.DeadlineExceeded)
// With the fix, Shutdown returns shortly after the deadline once
// the forced job unwinds. Without the fix, the blocking <-p.done
// would hang for the full 30s mockRunner delay.
assert.Less(t, elapsed, 5*time.Second,
"Shutdown must not block on the parked task; shutdownJobs() must run on timeout")
Loaded 100 of 131 files, more files were not shown because too many files have changed in this diff. Show more