mirror of
https://gitea.com/gitea/runner.git
synced 2026-11-13 19:33:41 +00:00
feat: add Kubernetes backend (#1260)
Adds a Kubernetes backend. A `name:kubernetes://image` label runs the job as one pod through the Kubernetes API, with services as containers of the same pod. Step env is exported by a script sent over exec stdin, keeping it out of the pod spec and process arguments, and service env goes into a per-job Secret. Docker actions and container options are rejected. See `docs/kubernetes.md`. Operators shape job pods with `kubernetes.pod_template`, with per-label `pod_templates` merged in `runs-on` order and a `$services` entry for service containers. The idle cleanup removes pods and Secrets a runner left behind. The client calls the Kubernetes API directly and runs step commands over `coder/websocket` (same dependency used in gitea) instead of pulling in the heavyweight `kubernetes/client-go`. Includes an e2e test against k3s. Replaces: https://gitea.com/gitea/runner/pulls/1000 Fixes: https://gitea.com/gitea/runner/issues/31 Assisted-by: Claude Code:claude-opus-5-5 Co-authored-by: Zettat123 <[email protected]> Co-authored-by: bircni <[email protected]> Reviewed-on: https://gitea.com/gitea/runner/pulls/1260 Reviewed-by: Lunny Xiao <[email protected]> Reviewed-by: Zettat123 <[email protected]> Co-authored-by: silverwind <[email protected]>
This commit is contained in:
1 parent
07ca4bfa7f
commit
0cedd98051
35 files changed
+2012
-226
No files matched your search
@@ -12,6 +12,7 @@ PODMAN_TEST_IMAGE ?= quay.io/containers/podman:v5.8.4-immutable@sha256:90405e3c6
|
|||||||
E2E_JOB_IMAGE ?= node:24-bookworm@sha256:64af3819f9275802414d7cdc38c27e9d82bd564dec4d4da87d008255d36c63b4 # renovate: datasource=docker
|
E2E_JOB_IMAGE ?= node:24-bookworm@sha256:64af3819f9275802414d7cdc38c27e9d82bd564dec4d4da87d008255d36c63b4 # renovate: datasource=docker
|
||||||
SERVICE_IMAGE ?= nginx:1.31.4-alpine@sha256:db35bfc6b2951e7f8a72db5db120288c127ffaeeb4a6d4b95a26fead017d5913 # renovate: datasource=docker
|
SERVICE_IMAGE ?= nginx:1.31.4-alpine@sha256:db35bfc6b2951e7f8a72db5db120288c127ffaeeb4a6d4b95a26fead017d5913 # renovate: datasource=docker
|
||||||
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
|
E2E_GITEA_IMAGE ?= gitea/gitea:main-nightly
|
||||||
|
E2E_K3S_IMAGE ?= rancher/k3s:v1.36.4-k3s1@sha256:edad48e12bf81c3a09ac1c05c0c0ffaaa22145980b989d6fae84543a76b83657 # renovate: datasource=docker
|
||||||
|
|
||||||
LINUX_ARCHS ?= linux/amd64,linux/arm64
|
LINUX_ARCHS ?= linux/amd64,linux/arm64
|
||||||
DARWIN_ARCHS ?= darwin-12/amd64,darwin-12/arm64
|
DARWIN_ARCHS ?= darwin-12/amd64,darwin-12/arm64
|
||||||
@@ -170,7 +171,7 @@ E2E_CONCURRENCY ?= 8
|
|||||||
|
|
||||||
.PHONY: test-e2e
|
.PHONY: test-e2e
|
||||||
test-e2e: ## run Gitea compatibility tests against E2E_GITEA_IMAGE
|
test-e2e: ## run Gitea compatibility tests against E2E_GITEA_IMAGE
|
||||||
@E2E_CONCURRENCY=$(E2E_CONCURRENCY) E2E_GITEA_IMAGE=$(E2E_GITEA_IMAGE) E2E_JOB_IMAGE=$(E2E_JOB_IMAGE) GO=$(GO) SERVICE_IMAGE=$(SERVICE_IMAGE) ./tools/test-e2e.sh
|
@E2E_CONCURRENCY=$(E2E_CONCURRENCY) E2E_GITEA_IMAGE=$(E2E_GITEA_IMAGE) E2E_K3S_IMAGE=$(E2E_K3S_IMAGE) E2E_JOB_IMAGE=$(E2E_JOB_IMAGE) GO=$(GO) SERVICE_IMAGE=$(SERVICE_IMAGE) ./tools/test-e2e.sh
|
||||||
|
|
||||||
.PHONY: install
|
.PHONY: install
|
||||||
install: $(GOFILES) ## install the runner binary via `go install`
|
install: $(GOFILES) ## install the runner binary via `go install`
|
||||||
|
|||||||
@@ -172,10 +172,10 @@ A label is written as:
|
|||||||
| Part | Meaning |
|
| Part | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `name` | The name a workflow refers to in `runs-on`, e.g. `ubuntu-latest`. |
|
| `name` | The name a workflow refers to in `runs-on`, e.g. `ubuntu-latest`. |
|
||||||
| `schema` | Either `docker` or `host`. Defaults to `host` when omitted. |
|
| `schema` | `docker`, `kubernetes` or `host`. Defaults to `host` when omitted. |
|
||||||
| `args` | Only used by the `docker` schema: the image to run the job in. |
|
| `args` | Only used by the `docker` and `kubernetes` schemas: the image to run the job in. |
|
||||||
|
|
||||||
Two schemas are supported:
|
Three schemas are supported:
|
||||||
|
|
||||||
- **`docker://<image>`** — the job runs inside a container created from `<image>`:
|
- **`docker://<image>`** — the job runs inside a container created from `<image>`:
|
||||||
|
|
||||||
@@ -183,6 +183,12 @@ Two schemas are supported:
|
|||||||
ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest
|
ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- **`kubernetes://<image>`** — the job runs as a Kubernetes pod created from `<image>`, see **[docs/kubernetes.md](docs/kubernetes.md)**:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ubuntu-latest:kubernetes://docker.gitea.com/runner-images:ubuntu-latest
|
||||||
|
```
|
||||||
|
|
||||||
- **`host`** — the job's steps run directly on the machine the runner is on, using the tools installed there:
|
- **`host`** — the job's steps run directly on the machine the runner is on, using the tools installed there:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
@@ -197,9 +203,9 @@ ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,macos:host
|
|||||||
|
|
||||||
a workflow with `runs-on: ubuntu-latest` is executed in the `runner-images:ubuntu-latest` container, and one with `runs-on: macos` is executed directly on the host.
|
a workflow with `runs-on: ubuntu-latest` is executed in the `runner-images:ubuntu-latest` container, and one with `runs-on: macos` is executed directly on the host.
|
||||||
|
|
||||||
Names may themselves contain a colon (for example `pool:e57e18d4-10d4-406f-93bf-60f127221bdd`); only `host` and `docker` are treated as schemas.
|
Names may themselves contain a colon, for example `pool:e57e18d4-10d4-406f-93bf-60f127221bdd`. Only `host`, `docker` and `kubernetes` are treated as schemas.
|
||||||
|
|
||||||
If a job's `runs-on` matches none of the runner's labels, or sets no `runs-on` at all, it still runs: in `runner.default_image` where docker is available, on the host where it is not. Images maintained for this purpose are listed at [gitea/runner-images](https://gitea.com/gitea/runner-images).
|
If a job's `runs-on` matches none of the runner's labels, or sets no `runs-on` at all, it still runs in `runner.default_image`: as a container when the runner has `docker` labels, as a pod when it has `kubernetes` labels, else as a container where docker is available and on the host where it is not. Images maintained for this purpose are listed at [gitea/runner-images](https://gitea.com/gitea/runner-images).
|
||||||
|
|
||||||
Labels are chosen at registration time (`--labels`, or the interactive prompt) and can be changed afterwards by editing `runner.labels` in the config file, or in the Gitea UI under the runner's settings.
|
Labels are chosen at registration time (`--labels`, or the interactive prompt) and can be changed afterwards by editing `runner.labels` in the config file, or in the Gitea UI under the runner's settings.
|
||||||
|
|
||||||
@@ -351,6 +357,7 @@ While the runner is idle it cleans up after earlier jobs:
|
|||||||
- only purely numeric subdirectories under `container.workdir_parent` are treated as task workspaces and may be removed
|
- only purely numeric subdirectories under `container.workdir_parent` are treated as task workspaces and may be removed
|
||||||
- cleanup assumes `container.workdir_parent` is not shared across multiple runners
|
- cleanup assumes `container.workdir_parent` is not shared across multiple runners
|
||||||
- on runners that use docker, per-job networks left behind by jobs the runner did not live to tear down are removed, identified by the `com.gitea.runner.uuid` label carrying this runner's uuid
|
- on runners that use docker, per-job networks left behind by jobs the runner did not live to tear down are removed, identified by the `com.gitea.runner.uuid` label carrying this runner's uuid
|
||||||
|
- on runners with `kubernetes` labels, job pods with that label older than `runner.workdir_cleanup_age` are removed the same way, together with their Secrets
|
||||||
- cleanup runs every `runner.idle_cleanup_interval` (default: `10m`; set `0` to disable), and setting either knob to `0` disables all of the above
|
- cleanup runs every `runner.idle_cleanup_interval` (default: `10m`; set `0` to disable), and setting either knob to `0` disables all of the above
|
||||||
|
|
||||||
#### Post-task script (`runner.post_task_script`)
|
#### Post-task script (`runner.post_task_script`)
|
||||||
|
|||||||
@@ -75,6 +75,9 @@ const (
|
|||||||
HealthUnhealthy = string(container.Unhealthy)
|
HealthUnhealthy = string(container.Unhealthy)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// marks what a runner creates for its jobs, telling its leftovers from another runner's
|
||||||
|
const runnerUUIDLabel = "com.gitea.runner.uuid"
|
||||||
|
|
||||||
// ErrContainerNotFound reports a container the daemon no longer knows. Its text is a
|
// ErrContainerNotFound reports a container the daemon no longer knows. Its text is a
|
||||||
// fragment, missingContainerError composes it into the message every operation shares.
|
// fragment, missingContainerError composes it into the message every operation shares.
|
||||||
var ErrContainerNotFound = errors.New("does not exist")
|
var ErrContainerNotFound = errors.New("does not exist")
|
||||||
|
|||||||
@@ -24,10 +24,6 @@ import (
|
|||||||
const (
|
const (
|
||||||
networkCreateAttempts = 3
|
networkCreateAttempts = 3
|
||||||
networkCreateRetryDelay = time.Second
|
networkCreateRetryDelay = time.Second
|
||||||
|
|
||||||
// marks the networks a runner creates for its jobs, so it can tell its own leftovers from
|
|
||||||
// those of another runner sharing the daemon
|
|
||||||
runnerUUIDLabel = "com.gitea.runner.uuid"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// RemoveOrphanNetworks removes the networks this runner created for jobs whose teardown did
|
// RemoveOrphanNetworks removes the networks this runner created for jobs whose teardown did
|
||||||
|
|||||||
@@ -7,7 +7,6 @@
|
|||||||
package container
|
package container
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"archive/tar"
|
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
@@ -25,16 +24,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.com/gitea/runner/act/common"
|
"gitea.com/gitea/runner/act/common"
|
||||||
"gitea.com/gitea/runner/act/filecollector"
|
|
||||||
|
|
||||||
"dario.cat/mergo"
|
"dario.cat/mergo"
|
||||||
"github.com/bmatcuk/doublestar/v4"
|
"github.com/bmatcuk/doublestar/v4"
|
||||||
cerrdefs "github.com/containerd/errdefs"
|
cerrdefs "github.com/containerd/errdefs"
|
||||||
"github.com/docker/cli/cli/compose/loader"
|
"github.com/docker/cli/cli/compose/loader"
|
||||||
"github.com/docker/cli/cli/connhelper"
|
"github.com/docker/cli/cli/connhelper"
|
||||||
"github.com/go-git/go-billy/v5/helper/polyfill"
|
|
||||||
"github.com/go-git/go-billy/v5/osfs"
|
|
||||||
"github.com/go-git/go-git/v5/plumbing/format/gitignore"
|
|
||||||
"github.com/joho/godotenv"
|
"github.com/joho/godotenv"
|
||||||
"github.com/moby/moby/api/pkg/stdcopy"
|
"github.com/moby/moby/api/pkg/stdcopy"
|
||||||
"github.com/moby/moby/api/types/container"
|
"github.com/moby/moby/api/types/container"
|
||||||
@@ -982,42 +977,7 @@ func (cr *containerReference) copyDir(dstPath, srcPath string, useGitIgnore, ski
|
|||||||
logger.Error(err)
|
logger.Error(err)
|
||||||
}
|
}
|
||||||
}(tarFile)
|
}(tarFile)
|
||||||
tw := tar.NewWriter(tarFile)
|
if err := writeDirTar(ctx, tarFile, dstPath, srcPath, useGitIgnore, skipGitDir, cr.UID, cr.GID); err != nil {
|
||||||
|
|
||||||
srcPrefix := filepath.Dir(srcPath)
|
|
||||||
if !strings.HasSuffix(srcPrefix, string(filepath.Separator)) {
|
|
||||||
srcPrefix += string(filepath.Separator)
|
|
||||||
}
|
|
||||||
logger.Debugf("Stripping prefix:%s src:%s", srcPrefix, srcPath)
|
|
||||||
|
|
||||||
var ignorer gitignore.Matcher
|
|
||||||
if useGitIgnore {
|
|
||||||
ps, err := gitignore.ReadPatterns(polyfill.New(osfs.New(srcPath)), nil)
|
|
||||||
if err != nil {
|
|
||||||
logger.Debugf("Error loading .gitignore: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ignorer = gitignore.NewMatcher(ps)
|
|
||||||
}
|
|
||||||
|
|
||||||
fc := &filecollector.FileCollector{
|
|
||||||
Ignorer: ignorer,
|
|
||||||
SrcPath: srcPath,
|
|
||||||
SrcPrefix: srcPrefix,
|
|
||||||
SkipGitDir: skipGitDir,
|
|
||||||
Handler: &filecollector.TarCollector{
|
|
||||||
TarWriter: tw,
|
|
||||||
UID: cr.UID,
|
|
||||||
GID: cr.GID,
|
|
||||||
DstDir: dstPath[1:],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
err = filepath.Walk(srcPath, fc.CollectFiles(ctx, []string{}))
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := tw.Close(); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1042,30 +1002,12 @@ func (cr *containerReference) copyContent(dstPath string, files ...*FileEntry) c
|
|||||||
if cr.id == "" {
|
if cr.id == "" {
|
||||||
return cr.missingContainerError("copy to %s", dstPath)
|
return cr.missingContainerError("copy to %s", dstPath)
|
||||||
}
|
}
|
||||||
logger := common.Logger(ctx)
|
|
||||||
var buf bytes.Buffer
|
var buf bytes.Buffer
|
||||||
tw := tar.NewWriter(&buf)
|
if err := writeFilesTar(ctx, &buf, cr.UID, cr.GID, files...); err != nil {
|
||||||
for _, file := range files {
|
|
||||||
logger.Debugf("Writing entry to tarball %s len:%d", file.Name, len(file.Body))
|
|
||||||
hdr := &tar.Header{
|
|
||||||
Name: file.Name,
|
|
||||||
Mode: file.Mode,
|
|
||||||
Size: int64(len(file.Body)),
|
|
||||||
Uid: cr.UID,
|
|
||||||
Gid: cr.GID,
|
|
||||||
}
|
|
||||||
if err := tw.WriteHeader(hdr); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if _, err := tw.Write([]byte(file.Body)); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if err := tw.Close(); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
logger.Debugf("Extracting content to '%s'", dstPath)
|
common.Logger(ctx).Debugf("Extracting content to '%s'", dstPath)
|
||||||
_, err := cr.cli.CopyToContainer(ctx, cr.id, client.CopyToContainerOptions{
|
_, err := cr.cli.CopyToContainer(ctx, cr.id, client.CopyToContainerOptions{
|
||||||
DestinationPath: dstPath,
|
DestinationPath: dstPath,
|
||||||
Content: &buf,
|
Content: &buf,
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,400 @@
|
|||||||
|
// Copyright 2026 The Gitea Authors. All rights reserved.
|
||||||
|
// SPDX-License-Identifier: MIT
|
||||||
|
|
||||||
|
package container
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"cmp"
|
||||||
|
"context"
|
||||||
|
"crypto/tls"
|
||||||
|
"crypto/x509"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/coder/websocket"
|
||||||
|
"go.yaml.in/yaml/v4"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
serviceAccountDir = "/var/run/secrets/kubernetes.io/serviceaccount"
|
||||||
|
execProtocol = "v5.channel.k8s.io"
|
||||||
|
)
|
||||||
|
|
||||||
|
type kubeClient struct {
|
||||||
|
server, namespace, token string
|
||||||
|
tokenFile string // re-read on every request, service account tokens rotate
|
||||||
|
inCluster bool
|
||||||
|
http *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
type apiError struct {
|
||||||
|
code int
|
||||||
|
message string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *apiError) Error() string { return e.message }
|
||||||
|
|
||||||
|
func isNotFound(err error) bool {
|
||||||
|
var apiErr *apiError
|
||||||
|
return errors.As(err, &apiErr) && apiErr.code == http.StatusNotFound
|
||||||
|
}
|
||||||
|
|
||||||
|
func newKubeClient(kubeconfig, namespace string) (*kubeClient, error) {
|
||||||
|
client := &kubeClient{namespace: namespace}
|
||||||
|
transport := &http.Transport{Proxy: http.ProxyFromEnvironment, TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12}}
|
||||||
|
if host := os.Getenv("KUBERNETES_SERVICE_HOST"); host != "" && kubeconfig == "" {
|
||||||
|
if err := client.loadInCluster(host, transport.TLSClientConfig); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
} else if err := client.loadKubeconfig(kubeconfig, transport); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
client.http = &http.Client{Transport: transport}
|
||||||
|
return client, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *kubeClient) loadInCluster(host string, tlsConfig *tls.Config) error {
|
||||||
|
c.server = "https://" + net.JoinHostPort(host, os.Getenv("KUBERNETES_SERVICE_PORT"))
|
||||||
|
c.tokenFile = serviceAccountDir + "/token"
|
||||||
|
ca, err := os.ReadFile(serviceAccountDir + "/ca.crt")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
tlsConfig.RootCAs = x509.NewCertPool()
|
||||||
|
tlsConfig.RootCAs.AppendCertsFromPEM(ca)
|
||||||
|
if c.namespace == "" {
|
||||||
|
namespace, err := os.ReadFile(serviceAccountDir + "/namespace")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
c.namespace = strings.TrimSpace(string(namespace))
|
||||||
|
}
|
||||||
|
c.inCluster = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *kubeClient) loadKubeconfig(path string, transport *http.Transport) error {
|
||||||
|
if paths := filepath.SplitList(os.Getenv("KUBECONFIG")); path == "" && len(paths) > 0 {
|
||||||
|
path = paths[0]
|
||||||
|
}
|
||||||
|
if path == "" {
|
||||||
|
home, err := os.UserHomeDir()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
path = filepath.Join(home, ".kube", "config")
|
||||||
|
}
|
||||||
|
content, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("no in-cluster service account and no kubeconfig: %w", err)
|
||||||
|
}
|
||||||
|
var config struct {
|
||||||
|
CurrentContext string `yaml:"current-context"`
|
||||||
|
Contexts []struct {
|
||||||
|
Name string
|
||||||
|
Context struct{ Cluster, User, Namespace string }
|
||||||
|
}
|
||||||
|
Clusters []struct {
|
||||||
|
Name string
|
||||||
|
Cluster struct {
|
||||||
|
Server string
|
||||||
|
TLSServerName string `yaml:"tls-server-name"`
|
||||||
|
InsecureSkipTLSVerify bool `yaml:"insecure-skip-tls-verify"`
|
||||||
|
ProxyURL string `yaml:"proxy-url"`
|
||||||
|
CertificateAuthority string `yaml:"certificate-authority"`
|
||||||
|
CertificateAuthorityData string `yaml:"certificate-authority-data"`
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Users []struct {
|
||||||
|
Name string
|
||||||
|
User struct {
|
||||||
|
Token string
|
||||||
|
TokenFile string `yaml:"tokenFile"`
|
||||||
|
ClientCertificate string `yaml:"client-certificate"`
|
||||||
|
ClientCertificateData string `yaml:"client-certificate-data"`
|
||||||
|
ClientKey string `yaml:"client-key"`
|
||||||
|
ClientKeyData string `yaml:"client-key-data"`
|
||||||
|
Exec any
|
||||||
|
AuthProvider any `yaml:"auth-provider"`
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := yaml.Unmarshal(content, &config); err != nil {
|
||||||
|
return fmt.Errorf("parse kubeconfig %s: %w", path, err)
|
||||||
|
}
|
||||||
|
var current struct{ Cluster, User, Namespace string }
|
||||||
|
for _, entry := range config.Contexts {
|
||||||
|
if entry.Name == config.CurrentContext {
|
||||||
|
current = entry.Context
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if current.Cluster == "" {
|
||||||
|
return fmt.Errorf("kubeconfig %s has no current context", path)
|
||||||
|
}
|
||||||
|
c.namespace = cmp.Or(c.namespace, current.Namespace, "default")
|
||||||
|
|
||||||
|
resolve := func(file string) string {
|
||||||
|
if file == "" || filepath.IsAbs(file) {
|
||||||
|
return file
|
||||||
|
}
|
||||||
|
return filepath.Join(filepath.Dir(path), file)
|
||||||
|
}
|
||||||
|
read := func(data, file string) ([]byte, error) {
|
||||||
|
if data != "" {
|
||||||
|
return base64.StdEncoding.DecodeString(data)
|
||||||
|
}
|
||||||
|
if file == "" {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return os.ReadFile(resolve(file))
|
||||||
|
}
|
||||||
|
|
||||||
|
tlsConfig := transport.TLSClientConfig
|
||||||
|
for _, cluster := range config.Clusters {
|
||||||
|
if cluster.Name != current.Cluster {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
c.server = strings.TrimSuffix(cluster.Cluster.Server, "/")
|
||||||
|
tlsConfig.ServerName, tlsConfig.InsecureSkipVerify = cluster.Cluster.TLSServerName, cluster.Cluster.InsecureSkipTLSVerify
|
||||||
|
if cluster.Cluster.ProxyURL != "" {
|
||||||
|
proxy, err := url.Parse(cluster.Cluster.ProxyURL)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("kubeconfig %s: %w", path, err)
|
||||||
|
}
|
||||||
|
transport.Proxy = http.ProxyURL(proxy)
|
||||||
|
}
|
||||||
|
ca, err := read(cluster.Cluster.CertificateAuthorityData, cluster.Cluster.CertificateAuthority)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if ca != nil {
|
||||||
|
tlsConfig.RootCAs = x509.NewCertPool()
|
||||||
|
tlsConfig.RootCAs.AppendCertsFromPEM(ca)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, user := range config.Users {
|
||||||
|
if user.Name != current.User {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if user.User.Exec != nil || user.User.AuthProvider != nil {
|
||||||
|
return fmt.Errorf("kubeconfig %s: user %q authenticates through a credential plugin, which the runner does not support, use a token or client certificate", path, user.Name)
|
||||||
|
}
|
||||||
|
c.token, c.tokenFile = user.User.Token, resolve(user.User.TokenFile)
|
||||||
|
cert, err := read(user.User.ClientCertificateData, user.User.ClientCertificate)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
key, err := read(user.User.ClientKeyData, user.User.ClientKey)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if cert != nil {
|
||||||
|
pair, err := tls.X509KeyPair(cert, key)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
tlsConfig.Certificates = []tls.Certificate{pair}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if c.server == "" {
|
||||||
|
return fmt.Errorf("kubeconfig %s names no server for context %q", path, config.CurrentContext)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *kubeClient) authorize(header http.Header) error {
|
||||||
|
token := c.token
|
||||||
|
if c.tokenFile != "" {
|
||||||
|
content, err := os.ReadFile(c.tokenFile)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
token = strings.TrimSpace(string(content))
|
||||||
|
}
|
||||||
|
if token != "" {
|
||||||
|
header.Set("Authorization", "Bearer "+token)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *kubeClient) path(resource string) string {
|
||||||
|
return "/api/v1/namespaces/" + c.namespace + "/" + resource
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *kubeClient) do(ctx context.Context, method, path string, in, out any) error {
|
||||||
|
var body io.Reader
|
||||||
|
if in != nil {
|
||||||
|
content, err := json.Marshal(in)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
body = bytes.NewReader(content)
|
||||||
|
}
|
||||||
|
req, err := http.NewRequestWithContext(ctx, method, c.server+path, body)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := c.authorize(req.Header); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
resp, err := c.http.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode >= http.StatusMultipleChoices {
|
||||||
|
return statusError(resp)
|
||||||
|
}
|
||||||
|
if out == nil {
|
||||||
|
out = io.Discard // draining lets the connection be reused
|
||||||
|
}
|
||||||
|
if writer, ok := out.(io.Writer); ok {
|
||||||
|
_, err = io.Copy(writer, resp.Body)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return json.NewDecoder(resp.Body).Decode(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *kubeClient) delete(ctx context.Context, resource, name string) error {
|
||||||
|
if name == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err := c.do(ctx, http.MethodDelete, c.path(resource+"/"+name), nil, nil); err != nil && !isNotFound(err) {
|
||||||
|
return fmt.Errorf("delete %s %s: %w", resource, name, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func statusError(resp *http.Response) error {
|
||||||
|
content, _ := io.ReadAll(resp.Body)
|
||||||
|
var status struct{ Message string }
|
||||||
|
if json.Unmarshal(content, &status) != nil || status.Message == "" {
|
||||||
|
status.Message = strings.TrimSpace(string(content))
|
||||||
|
}
|
||||||
|
return &apiError{code: resp.StatusCode, message: status.Message}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *kubeClient) exec(ctx context.Context, pod, containerName string, command []string, stdin io.Reader, stdout, stderr io.Writer) error {
|
||||||
|
query := url.Values{"container": {containerName}, "command": command, "stdout": {"true"}, "stderr": {"true"}}
|
||||||
|
if stdin != nil {
|
||||||
|
query.Set("stdin", "true")
|
||||||
|
}
|
||||||
|
header := http.Header{}
|
||||||
|
if err := c.authorize(header); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancelCause(ctx)
|
||||||
|
defer cancel(nil)
|
||||||
|
conn, resp, err := websocket.Dial(ctx, c.server+c.path("pods/"+pod+"/exec?")+query.Encode(), &websocket.DialOptions{
|
||||||
|
HTTPClient: c.http,
|
||||||
|
HTTPHeader: header,
|
||||||
|
Subprotocols: []string{execProtocol},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
if resp != nil && resp.StatusCode >= http.StatusMultipleChoices {
|
||||||
|
defer resp.Body.Close()
|
||||||
|
err = statusError(resp)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("exec in pod %s: %w", pod, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = conn.CloseNow() }()
|
||||||
|
if conn.Subprotocol() != execProtocol {
|
||||||
|
return fmt.Errorf("the Kubernetes API server does not speak %s, which needs Kubernetes 1.30 or later", execProtocol)
|
||||||
|
}
|
||||||
|
conn.SetReadLimit(-1)
|
||||||
|
|
||||||
|
if stdin != nil {
|
||||||
|
go func() {
|
||||||
|
if err := sendStdin(ctx, conn, stdin); err != nil {
|
||||||
|
cancel(err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
go func() {
|
||||||
|
ticker := time.NewTicker(5 * time.Second)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
if conn.Ping(ctx) != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
for {
|
||||||
|
_, frame, err := conn.Read(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return cmp.Or(context.Cause(ctx), fmt.Errorf("exec in pod %s ended without an exit status: %w", pod, err))
|
||||||
|
}
|
||||||
|
if len(frame) < 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
switch frame[0] {
|
||||||
|
case 1:
|
||||||
|
_, err = stdout.Write(frame[1:])
|
||||||
|
case 2:
|
||||||
|
_, err = stderr.Write(frame[1:])
|
||||||
|
case 3:
|
||||||
|
return cmp.Or(context.Cause(ctx), execStatus(frame[1:]))
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func sendStdin(ctx context.Context, conn *websocket.Conn, stdin io.Reader) error {
|
||||||
|
frame := make([]byte, 1+32*1024) // frame[0] is the stdin channel, 0
|
||||||
|
for {
|
||||||
|
n, err := stdin.Read(frame[1:])
|
||||||
|
if n > 0 {
|
||||||
|
if err := conn.Write(ctx, websocket.MessageBinary, frame[:1+n]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if errors.Is(err, io.EOF) {
|
||||||
|
return conn.Write(ctx, websocket.MessageBinary, []byte{255, 0}) // v5 close signal for stdin
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func execStatus(content []byte) error {
|
||||||
|
var status struct {
|
||||||
|
Status, Message string
|
||||||
|
Details struct {
|
||||||
|
Causes []struct{ Reason, Message string }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(content, &status); err != nil {
|
||||||
|
return fmt.Errorf("decode exec status: %w", err)
|
||||||
|
}
|
||||||
|
if status.Status == "Success" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for _, cause := range status.Details.Causes {
|
||||||
|
if code, err := strconv.Atoi(cause.Message); cause.Reason == "ExitCode" && err == nil {
|
||||||
|
return ExitCodeError(code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fmt.Errorf("exec failed: %s", status.Message)
|
||||||
|
}
|
||||||
@@ -0,0 +1,290 @@
|
|||||||
|
// Copyright 2026 The Gitea Authors. All rights reserved.
|
||||||
|
// SPDX-License-Identifier: MIT
|
||||||
|
|
||||||
|
package container
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/coder/websocket"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
type podManifest struct {
|
||||||
|
Metadata struct{ Labels map[string]string }
|
||||||
|
Spec struct {
|
||||||
|
AutomountServiceAccountToken bool
|
||||||
|
EnableServiceLinks bool
|
||||||
|
HostAliases []struct{ Hostnames []string }
|
||||||
|
Volumes []map[string]any
|
||||||
|
Containers []struct {
|
||||||
|
Name, ImagePullPolicy string
|
||||||
|
Resources map[string]any
|
||||||
|
StartupProbe *struct{ PeriodSeconds, TimeoutSeconds, FailureThreshold int }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeCluster struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
requests map[string][]string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeCluster) record(request, content string) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
f.requests[request] = append(f.requests[request], content)
|
||||||
|
}
|
||||||
|
|
||||||
|
func fakeKubernetes(t *testing.T, podStatus string) (string, *fakeCluster) {
|
||||||
|
cluster := &fakeCluster{requests: map[string][]string{}}
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Header.Get("Authorization") != "Bearer test-token" {
|
||||||
|
w.WriteHeader(http.StatusUnauthorized)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
body, err := io.ReadAll(r.Body)
|
||||||
|
assert.NoError(t, err)
|
||||||
|
cluster.record(r.Method+" "+r.URL.Path, string(body))
|
||||||
|
pods, secrets := "/api/v1/namespaces/ci/pods", "/api/v1/namespaces/ci/secrets"
|
||||||
|
switch {
|
||||||
|
case r.Method == http.MethodGet && r.URL.Path == pods:
|
||||||
|
cluster.record("list", r.URL.Query().Get("labelSelector"))
|
||||||
|
_, _ = io.WriteString(w, `{"items":[{"metadata":{"name":"old","creationTimestamp":"2020-01-01T00:00:00Z"}},{"metadata":{"name":"new","creationTimestamp":"2999-01-01T00:00:00Z"}}]}`)
|
||||||
|
case strings.HasSuffix(r.URL.Path, "/exec"):
|
||||||
|
cluster.record("exec", r.URL.RawQuery)
|
||||||
|
serveLocalExec(t, w, r)
|
||||||
|
case strings.HasSuffix(r.URL.Path, "/log"):
|
||||||
|
_, _ = io.WriteString(w, "log of "+r.URL.Query().Get("container"))
|
||||||
|
case r.Method == http.MethodGet && strings.HasPrefix(r.URL.Path, pods+"/"):
|
||||||
|
_, _ = io.WriteString(w, podStatus)
|
||||||
|
case r.Method == http.MethodPost && (r.URL.Path == pods || r.URL.Path == secrets), r.Method == http.MethodDelete:
|
||||||
|
default:
|
||||||
|
w.WriteHeader(http.StatusNotFound)
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
t.Cleanup(server.Close)
|
||||||
|
kubeconfig := filepath.Join(t.TempDir(), "kubeconfig")
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(filepath.Dir(kubeconfig), "token"), []byte("test-token\n"), 0o600))
|
||||||
|
require.NoError(t, os.WriteFile(kubeconfig, fmt.Appendf(nil, `
|
||||||
|
current-context: test
|
||||||
|
contexts: [{name: test, context: {cluster: test, user: test, namespace: ci}}]
|
||||||
|
clusters: [{name: test, cluster: {server: "http://kubernetes.invalid", proxy-url: %q}}]
|
||||||
|
users: [{name: test, user: {tokenFile: token}}]
|
||||||
|
`, server.URL), 0o600))
|
||||||
|
return kubeconfig, cluster
|
||||||
|
}
|
||||||
|
|
||||||
|
func serveLocalExec(t *testing.T, w http.ResponseWriter, r *http.Request) {
|
||||||
|
conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{Subprotocols: []string{execProtocol}})
|
||||||
|
if err != nil {
|
||||||
|
t.Error(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer func() { _ = conn.CloseNow() }()
|
||||||
|
conn.SetReadLimit(-1)
|
||||||
|
command := r.URL.Query()["command"]
|
||||||
|
var stdin bytes.Buffer
|
||||||
|
for r.URL.Query().Get("stdin") == "true" {
|
||||||
|
_, frame, err := conn.Read(r.Context())
|
||||||
|
if err != nil {
|
||||||
|
t.Error(err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if frame[0] == 255 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
stdin.Write(frame[1:])
|
||||||
|
}
|
||||||
|
cmd := exec.Command(command[0], command[1:]...)
|
||||||
|
cmd.Env = append(os.Environ(), "COPYFILE_DISABLE=1")
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
cmd.Stdin, cmd.Stdout, cmd.Stderr = &stdin, &stdout, &stderr
|
||||||
|
status := `{"status":"Success"}`
|
||||||
|
var exitErr *exec.ExitError
|
||||||
|
if err := cmd.Run(); errors.As(err, &exitErr) {
|
||||||
|
status = fmt.Sprintf(`{"status":"Failure","reason":"NonZeroExitCode","details":{"causes":[{"reason":"ExitCode","message":"%d"}]}}`, exitErr.ExitCode())
|
||||||
|
} else {
|
||||||
|
assert.NoError(t, err)
|
||||||
|
}
|
||||||
|
for _, frame := range [][]byte{append([]byte{1}, stdout.Bytes()...), append([]byte{2}, stderr.Bytes()...), append([]byte{3}, status...)} {
|
||||||
|
_ = conn.Write(r.Context(), websocket.MessageBinary, frame)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestKubernetesPodRunsJobThroughAPI(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("uses POSIX shell")
|
||||||
|
}
|
||||||
|
ctx := t.Context()
|
||||||
|
kubeconfig, cluster := fakeKubernetes(t, `{"spec":{"containers":[{},{}]},"status":{"phase":"Running","containerStatuses":[
|
||||||
|
{"name":"job","ready":true,"state":{"running":{}}},
|
||||||
|
{"name":"my-db","ready":true,"state":{"running":{}}}]}}`)
|
||||||
|
workdir, err := filepath.EvalSymlinks(t.TempDir())
|
||||||
|
require.NoError(t, err)
|
||||||
|
options := KubernetesOptions{Kubeconfig: kubeconfig, RunnerUUID: "runner-uuid", PodTemplates: []map[string]any{
|
||||||
|
{"spec": map[string]any{
|
||||||
|
"automountServiceAccountToken": true,
|
||||||
|
"containers": []any{
|
||||||
|
map[string]any{"name": "job", "resources": map[string]any{"limits": map[string]any{"cpu": "2"}}},
|
||||||
|
map[string]any{
|
||||||
|
"name": "$services", "imagePullPolicy": "Always", "resources": map[string]any{"limits": map[string]any{"memory": "256Mi"}},
|
||||||
|
"env": []any{map[string]any{"name": "POSTGRES_PASSWORD", "value": "template"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}},
|
||||||
|
{"spec": map[string]any{
|
||||||
|
"volumes": []any{map[string]any{"name": "workspace", "emptyDir": map[string]any{"sizeLimit": "1Gi"}}},
|
||||||
|
"containers": []any{map[string]any{"name": "job", "resources": map[string]any{"limits": map[string]any{"memory": "1Gi"}}}},
|
||||||
|
}},
|
||||||
|
}}
|
||||||
|
pod, err := NewKubernetesPod(options)
|
||||||
|
require.NoError(t, err)
|
||||||
|
var jobLog, serviceLog bytes.Buffer
|
||||||
|
job := pod.JobContainer(&NewContainerInput{Name: "Job-Name", Image: "node", WorkingDir: workdir, Stdout: &jobLog, Stderr: &jobLog})
|
||||||
|
service := pod.ServiceContainer("My_DB", &NewContainerInput{
|
||||||
|
Image: "postgres",
|
||||||
|
Cmd: []string{"postgres", "-D", "$(PGDATA)"},
|
||||||
|
Env: []string{"POSTGRES_PASSWORD=service-secret"},
|
||||||
|
WorkflowOptions: "--health-cmd pg_isready --health-interval 2s --health-start-period 4s",
|
||||||
|
Stdout: &serviceLog,
|
||||||
|
})
|
||||||
|
|
||||||
|
require.NoError(t, job.Create(nil, nil)(ctx))
|
||||||
|
var manifest podManifest
|
||||||
|
require.NoError(t, json.Unmarshal([]byte(cluster.requests["POST /api/v1/namespaces/ci/pods"][0]), &manifest))
|
||||||
|
assert.Equal(t, "gitea-runner", manifest.Metadata.Labels["app.kubernetes.io/managed-by"])
|
||||||
|
assert.Equal(t, "runner-uuid", manifest.Metadata.Labels[runnerUUIDLabel])
|
||||||
|
assert.True(t, manifest.Spec.AutomountServiceAccountToken)
|
||||||
|
assert.False(t, manifest.Spec.EnableServiceLinks)
|
||||||
|
assert.Equal(t, []string{"my-db"}, manifest.Spec.HostAliases[0].Hostnames)
|
||||||
|
require.Len(t, manifest.Spec.Containers, 2)
|
||||||
|
assert.Equal(t, map[string]any{"limits": map[string]any{"cpu": "2", "memory": "1Gi"}}, manifest.Spec.Containers[0].Resources)
|
||||||
|
assert.Empty(t, manifest.Spec.Containers[0].ImagePullPolicy)
|
||||||
|
assert.Equal(t, "my-db", manifest.Spec.Containers[1].Name)
|
||||||
|
assert.Equal(t, "Always", manifest.Spec.Containers[1].ImagePullPolicy)
|
||||||
|
assert.Equal(t, map[string]any{"limits": map[string]any{"memory": "256Mi"}}, manifest.Spec.Containers[1].Resources)
|
||||||
|
assert.Equal(t, []map[string]any{
|
||||||
|
{"name": "workspace", "emptyDir": map[string]any{"sizeLimit": "1Gi"}},
|
||||||
|
{"name": "act", "emptyDir": map[string]any{}},
|
||||||
|
}, manifest.Spec.Volumes)
|
||||||
|
assert.Equal(t, struct{ PeriodSeconds, TimeoutSeconds, FailureThreshold int }{2, 30, 5}, *manifest.Spec.Containers[1].StartupProbe)
|
||||||
|
assert.NotContains(t, cluster.requests["POST /api/v1/namespaces/ci/pods"][0], "service-secret")
|
||||||
|
assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/secrets"][0], "service-secret")
|
||||||
|
assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/secrets"][0], `"com.gitea.runner.uuid":"runner-uuid"`)
|
||||||
|
assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/pods"][0], `{"name":"POSTGRES_PASSWORD","valueFrom":{"secretKeyRef":{"key":"my-db.0","name":"`+pod.secretName+`"}}}`)
|
||||||
|
assert.Equal(t, []any{"new"}, mergeManifest([]any{"old"}, []any{"new"}, ""))
|
||||||
|
mount := func(path string) any { return map[string]any{"name": "config", "mountPath": path} }
|
||||||
|
assert.Equal(t, []any{mount("/a"), mount("/b")}, mergeManifest([]any{mount("/a")}, []any{mount("/b")}, "volumeMounts"))
|
||||||
|
tmpfs := map[string]any{"name": "workspace", "emptyDir": map[string]any{"medium": "Memory"}}
|
||||||
|
ephemeral := map[string]any{"name": "workspace", "ephemeral": map[string]any{}}
|
||||||
|
assert.Equal(t, []any{ephemeral}, mergeManifest([]any{tmpfs}, []any{ephemeral}, "volumes"))
|
||||||
|
assert.Contains(t, cluster.requests["POST /api/v1/namespaces/ci/pods"][0], `"args":["postgres","-D","$$(PGDATA)"]`)
|
||||||
|
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(workdir, "env"), []byte("#!/bin/sh\necho \"$@\" > \"$0-args\"\nexec /usr/bin/env \"$@\"\n"), 0o755))
|
||||||
|
t.Setenv("PATH", workdir+":"+os.Getenv("PATH"))
|
||||||
|
require.NoError(t, job.Start(false)(ctx))
|
||||||
|
|
||||||
|
stepEnv := map[string]string{"INPUT_WHO-TO-GREET": "it's\nme", "SECRET": "hunter2", "UID": "5"}
|
||||||
|
require.NoError(t, job.Exec([]string{"printenv", "SECRET"}, stepEnv, "", "")(ctx))
|
||||||
|
envArgs, err := os.ReadFile(filepath.Join(workdir, "env-args"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.NotContains(t, string(envArgs), "hunter2")
|
||||||
|
require.NoError(t, job.Exec([]string{"printenv", "INPUT_WHO-TO-GREET"}, stepEnv, "", "")(ctx))
|
||||||
|
require.NoError(t, job.Exec([]string{"printenv", "UID"}, stepEnv, "", "")(ctx))
|
||||||
|
require.NoError(t, job.Exec([]string{"pwd", "-P"}, nil, "", workdir)(ctx))
|
||||||
|
assert.Equal(t, "hunter2\nit's\nme\n5\n"+workdir+"\n", jobLog.String())
|
||||||
|
assert.Equal(t, ExitCodeError(3), job.Exec([]string{"sh", "-c", "echo failing >&2; exit 3"}, nil, "", "")(ctx))
|
||||||
|
assert.Contains(t, jobLog.String(), "failing\n")
|
||||||
|
for _, query := range cluster.requests["exec"] {
|
||||||
|
assert.NotContains(t, query, "hunter2")
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, job.Copy(workdir+"/act/", &FileEntry{Name: "workflow/event.json", Mode: 0o644, Body: `{"ref":"main"}`})(ctx))
|
||||||
|
archive, err := job.GetContainerArchive(ctx, workdir+"/act/workflow/event.json")
|
||||||
|
require.NoError(t, err)
|
||||||
|
reader := tar.NewReader(archive)
|
||||||
|
header, err := reader.Next()
|
||||||
|
require.NoError(t, err)
|
||||||
|
content, err := io.ReadAll(reader)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "event.json", header.Name)
|
||||||
|
assert.JSONEq(t, `{"ref":"main"}`, string(content))
|
||||||
|
require.NoError(t, archive.Close())
|
||||||
|
_, err = job.GetContainerArchive(ctx, workdir+"/missing")
|
||||||
|
require.Error(t, err)
|
||||||
|
|
||||||
|
pathEnv := map[string]string{}
|
||||||
|
require.NoError(t, job.UpdateFromImageEnv(&pathEnv)(ctx))
|
||||||
|
assert.Equal(t, os.Getenv("PATH"), pathEnv["PATH"])
|
||||||
|
|
||||||
|
info, err := service.Inspect(ctx)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, HealthHealthy, info.Health)
|
||||||
|
require.NoError(t, service.DumpLogs(ctx))
|
||||||
|
assert.Equal(t, "log of my-db", serviceLog.String())
|
||||||
|
|
||||||
|
require.NoError(t, job.Remove()(ctx))
|
||||||
|
assert.Len(t, cluster.requests["DELETE /api/v1/namespaces/ci/pods/"+pod.name], 1)
|
||||||
|
assert.Len(t, cluster.requests["DELETE /api/v1/namespaces/ci/secrets/"+pod.secretName], 1)
|
||||||
|
|
||||||
|
require.NoError(t, RemoveOrphanKubernetesResources(ctx, options, time.Now()))
|
||||||
|
assert.Equal(t, []string{runnerUUIDLabel + "=runner-uuid"}, cluster.requests["list"])
|
||||||
|
assert.Len(t, cluster.requests["DELETE /api/v1/namespaces/ci/pods/old"], 1)
|
||||||
|
assert.Len(t, cluster.requests["DELETE /api/v1/namespaces/ci/secrets/old"], 1)
|
||||||
|
assert.Empty(t, cluster.requests["DELETE /api/v1/namespaces/ci/pods/new"])
|
||||||
|
assert.Empty(t, cluster.requests["DELETE /api/v1/namespaces/ci/secrets/new"])
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestKubernetesPodFailsBeforeSteps(t *testing.T) {
|
||||||
|
kubeconfig, _ := fakeKubernetes(t, `{"spec":{"containers":[{}]},"status":{"containerStatuses":[
|
||||||
|
{"name":"job","state":{"waiting":{"reason":"ErrImagePull","message":"no such image"}}}]}}`)
|
||||||
|
template := map[string]any{"spec": map[string]any{"containers": []any{map[string]any{"name": "sidecar"}}}}
|
||||||
|
for _, test := range []struct {
|
||||||
|
services []string
|
||||||
|
options, err string
|
||||||
|
}{
|
||||||
|
{[]string{"db"}, "--hostname db", "only --health-* options are supported"},
|
||||||
|
{[]string{"sidecar"}, "", "named like another container"},
|
||||||
|
{[]string{"my_db", "My-DB"}, "", `service My-DB maps to container "my-db"`},
|
||||||
|
} {
|
||||||
|
pod, err := NewKubernetesPod(KubernetesOptions{Kubeconfig: kubeconfig, Namespace: "ci", PodTemplates: []map[string]any{template}})
|
||||||
|
require.NoError(t, err)
|
||||||
|
job := pod.JobContainer(&NewContainerInput{Image: "node"})
|
||||||
|
for _, service := range test.services {
|
||||||
|
pod.ServiceContainer(service, &NewContainerInput{Image: "postgres", WorkflowOptions: test.options})
|
||||||
|
}
|
||||||
|
require.ErrorContains(t, job.Create(nil, nil)(t.Context()), test.err)
|
||||||
|
}
|
||||||
|
pod, err := NewKubernetesPod(KubernetesOptions{Kubeconfig: kubeconfig, Namespace: "ci"})
|
||||||
|
require.NoError(t, err)
|
||||||
|
job := pod.JobContainer(&NewContainerInput{Image: "missing"})
|
||||||
|
require.NoError(t, job.Create(nil, nil)(t.Context()))
|
||||||
|
require.ErrorContains(t, job.Start(false)(t.Context()), "ErrImagePull: no such image")
|
||||||
|
|
||||||
|
plugin := filepath.Join(t.TempDir(), "kubeconfig")
|
||||||
|
require.NoError(t, os.WriteFile(plugin, []byte(`
|
||||||
|
current-context: eks
|
||||||
|
contexts: [{name: eks, context: {cluster: eks, user: eks}}]
|
||||||
|
clusters: [{name: eks, cluster: {server: "https://eks.example"}}]
|
||||||
|
users: [{name: eks, user: {exec: {command: aws}}}]
|
||||||
|
`), 0o600))
|
||||||
|
_, err = NewKubernetesPod(KubernetesOptions{Kubeconfig: plugin})
|
||||||
|
require.ErrorContains(t, err, "credential plugin")
|
||||||
|
}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
// Copyright 2026 The Gitea Authors. All rights reserved.
|
||||||
|
// Copyright 2020 The nektos/act Authors. All rights reserved.
|
||||||
|
// SPDX-License-Identifier: MIT
|
||||||
|
|
||||||
|
package container
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.com/gitea/runner/act/common"
|
||||||
|
"gitea.com/gitea/runner/act/filecollector"
|
||||||
|
|
||||||
|
"github.com/go-git/go-billy/v5/helper/polyfill"
|
||||||
|
"github.com/go-git/go-billy/v5/osfs"
|
||||||
|
"github.com/go-git/go-git/v5/plumbing/format/gitignore"
|
||||||
|
)
|
||||||
|
|
||||||
|
func writeFilesTar(ctx context.Context, w io.Writer, uid, gid int, files ...*FileEntry) error {
|
||||||
|
tw := tar.NewWriter(w)
|
||||||
|
for _, file := range files {
|
||||||
|
common.Logger(ctx).Debugf("Writing entry to tarball %s len:%d", file.Name, len(file.Body))
|
||||||
|
if err := tw.WriteHeader(&tar.Header{
|
||||||
|
Name: file.Name,
|
||||||
|
Mode: file.Mode,
|
||||||
|
Size: int64(len(file.Body)),
|
||||||
|
Uid: uid,
|
||||||
|
Gid: gid,
|
||||||
|
}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, err := tw.Write([]byte(file.Body)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return tw.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeDirTar prefixes entries with dstPath for extraction at root.
|
||||||
|
func writeDirTar(ctx context.Context, w io.Writer, dstPath, srcPath string, useGitIgnore, skipGitDir bool, uid, gid int) error {
|
||||||
|
logger := common.Logger(ctx)
|
||||||
|
tw := tar.NewWriter(w)
|
||||||
|
srcPrefix := strings.TrimSuffix(filepath.Dir(srcPath), string(filepath.Separator)) + string(filepath.Separator)
|
||||||
|
logger.Debugf("Stripping prefix:%s src:%s", srcPrefix, srcPath)
|
||||||
|
|
||||||
|
var ignorer gitignore.Matcher
|
||||||
|
if useGitIgnore {
|
||||||
|
ps, err := gitignore.ReadPatterns(polyfill.New(osfs.New(srcPath)), nil)
|
||||||
|
if err != nil {
|
||||||
|
logger.Debugf("Error loading .gitignore: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
ignorer = gitignore.NewMatcher(ps)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := filepath.Walk(srcPath, (&filecollector.FileCollector{
|
||||||
|
Ignorer: ignorer,
|
||||||
|
SrcPath: srcPath,
|
||||||
|
SrcPrefix: srcPrefix,
|
||||||
|
SkipGitDir: skipGitDir,
|
||||||
|
Handler: &filecollector.TarCollector{TarWriter: tw, UID: uid, GID: gid, DstDir: dstPath[1:]},
|
||||||
|
}).CollectFiles(ctx, []string{})); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return tw.Close()
|
||||||
|
}
|
||||||
@@ -292,6 +292,9 @@ func execAsDocker(ctx context.Context, step actionStep, actionName, actionDir, b
|
|||||||
logger := common.Logger(ctx)
|
logger := common.Logger(ctx)
|
||||||
rc := step.getRunContext()
|
rc := step.getRunContext()
|
||||||
action := step.getActionModel()
|
action := step.getActionModel()
|
||||||
|
if rc.kubernetes {
|
||||||
|
return errKubernetesDocker
|
||||||
|
}
|
||||||
|
|
||||||
var prepImage common.Executor
|
var prepImage common.Executor
|
||||||
var image string
|
var image string
|
||||||
|
|||||||
@@ -104,6 +104,7 @@ func newCompositeRunContext(ctx context.Context, parent *RunContext, step action
|
|||||||
Parent: parent,
|
Parent: parent,
|
||||||
EventJSON: parent.EventJSON,
|
EventJSON: parent.EventJSON,
|
||||||
platformImage: parent.platformImage,
|
platformImage: parent.platformImage,
|
||||||
|
kubernetes: parent.kubernetes,
|
||||||
jobIndex: parent.jobIndex,
|
jobIndex: parent.jobIndex,
|
||||||
jobTotal: parent.jobTotal,
|
jobTotal: parent.jobTotal,
|
||||||
}
|
}
|
||||||
|
|||||||
+166
-103
@@ -7,6 +7,7 @@ package runner
|
|||||||
import (
|
import (
|
||||||
"archive/tar"
|
"archive/tar"
|
||||||
"bufio"
|
"bufio"
|
||||||
|
"cmp"
|
||||||
"context"
|
"context"
|
||||||
"crypto/rand"
|
"crypto/rand"
|
||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
@@ -22,6 +23,7 @@ import (
|
|||||||
"regexp"
|
"regexp"
|
||||||
"runtime"
|
"runtime"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
@@ -64,6 +66,7 @@ type RunContext struct {
|
|||||||
ExprEval *expressionEvaluator
|
ExprEval *expressionEvaluator
|
||||||
JobContainer container.ExecutionsEnvironment
|
JobContainer container.ExecutionsEnvironment
|
||||||
serviceContainers []*serviceContainer
|
serviceContainers []*serviceContainer
|
||||||
|
kubernetes bool
|
||||||
containerSpec model.ContainerSpec // container:, resolved once with platformImage
|
containerSpec model.ContainerSpec // container:, resolved once with platformImage
|
||||||
containerEnv yaml.Node // container.env of the kept containerSpec, decoded once the job env resolves
|
containerEnv yaml.Node // container.env of the kept containerSpec, decoded once the job env resolves
|
||||||
JobName string
|
JobName string
|
||||||
@@ -417,11 +420,7 @@ func (rc *RunContext) startHostEnvironment() common.Executor {
|
|||||||
AllocatePTY: rc.Config.AllocatePTY,
|
AllocatePTY: rc.Config.AllocatePTY,
|
||||||
}
|
}
|
||||||
rc.cleanUpJobContainer = rc.JobContainer.Remove()
|
rc.cleanUpJobContainer = rc.JobContainer.Remove()
|
||||||
for k, v := range rc.getRunnerContext(ctx) {
|
rc.setRunnerContextEnv(ctx)
|
||||||
if v, ok := v.(string); ok {
|
|
||||||
rc.Env["RUNNER_"+strings.ToUpper(k)] = v
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for _, env := range os.Environ() {
|
for _, env := range os.Environ() {
|
||||||
if k, v, ok := strings.Cut(env, "="); ok {
|
if k, v, ok := strings.Cut(env, "="); ok {
|
||||||
// don't override
|
// don't override
|
||||||
@@ -431,16 +430,70 @@ func (rc *RunContext) startHostEnvironment() common.Executor {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return rc.copyWorkflowFiles()(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (rc *RunContext) setRunnerContextEnv(ctx context.Context) {
|
||||||
|
for k, v := range rc.getRunnerContext(ctx) {
|
||||||
|
if v, ok := v.(string); ok {
|
||||||
|
rc.Env["RUNNER_"+strings.ToUpper(k)] = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var errKubernetesDocker = errors.New("docker actions and docker:// steps are not supported on Kubernetes")
|
||||||
|
|
||||||
|
func (rc *RunContext) startKubernetesPod() common.Executor {
|
||||||
|
return func(ctx context.Context) error {
|
||||||
|
if rc.containerSpec.Options != "" || len(rc.containerSpec.Volumes) > 0 || rc.containerSpec.Credentials != nil {
|
||||||
|
return errors.New("container options, volumes and credentials are not supported on Kubernetes")
|
||||||
|
}
|
||||||
|
var options container.KubernetesOptions
|
||||||
|
if rc.Config.KubernetesPicker != nil {
|
||||||
|
options = rc.Config.KubernetesPicker(rc.runsOnPlatformNames(ctx))
|
||||||
|
}
|
||||||
|
options.MaxLifetime, options.ForcePull = rc.Config.ContainerMaxLifetime, rc.Config.ForcePull
|
||||||
|
pod, err := container.NewKubernetesPod(options)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
services, err := rc.jobServices(ctx, "")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, serviceID := range slices.Sorted(maps0.Keys(services)) {
|
||||||
|
input := services[serviceID]
|
||||||
|
if input.Username != "" || len(input.Binds) > 0 || len(input.Mounts) > 0 {
|
||||||
|
return fmt.Errorf("service %s: volumes and credentials are not supported on Kubernetes", serviceID)
|
||||||
|
}
|
||||||
|
rc.serviceContainers = append(rc.serviceContainers, &serviceContainer{name: serviceID, image: input.Image, container: pod.ServiceContainer(serviceID, input)})
|
||||||
|
}
|
||||||
|
|
||||||
|
logWriter := rc.commandLogWriter(ctx)
|
||||||
|
rc.JobContainer = pod.JobContainer(&container.NewContainerInput{
|
||||||
|
Entrypoint: []string{"tail", "-f", "/dev/null"},
|
||||||
|
WorkingDir: (&container.LinuxContainerEnvironmentExtensions{}).ToContainerPath(rc.Config.Workdir),
|
||||||
|
Image: rc.platformImage,
|
||||||
|
Name: rc.jobContainerName(),
|
||||||
|
Env: []string{"LANG=C.UTF-8"},
|
||||||
|
Stdout: logWriter,
|
||||||
|
Stderr: logWriter,
|
||||||
|
})
|
||||||
|
rc.cleanUpJobContainer = func(ctx context.Context) error {
|
||||||
|
rc.printServiceLogs(ctx)
|
||||||
|
return rc.JobContainer.Remove()(ctx)
|
||||||
|
}
|
||||||
|
defer printStartJobContainerGroup(ctx, rc.platformImage, rc.jobContainerName(), "")()
|
||||||
return common.NewPipelineExecutor(
|
return common.NewPipelineExecutor(
|
||||||
rc.JobContainer.Copy(rc.JobContainer.GetActPath()+"/", &container.FileEntry{
|
rc.JobContainer.Create(nil, nil),
|
||||||
Name: "workflow/event.json",
|
rc.startServiceContainers(),
|
||||||
Mode: 0o644,
|
rc.JobContainer.Start(false),
|
||||||
Body: rc.EventJSON,
|
func(ctx context.Context) error { rc.setRunnerContextEnv(ctx); return nil },
|
||||||
}, &container.FileEntry{
|
rc.captureJobContainerInfo(),
|
||||||
Name: "workflow/envs.txt",
|
rc.copyWorkflowFiles(),
|
||||||
Mode: 0o666,
|
rc.reportUnstartedServices(),
|
||||||
Body: "",
|
rc.waitForServiceContainers(),
|
||||||
}),
|
|
||||||
)(ctx)
|
)(ctx)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -470,7 +523,6 @@ func WithJobVolumeCleanup(ctx context.Context, deferCleanup func(common.Executor
|
|||||||
func (rc *RunContext) startJobContainer() common.Executor {
|
func (rc *RunContext) startJobContainer() common.Executor {
|
||||||
return func(ctx context.Context) error {
|
return func(ctx context.Context) error {
|
||||||
rc.deferVolumeCleanup, _ = ctx.Value(jobVolumeCleanupKey{}).(func(common.Executor))
|
rc.deferVolumeCleanup, _ = ctx.Value(jobVolumeCleanupKey{}).(func(common.Executor))
|
||||||
logger := common.Logger(ctx)
|
|
||||||
image := rc.platformImage
|
image := rc.platformImage
|
||||||
logWriter := rc.commandLogWriter(ctx)
|
logWriter := rc.commandLogWriter(ctx)
|
||||||
|
|
||||||
@@ -495,78 +547,13 @@ func (rc *RunContext) startJobContainer() common.Executor {
|
|||||||
networkName, createAndDeleteNetwork := rc.networkNameForGitea()
|
networkName, createAndDeleteNetwork := rc.networkNameForGitea()
|
||||||
rc.cleanUpJobContainer = rc.cleanupJobResources(networkName, createAndDeleteNetwork, false)
|
rc.cleanUpJobContainer = rc.cleanupJobResources(networkName, createAndDeleteNetwork, false)
|
||||||
|
|
||||||
var services map[string]*model.ContainerSpec
|
services, err := rc.jobServices(ctx, networkName)
|
||||||
if err := model.DecodeEvaluated("job services", rc.Run.Job().RawServices, rc.ExprEval.shared(ctx).EvaluateYamlNode, &services); err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
for _, serviceID := range slices.Sorted(maps0.Keys(services)) {
|
for _, serviceID := range slices.Sorted(maps0.Keys(services)) {
|
||||||
spec := services[serviceID]
|
input := services[serviceID]
|
||||||
if spec == nil {
|
rc.serviceContainers = append(rc.serviceContainers, &serviceContainer{name: serviceID, image: input.Image, container: newContainer(input)})
|
||||||
return fmt.Errorf("service %s has no container definition", serviceID)
|
|
||||||
}
|
|
||||||
spec.Image = strings.TrimPrefix(spec.Image, "docker://")
|
|
||||||
// GitHub compatibility: skip services whose image evaluates to an
|
|
||||||
// empty string, enabling conditional services via expressions
|
|
||||||
if spec.Image == "" {
|
|
||||||
logger.Infof("The service '%s' will not be started because the container definition has an empty image.", serviceID)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
envs := make([]string, 0, len(spec.Env)+len(rc.Config.ProxyEnv))
|
|
||||||
// a service reaches the internet the way the job does; its own env still wins
|
|
||||||
for k, v := range mergeMaps(rc.Config.ProxyEnv, spec.Env) {
|
|
||||||
envs = append(envs, fmt.Sprintf("%s=%s", k, v))
|
|
||||||
}
|
|
||||||
// keep these local: reusing username/password would overwrite the
|
|
||||||
// credentials the job container is pulled with further down
|
|
||||||
serviceUsername, servicePassword, err := registryCredentials(spec.Credentials, "")
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to handle service %s credentials: %w", serviceID, err)
|
|
||||||
}
|
|
||||||
cmd := spec.Cmd
|
|
||||||
if len(cmd) == 0 && spec.Command != "" {
|
|
||||||
if cmd, err = shellquote.Split(spec.Command); err != nil {
|
|
||||||
return fmt.Errorf("failed to parse service %s command: %w", serviceID, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
var entrypoint []string
|
|
||||||
if spec.Entrypoint != "" {
|
|
||||||
entrypoint = []string{spec.Entrypoint}
|
|
||||||
}
|
|
||||||
|
|
||||||
serviceBinds, serviceMounts, _ := splitVolumes(spec.Volumes)
|
|
||||||
|
|
||||||
exposedPorts, portBindings, err := nat.ParsePortSpecs(spec.Ports)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to parse service %s ports: %w", serviceID, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
serviceContainerName := createContainerName(rc.jobContainerName(), serviceID)
|
|
||||||
serviceLog := common.NewLineWriter(rawLogLine(ctx)) // raw, so a service's output cannot set the job's env, outputs or state
|
|
||||||
c := newContainer(&container.NewContainerInput{
|
|
||||||
Name: serviceContainerName,
|
|
||||||
Image: spec.Image,
|
|
||||||
Username: serviceUsername,
|
|
||||||
Password: servicePassword,
|
|
||||||
Entrypoint: entrypoint,
|
|
||||||
Cmd: cmd,
|
|
||||||
Env: envs,
|
|
||||||
Mounts: serviceMounts,
|
|
||||||
Binds: serviceBinds,
|
|
||||||
Stdout: serviceLog,
|
|
||||||
Stderr: serviceLog,
|
|
||||||
Privileged: rc.Config.Privileged,
|
|
||||||
UsernsMode: rc.Config.UsernsMode,
|
|
||||||
Platform: rc.Config.ContainerArchitecture,
|
|
||||||
AutoRemove: false, // so a dead service's log survives, cleanupJobResources removes it
|
|
||||||
WorkflowOptions: spec.Options,
|
|
||||||
NetworkMode: networkName,
|
|
||||||
NetworkAliases: []string{serviceID},
|
|
||||||
ExposedPorts: exposedPorts,
|
|
||||||
PortBindings: portBindings,
|
|
||||||
ValidVolumes: rc.Config.ValidVolumes, // not validVolumes(), a service gets no docker socket
|
|
||||||
AllocatePTY: rc.Config.AllocatePTY,
|
|
||||||
})
|
|
||||||
rc.serviceContainers = append(rc.serviceContainers, &serviceContainer{name: serviceID, image: spec.Image, container: c})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// For Gitea, `jobContainerNetwork` should be the same as `networkName`
|
// For Gitea, `jobContainerNetwork` should be the same as `networkName`
|
||||||
@@ -575,7 +562,7 @@ func (rc *RunContext) startJobContainer() common.Executor {
|
|||||||
ext := container.LinuxContainerEnvironmentExtensions{}
|
ext := container.LinuxContainerEnvironmentExtensions{}
|
||||||
containerInput := &container.NewContainerInput{
|
containerInput := &container.NewContainerInput{
|
||||||
Cmd: nil,
|
Cmd: nil,
|
||||||
Entrypoint: []string{"sleep", fmt.Sprint(rc.Config.ContainerMaxLifetime.Round(time.Second).Seconds())},
|
Entrypoint: []string{"sleep", strconv.FormatInt(int64(rc.Config.ContainerMaxLifetime.Round(time.Second).Seconds()), 10)},
|
||||||
WorkingDir: ext.ToContainerPath(rc.Config.Workdir),
|
WorkingDir: ext.ToContainerPath(rc.Config.Workdir),
|
||||||
Image: image,
|
Image: image,
|
||||||
Username: username,
|
Username: username,
|
||||||
@@ -623,20 +610,86 @@ func (rc *RunContext) startJobContainer() common.Executor {
|
|||||||
rc.JobContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop),
|
rc.JobContainer.Create(rc.Config.ContainerCapAdd, rc.Config.ContainerCapDrop),
|
||||||
rc.JobContainer.Start(false),
|
rc.JobContainer.Start(false),
|
||||||
rc.captureJobContainerInfo(),
|
rc.captureJobContainerInfo(),
|
||||||
rc.JobContainer.Copy(rc.JobContainer.GetActPath()+"/", &container.FileEntry{
|
rc.copyWorkflowFiles(),
|
||||||
Name: "workflow/event.json",
|
|
||||||
Mode: 0o644,
|
|
||||||
Body: rc.EventJSON,
|
|
||||||
}, &container.FileEntry{
|
|
||||||
Name: "workflow/envs.txt",
|
|
||||||
Mode: 0o666,
|
|
||||||
Body: "",
|
|
||||||
}),
|
|
||||||
rc.waitForServiceContainers(),
|
rc.waitForServiceContainers(),
|
||||||
)(ctx)
|
)(ctx)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (rc *RunContext) jobServices(ctx context.Context, networkName string) (map[string]*container.NewContainerInput, error) {
|
||||||
|
var services map[string]*model.ContainerSpec
|
||||||
|
if err := model.DecodeEvaluated("job services", rc.Run.Job().RawServices, rc.ExprEval.shared(ctx).EvaluateYamlNode, &services); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
inputs := map[string]*container.NewContainerInput{}
|
||||||
|
for _, serviceID := range slices.Sorted(maps0.Keys(services)) {
|
||||||
|
spec := services[serviceID]
|
||||||
|
if spec == nil {
|
||||||
|
return nil, fmt.Errorf("service %s has no container definition", serviceID)
|
||||||
|
}
|
||||||
|
spec.Image = strings.TrimPrefix(spec.Image, "docker://")
|
||||||
|
// GitHub compatibility: skip services whose image evaluates to an
|
||||||
|
// empty string, enabling conditional services via expressions
|
||||||
|
if spec.Image == "" {
|
||||||
|
common.Logger(ctx).Infof("The service '%s' will not be started because the container definition has an empty image.", serviceID)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
envs := make([]string, 0, len(spec.Env)+len(rc.Config.ProxyEnv))
|
||||||
|
// a service reaches the internet the way the job does; its own env still wins
|
||||||
|
for k, v := range mergeMaps(rc.Config.ProxyEnv, spec.Env) {
|
||||||
|
envs = append(envs, fmt.Sprintf("%s=%s", k, v))
|
||||||
|
}
|
||||||
|
serviceUsername, servicePassword, err := registryCredentials(spec.Credentials, "")
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to handle service %s credentials: %w", serviceID, err)
|
||||||
|
}
|
||||||
|
cmd := spec.Cmd
|
||||||
|
if len(cmd) == 0 && spec.Command != "" {
|
||||||
|
if cmd, err = shellquote.Split(spec.Command); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to parse service %s command: %w", serviceID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var entrypoint []string
|
||||||
|
if spec.Entrypoint != "" {
|
||||||
|
entrypoint = []string{spec.Entrypoint}
|
||||||
|
}
|
||||||
|
|
||||||
|
serviceBinds, serviceMounts, _ := splitVolumes(spec.Volumes)
|
||||||
|
|
||||||
|
exposedPorts, portBindings, err := nat.ParsePortSpecs(spec.Ports)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to parse service %s ports: %w", serviceID, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
serviceLog := common.NewLineWriter(rawLogLine(ctx)) // raw, so a service's output cannot set the job's env, outputs or state
|
||||||
|
inputs[serviceID] = &container.NewContainerInput{
|
||||||
|
Name: createContainerName(rc.jobContainerName(), serviceID),
|
||||||
|
Image: spec.Image,
|
||||||
|
Username: serviceUsername,
|
||||||
|
Password: servicePassword,
|
||||||
|
Entrypoint: entrypoint,
|
||||||
|
Cmd: cmd,
|
||||||
|
Env: envs,
|
||||||
|
Mounts: serviceMounts,
|
||||||
|
Binds: serviceBinds,
|
||||||
|
Stdout: serviceLog,
|
||||||
|
Stderr: serviceLog,
|
||||||
|
Privileged: rc.Config.Privileged,
|
||||||
|
UsernsMode: rc.Config.UsernsMode,
|
||||||
|
Platform: rc.Config.ContainerArchitecture,
|
||||||
|
AutoRemove: false, // so a dead service's log survives, cleanupJobResources removes it
|
||||||
|
WorkflowOptions: spec.Options,
|
||||||
|
NetworkMode: networkName,
|
||||||
|
NetworkAliases: []string{serviceID},
|
||||||
|
ExposedPorts: exposedPorts,
|
||||||
|
PortBindings: portBindings,
|
||||||
|
ValidVolumes: rc.Config.ValidVolumes, // not validVolumes(), a service gets no docker socket
|
||||||
|
AllocatePTY: rc.Config.AllocatePTY,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return inputs, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (rc *RunContext) commandLogWriter(ctx context.Context) io.Writer {
|
func (rc *RunContext) commandLogWriter(ctx context.Context) io.Writer {
|
||||||
return common.NewLineWriter(rc.commandHandler(ctx), rawLogLine(ctx))
|
return common.NewLineWriter(rc.commandHandler(ctx), rawLogLine(ctx))
|
||||||
}
|
}
|
||||||
@@ -975,6 +1028,13 @@ func (svc *serviceContainer) healthOutputSuffix() string {
|
|||||||
return ": " + svc.info.HealthOutput
|
return ": " + svc.info.HealthOutput
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (rc *RunContext) copyWorkflowFiles() common.Executor {
|
||||||
|
return rc.JobContainer.Copy(rc.JobContainer.GetActPath()+"/",
|
||||||
|
&container.FileEntry{Name: "workflow/event.json", Mode: 0o644, Body: rc.EventJSON},
|
||||||
|
&container.FileEntry{Name: "workflow/envs.txt", Mode: 0o666},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
// captureJobContainerInfo is a convenience: failing to describe the container must not
|
// captureJobContainerInfo is a convenience: failing to describe the container must not
|
||||||
// fail the job.
|
// fail the job.
|
||||||
func (rc *RunContext) captureJobContainerInfo() common.Executor {
|
func (rc *RunContext) captureJobContainerInfo() common.Executor {
|
||||||
@@ -1043,9 +1103,12 @@ func (rc *RunContext) interpolateOutputs() common.Executor {
|
|||||||
func (rc *RunContext) startContainer() common.Executor {
|
func (rc *RunContext) startContainer() common.Executor {
|
||||||
return func(ctx context.Context) error {
|
return func(ctx context.Context) error {
|
||||||
var err error
|
var err error
|
||||||
if rc.IsHostEnv() {
|
switch {
|
||||||
|
case rc.kubernetes:
|
||||||
|
err = rc.startKubernetesPod()(ctx)
|
||||||
|
case rc.IsHostEnv():
|
||||||
err = rc.startHostEnvironment()(ctx)
|
err = rc.startHostEnvironment()(ctx)
|
||||||
} else {
|
default:
|
||||||
err = rc.startJobContainer()(ctx)
|
err = rc.startJobContainer()(ctx)
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1154,7 +1217,7 @@ func (rc *RunContext) Executor() (common.Executor, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (rc *RunContext) runsOnImage(ctx context.Context) (string, error) {
|
func (rc *RunContext) runsOnImage(ctx context.Context) (string, error) {
|
||||||
if rc.Run.Job().RunsOn() == nil {
|
if rc.Run.Job().RunsOn() == nil && rc.containerSpec.Image == "" {
|
||||||
common.Logger(ctx).Errorf("'runs-on' key not defined in %s", rc.String())
|
common.Logger(ctx).Errorf("'runs-on' key not defined in %s", rc.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1201,13 +1264,13 @@ func (rc *RunContext) resolvePlatformImage(ctx context.Context) error {
|
|||||||
if spec.Image != "" || imagelessContainer(withoutEnv) {
|
if spec.Image != "" || imagelessContainer(withoutEnv) {
|
||||||
rc.containerSpec, rc.containerEnv = spec, env
|
rc.containerSpec, rc.containerEnv = spec, env
|
||||||
}
|
}
|
||||||
if rc.containerSpec.Image != "" {
|
|
||||||
rc.platformImage = rc.containerSpec.Image
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
image, err := rc.runsOnImage(ctx)
|
image, err := rc.runsOnImage(ctx)
|
||||||
rc.platformImage = image
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
|
}
|
||||||
|
image, rc.kubernetes = strings.CutPrefix(image, "kubernetes://")
|
||||||
|
rc.platformImage = cmp.Or(rc.containerSpec.Image, image)
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// imagelessContainer reports a container: mapping without an image key, which configures the runs-on image.
|
// imagelessContainer reports a container: mapping without an image key, which configures the runs-on image.
|
||||||
|
|||||||
@@ -1629,10 +1629,13 @@ func createRunsOnRunContext(t *testing.T, runsOn string) *RunContext {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunContextContainerImageCannotSelectHost(t *testing.T) {
|
func TestRunContextContainerImageCannotSelectBackend(t *testing.T) {
|
||||||
rc := createIfTestRunContext(map[string]*model.Job{"job1": createJob(t, "container: -self-hosted", "")})
|
rc := createIfTestRunContext(map[string]*model.Job{"job1": createJob(t, "container: -self-hosted", "")})
|
||||||
require.NoError(t, rc.resolvePlatformImage(t.Context()))
|
require.NoError(t, rc.resolvePlatformImage(t.Context()))
|
||||||
assert.False(t, rc.IsHostEnv())
|
assert.False(t, rc.IsHostEnv())
|
||||||
|
|
||||||
|
rc = createIfTestRunContext(map[string]*model.Job{"job1": createJob(t, "runs-on: ${{ fromJSON('x') }}\ncontainer: node", "")})
|
||||||
|
require.ErrorContains(t, rc.resolvePlatformImage(t.Context()), "runs-on")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunContextImageOS(t *testing.T) {
|
func TestRunContextImageOS(t *testing.T) {
|
||||||
|
|||||||
@@ -75,6 +75,7 @@ type Config struct {
|
|||||||
// AppURL. It is never set for github.com or a GithubMirror, so the token stays on-instance.
|
// AppURL. It is never set for github.com or a GithubMirror, so the token stays on-instance.
|
||||||
DefaultActionInstanceIsSelfHosted bool
|
DefaultActionInstanceIsSelfHosted bool
|
||||||
PlatformPicker func(labels []string) string
|
PlatformPicker func(labels []string) string
|
||||||
|
KubernetesPicker func(labels []string) container.KubernetesOptions
|
||||||
JobLoggerLevel *log.Level // the level of job logger
|
JobLoggerLevel *log.Level // the level of job logger
|
||||||
ValidVolumes []string // only volumes (and bind mounts) in this slice can be mounted on the job container or service containers
|
ValidVolumes []string // only volumes (and bind mounts) in this slice can be mounted on the job container or service containers
|
||||||
SharedToolCache bool // one tool cache for all jobs instead of one per job
|
SharedToolCache bool // one tool cache for all jobs instead of one per job
|
||||||
|
|||||||
@@ -61,11 +61,14 @@ func (sal *stepActionLocal) main() common.Executor {
|
|||||||
treader := tar.NewReader(tars)
|
treader := tar.NewReader(tars)
|
||||||
header, err := treader.Next()
|
header, err := treader.Next()
|
||||||
if errors.Is(err, io.EOF) {
|
if errors.Is(err, io.EOF) {
|
||||||
|
tars.Close()
|
||||||
return nil, nil, os.ErrNotExist
|
return nil, nil, os.ErrNotExist
|
||||||
} else if err != nil {
|
} else if err != nil {
|
||||||
|
tars.Close()
|
||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
if header.FileInfo().Mode()&os.ModeSymlink == os.ModeSymlink {
|
if header.FileInfo().Mode()&os.ModeSymlink == os.ModeSymlink {
|
||||||
|
tars.Close()
|
||||||
spath, err = symlinkJoin(spath, header.Linkname, containerActionPath)
|
spath, err = symlinkJoin(spath, header.Linkname, containerActionPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
|
|||||||
@@ -56,6 +56,9 @@ func (sd *stepDocker) runUsesContainer() common.Executor {
|
|||||||
step := sd.Step
|
step := sd.Step
|
||||||
|
|
||||||
return func(ctx context.Context) error {
|
return func(ctx context.Context) error {
|
||||||
|
if rc.kubernetes {
|
||||||
|
return errKubernetesDocker
|
||||||
|
}
|
||||||
image := strings.TrimPrefix(step.Uses, "docker://")
|
image := strings.TrimPrefix(step.Uses, "docker://")
|
||||||
cmd, err := shellquote.Split(step.With["args"])
|
cmd, err := shellquote.Split(step.With["args"])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# Jobs as Kubernetes pods
|
||||||
|
|
||||||
|
A `kubernetes` label runs each job as a pod, without a Docker daemon:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ubuntu-latest:kubernetes://docker.gitea.com/runner-images:ubuntu-latest
|
||||||
|
```
|
||||||
|
|
||||||
|
The pod has a `job` container from the label's image or the job's `container.image`, plus one container per service.
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
In a cluster, the runner uses its pod's service account and namespace. [`examples/kubernetes/kubernetes-pods.yaml`](../examples/kubernetes/kubernetes-pods.yaml) deploys a runner with these permissions:
|
||||||
|
|
||||||
|
| Resource | Verbs |
|
||||||
|
| --- | --- |
|
||||||
|
| `pods` | `create`, `get`, `list`, `delete` |
|
||||||
|
| `pods/exec` | `create`, `get` |
|
||||||
|
| `pods/log` | `get` |
|
||||||
|
| `secrets` | `create`, `delete` |
|
||||||
|
|
||||||
|
Outside a cluster, it uses the current context of the first file in `$KUBECONFIG`, or of `~/.kube/config`, which must authenticate with a token, token file or client certificate. The cluster must run Kubernetes 1.30 or later, and job images need `sh`, `tar`, `tail` and `env`.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
kubernetes:
|
||||||
|
kubeconfig: "" # instead of the service account
|
||||||
|
namespace: "" # defaults to the service account's, the kubeconfig context's, or default
|
||||||
|
pod_template: # merged under each job pod, a container named job merges into the job container, one named $services into each service container
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: job
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
memory: 4Gi
|
||||||
|
- name: $services
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
memory: 1Gi
|
||||||
|
pod_templates: # merged over pod_template for jobs whose runs-on has the label they are keyed by
|
||||||
|
gpu:
|
||||||
|
spec:
|
||||||
|
nodeSelector:
|
||||||
|
nvidia.com/gpu.present: "true"
|
||||||
|
```
|
||||||
|
|
||||||
|
A job with `runs-on: [linux, gpu]` gets the `linux` and then the `gpu` entry of `pod_templates` merged over `pod_template`, while its image comes from the first of its labels the runner has. A template volume named `workspace` or `act` replaces the `emptyDir` the runner mounts at the workspace or at `/var/run/act`, for example to limit its size. Images are pulled by the cluster's `imagePullPolicy` default, a template's, or always with `container.force_pull`.
|
||||||
|
|
||||||
|
Job pods mount no service account token unless the template sets `automountServiceAccountToken: true`. Job pods and their Secrets are labeled `app.kubernetes.io/managed-by: gitea-runner` and `com.gitea.runner.uuid: <runner UUID>`. The idle cleanup removes those the runner did not get to remove, once they are older than `runner.workdir_cleanup_age`.
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
|
Services share the pod's network, so a step reaches one by its id or `localhost`, and two services cannot listen on the same port. An id is lowercased and has characters other than letters, digits and `-` replaced by `-`, so the service `my_db` is reached as `my-db`. Pass secrets to services through `env`, as `command` is visible in the pod spec. Of `options`, only `--health-*` is supported. They become a startup probe with Docker's defaults, which stops a service that fails its retries, so the job fails as on Docker.
|
||||||
|
|
||||||
|
## Unsupported
|
||||||
|
|
||||||
|
Docker actions, `docker://` steps, and the `options`, `volumes` and `credentials` of the job container fail the job, as do `volumes` and `credentials` of services. Run such workflows on a [Docker in Docker](../examples/kubernetes/README.md) runner. Pull private images with `imagePullSecrets` in `pod_template`.
|
||||||
@@ -31,6 +31,7 @@ func TestCompatibility(t *testing.T) {
|
|||||||
t.Run("cancellation_and_log_streaming", testRunCancellation)
|
t.Run("cancellation_and_log_streaming", testRunCancellation)
|
||||||
t.Run("dispatch", testWorkflowDispatch)
|
t.Run("dispatch", testWorkflowDispatch)
|
||||||
t.Run("ephemeral", testEphemeralRunner)
|
t.Run("ephemeral", testEphemeralRunner)
|
||||||
|
t.Run("kubernetes", testKubernetes)
|
||||||
}
|
}
|
||||||
|
|
||||||
func testPayloads(t *testing.T) {
|
func testPayloads(t *testing.T) {
|
||||||
|
|||||||
+17
-5
@@ -23,6 +23,7 @@ import (
|
|||||||
|
|
||||||
"gitea.com/gitea/runner/act/container"
|
"gitea.com/gitea/runner/act/container"
|
||||||
|
|
||||||
|
"github.com/moby/moby/api/pkg/stdcopy"
|
||||||
apicontainer "github.com/moby/moby/api/types/container"
|
apicontainer "github.com/moby/moby/api/types/container"
|
||||||
"github.com/moby/moby/api/types/network"
|
"github.com/moby/moby/api/types/network"
|
||||||
mobyclient "github.com/moby/moby/client"
|
mobyclient "github.com/moby/moby/client"
|
||||||
@@ -306,9 +307,14 @@ func randomToken(n int) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (f *GiteaFixture) exec(ctx context.Context, cmd []string) (string, error) {
|
func (f *GiteaFixture) exec(ctx context.Context, cmd []string) (string, error) {
|
||||||
created, err := f.cli.ExecCreate(ctx, f.id, mobyclient.ExecCreateOptions{
|
return containerExec(ctx, f.cli, f.id, "git", nil, cmd...) // gitea refuses root
|
||||||
|
}
|
||||||
|
|
||||||
|
func containerExec(ctx context.Context, cli mobyclient.APIClient, id, user string, stdin io.Reader, cmd ...string) (string, error) {
|
||||||
|
created, err := cli.ExecCreate(ctx, id, mobyclient.ExecCreateOptions{
|
||||||
Cmd: cmd,
|
Cmd: cmd,
|
||||||
User: "git", // gitea refuses root
|
User: user,
|
||||||
|
AttachStdin: stdin != nil,
|
||||||
AttachStdout: true,
|
AttachStdout: true,
|
||||||
AttachStderr: true,
|
AttachStderr: true,
|
||||||
})
|
})
|
||||||
@@ -316,18 +322,24 @@ func (f *GiteaFixture) exec(ctx context.Context, cmd []string) (string, error) {
|
|||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
|
|
||||||
attached, err := f.cli.ExecAttach(ctx, created.ID, mobyclient.ExecAttachOptions{})
|
attached, err := cli.ExecAttach(ctx, created.ID, mobyclient.ExecAttachOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
defer attached.Close()
|
defer attached.Close()
|
||||||
|
if stdin != nil {
|
||||||
|
go func() {
|
||||||
|
_, _ = io.Copy(attached.Conn, stdin)
|
||||||
|
_ = attached.CloseWrite()
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
if _, err := io.Copy(&out, attached.Reader); err != nil {
|
if _, err := stdcopy.StdCopy(&out, &out, attached.Reader); err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
|
|
||||||
inspected, err := f.cli.ExecInspect(ctx, created.ID, mobyclient.ExecInspectOptions{})
|
inspected, err := cli.ExecInspect(ctx, created.ID, mobyclient.ExecInspectOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,166 @@
|
|||||||
|
// Copyright 2026 The Gitea Authors. All rights reserved.
|
||||||
|
// SPDX-License-Identifier: MIT
|
||||||
|
|
||||||
|
//go:build e2e
|
||||||
|
|
||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"net/netip"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.com/gitea/runner/internal/pkg/config"
|
||||||
|
|
||||||
|
"github.com/moby/moby/api/pkg/stdcopy"
|
||||||
|
apicontainer "github.com/moby/moby/api/types/container"
|
||||||
|
"github.com/moby/moby/api/types/network"
|
||||||
|
mobyclient "github.com/moby/moby/client"
|
||||||
|
)
|
||||||
|
|
||||||
|
func testKubernetes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
ctx := t.Context()
|
||||||
|
k3sID, kubernetes := startK3s(t)
|
||||||
|
|
||||||
|
images := preloadImages(t, k3sID, os.Getenv("E2E_JOB_IMAGE"), os.Getenv("SERVICE_IMAGE"))
|
||||||
|
|
||||||
|
api, repo := newScenario(t)
|
||||||
|
if err := api.CreateVariable(ctx, repo, "E2E_SERVICE_IMAGE", images[1]); err != nil {
|
||||||
|
t.Fatalf("create service image variable: %v", err)
|
||||||
|
}
|
||||||
|
startRunner(t, repo, "e2e-kubernetes", runnerOptions{image: images[0], kubernetes: kubernetes})
|
||||||
|
pushWorkflow(t, api, repo, "kubernetes.yml")
|
||||||
|
|
||||||
|
wfRun := waitForRun(t, api, repo)
|
||||||
|
requireSuccess(t, api, repo, wfRun.ID)
|
||||||
|
|
||||||
|
if out, err := containerExec(ctx, fixture.cli, k3sID, "", nil,
|
||||||
|
"kubectl", "wait", "--for=delete", "pods,secrets", "--selector=app.kubernetes.io/managed-by=gitea-runner", "--timeout=1m",
|
||||||
|
); err != nil {
|
||||||
|
t.Errorf("job pods or secrets left behind: %v: %s", err, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func startK3s(t *testing.T) (string, *config.Kubernetes) {
|
||||||
|
t.Helper()
|
||||||
|
ctx := t.Context()
|
||||||
|
cli := fixture.cli
|
||||||
|
|
||||||
|
name := fmt.Sprintf("gitea-runner-e2e-k3s-%d", time.Now().UnixNano())
|
||||||
|
apiPort := network.MustParsePort("6443/tcp")
|
||||||
|
server := "https://" + name + ":6443"
|
||||||
|
hostConfig := &apicontainer.HostConfig{Privileged: true}
|
||||||
|
var netConfig *network.NetworkingConfig
|
||||||
|
if fixture.network != "" {
|
||||||
|
netConfig = &network.NetworkingConfig{EndpointsConfig: map[string]*network.EndpointSettings{fixture.network: {}}}
|
||||||
|
} else {
|
||||||
|
loopback := netip.MustParseAddr("127.0.0.1")
|
||||||
|
port, err := freeHostPort(loopback.String())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("find a free host port: %v", err)
|
||||||
|
}
|
||||||
|
server = "https://" + net.JoinHostPort(loopback.String(), strconv.Itoa(port))
|
||||||
|
hostConfig.PortBindings = network.PortMap{apiPort: []network.PortBinding{{HostIP: loopback, HostPort: strconv.Itoa(port)}}}
|
||||||
|
}
|
||||||
|
|
||||||
|
created, err := cli.ContainerCreate(ctx, mobyclient.ContainerCreateOptions{
|
||||||
|
Config: &apicontainer.Config{
|
||||||
|
Image: os.Getenv("E2E_K3S_IMAGE"),
|
||||||
|
Cmd: []string{"server", "--disable=traefik,metrics-server,servicelb,local-storage,coredns", "--tls-san=" + name, "--kubelet-arg=fail-cgroupv1=false"},
|
||||||
|
ExposedPorts: network.PortSet{apiPort: struct{}{}},
|
||||||
|
},
|
||||||
|
HostConfig: hostConfig,
|
||||||
|
NetworkingConfig: netConfig,
|
||||||
|
Name: name,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("create k3s container: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
_, _ = cli.ContainerRemove(context.Background(), created.ID, mobyclient.ContainerRemoveOptions{Force: true, RemoveVolumes: true})
|
||||||
|
})
|
||||||
|
if _, err := cli.ContainerStart(ctx, created.ID, mobyclient.ContainerStartOptions{}); err != nil {
|
||||||
|
t.Fatalf("start k3s container: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
waitCtx, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
for {
|
||||||
|
_, err := containerExec(waitCtx, cli, created.ID, "", nil, "kubectl", "get", "serviceaccount", "default")
|
||||||
|
if err == nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
inspected, inspectErr := cli.ContainerInspect(ctx, created.ID, mobyclient.ContainerInspectOptions{})
|
||||||
|
if inspectErr != nil || !inspected.Container.State.Running || waitCtx.Err() != nil {
|
||||||
|
var logs bytes.Buffer
|
||||||
|
if reader, logsErr := cli.ContainerLogs(ctx, created.ID, mobyclient.ContainerLogsOptions{ShowStdout: true, ShowStderr: true, Tail: "30"}); logsErr == nil {
|
||||||
|
_, _ = stdcopy.StdCopy(&logs, &logs, reader)
|
||||||
|
_ = reader.Close()
|
||||||
|
}
|
||||||
|
t.Fatalf("k3s did not start: %v, state %+v %v, last log lines:\n%s", err, inspected.Container.State, inspectErr, logs.String())
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-waitCtx.Done():
|
||||||
|
case <-time.After(pollInterval):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
kubeconfig, err := containerExec(ctx, cli, created.ID, "", nil, "cat", "/etc/rancher/k3s/k3s.yaml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read kubeconfig: %v", err)
|
||||||
|
}
|
||||||
|
path := filepath.Join(t.TempDir(), "kubeconfig")
|
||||||
|
if err := os.WriteFile(path, []byte(strings.ReplaceAll(kubeconfig, "https://127.0.0.1:6443", server)), 0o600); err != nil {
|
||||||
|
t.Fatalf("write kubeconfig: %v", err)
|
||||||
|
}
|
||||||
|
kubernetes := &config.Kubernetes{Kubeconfig: path, Namespace: "default"}
|
||||||
|
|
||||||
|
if fixture.network != "" {
|
||||||
|
gitea, err := cli.ContainerInspect(ctx, fixture.id, mobyclient.ContainerInspectOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("inspect gitea container: %v", err)
|
||||||
|
}
|
||||||
|
giteaURL, err := url.Parse(fixture.baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse gitea url: %v", err)
|
||||||
|
}
|
||||||
|
kubernetes.PodTemplate = map[string]any{"spec": map[string]any{"hostAliases": []any{map[string]any{
|
||||||
|
"ip": gitea.Container.NetworkSettings.Networks[fixture.network].IPAddress.String(),
|
||||||
|
"hostnames": []any{giteaURL.Hostname()},
|
||||||
|
}}}}
|
||||||
|
}
|
||||||
|
return created.ID, kubernetes
|
||||||
|
}
|
||||||
|
|
||||||
|
func preloadImages(t *testing.T, k3sID string, images ...string) []string {
|
||||||
|
t.Helper()
|
||||||
|
ctx := t.Context()
|
||||||
|
var aliases []string
|
||||||
|
for index, image := range images {
|
||||||
|
alias := fmt.Sprintf("gitea-runner-e2e/preload-%d:%d", index, time.Now().UnixNano())
|
||||||
|
if _, err := fixture.cli.ImageTag(ctx, mobyclient.ImageTagOptions{Source: image, Target: alias}); err != nil {
|
||||||
|
t.Fatalf("tag %s: %v", image, err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _, _ = fixture.cli.ImageRemove(context.Background(), alias, mobyclient.ImageRemoveOptions{}) })
|
||||||
|
aliases = append(aliases, alias)
|
||||||
|
}
|
||||||
|
archive, err := fixture.cli.ImageSave(ctx, aliases)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("save %v: %v", images, err)
|
||||||
|
}
|
||||||
|
defer archive.Close()
|
||||||
|
if out, err := containerExec(ctx, fixture.cli, k3sID, "", archive, "ctr", "--namespace=k8s.io", "images", "import", "-"); err != nil {
|
||||||
|
t.Fatalf("import %v into k3s: %v: %s", images, err, out)
|
||||||
|
}
|
||||||
|
return aliases
|
||||||
|
}
|
||||||
+12
-5
@@ -6,6 +6,7 @@
|
|||||||
package e2e
|
package e2e
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"cmp"
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
@@ -24,10 +25,12 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type runnerOptions struct {
|
type runnerOptions struct {
|
||||||
capacity int
|
capacity int
|
||||||
ephemeral bool
|
ephemeral bool
|
||||||
cacheV2 *bool
|
cacheV2 *bool
|
||||||
cacheHost string
|
cacheHost string
|
||||||
|
image string
|
||||||
|
kubernetes *config.Kubernetes
|
||||||
}
|
}
|
||||||
|
|
||||||
func startRunner(t *testing.T, repo, labelName string, options runnerOptions) *poll.Poller {
|
func startRunner(t *testing.T, repo, labelName string, options runnerOptions) *poll.Poller {
|
||||||
@@ -61,7 +64,11 @@ func startRunner(t *testing.T, repo, labelName string, options runnerOptions) *p
|
|||||||
cfg.Container.Network = fixture.network
|
cfg.Container.Network = fixture.network
|
||||||
}
|
}
|
||||||
|
|
||||||
rawLabel := labelName + ":docker://" + os.Getenv("E2E_JOB_IMAGE")
|
scheme := labels.SchemeDocker
|
||||||
|
if options.kubernetes != nil {
|
||||||
|
scheme, cfg.Kubernetes = labels.SchemeKubernetes, *options.kubernetes
|
||||||
|
}
|
||||||
|
rawLabel := labelName + ":" + scheme + "://" + cmp.Or(options.image, os.Getenv("E2E_JOB_IMAGE"))
|
||||||
label, err := labels.Parse(rawLabel)
|
label, err := labels.Parse(rawLabel)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("parse label %q: %v", rawLabel, err)
|
t.Fatalf("parse label %q: %v", rawLabel, err)
|
||||||
|
|||||||
+20
@@ -0,0 +1,20 @@
|
|||||||
|
name: kubernetes
|
||||||
|
on: push
|
||||||
|
jobs:
|
||||||
|
pod:
|
||||||
|
runs-on: e2e-kubernetes
|
||||||
|
services:
|
||||||
|
web:
|
||||||
|
image: ${{ vars.E2E_SERVICE_IMAGE }}
|
||||||
|
env:
|
||||||
|
NGINX_ENTRYPOINT_QUIET_LOGS: 1
|
||||||
|
options: --health-cmd "wget -q -O /dev/null http://localhost/" --health-interval 1s
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- run: |
|
||||||
|
test -f .gitea/workflows/kubernetes.yml
|
||||||
|
echo "FROM_ENV=exported" >> "$GITHUB_ENV"
|
||||||
|
- run: |
|
||||||
|
test "$FROM_ENV" = exported
|
||||||
|
test -n "$KUBERNETES_SERVICE_HOST" -a -n "${{ runner.arch }}"
|
||||||
|
curl -fsS -o /dev/null http://web/
|
||||||
@@ -1,25 +1,8 @@
|
|||||||
## Kubernetes Docker in Docker Deployment with `gitea-runner`
|
## Kubernetes deployments of `gitea-runner`
|
||||||
|
|
||||||
NOTE: Docker in Docker (dind) requires elevated privileges on Kubernetes. The current way to achieve this is to set the pod `SecurityContext` to `privileged`. Keep in mind that this is a potential security issue that has the potential for a malicious application to break out of the container context.
|
Jobs can run on Kubernetes in two ways:
|
||||||
|
|
||||||
NOTE: `dind-docker.yaml` uses the native sidecar pattern (init container with `restartPolicy: Always`), which requires Kubernetes 1.29+ (or 1.28 with the `SidecarContainers` feature gate).
|
- **Job pods**: each job runs as its own pod, without privileged containers and under the cluster's quotas and policies. Docker actions and `docker://` steps are not supported. Deploy with [`kubernetes-pods.yaml`](kubernetes-pods.yaml) and see [docs/kubernetes.md](../../docs/kubernetes.md).
|
||||||
|
- **Docker in Docker**: jobs run in a Docker daemon next to the runner. Every workflow feature works, but the daemon is privileged, so a job can break out to the node. Deploy with [`dind-docker.yaml`](dind-docker.yaml), [`statefulset-dind.yaml`](statefulset-dind.yaml) for several replicas, [`rootless-docker.yaml`](rootless-docker.yaml) for a rootless daemon, or the [Helm chart](https://gitea.com/gitea/helm-actions). The sidecars in the first two need Kubernetes 1.29 or later.
|
||||||
|
|
||||||
NOTE: A helm chart for `gitea-runner` also exists for easier deployments https://gitea.com/gitea/helm-actions
|
All examples read the registration token from the `runner-secret` Secret and the Gitea URL from `GITEA_INSTANCE_URL`, and keep the registration in `/data`. They give a stopping pod three hours to finish its jobs, so keep `runner.shutdown_timeout` below `terminationGracePeriodSeconds`.
|
||||||
|
|
||||||
Each example persists **two** things, and it is worth knowing which is which:
|
|
||||||
|
|
||||||
- `/data` is the runner's working directory. It holds the `.runner` registration file and, optionally, the config file — so the runner re-attaches to the server instead of registering again.
|
|
||||||
- The Docker daemon's data root holds the images pulled for jobs (`/var/lib/docker` for the dind sidecar, `/home/rootless/.local/share/docker` for `dind-rootless`). It is *not* under `/data`. If you drop this volume, the examples still work, but the image cache is discarded whenever the pod is recreated and every job re-pulls its images.
|
|
||||||
|
|
||||||
- Kubernetes SIGKILLs a pod 30s after SIGTERM by default, long before a job finishes and reports its result, which leaves tasks the server can only reap as zombies. The manifests raise `terminationGracePeriodSeconds` to three hours, matching the systemd example and the `runner.timeout` job ceiling; set `runner.shutdown_timeout` below that so the runner drains jobs within the window rather than being killed mid-cleanup.
|
|
||||||
|
|
||||||
Files in this directory:
|
|
||||||
|
|
||||||
- [`dind-docker.yaml`](dind-docker.yaml)
|
|
||||||
How to create a Deployment and Persistent Volume for Kubernetes to act as a runner. The Docker credentials are re-generated each time the pod connects and does not need to be persisted.
|
|
||||||
|
|
||||||
- [`rootless-docker.yaml`](rootless-docker.yaml)
|
|
||||||
How to create a rootless Deployment and Persistent Volume for Kubernetes to act as a runner. The Docker credentials are re-generated each time the pod connects and does not need to be persisted.
|
|
||||||
|
|
||||||
- [`statefulset-dind.yaml`](statefulset-dind.yaml)
|
|
||||||
StatefulSet variant of the dind example. Each replica gets a stable identity and its own persistent volume via `volumeClaimTemplates`, so the runner keeps its `.runner` registration across restarts and reschedules instead of trying to register again.
|
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
data:
|
||||||
|
token: << base64 encoded registration token >>
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: runner-secret
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: runner
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: Role
|
||||||
|
metadata:
|
||||||
|
name: runner
|
||||||
|
rules:
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods"]
|
||||||
|
verbs: ["create", "get", "list", "delete"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods/exec"]
|
||||||
|
verbs: ["create", "get"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods/log"]
|
||||||
|
verbs: ["get"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["secrets"]
|
||||||
|
verbs: ["create", "delete"]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: RoleBinding
|
||||||
|
metadata:
|
||||||
|
name: runner
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: Role
|
||||||
|
name: runner
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: runner
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: StatefulSet
|
||||||
|
metadata:
|
||||||
|
name: runner
|
||||||
|
spec:
|
||||||
|
serviceName: runner
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: runner
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: runner
|
||||||
|
spec:
|
||||||
|
serviceAccountName: runner
|
||||||
|
terminationGracePeriodSeconds: 10800 # keep above runner.shutdown_timeout, see README
|
||||||
|
containers:
|
||||||
|
- name: runner
|
||||||
|
image: gitea/runner:nightly
|
||||||
|
env:
|
||||||
|
- name: GITEA_INSTANCE_URL
|
||||||
|
value: http://gitea-http.gitea.svc.cluster.local:3000
|
||||||
|
- name: GITEA_RUNNER_REGISTRATION_TOKEN
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: runner-secret
|
||||||
|
key: token
|
||||||
|
- name: GITEA_RUNNER_LABELS
|
||||||
|
value: ubuntu-latest:kubernetes://docker.gitea.com/runner-images:ubuntu-latest
|
||||||
|
volumeMounts:
|
||||||
|
- name: runner-data
|
||||||
|
mountPath: /data
|
||||||
|
volumeClaimTemplates:
|
||||||
|
- metadata:
|
||||||
|
name: runner-data
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 1Gi
|
||||||
@@ -10,6 +10,7 @@ require (
|
|||||||
gitea.dev/actionslib v1.2.1
|
gitea.dev/actionslib v1.2.1
|
||||||
github.com/avast/retry-go/v5 v5.0.0
|
github.com/avast/retry-go/v5 v5.0.0
|
||||||
github.com/bmatcuk/doublestar/v4 v4.10.2
|
github.com/bmatcuk/doublestar/v4 v4.10.2
|
||||||
|
github.com/coder/websocket v1.8.15
|
||||||
github.com/containerd/errdefs v1.0.0
|
github.com/containerd/errdefs v1.0.0
|
||||||
github.com/creack/pty v1.1.24
|
github.com/creack/pty v1.1.24
|
||||||
github.com/distribution/reference v0.6.0
|
github.com/distribution/reference v0.6.0
|
||||||
|
|||||||
@@ -55,6 +55,8 @@ github.com/cevatbarisyilmaz/ara v0.0.4 h1:SGH10hXpBJhhTlObuZzTuFn1rrdmjQImITXnZV
|
|||||||
github.com/cevatbarisyilmaz/ara v0.0.4/go.mod h1:BfFOxnUd6Mj6xmcvRxHN3Sr21Z1T3U2MYkYOmoQe4Ts=
|
github.com/cevatbarisyilmaz/ara v0.0.4/go.mod h1:BfFOxnUd6Mj6xmcvRxHN3Sr21Z1T3U2MYkYOmoQe4Ts=
|
||||||
github.com/cloudflare/circl v1.6.5 h1:O64F26HEqNhznd/hrC5KZXVKYuKM2rx4deZDTc4ihQA=
|
github.com/cloudflare/circl v1.6.5 h1:O64F26HEqNhznd/hrC5KZXVKYuKM2rx4deZDTc4ihQA=
|
||||||
github.com/cloudflare/circl v1.6.5/go.mod h1:h5LNyxAc5nTue9DS5jT+48en2PSDYt3zdGnz5OstK6c=
|
github.com/cloudflare/circl v1.6.5/go.mod h1:h5LNyxAc5nTue9DS5jT+48en2PSDYt3zdGnz5OstK6c=
|
||||||
|
github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNUA=
|
||||||
|
github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg=
|
||||||
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
|
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
|
||||||
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
|
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
|
||||||
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
|
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
|
||||||
|
|||||||
@@ -618,7 +618,7 @@ func loadExecCmd(ctx context.Context) *cobra.Command {
|
|||||||
execCmd.PersistentFlags().BoolVarP(&execArg.noSkipCheckout, "no-skip-checkout", "", false, "Do not skip actions/checkout")
|
execCmd.PersistentFlags().BoolVarP(&execArg.noSkipCheckout, "no-skip-checkout", "", false, "Do not skip actions/checkout")
|
||||||
execCmd.PersistentFlags().BoolVarP(&execArg.debug, "debug", "d", false, "enable debug log")
|
execCmd.PersistentFlags().BoolVarP(&execArg.debug, "debug", "d", false, "enable debug log")
|
||||||
execCmd.PersistentFlags().BoolVarP(&execArg.dryrun, "dryrun", "n", false, "dryrun mode")
|
execCmd.PersistentFlags().BoolVarP(&execArg.dryrun, "dryrun", "n", false, "dryrun mode")
|
||||||
execCmd.PersistentFlags().StringVarP(&execArg.image, "image", "i", config.DefaultImage, "Docker image to use. Use \"-self-hosted\" to run directly on the host.")
|
execCmd.PersistentFlags().StringVarP(&execArg.image, "image", "i", config.DefaultImage, "Docker image to use. Use \"kubernetes://<image>\" to run it as a pod, \"-self-hosted\" to run directly on the host.")
|
||||||
execCmd.PersistentFlags().StringVarP(&execArg.toolCacheMode, "tool-cache-mode", "", config.ToolCacheModeNone, "What to mount at RUNNER_TOOL_CACHE: none, or shared to reuse one tool cache across runs")
|
execCmd.PersistentFlags().StringVarP(&execArg.toolCacheMode, "tool-cache-mode", "", config.ToolCacheModeNone, "What to mount at RUNNER_TOOL_CACHE: none, or shared to reuse one tool cache across runs")
|
||||||
execCmd.PersistentFlags().StringVarP(&execArg.network, "network", "", "", "Specify the network to which the container will connect")
|
execCmd.PersistentFlags().StringVarP(&execArg.network, "network", "", "", "Specify the network to which the container will connect")
|
||||||
execCmd.PersistentFlags().StringVarP(&execArg.githubInstance, "gitea-instance", "", "", "Gitea instance to use.")
|
execCmd.PersistentFlags().StringVarP(&execArg.githubInstance, "gitea-instance", "", "", "Gitea instance to use.")
|
||||||
|
|||||||
@@ -153,6 +153,8 @@ var removeOrphanNetworks = container.RemoveOrphanNetworks
|
|||||||
|
|
||||||
var removeOrphanJobVolumes = container.RemoveOrphanJobVolumes
|
var removeOrphanJobVolumes = container.RemoveOrphanJobVolumes
|
||||||
|
|
||||||
|
var removeOrphanKubernetesResources = container.RemoveOrphanKubernetesResources
|
||||||
|
|
||||||
// OnIdle performs lightweight maintenance during polling idle windows.
|
// OnIdle performs lightweight maintenance during polling idle windows.
|
||||||
// It runs synchronously on the poller goroutine; shouldRunIdleCleanup
|
// It runs synchronously on the poller goroutine; shouldRunIdleCleanup
|
||||||
// throttles invocations to runner.idle_cleanup_interval so the impact on
|
// throttles invocations to runner.idle_cleanup_interval so the impact on
|
||||||
@@ -175,6 +177,13 @@ func (r *Runner) OnIdle(ctx context.Context) {
|
|||||||
r.cleanupStaleDirs(ctx, hostRoot, isHostScratchDir)
|
r.cleanupStaleDirs(ctx, hostRoot, isHostScratchDir)
|
||||||
}
|
}
|
||||||
r.cleanupOrphanDockerResources(ctx)
|
r.cleanupOrphanDockerResources(ctx)
|
||||||
|
if r.uuid != "" && r.usesKubernetes() {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
if err := removeOrphanKubernetesResources(ctx, r.kubernetesOptions(nil), r.now().Add(-r.cfg.Runner.WorkdirCleanupAge)); err != nil {
|
||||||
|
log.Warnf("failed to clean up job pods left behind by earlier jobs: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *Runner) cleanupOrphanDockerResources(ctx context.Context) {
|
func (r *Runner) cleanupOrphanDockerResources(ctx context.Context) {
|
||||||
@@ -402,10 +411,36 @@ func (r *Runner) requiresDocker() bool {
|
|||||||
return r.labels.RequireDocker() || r.cfg.Container.RequireDocker
|
return r.labels.RequireDocker() || r.cfg.Container.RequireDocker
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (r *Runner) usesKubernetes() bool {
|
||||||
|
return slices.ContainsFunc(r.labels, func(label *labels.Label) bool { return label.Schema == labels.SchemeKubernetes })
|
||||||
|
}
|
||||||
|
|
||||||
|
// kubernetesOptions layers the pod templates of the runner labels a job runs on, in runs-on order, over the shared one.
|
||||||
|
func (r *Runner) kubernetesOptions(runsOn []string) container.KubernetesOptions {
|
||||||
|
options := container.KubernetesOptions{
|
||||||
|
Kubeconfig: r.cfg.Kubernetes.Kubeconfig,
|
||||||
|
Namespace: r.cfg.Kubernetes.Namespace,
|
||||||
|
PodTemplates: []map[string]any{r.cfg.Kubernetes.PodTemplate},
|
||||||
|
RunnerUUID: r.uuid,
|
||||||
|
}
|
||||||
|
names := r.labels.Names()
|
||||||
|
for index, name := range runsOn {
|
||||||
|
if template, ok := r.cfg.Kubernetes.PodTemplates[name]; ok && slices.Contains(names, name) && !slices.Contains(runsOn[:index], name) {
|
||||||
|
options.PodTemplates = append(options.PodTemplates, template)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return options
|
||||||
|
}
|
||||||
|
|
||||||
// fallbackPlatform is where a job runs whose runs-on matches no label, as any job without a
|
// fallbackPlatform is where a job runs whose runs-on matches no label, as any job without a
|
||||||
// runs-on does, since Gitea sends those to every runner.
|
// runs-on does, since Gitea sends those to every runner.
|
||||||
func (r *Runner) fallbackPlatform(ctx context.Context) string {
|
func (r *Runner) fallbackPlatform(ctx context.Context) string {
|
||||||
if r.requiresDocker() || dockerReachable(ctx) {
|
switch {
|
||||||
|
case r.requiresDocker():
|
||||||
|
return r.cfg.Runner.DefaultImage
|
||||||
|
case r.usesKubernetes():
|
||||||
|
return labels.SchemeKubernetes + "://" + r.cfg.Runner.DefaultImage
|
||||||
|
case dockerReachable(ctx):
|
||||||
return r.cfg.Runner.DefaultImage
|
return r.cfg.Runner.DefaultImage
|
||||||
}
|
}
|
||||||
return labels.SelfHostedPlatform
|
return labels.SelfHostedPlatform
|
||||||
@@ -449,7 +484,8 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
|
|||||||
}
|
}
|
||||||
return fallbackPlatform()
|
return fallbackPlatform()
|
||||||
}
|
}
|
||||||
cacheURL, cacheContainer := r.cacheForJob(job, platformPicker)
|
platform := platformPicker(job.RunsOn())
|
||||||
|
cacheURL, cacheContainer := r.cacheForJob(job, platform)
|
||||||
if cacheURL != "" {
|
if cacheURL != "" {
|
||||||
envs["ACTIONS_CACHE_URL"] = cacheURL + "/"
|
envs["ACTIONS_CACHE_URL"] = cacheURL + "/"
|
||||||
}
|
}
|
||||||
@@ -458,7 +494,13 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
|
|||||||
// act reaches them by their workflow key.
|
// act reaches them by their workflow key.
|
||||||
var services map[string]yaml.Node
|
var services map[string]yaml.Node
|
||||||
_ = job.RawServices.Decode(&services) // a whole-value expression names no service before evaluation
|
_ = job.RawServices.Decode(&services) // a whole-value expression names no service before evaluation
|
||||||
proxyEnv := JobProxyEnv(envs, cacheURL, slices.Sorted(maps.Keys(services)))
|
serviceNames := slices.Sorted(maps.Keys(services))
|
||||||
|
if strings.HasPrefix(platform, labels.SchemeKubernetes+"://") {
|
||||||
|
for index, name := range serviceNames {
|
||||||
|
serviceNames[index] = container.KubernetesServiceName(name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
proxyEnv := JobProxyEnv(envs, cacheURL, serviceNames)
|
||||||
maps.Copy(envs, proxyEnv)
|
maps.Copy(envs, proxyEnv)
|
||||||
|
|
||||||
if r.capabilities != "" {
|
if r.capabilities != "" {
|
||||||
@@ -585,6 +627,7 @@ func (r *Runner) run(ctx context.Context, task *runnerv1.Task, reporter *report.
|
|||||||
SharedToolCache: r.cfg.Runner.ToolCacheMode == config.ToolCacheModeShared,
|
SharedToolCache: r.cfg.Runner.ToolCacheMode == config.ToolCacheModeShared,
|
||||||
InsecureSkipTLS: r.cfg.Runner.Insecure,
|
InsecureSkipTLS: r.cfg.Runner.Insecure,
|
||||||
RunnerName: r.name,
|
RunnerName: r.name,
|
||||||
|
KubernetesPicker: r.kubernetesOptions,
|
||||||
}
|
}
|
||||||
|
|
||||||
rr, err := runner.New(runnerConfig)
|
rr, err := runner.New(runnerConfig)
|
||||||
@@ -867,10 +910,10 @@ func warnIgnoredCacheSecret(cfg *config.Config) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// cacheForJob returns the cache address for the job, and the container to attach to its network to reach it there.
|
// cacheForJob returns the cache address for the job, and the container to attach to its network to reach it there.
|
||||||
func (r *Runner) cacheForJob(job *model.Job, pickPlatform func([]string) string) (string, string) {
|
func (r *Runner) cacheForJob(job *model.Job, platform string) (string, string) {
|
||||||
jobContainer := job.Container()
|
jobContainer := job.Container()
|
||||||
cacheContainer := ""
|
cacheContainer := ""
|
||||||
if jobContainer != nil && jobContainer.Image != "" || pickPlatform(job.RunsOn()) != labels.SelfHostedPlatform {
|
if !strings.HasPrefix(platform, labels.SchemeKubernetes+"://") && (jobContainer != nil && jobContainer.Image != "" || platform != labels.SelfHostedPlatform) {
|
||||||
cacheContainer = r.isolatedCacheContainer()
|
cacheContainer = r.isolatedCacheContainer()
|
||||||
}
|
}
|
||||||
if cacheContainer == "" {
|
if cacheContainer == "" {
|
||||||
|
|||||||
@@ -11,7 +11,9 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.com/gitea/runner/act/container"
|
||||||
"gitea.com/gitea/runner/internal/pkg/config"
|
"gitea.com/gitea/runner/internal/pkg/config"
|
||||||
|
"gitea.com/gitea/runner/internal/pkg/labels"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
@@ -329,8 +331,20 @@ func TestRunnerOnIdleRemovesOrphanNetworks(t *testing.T) {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
t.Cleanup(func() { removeOrphanJobVolumes = origRemoveOrphanJobVolumes })
|
t.Cleanup(func() { removeOrphanJobVolumes = origRemoveOrphanJobVolumes })
|
||||||
|
var sweptPods []string
|
||||||
|
origRemoveOrphanKubernetesResources := removeOrphanKubernetesResources
|
||||||
|
removeOrphanKubernetesResources = func(ctx context.Context, options container.KubernetesOptions, createdBefore time.Time) error {
|
||||||
|
_, hasDeadline := ctx.Deadline()
|
||||||
|
assert.True(t, hasDeadline)
|
||||||
|
sweptPods = append(sweptPods, options.RunnerUUID)
|
||||||
|
assert.Equal(t, now.Add(-24*time.Hour), createdBefore)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { removeOrphanKubernetesResources = origRemoveOrphanKubernetesResources })
|
||||||
|
kubernetesLabel, err := labels.Parse("k8s:kubernetes://node")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
r := &Runner{uuid: "runner-1", cfg: cfg, now: func() time.Time { return now }}
|
r := &Runner{uuid: "runner-1", cfg: cfg, labels: labels.Labels{kubernetesLabel}, now: func() time.Time { return now }}
|
||||||
r.isolatedCacheContainer = func() string { return "a1b2c3d4e5f6" }
|
r.isolatedCacheContainer = func() string { return "a1b2c3d4e5f6" }
|
||||||
r.OnIdle(context.Background())
|
r.OnIdle(context.Background())
|
||||||
assert.Equal(t, []string{"runner-1"}, swept)
|
assert.Equal(t, []string{"runner-1"}, swept)
|
||||||
@@ -353,4 +367,5 @@ func TestRunnerOnIdleRemovesOrphanNetworks(t *testing.T) {
|
|||||||
hostOnly.OnIdle(context.Background())
|
hostOnly.OnIdle(context.Background())
|
||||||
assert.Equal(t, []string{"runner-1", "runner-2"}, sweptVolumes)
|
assert.Equal(t, []string{"runner-1", "runner-2"}, sweptVolumes)
|
||||||
assert.Equal(t, []string{"a1b2c3d4e5f6", ""}, detached)
|
assert.Equal(t, []string{"a1b2c3d4e5f6", ""}, detached)
|
||||||
|
assert.Equal(t, []string{"runner-1"}, sweptPods)
|
||||||
}
|
}
|
||||||
@@ -245,6 +245,16 @@ func TestRunnerFallbackPlatform(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunnerKubernetesOptionsLayerTheLabelPodTemplatesInRunsOnOrder(t *testing.T) {
|
||||||
|
cfg := &config.Config{Kubernetes: config.Kubernetes{
|
||||||
|
PodTemplate: map[string]any{"base": true},
|
||||||
|
PodTemplates: map[string]map[string]any{"gpu": {"gpu": true}, "large": {"large": true}, "other": {"other": true}},
|
||||||
|
}}
|
||||||
|
r := &Runner{cfg: cfg, labels: labels.Labels{{Name: "linux"}, {Name: "gpu"}, {Name: "large"}}}
|
||||||
|
|
||||||
|
require.Equal(t, []map[string]any{{"base": true}, {"large": true}, {"gpu": true}}, r.kubernetesOptions([]string{"linux", "large", "other", "gpu", "large"}).PodTemplates)
|
||||||
|
}
|
||||||
|
|
||||||
// Proxy variables are assembled per task, because a job's service containers have to be
|
// Proxy variables are assembled per task, because a job's service containers have to be
|
||||||
// reached directly and they are only known once the workflow is parsed.
|
// reached directly and they are only known once the workflow is parsed.
|
||||||
func TestNewRunnerLeavesProxyToTheTask(t *testing.T) {
|
func TestNewRunnerLeavesProxyToTheTask(t *testing.T) {
|
||||||
@@ -350,7 +360,7 @@ func TestNewRunnerCacheServiceV2(t *testing.T) {
|
|||||||
r.isolatedCacheContainer = func() string { return "a1b2c3d4e5f6" }
|
r.isolatedCacheContainer = func() string { return "a1b2c3d4e5f6" }
|
||||||
pickPlatform := func(runsOn []string) string { return map[string]string{"native": labels.SelfHostedPlatform}[runsOn[0]] }
|
pickPlatform := func(runsOn []string) string { return map[string]string{"native": labels.SelfHostedPlatform}[runsOn[0]] }
|
||||||
for job, isolated := range map[string]bool{"native": false, "linux": true, "containerized": true, "empty": false} {
|
for job, isolated := range map[string]bool{"native": false, "linux": true, "containerized": true, "empty": false} {
|
||||||
cacheURL, cacheContainer := r.cacheForJob(workflow.GetJob(job), pickPlatform)
|
cacheURL, cacheContainer := r.cacheForJob(workflow.GetJob(job), pickPlatform(workflow.GetJob(job).RunsOn()))
|
||||||
if isolated {
|
if isolated {
|
||||||
assert.Equal(t, "a1b2c3d4e5f6", cacheContainer, job)
|
assert.Equal(t, "a1b2c3d4e5f6", cacheContainer, job)
|
||||||
assert.Equal(t, strings.Replace(r.cacheHandler.ExternalURL(), "127.0.0.1", cacheContainer, 1), cacheURL, job)
|
assert.Equal(t, strings.Replace(r.cacheHandler.ExternalURL(), "127.0.0.1", cacheContainer, 1), cacheURL, job)
|
||||||
|
|||||||
@@ -53,12 +53,13 @@ runner:
|
|||||||
# scratch directories (left behind when a host cleanup delete stalls) older than
|
# scratch directories (left behind when a host cleanup delete stalls) older than
|
||||||
# this duration. Setting either workdir_cleanup_age or idle_cleanup_interval to 0
|
# this duration. Setting either workdir_cleanup_age or idle_cleanup_interval to 0
|
||||||
# (or any non-positive value) disables stale-directory cleanup entirely, along with
|
# (or any non-positive value) disables stale-directory cleanup entirely, along with
|
||||||
# the docker network and volume cleanup below.
|
# the docker and Kubernetes cleanup below.
|
||||||
#workdir_cleanup_age: 24h
|
#workdir_cleanup_age: 24h
|
||||||
# Cadence for the idle cleanup pass. Besides the directories above, on runners that use
|
# Cadence for the idle cleanup pass. Besides the directories above, on runners that use
|
||||||
# docker it removes the per-job networks and volumes of jobs this runner did not live to
|
# docker it removes the per-job networks and volumes of jobs this runner did not live to
|
||||||
# tear down, which would otherwise hold a subnet of the daemon address pool and the job's
|
# tear down, which would otherwise hold a subnet of the daemon address pool and the job's
|
||||||
# workspace contents until the host is rebuilt.
|
# workspace contents until the host is rebuilt. On runners with kubernetes labels it
|
||||||
|
# removes such jobs' pods and Secrets.
|
||||||
#idle_cleanup_interval: 10m
|
#idle_cleanup_interval: 10m
|
||||||
# The base interval for periodic log flush to the Gitea instance.
|
# The base interval for periodic log flush to the Gitea instance.
|
||||||
# Logs may be sent earlier if the buffer reaches log_report_batch_size
|
# Logs may be sent earlier if the buffer reaches log_report_batch_size
|
||||||
@@ -106,8 +107,7 @@ runner:
|
|||||||
# terminal; tools like `docker build` emit redrawing progress frames into the captured log
|
# terminal; tools like `docker build` emit redrawing progress frames into the captured log
|
||||||
# when a TTY is present.
|
# when a TTY is present.
|
||||||
#allocate_pty: false
|
#allocate_pty: false
|
||||||
# Image for a job whose runs-on matches none of the labels above. A runner without a
|
# Image for a job whose runs-on matches none of the labels above, see the Labels section of the README for where it runs.
|
||||||
# docker daemon runs such a job on the host instead.
|
|
||||||
#default_image: "docker.gitea.com/runner-images:ubuntu-latest"
|
#default_image: "docker.gitea.com/runner-images:ubuntu-latest"
|
||||||
# What to mount at RUNNER_TOOL_CACHE (/opt/hostedtoolcache), where setup actions install tools:
|
# What to mount at RUNNER_TOOL_CACHE (/opt/hostedtoolcache), where setup actions install tools:
|
||||||
# none: nothing. A docker job sees what its image ships there, a host job an empty dir, and
|
# none: nothing. A docker job sees what its image ships there, a host job an empty dir, and
|
||||||
@@ -280,6 +280,14 @@ host:
|
|||||||
# If it's empty, $HOME/.cache/act/ will be used.
|
# If it's empty, $HOME/.cache/act/ will be used.
|
||||||
#workdir_parent:
|
#workdir_parent:
|
||||||
|
|
||||||
|
# Jobs that run as Kubernetes pods, see docs/kubernetes.md.
|
||||||
|
kubernetes:
|
||||||
|
#kubeconfig: ""
|
||||||
|
#namespace: ""
|
||||||
|
#pod_template: {}
|
||||||
|
# Pod templates merged over pod_template, in runs-on order, for jobs whose runs-on has the label they are keyed by.
|
||||||
|
#pod_templates: {}
|
||||||
|
|
||||||
# Optional local task-admission checks. Disabled by default. When enabled, low
|
# Optional local task-admission checks. Disabled by default. When enabled, low
|
||||||
# disk space or a failing script pauses new task fetching; existing jobs continue.
|
# disk space or a failing script pauses new task fetching; existing jobs continue.
|
||||||
# No health checks run while any job is active; the last result is reused until idle.
|
# No health checks run while any job is active; the last result is reused until idle.
|
||||||
|
|||||||
@@ -63,8 +63,8 @@ type Runner struct {
|
|||||||
FetchTimeout time.Duration `yaml:"fetch_timeout"` // FetchTimeout specifies the timeout duration for fetching resources.
|
FetchTimeout time.Duration `yaml:"fetch_timeout"` // FetchTimeout specifies the timeout duration for fetching resources.
|
||||||
FetchInterval time.Duration `yaml:"fetch_interval"` // FetchInterval specifies the interval duration for fetching resources.
|
FetchInterval time.Duration `yaml:"fetch_interval"` // FetchInterval specifies the interval duration for fetching resources.
|
||||||
FetchIntervalMax time.Duration `yaml:"fetch_interval_max"` // FetchIntervalMax specifies the maximum backoff interval when idle.
|
FetchIntervalMax time.Duration `yaml:"fetch_interval_max"` // FetchIntervalMax specifies the maximum backoff interval when idle.
|
||||||
WorkdirCleanupAge time.Duration `yaml:"workdir_cleanup_age"` // WorkdirCleanupAge removes stale bind-workdir task directories, orphaned host-mode scratch dirs and orphaned docker job resources older than this duration during idle cleanup.
|
WorkdirCleanupAge time.Duration `yaml:"workdir_cleanup_age"` // WorkdirCleanupAge removes stale bind-workdir task directories, orphaned host-mode scratch dirs and orphaned docker and Kubernetes job resources older than this duration during idle cleanup.
|
||||||
IdleCleanupInterval time.Duration `yaml:"idle_cleanup_interval"` // IdleCleanupInterval runs the idle cleanup (stale directories and orphaned docker networks and volumes) periodically while the runner is idle. Set to 0 to disable cleanup cadence.
|
IdleCleanupInterval time.Duration `yaml:"idle_cleanup_interval"` // IdleCleanupInterval runs the idle cleanup (stale directories, orphaned docker networks and volumes, orphaned job pods) periodically while the runner is idle. Set to 0 to disable cleanup cadence.
|
||||||
LogReportInterval time.Duration `yaml:"log_report_interval"` // LogReportInterval specifies the base interval for periodic log flush.
|
LogReportInterval time.Duration `yaml:"log_report_interval"` // LogReportInterval specifies the base interval for periodic log flush.
|
||||||
LogReportMaxLatency time.Duration `yaml:"log_report_max_latency"` // LogReportMaxLatency specifies the max time a log row can wait before being sent.
|
LogReportMaxLatency time.Duration `yaml:"log_report_max_latency"` // LogReportMaxLatency specifies the max time a log row can wait before being sent.
|
||||||
LogReportBatchSize int `yaml:"log_report_batch_size"` // LogReportBatchSize triggers immediate log flush when buffer reaches this size.
|
LogReportBatchSize int `yaml:"log_report_batch_size"` // LogReportBatchSize triggers immediate log flush when buffer reaches this size.
|
||||||
@@ -194,6 +194,14 @@ type Host struct {
|
|||||||
WorkdirParent string `yaml:"workdir_parent"` // WorkdirParent specifies the parent directory for the host's working directory.
|
WorkdirParent string `yaml:"workdir_parent"` // WorkdirParent specifies the parent directory for the host's working directory.
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Kubernetes configures the pods jobs run in.
|
||||||
|
type Kubernetes struct {
|
||||||
|
Kubeconfig string `yaml:"kubeconfig"` // Kubeconfig to use instead of the in-cluster service account.
|
||||||
|
Namespace string `yaml:"namespace"` // Namespace of job pods, empty uses the service account's, else the kubeconfig context's, else default.
|
||||||
|
PodTemplate map[string]any `yaml:"pod_template"` // PodTemplate is a pod manifest the runner merges its own onto.
|
||||||
|
PodTemplates map[string]map[string]any `yaml:"pod_templates"` // PodTemplates are merged over PodTemplate, in runs-on order, for jobs whose runs-on has the label they are keyed by.
|
||||||
|
}
|
||||||
|
|
||||||
// Metrics represents the configuration for the Prometheus metrics endpoint.
|
// Metrics represents the configuration for the Prometheus metrics endpoint.
|
||||||
type Metrics struct {
|
type Metrics struct {
|
||||||
Enabled bool `yaml:"enabled"` // Enabled indicates whether the metrics endpoint is exposed.
|
Enabled bool `yaml:"enabled"` // Enabled indicates whether the metrics endpoint is exposed.
|
||||||
@@ -220,6 +228,7 @@ type Config struct {
|
|||||||
Host Host `yaml:"host"` // Host represents the configuration for the host.
|
Host Host `yaml:"host"` // Host represents the configuration for the host.
|
||||||
Metrics Metrics `yaml:"metrics"` // Metrics represents the configuration for the Prometheus metrics endpoint.
|
Metrics Metrics `yaml:"metrics"` // Metrics represents the configuration for the Prometheus metrics endpoint.
|
||||||
HealthCheck HealthCheck `yaml:"health_check"` // HealthCheck controls opt-in local task-admission checks.
|
HealthCheck HealthCheck `yaml:"health_check"` // HealthCheck controls opt-in local task-admission checks.
|
||||||
|
Kubernetes Kubernetes `yaml:"kubernetes"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// LoadDefault returns the default configuration.
|
// LoadDefault returns the default configuration.
|
||||||
|
|||||||
@@ -9,8 +9,9 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
SchemeHost = "host"
|
SchemeHost = "host"
|
||||||
SchemeDocker = "docker"
|
SchemeDocker = "docker"
|
||||||
|
SchemeKubernetes = "kubernetes"
|
||||||
|
|
||||||
// SelfHostedPlatform is the platform marker act treats as "run on the host".
|
// SelfHostedPlatform is the platform marker act treats as "run on the host".
|
||||||
SelfHostedPlatform = "-self-hosted"
|
SelfHostedPlatform = "-self-hosted"
|
||||||
@@ -42,8 +43,9 @@ func Parse(str string) (*Label, error) {
|
|||||||
if len(splits) >= 3 {
|
if len(splits) >= 3 {
|
||||||
label.Arg = splits[2]
|
label.Arg = splits[2]
|
||||||
}
|
}
|
||||||
if label.Schema != SchemeHost && label.Schema != SchemeDocker {
|
kubernetes := label.Schema == SchemeKubernetes && strings.HasPrefix(label.Arg, "//") && len(label.Arg) > 2
|
||||||
// Not a schema we know: the colon belongs to the label name itself.
|
if label.Schema != SchemeHost && label.Schema != SchemeDocker && !kubernetes {
|
||||||
|
// Not a schema we know, or kubernetes without an image: the colon belongs to the label name itself.
|
||||||
return &Label{
|
return &Label{
|
||||||
Name: str,
|
Name: str,
|
||||||
Schema: SchemeHost,
|
Schema: SchemeHost,
|
||||||
@@ -72,10 +74,11 @@ func (l Labels) PickPlatform(runsOn []string) string {
|
|||||||
case SchemeDocker:
|
case SchemeDocker:
|
||||||
// "//" will be ignored
|
// "//" will be ignored
|
||||||
platforms[label.Name] = strings.TrimPrefix(label.Arg, "//")
|
platforms[label.Name] = strings.TrimPrefix(label.Arg, "//")
|
||||||
|
case SchemeKubernetes:
|
||||||
|
platforms[label.Name] = SchemeKubernetes + ":" + label.Arg
|
||||||
case SchemeHost:
|
case SchemeHost:
|
||||||
platforms[label.Name] = SelfHostedPlatform
|
platforms[label.Name] = SelfHostedPlatform
|
||||||
default:
|
default:
|
||||||
// unreachable: Parse only produces host or docker schemas
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -115,6 +115,7 @@ func TestPickPlatform(t *testing.T) {
|
|||||||
ls := mustParse(t,
|
ls := mustParse(t,
|
||||||
"ubuntu:docker://node:18",
|
"ubuntu:docker://node:18",
|
||||||
"self-hosted:host",
|
"self-hosted:host",
|
||||||
|
"k8s:kubernetes://node:24",
|
||||||
)
|
)
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
@@ -123,6 +124,7 @@ func TestPickPlatform(t *testing.T) {
|
|||||||
want string
|
want string
|
||||||
}{
|
}{
|
||||||
{"docker strips leading slashes", []string{"ubuntu"}, "node:18"},
|
{"docker strips leading slashes", []string{"ubuntu"}, "node:18"},
|
||||||
|
{"kubernetes keeps its scheme", []string{"k8s"}, "kubernetes://node:24"},
|
||||||
{"host maps to self-hosted marker", []string{"self-hosted"}, SelfHostedPlatform},
|
{"host maps to self-hosted marker", []string{"self-hosted"}, SelfHostedPlatform},
|
||||||
{"first match wins", []string{"self-hosted", "ubuntu"}, SelfHostedPlatform},
|
{"first match wins", []string{"self-hosted", "ubuntu"}, SelfHostedPlatform},
|
||||||
{"unknown label picks nothing", []string{"windows"}, ""},
|
{"unknown label picks nothing", []string{"windows"}, ""},
|
||||||
|
|||||||
@@ -4,10 +4,12 @@ set -euo pipefail
|
|||||||
: "${E2E_CONCURRENCY:?E2E_CONCURRENCY is required}"
|
: "${E2E_CONCURRENCY:?E2E_CONCURRENCY is required}"
|
||||||
: "${E2E_JOB_IMAGE:?E2E_JOB_IMAGE is required}"
|
: "${E2E_JOB_IMAGE:?E2E_JOB_IMAGE is required}"
|
||||||
: "${E2E_GITEA_IMAGE:?E2E_GITEA_IMAGE is required}"
|
: "${E2E_GITEA_IMAGE:?E2E_GITEA_IMAGE is required}"
|
||||||
|
: "${E2E_K3S_IMAGE:?E2E_K3S_IMAGE is required}"
|
||||||
: "${SERVICE_IMAGE:?SERVICE_IMAGE is required}"
|
: "${SERVICE_IMAGE:?SERVICE_IMAGE is required}"
|
||||||
|
|
||||||
docker pull "$E2E_JOB_IMAGE"
|
docker pull "$E2E_JOB_IMAGE"
|
||||||
docker pull "$SERVICE_IMAGE"
|
docker pull "$SERVICE_IMAGE"
|
||||||
docker pull "$E2E_GITEA_IMAGE"
|
docker pull "$E2E_GITEA_IMAGE"
|
||||||
|
docker pull "$E2E_K3S_IMAGE"
|
||||||
|
|
||||||
exec "${GO:-go}" test -tags e2e -count=1 -parallel "$E2E_CONCURRENCY" -timeout 20m -v ./e2e/...
|
exec "${GO:-go}" test -tags e2e -count=1 -parallel "$E2E_CONCURRENCY" -timeout 20m -v ./e2e/...
|
||||||
Reference in new issue
Block a user