mirror of
https://gitea.com/gitea/docs.git
synced 2026-11-05 01:25:46 +00:00
Simplify the runner docs and fix their outdated facts (#600)
The runner pages had drifted from the runner. They claimed Podman is unsupported, a registration retry limit of 10, a `container.env` proxy override that has no effect, and they used the deprecated `generate-config` command, while the actual instructions were buried under implementation details. The develop and 5.x pages are rewritten from the current runner sources (`main` is `v5.2.0`). They are shorter and aimed at operators, with a new page covering what jobs get. The reference generator now documents the `config` command, and the quickstart pages in every language use the `gitea-runner` binary name. Reviewed-on: https://gitea.com/gitea/docs/pulls/600 Reviewed-by: Lunny Xiao <[email protected]> Co-authored-by: silverwind <[email protected]>
This commit is contained in:
silverwind authored and 
silverwind committed
1 parent
a874918bb7
commit
be6aefc7e8
53 files changed
+1531
-1661
No files matched your search
@@ -1,93 +1,81 @@
|
||||
---
|
||||
sidebar_position: 7
|
||||
sidebar_position: 8
|
||||
---
|
||||
|
||||
# Monitoring and health
|
||||
# Monitoring
|
||||
|
||||
## Prometheus metrics
|
||||
## Metrics
|
||||
|
||||
```yaml
|
||||
metrics:
|
||||
enabled: true
|
||||
addr: "127.0.0.1:9101"
|
||||
readiness_grace: 30s
|
||||
```
|
||||
|
||||
With `metrics.enabled`, the runner serves three endpoints on `metrics.addr`:
|
||||
With metrics enabled, the runner serves these endpoints:
|
||||
|
||||
| Endpoint | Purpose |
|
||||
| --- | --- |
|
||||
| `/metrics` | Prometheus exposition of the runner's own metrics |
|
||||
| `/healthz` | liveness: the process is up |
|
||||
| `/readyz` | task admission: `503` while a [health check](#local-health-checks) reports the machine as unavailable, once polling has been failing for longer than `readiness_grace`, or after the runner has been deleted on the instance |
|
||||
| `/metrics` | Prometheus metrics |
|
||||
| `/healthz` | Liveness, always `200` while the process runs |
|
||||
| `/readyz` | Readiness, `503` when a [health check](#health-checks) fails, the runner cannot reach Gitea for longer than `metrics.readiness_grace` (`30s`), or the runner is shutting down |
|
||||
|
||||
There is **no authentication** on this listener. The default address binds to localhost only; expose it more widely (`":9101"`) only behind a firewall or a scrape-only network.
|
||||
The listener has no authentication. If you expose it beyond localhost, for example with `addr: ":9101"`, protect it with a firewall.
|
||||
|
||||
All series are prefixed with `gitea_runner_`, among them:
|
||||
The runner exports these metrics in addition to the standard Go and process metrics:
|
||||
|
||||
| Metric | Meaning |
|
||||
| --- | --- |
|
||||
| `gitea_runner_info` | always `1`, with the version and runner name as labels |
|
||||
| `gitea_runner_uptime_seconds` | seconds since the daemon started |
|
||||
| `gitea_runner_capacity` / `gitea_runner_job_running` / `gitea_runner_job_capacity_utilization_ratio` | configured capacity, jobs in flight, and their ratio |
|
||||
| `gitea_runner_job_total` | jobs by result (`success`, `failure`, `cancelled`, `skipped`, `unknown`) |
|
||||
| `gitea_runner_job_duration_seconds` | job duration histogram |
|
||||
| `gitea_runner_poll_fetch_total` / `gitea_runner_poll_fetch_duration_seconds` | task fetches by result (`task`, `empty`, `error`) and their latency |
|
||||
| `gitea_runner_poll_backoff_seconds` | last polling backoff interval |
|
||||
| `gitea_runner_report_log_total` / `gitea_runner_report_state_total` | log and state reports by result |
|
||||
| `gitea_runner_report_log_buffer_rows` | log rows buffered but not yet sent |
|
||||
| `gitea_runner_client_errors_total` | RPC errors by method |
|
||||
| `gitea_runner_state` | `1` for the current `state`: `busy`, `idle` or `unavailable` |
|
||||
| `gitea_runner_job_running`, `gitea_runner_capacity` | Running jobs and configured capacity |
|
||||
| `gitea_runner_job_capacity_utilization_ratio` | Running jobs divided by capacity |
|
||||
| `gitea_runner_job_total` | Finished jobs by `status`: `success`, `failure`, `cancelled`, `skipped`, `timeout` or `unknown` |
|
||||
| `gitea_runner_job_duration_seconds` | Job duration histogram |
|
||||
| `gitea_runner_poll_fetch_total` | Job fetches by `result`: `task`, `empty` or `error` |
|
||||
| `gitea_runner_client_errors_total` | Failed requests to Gitea by `method` and `code` |
|
||||
|
||||
Useful things to alert on: `gitea_runner_poll_fetch_total{result="error"}` rising (the instance is unreachable or the runner was deleted), a `capacity_utilization_ratio` pinned at `1` (the pool is too small), and a growing `report_log_buffer_rows` (log delivery is falling behind).
|
||||
Consider alerts for `gitea_runner_state{state="unavailable"}`, a rising `gitea_runner_poll_fetch_total{result="error"}`, and a utilization ratio that stays at `1`, which means jobs may be waiting for a free slot.
|
||||
|
||||
## OpenTelemetry tracing
|
||||
## OpenTelemetry
|
||||
|
||||
Setting `OTEL_EXPORTER_OTLP_ENDPOINT` in the runner's environment sends a trace of every job, with a span per step, to an OTLP collector over `http/protobuf`:
|
||||
Set `OTEL_EXPORTER_OTLP_ENDPOINT` in the runner's environment to send traces and the metrics above to an OpenTelemetry collector. Each job has a trace with a span for each step:
|
||||
|
||||
```sh
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
|
||||
```
|
||||
|
||||
Jobs receive `TRACEPARENT`, so instrumented tools in them join the job's trace. See [docs/telemetry.md](https://gitea.com/gitea/runner/src/branch/main/docs/telemetry.md) for headers, sampling and the security notes.
|
||||
Jobs receive `TRACEPARENT`, so instrumented tools can add spans to the job's trace. See [telemetry.md](https://gitea.com/gitea/runner/src/branch/main/docs/telemetry.md) for all variables and security notes.
|
||||
|
||||
## Local health checks
|
||||
## Health checks
|
||||
|
||||
Health checks let a runner take itself out of rotation when the machine it runs on is not fit for work — most commonly when the disk is full.
|
||||
Health checks stop a runner from accepting new jobs while its machine has a problem, such as a full disk:
|
||||
|
||||
```yaml
|
||||
health_check:
|
||||
enabled: false
|
||||
enabled: true
|
||||
min_free_disk_space_mb: 1024
|
||||
script: ''
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
script: /usr/local/bin/check-runner.sh
|
||||
```
|
||||
|
||||
- while a check fails, the runner stops fetching **new** tasks; jobs already running are unaffected and finish normally;
|
||||
- no check runs while a job is active — the last result is reused until the runner is idle again;
|
||||
- `min_free_disk_space_mb` is measured on the filesystem holding the runner's workspaces, and defaults to 1024 MiB when omitted or zero;
|
||||
- `script` is any executable. A non-zero exit, a timeout, or a failure to start marks the runner unavailable. Its result is cached for `interval` and it is killed after `timeout`;
|
||||
- recovery is automatic and logged (`runner local health recovered, resuming task polling`), and the state is reflected by `/readyz`.
|
||||
- The disk check measures free space on the filesystem containing `host.workdir_parent`, or `container.workdir_parent` when `container.bind_workdir` is enabled.
|
||||
- `script` is optional. The runner becomes unavailable if the script exits with a non-zero status, fails to start, or runs longer than `health_check.timeout` (`10s`). The runner reuses the script's result for `health_check.interval` (`30s`).
|
||||
- Checks only run while the runner is idle. Running jobs are never interrupted.
|
||||
- The runner resumes accepting jobs once the checks pass again.
|
||||
|
||||
## Logs
|
||||
|
||||
The runner logs to stderr; `log.level` controls the verbosity, with `debug` and `trace` adding `file:line` to each line. Under systemd the log ends up in the journal (`journalctl -u gitea-runner`), in Docker in `docker logs`.
|
||||
The runner writes logs to stderr. View them in the systemd journal or with `docker logs` for a runner container. Set `log.level` to control verbosity, from `trace` to `error`.
|
||||
|
||||
The runner log is not the job log: step output is streamed to Gitea and is tuned with the `runner.log_report_*` settings, while things like [post-task script](hooks/post-task-script.md) output only ever appear in the runner log.
|
||||
The runner streams step output to Gitea without including it in its own log. To also keep a local copy of every job log, set a directory:
|
||||
|
||||
```yaml
|
||||
log:
|
||||
job:
|
||||
dir: /var/log/gitea-runner
|
||||
```
|
||||
|
||||
The runner writes each job log to `<start time>-task-<id>.log` in this directory and masks secrets. It keeps the files for `log.job.retention` (`168h`). Only the user running the runner can read them.
|
||||
|
||||
## Reporting a problem
|
||||
|
||||
`gitea-runner bug-report` prints the version, Go version, OS/architecture and CPU count — paste its output into an issue at [gitea/runner](https://gitea.com/gitea/runner/issues):
|
||||
|
||||
```bash
|
||||
gitea-runner bug-report
|
||||
```
|
||||
|
||||
```text
|
||||
Runner version: 5.1.1
|
||||
Go version: go1.26.5
|
||||
OS/Arch: linux/amd64
|
||||
NumCPU: 8
|
||||
```
|
||||
|
||||
To reproduce a workflow locally, without a Gitea instance and without touching the runner's registration, use `gitea-runner exec` — see the [command line reference](reference/cli.md#exec).
|
||||
`gitea-runner bug-report` prints the runner version, Go version, platform and CPU count. Include its output when you open an issue at [gitea/runner](https://gitea.com/gitea/runner/issues).
|
||||
Reference in new issue
Block a user