bircni
a874918bb7
The 28.2.0 bump in #594 only changed the version string of the api documents, so they still listed the 28.1.0 endpoints. They are now regenerated from v28.2.0 (mostly new `423 repoArchivedError` responses). The runner docs stopped at 3.3.0 while the runner is at 5.1.1: - cut the runner `4` and `5` series, `/runner/` now serves 5.1.1 and `/runner/4/` 4.1.0, the `3` series is bumped to 3.5.0 - series specific image tags and versions in the new copies, the `3` series no longer claims the `latest` tag - 4.0 and 5.0 sections in the upgrade guide, and the Git 2.34.1 requirement of 5.0 in the requirements - regenerated every reference page from the newest stable tag of its series Reviewed-on: https://gitea.com/gitea/docs/pulls/599 Reviewed-by: TheFox0x7 <[email protected]>
4.5 KiB
sidebar_position
| sidebar_position |
|---|
| 7 |
Monitoring and health
Prometheus metrics
metrics:
enabled: true
addr: "127.0.0.1:9101"
readiness_grace: 30s
With metrics.enabled, the runner serves three endpoints on metrics.addr:
| Endpoint | Purpose |
|---|---|
/metrics |
Prometheus exposition of the runner's own metrics |
/healthz |
liveness: the process is up |
/readyz |
task admission: 503 while a health check reports the machine as unavailable, once polling has been failing for longer than readiness_grace, or after the runner has been deleted on the instance |
There is no authentication on this listener. The default address binds to localhost only; expose it more widely (":9101") only behind a firewall or a scrape-only network.
All series are prefixed with gitea_runner_, among them:
| Metric | Meaning |
|---|---|
gitea_runner_info |
always 1, with the version and runner name as labels |
gitea_runner_uptime_seconds |
seconds since the daemon started |
gitea_runner_capacity / gitea_runner_job_running / gitea_runner_job_capacity_utilization_ratio |
configured capacity, jobs in flight, and their ratio |
gitea_runner_job_total |
jobs by result (success, failure, cancelled, skipped, unknown) |
gitea_runner_job_duration_seconds |
job duration histogram |
gitea_runner_poll_fetch_total / gitea_runner_poll_fetch_duration_seconds |
task fetches by result (task, empty, error) and their latency |
gitea_runner_poll_backoff_seconds |
last polling backoff interval |
gitea_runner_report_log_total / gitea_runner_report_state_total |
log and state reports by result |
gitea_runner_report_log_buffer_rows |
log rows buffered but not yet sent |
gitea_runner_client_errors_total |
RPC errors by method |
Useful things to alert on: gitea_runner_poll_fetch_total{result="error"} rising (the instance is unreachable or the runner was deleted), a capacity_utilization_ratio pinned at 1 (the pool is too small), and a growing report_log_buffer_rows (log delivery is falling behind).
OpenTelemetry tracing
Setting OTEL_EXPORTER_OTLP_ENDPOINT in the runner's environment sends a trace of every job, with a span per step, to an OTLP collector over http/protobuf:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
Jobs receive TRACEPARENT, so instrumented tools in them join the job's trace. See docs/telemetry.md for headers, sampling and the security notes.
Local health checks
Health checks let a runner take itself out of rotation when the machine it runs on is not fit for work — most commonly when the disk is full.
health_check:
enabled: false
min_free_disk_space_mb: 1024
script: ''
interval: 30s
timeout: 10s
- while a check fails, the runner stops fetching new tasks; jobs already running are unaffected and finish normally;
- no check runs while a job is active — the last result is reused until the runner is idle again;
min_free_disk_space_mbis measured on the filesystem holding the runner's workspaces, and defaults to 1024 MiB when omitted or zero;scriptis any executable. A non-zero exit, a timeout, or a failure to start marks the runner unavailable. Its result is cached forintervaland it is killed aftertimeout;- recovery is automatic and logged (
runner local health recovered, resuming task polling), and the state is reflected by/readyz.
Logs
The runner logs to stderr; log.level controls the verbosity, with debug and trace adding file:line to each line. Under systemd the log ends up in the journal (journalctl -u gitea-runner), in Docker in docker logs.
The runner log is not the job log: step output is streamed to Gitea and is tuned with the runner.log_report_* settings, while things like post-task script output only ever appear in the runner log.
Reporting a problem
gitea-runner bug-report prints the version, Go version, OS/architecture and CPU count — paste its output into an issue at gitea/runner:
gitea-runner bug-report
Runner version: 4.1.0
Go version: go1.26.5
OS/Arch: linux/amd64
NumCPU: 8
To reproduce a workflow locally, without a Gitea instance and without touching the runner's registration, use gitea-runner exec — see the command line reference.