enhance!: send OTLP metrics under OpenTelemetry semantic conventions, add runner state (#1282)

Sends the OTLP metrics under [OpenTelemetry semantic convention](https://opentelemetry.io/docs/specs/semconv/) names where the runner's data matches the definition, like `cicd.worker.count`, `cicd.system.errors` and the `go.*` and `process.*` runtime metrics, and under `gitea.runner.*` otherwise. OTel dashboards then work without remapping, as they already do for the traces from https://gitea.com/gitea/runner/pulls/1206.

Adds a `gitea_runner_state` gauge (`busy`, `idle`, `unavailable`), and `/readyz` now fails once the runner shuts down.

**Breaking**

1. OTLP metric names, units, types and attribute values change, and the info and start time metrics move to resource attributes.
1. Prometheus `gitea_runner_client_errors_total` gains a `code` label, and `gitea_runner_job_total` counts jobs stopped by `runner.timeout` as `timeout` instead of `failure`.

---------

Co-authored-by: silverwind <[email protected]>
Co-authored-by: silverwind <[email protected]>
Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/runner/pulls/1282
Reviewed-by: silverwind <[email protected]>
Co-authored-by: wmTJc9IK0Q <[email protected]>

internal/app/cmd/daemon.go

@@ -209,6 +209,15 @@ func runDaemon(ctx context.Context, daemArgs *daemonArgs, configFile *string) fu
 		metrics.RunnerCapacity.Set(float64(cfg.Runner.Capacity))
 		metrics.RegisterUptimeFunc(time.Now())
 		metrics.RegisterRunningJobsFunc(runner.RunningCount, cfg.Runner.Capacity)
+		metrics.RegisterStateFunc(func() string {
+			if runner.RunningCount() > 0 {
+				return metrics.LabelStateBusy
+			}
+			if ready, _ := poller.Ready(cfg.Metrics.ReadinessGrace); !ready {
+				return metrics.LabelStateUnavailable
+			}
+			return metrics.LabelStateIdle
+		})
 		if cfg.Metrics.Enabled {
 			metrics.StartServer(ctx, cfg.Metrics.Addr, func() (bool, string) {
 				return poller.Ready(cfg.Metrics.ReadinessGrace)