feat(api): render and publish the openapi 3.0 documents (#514)

Closes #37.

Gitea generates an OpenAPI 3.0 document since 1.27 (`templates/swagger/v1_openapi3_json.tmpl`, renamed to `templates/swagger/v1-openapi3.generated.json` on main, served by an instance at `/openapi3.v1.json`), but this site never picked it up and still published Swagger 2.0 only. That is what the issue is about: code generators such as `openapi-python-client` reject the document with "You may be trying to use a Swagger document; this is not supported by this project".

**What this does**

- `update_api_docs.sh` downloads the OpenAPI 3.0 document next to the Swagger 2.0 one, rewrites the same placeholders (they sit in `servers` instead of `basePath`) and writes `static/openapi3-latest.json` and `static/openapi3-<minor>.json`. Versions that do not have one (1.26 and older) are skipped with a message and keep their Swagger 2.0 document.
- The OpenAPI 3.0 document is rendered where it exists (`/api/` and `/api/next/`), the older versions keep rendering Swagger 2.0. Both documents describe the same api and carry the same operation ids, so no url of an operation page changes; the `basePath` normalization is now only applied to the Swagger 2.0 documents, since a full url in `servers` is valid.
- The overview page of each version links both documents for download, which is what the issue actually needs: `https://docs.gitea.com/openapi3-27.json` for the current release, `https://docs.gitea.com/openapi3-latest.json` for gitea main.
- The scheduled workflow now refreshes and proposes both documents of gitea main.

**Verification**

- `pnpm dlx @redocly/cli lint static/openapi3-latest.json`: valid, only content warnings that come from upstream (missing 4xx responses, ambiguous paths).
- `openapi-python-client generate --path static/openapi3-latest.json`, the tool from the issue, generates a client. The only warning left is `POST /markdown/raw`, whose request body is `text/plain`.
- Operation ids of `static/swagger-27.json` and `static/openapi3-27.json` are identical (481 on both sides), so the operation pages keep their urls.
- Built the api product and compared a rendered operation page against the Swagger 2.0 one: same sections, the request body is now marked required where the document says so.

The released Swagger 2.0 documents are kept: they are the only description 1.26 and older ever produced, and some tooling still expects them.

<!-- cloudflare-preview --> Preview: https://pr-514.docs-gitea-com.pages.dev

Reviewed-on: https://gitea.com/gitea/docs/pulls/514
Reviewed-by: bircni <[email protected]>

.gitea/workflows/update_swagger.yaml

@@ -1,4 +1,4 @@
-name: update swagger files
+name: update api spec files
 
 on:
   schedule:
@@ -9,6 +9,8 @@ env:
   # main is protected, so the update is proposed as a pull request from this branch
   BOT_BRANCH: bot/update-swagger-latest
   BASE_BRANCH: main
+  # the documents of the gitea main branch, refreshed by this job
+  LATEST_FILES: static/swagger-latest.json static/openapi3-latest.json
 
 jobs:
   update-swagger:
@@ -20,19 +22,21 @@ jobs:
           ref: main
           # pushing uses DEPLOY_TOKEN, keep the ephemeral job token out of .git/config
           persist-credentials: false
-      - name: regenerate swagger-latest.json
+      - name: regenerate the latest api documents
         run: make update-api-docs-latest
       - name: verify the generated file
         run: |
           set -euo pipefail
-          # placeholders must have been replaced, otherwise upstream changed them again
+          for file in $LATEST_FILES; do
-          if grep -q -e 'GITEA-API-APP' -e '{{' static/swagger-latest.json; then
+            # placeholders must have been replaced, otherwise upstream changed them again
-            echo "static/swagger-latest.json still contains template placeholders"
+            if grep -q -e 'GITEA-API-APP' -e '{{' "$file"; then
-            exit 1
+              echo "$file still contains template placeholders"
-          fi
+              exit 1
-          if command -v python3 >/dev/null 2>&1; then
+            fi
-            python3 -c "import json; json.load(open('static/swagger-latest.json'))"
+            if command -v python3 >/dev/null 2>&1; then
-          fi
+              python3 -c "import json,sys; json.load(open(sys.argv[1]))" "$file"
+            fi
+          done
       - name: push bot branch
         id: bot_branch
         env:
@@ -40,8 +44,8 @@ jobs:
         run: |
           set -euo pipefail
 
-          if git diff --quiet -- static/swagger-latest.json; then
+          if git diff --quiet -- $LATEST_FILES; then
-            echo "swagger-latest.json is already up to date"
+            echo "the api documents are already up to date"
             echo "changed=false" >> "$GITHUB_OUTPUT"
             exit 0
           fi
@@ -54,9 +58,16 @@ jobs:
 
           # skip if an open bot branch already carries exactly this file
           if git fetch --quiet --depth=1 "$remote" "refs/heads/$BOT_BRANCH" 2>/dev/null; then
-            old_blob="$(git rev-parse --quiet --verify "FETCH_HEAD:static/swagger-latest.json" || true)"
+            same=true
-            if [ "$old_blob" = "$(git hash-object static/swagger-latest.json)" ]; then
+            for file in $LATEST_FILES; do
-              echo "$BOT_BRANCH already proposes this swagger-latest.json"
+              old_blob="$(git rev-parse --quiet --verify "FETCH_HEAD:$file" || true)"
+              if [ "$old_blob" != "$(git hash-object "$file")" ]; then
+                same=false
+                break
+              fi
+            done
+            if [ "$same" = true ]; then
+              echo "$BOT_BRANCH already proposes these api documents"
               echo "changed=false" >> "$GITHUB_OUTPUT"
               exit 0
             fi
@@ -65,8 +76,8 @@ jobs:
           git config user.name "Gitea Bot"
           git config user.email "[email protected]"
           git switch --create "$BOT_BRANCH"
-          git add static/swagger-latest.json
+          git add $LATEST_FILES
-          git commit -m "Update swagger-latest.json"
+          git commit -m "Update the latest api documents"
           # force push: the branch is always rebuilt on top of the current main
           git push --force "$remote" "HEAD:refs/heads/$BOT_BRANCH"
           echo "changed=true" >> "$GITHUB_OUTPUT"
@@ -81,8 +92,8 @@ jobs:
           {
             "base": "$BASE_BRANCH",
             "head": "$BOT_BRANCH",
-            "title": "Update swagger-latest.json",
+            "title": "Update the latest api documents",
-            "body": "Automated update of static/swagger-latest.json from the gitea main branch, opened by the \"update swagger files\" scheduled workflow."
+            "body": "Automated update of static/swagger-latest.json and static/openapi3-latest.json from the gitea main branch, opened by the \"update api spec files\" scheduled workflow."
           }
           EOF
 

Makefile

@@ -59,12 +59,13 @@ clean:
 	rm -rf sites/docs/.cache
 	rm -rf sites/docs/node_modules/.astro
 
-# static/swagger-*.json are committed files, use update-api-docs to refresh them
+# static/swagger-*.json and static/openapi3-*.json are committed files, use
+# update-api-docs to refresh them
 .PHONY: update-api-docs
 update-api-docs:
 	./update_api_docs.sh
 
-# only refresh static/swagger-latest.json (used by the update swagger cron job)
+# only refresh the documents of the gitea main branch (used by the cron job)
 .PHONY: update-api-docs-latest
 update-api-docs-latest:
 	./update_api_docs.sh --latest-only

README.md

@@ -8,7 +8,7 @@ The site covers three products, all served from this repository:
 | Product | Content | Versions | Languages |
 | --- | --- | --- | --- |
 | Docs | `docs/`, `versioned_docs/`, `i18n/` | next, 1.27 … 1.22 | English, 简体中文, 繁體中文 |
-| API | `static/swagger-*.json` | next, 1.27 … 1.22 | English |
+| API | `static/swagger-*.json`, `static/openapi3-*.json` | next, 1.27 … 1.22 | English |
 | Runner | `runner-docs/`, `runner-docs_versioned_docs/` | develop, 3 … 0 | English |
 
 The enterprise documentation is built and deployed from its own repository and
@@ -64,16 +64,21 @@ matrix.
 
 ## API docs
 
-The swagger definitions rendered under `/api/` live in
+The definitions rendered under `/api/` live in `static/`:
-`static/swagger-latest.json` (gitea main) and `static/swagger-<minor>.json`
+`swagger-latest.json` (gitea main) and `swagger-<minor>.json` (released
-(released versions).
+versions) hold the swagger 2.0 document every gitea version ships,
+`openapi3-latest.json` and `openapi3-<minor>.json` the openapi 3.0 document
+gitea generates since 1.27. The openapi 3.0 document is what gets rendered
+where it exists, and both are offered for download on the overview page of the
+version, since code generators cannot read the rendered pages (and several of
+them reject swagger 2.0).
 
 ```shell
 make update-api-docs         # refresh latest + every released version
-make update-api-docs-latest  # refresh only static/swagger-latest.json
+make update-api-docs-latest  # refresh only the documents of gitea main
 ```
 
-`static/swagger-latest.json` is refreshed automatically: the `update swagger
+The documents of gitea main are refreshed automatically: the `update api spec
 files` workflow runs every 12 hours and opens a pull request whenever gitea main
 changed. Released versions are updated by hand when a new gitea version is
 documented.

packages/content-loader/src/products.ts

@@ -38,8 +38,17 @@ export interface VersionDef {
   banner?: 'unreleased' | 'unmaintained';
   /** Content directories, relative to the repository root, per language. */
   sources: Partial<Record<LocaleId, string>>;
-  /** OpenAPI document, relative to the repository root (api product only). */
+  /**
+   * Swagger 2.0 document, relative to the repository root (api product only).
+   * Every documented gitea version ships one.
+   */
   schema?: string;
+  /**
+   * OpenAPI 3.0 document, relative to the repository root (api product only).
+   * Gitea generates it since 1.27; it is what gets rendered and what the code
+   * generators that rejected the swagger 2.0 document can read.
+   */
+  openapi3?: string;
   variables?: VersionVariables;
 }
 
@@ -239,6 +248,7 @@ const apiProduct: ProductDef = {
       banner: 'unreleased',
       sources: {},
       schema: 'static/swagger-latest.json',
+      openapi3: 'static/openapi3-latest.json',
     },
     {
       id: '1.27',
@@ -247,6 +257,7 @@ const apiProduct: ProductDef = {
       latest: true,
       sources: {},
       schema: 'static/swagger-27.json',
+      openapi3: 'static/openapi3-27.json',
     },
     ...['1.26', '1.25', '1.24', '1.23', '1.22'].map(
       (id): VersionDef => ({

sites/docs/README.md

@@ -7,8 +7,8 @@ deployment and are linked from the header.
 
 The content directories are the ones the site has always used — `docs/`,
 `versioned_docs/`, `i18n/`, `runner-docs/`, `runner-docs_versioned_docs/` and
-`static/swagger-*.json` — so the release scripts and the translation workflow
+`static/swagger-*.json` (plus `static/openapi3-*.json` since gitea 1.27) — so
-stay unchanged.
+the release scripts and the translation workflow stay unchanged.
 
 ## Running it
 
@@ -40,8 +40,9 @@ pickers, the version banner, the search facets and the api schemas.
 | `i18n/zh-cn/docusaurus-plugin-content-docs/version-1.27/` | `/zh-cn/` |
 | `runner-docs_versioned_docs/version-3/` | `/runner/` |
 | `runner-docs/` | `/runner/develop/` |
-| `static/swagger-27.json` | `/api/` |
+| `static/openapi3-27.json` | `/api/` |
-| `static/swagger-latest.json` | `/api/next/` |
+| `static/openapi3-latest.json` | `/api/next/` |
+| `static/swagger-26.json` | `/api/1.26/` |
 
 The loader reads the markdown in two passes. The first one works out the route
 of every file, the second normalizes and renders it:
@@ -81,12 +82,21 @@ and `/api/1.27/` are redirected onto them in `cloudflare/_redirects`.
 
 ## Api reference
 
-`starlight-openapi` generates a page per operation from the seven swagger
+`starlight-openapi` generates a page per operation from the documents
-documents `update_api_docs.sh` maintains. Two adjustments happen at build time:
+`update_api_docs.sh` maintains: the openapi 3.0 document for the versions that
+have one (gitea >= 1.27), the swagger 2.0 one for the older versions. Both
+describe the same api and use the same operation ids, so the urls of the
+operation pages do not depend on which one is rendered. Three adjustments
+happen at build time:
 
-- the documents set `basePath` to the full `https://gitea.com/api/v1` url, which
+- the swagger 2.0 documents set `basePath` to the full
-  redoc accepted but is not valid swagger 2.0. A normalized copy is written to
+  `https://gitea.com/api/v1` url, which redoc accepted but is not valid swagger
-  `.cache/openapi/` instead of touching the sources.
+  2.0. A normalized copy is written to `.cache/openapi/` instead of touching the
+  sources. The openapi 3.0 documents keep that url in `servers`, where it is
+  valid, and only get the operation ids and the tag descriptions adjusted.
+- both documents of a version are linked for download from its overview page,
+  see `specDownloads()` in `src/config/api.ts` and
+  `src/components/ApiOverviewTags.astro`.
 - the plugin slugifies the base path, so `/api/1.26/` is generated as
   `/api/126/`. `src/integrations/postbuild.ts` renames the directories and
   rewrites the links once the build is done.

sites/docs/src/components/ApiOverviewTags.astro

@@ -1,5 +1,6 @@
 ---
 import { Card, CardGrid } from '@astrojs/starlight/components';
+import { specDownloads } from '../config/api';
 
 /**
  * Replaces the "Operations" section of the api overview, which lists every
@@ -9,6 +10,9 @@ import { Card, CardGrid } from '@astrojs/starlight/components';
  * Substituted for `starlight-openapi/components/overview/OverviewNavigationLinks.astro`
  * by the `gitea-openapi-overview` vite plugin in `astro.config.mjs`, so the
  * props are the ones that component receives.
+ *
+ * The documents of the version are offered for download above the cards, the
+ * rendered pages are of no use to a code generator.
  */
 interface NavigationGroup {
   label: string;
@@ -20,6 +24,9 @@ interface NavigationGroup {
 const { groups } = Astro.props as { groups: NavigationGroup[] };
 const visible = groups.filter((group) => group.links.length > 0);
 
+const meta = Astro.locals.starlightRoute.entry.data.gitea;
+const downloads = meta ? specDownloads(meta.version) : [];
+
 /**
  * Page of a tag, `/api/operations/tags/repository/`. Derived from the first
  * operation link of the group, which is `<base>/operations/<operation>/`, so it
@@ -43,6 +50,21 @@ function count(group: NavigationGroup): string {
 }
 ---
 
+{
+  downloads.length > 0 && (
+    <p class="gitea-api-downloads">
+      Download the specification:
+      {downloads.map((download, index) => (
+        <>
+          {index > 0 && ', '}
+          <a href={download.href} download>
+            {download.label}
+          </a>
+        </>
+      ))}
+    </p>
+  )
+}
 {
   visible.length > 0 && (
     <div class="gitea-api-overview">
@@ -73,4 +95,12 @@ function count(group: NavigationGroup): string {
     font-size: var(--sl-text-sm);
     color: var(--sl-color-gray-3);
   }
+
+  .gitea-api-downloads {
+    margin-top: 1.5rem;
+  }
+
+  .gitea-api-downloads a {
+    margin-inline-start: 0.35rem;
+  }
 </style>

sites/docs/src/config/api.ts

@@ -28,6 +28,7 @@ interface Operation {
 }
 
 interface SwaggerDocument {
+  openapi?: string;
   basePath?: string;
   host?: string;
   schemes?: string[];
@@ -75,6 +76,9 @@ function hyphenateOperationIds(document: SwaggerDocument): void {
  * `https://gitea.com/api/v1` url, which redoc accepted but is not valid swagger
  * 2.0: the host belongs into `host` and the scheme into `schemes`. Normalize a
  * copy instead of touching the sources, which the docusaurus site still reads.
+ *
+ * The openapi 3.0 documents keep the url in `servers`, where a full url is
+ * valid, so only the operation ids and the tags are touched for those.
  */
 function normalizeSchema(source: string, name: string): string {
   const document = JSON.parse(readFileSync(source, 'utf-8')) as SwaggerDocument;
@@ -129,7 +133,12 @@ export const apiSchemas = getProduct('api')
   .map((version) => ({
     base: ['api', version.path].filter(Boolean).join('/'),
     label: `API ${version.label}`,
-    schema: normalizeSchema(path.join(repoRoot, version.schema!), `${version.id}.json`),
+    // gitea generates an openapi 3.0 document since 1.27, render it where it
+    // exists: it is the same api, described in the format the tooling expects
+    schema: normalizeSchema(
+      path.join(repoRoot, version.openapi3 ?? version.schema!),
+      `${version.id}.json`,
+    ),
     // the schema group is lifted to the top level of the sidebar in
     // src/middleware/api.ts, so the tag groups below it start collapsed and
     // starlight opens the one holding the current operation
@@ -140,3 +149,28 @@ export const apiSchemas = getProduct('api')
       operations: { badges: true },
     },
   }));
+
+export interface SpecDownload {
+  label: string;
+  href: string;
+}
+
+/**
+ * The documents of a version, as they are served from `static/`. Shown on the
+ * overview page of the version: the rendered reference is not something a code
+ * generator can read, the documents are.
+ */
+export function specDownloads(versionId: string): SpecDownload[] {
+  const version = getProduct('api').versions.find((candidate) => candidate.id === versionId);
+  if (!version) return [];
+
+  const downloads: SpecDownload[] = [];
+  if (version.openapi3) downloads.push({ label: 'OpenAPI 3.0', href: staticHref(version.openapi3) });
+  if (version.schema) downloads.push({ label: 'Swagger 2.0', href: staticHref(version.schema) });
+  return downloads;
+}
+
+/** `static/openapi3-27.json` is published as `/openapi3-27.json`. */
+function staticHref(file: string): string {
+  return `/${file.replace(/^static\//, '')}`;
+}

update_api_docs.sh

@@ -1,11 +1,14 @@
 #!/bin/bash
 #
-# Regenerates the swagger definitions served by this site.
+# Regenerates the api definitions served by this site: the swagger 2.0 document
+# every gitea version ships, and the openapi 3.0 document gitea generates since
+# 1.27.
 #
 # Usage: ./update_api_docs.sh [--latest-only] [sed -i suffix] [extra sed args...]
 #
-#   --latest-only   only refresh static/swagger-latest.json from gitea main,
+#   --latest-only   only refresh static/swagger-latest.json and
-#                   leaving the released static/swagger-<minor>.json untouched.
+#                   static/openapi3-latest.json from gitea main, leaving the
+#                   released documents untouched.
 
 set -euo pipefail
 
@@ -42,6 +45,13 @@ SWAGGER_PATHS=(
   'templates/swagger/v1_json.tmpl'
 )
 
+# the openapi 3.0 document exists since gitea 1.27, with the same split between
+# a pre-generated json and a go template
+OPENAPI3_PATHS=(
+  'templates/swagger/v1-openapi3.generated.json'
+  'templates/swagger/v1_openapi3_json.tmpl'
+)
+
 # download_swagger <git ref> <output file>
 download_swagger() {
   local ref="$1" output="$2" path
@@ -55,6 +65,18 @@ download_swagger() {
   return 1
 }
 
+# download_openapi3 <git ref> <output file>, returns 1 for a version without one
+download_openapi3() {
+  local ref="$1" output="$2" path
+  for path in "${OPENAPI3_PATHS[@]}"; do
+    if curl --silent --fail --location --output "$output" \
+        "https://raw.githubusercontent.com/go-gitea/gitea/${ref}/${path}"; then
+      return 0
+    fi
+  done
+  return 1
+}
+
 # rewrite_swagger <file> <version to display>
 rewrite_swagger() {
   local file="$1" version="$2"
@@ -69,17 +91,43 @@ rewrite_swagger() {
   inplace_sed "s#\"basePath\": \"{{AppSubUrl | JSEscape}}/api/v1\"#\"basePath\": \"https://gitea.com/api/v1\"#" "$file"
 }
 
-download_swagger 'refs/heads/main' v1_json.tmpl
+# rewrite_openapi3 <file> <version to display>, the placeholders sit in `servers`
-rewrite_swagger v1_json.tmpl 'dev'
+# instead of `basePath`
-mv v1_json.tmpl static/swagger-latest.json
+rewrite_openapi3() {
+  local file="$1" version="$2"
+  # gitea >= 1.28
+  inplace_sed "s|\"version\": \"0.0.0+GITEA-API-APP-VERSION\"|\"version\": \"${version}\"|" "$file"
+  inplace_sed 's|"url": "/GITEA-API-APP-SUBURL/api/v1"|"url": "https://gitea.com/api/v1"|' "$file"
+  # gitea 1.27
+  inplace_sed "s|\"version\": \"{{.SwaggerAppVer}}\"|\"version\": \"${version}\"|" "$file"
+  inplace_sed 's|"url": "{{.SwaggerAppSubUrl}}/api/v1"|"url": "https://gitea.com/api/v1"|' "$file"
+}
+
+# update_version <git ref> <version to display> <suffix of the static files>
+update_version() {
+  local ref="$1" version="$2" suffix="$3"
+
+  download_swagger "$ref" v1_json.tmpl
+  rewrite_swagger v1_json.tmpl "$version"
+  mv v1_json.tmpl "static/swagger-${suffix}.json"
+
+  if download_openapi3 "$ref" v1_openapi3_json.tmpl; then
+    rewrite_openapi3 v1_openapi3_json.tmpl "$version"
+    mv v1_openapi3_json.tmpl "static/openapi3-${suffix}.json"
+  else
+    # gitea < 1.27 only has the swagger 2.0 document
+    rm -f v1_openapi3_json.tmpl
+    echo "no openapi 3.0 document in ${ref}, keeping the swagger 2.0 one only" >&2
+  fi
+}
+
+update_version 'refs/heads/main' 'dev' 'latest'
 
 if [ "$LATEST_ONLY" -eq 1 ]; then
   exit 0
 fi
 
 for ver in '1.27.2' '1.26.4' '1.25.5' '1.24.7' '1.23.8' '1.22.6'; do
-  download_swagger "refs/tags/v${ver}" v1_json.tmpl
-  rewrite_swagger v1_json.tmpl "${ver}"
   minor=$(echo "$ver" | cut -d '.' -f 2)
-  mv v1_json.tmpl "static/swagger-$minor.json"
+  update_version "refs/tags/v${ver}" "${ver}" "${minor}"
 done