Rebuild the site with Astro and Starlight (#496)

Replace [#307](https://gitea.com/gitea/docs/issues/307)
Closes [#237](https://gitea.com/gitea/docs/issues/237)

## Summary

Rebuilds docs.gitea.com with [Astro](https://astro.build) and [Starlight](https://starlight.astro.build), replacing Docusaurus. Every
published url keeps working and no content file was moved: the site is a new rendering layer over the existing `docs/`, `versioned_docs/`, `i18n/`, `runner-docs/` and `static/swagger-*.json` trees.

For the first time the API reference is part of the site rather than a Redoc
bundle: all seven swagger documents are rendered into real pages, one per
operation, so they are linkable, crawlable and searchable.

## Why

- the Docusaurus build needs an 8 GB heap and about 2 minutes for 2478 pages;
  the Astro build produces 5751 pages, API reference included, in about 100
  seconds
- the API reference was a single client rendered Redoc page per version, absent
  from the site search and from search engines
- sidebars, version lists and language lists were configured in three different
  places and drifted apart

## What is in here

- **`packages/content-loader`** — the product × version × language matrix
  (`products.ts`) and an Astro content loader that reads the existing markdown
  trees directly. The matrix is the single source of truth: content loading,
  sidebars, the version and language pickers, the version banner, the search
  facets and the API schemas are all derived from it.
- **Content compatibility** — frontmatter `slug`, `sidebar_position` and
  `sidebar_label` keep working, `_category_.json` still drives sidebar labels,
  order and the generated category pages, `@version@` style release variables are
  still substituted, `:::note` admonitions become Starlight asides and the 2778
  relative `*.md` links are rewritten to urls while the site is built.
- **API reference** — `starlight-openapi` renders the seven swagger documents.
  Operation urls are hyphenated (`/api/operations/list-admin-workflow-jobs/`),
  the sidebar shows the operations of the version being read with a coloured
  HTTP method badge, and the overview page links to one page per tag instead of
  repeating all 484 operations.
- **Navigation** — a version picker that follows the reader to the same page in
  another version, a language picker that only offers the languages the current
  product is published in, and product links for Docs, API, Runner and
  Enterprise.
- **Theme** — the light and dark palettes of about.gitea.com, mapped onto the
  Starlight variables.
- **Search** — Pagefind by default; Algolia DocSearch takes over when
  `PUBLIC_DOCSEARCH_APP_ID` and `PUBLIC_DOCSEARCH_API_KEY` are set. Every page
  carries `docsearch:product`, `docsearch:version` and `docsearch:language` meta
  tags, so a search stays inside what is being read.
  `cloudflare/docsearch-crawler.json` holds the crawler configuration.

## Bugs fixed along the way

- `GET /user/applications/oauth2` and `GET /user/applications/oauth2/{id}` have
  operation ids that only differ in case, so they collapsed onto the same url
  and one of the two pages was silently dropped. They are now
  `user-get-oauth2-application` and `user-get-oauth2-application-by-id`.
- The API overview and every tag page were all titled "Overview".

## Url compatibility

The routes of both builds were compared page by page during the migration with
`scripts/url-diff.mjs`:

```
identical: 2368, missing: 0, accepted: 110, added: 3292
```

The 110 accepted ones are all redirected in `cloudflare/_redirects`: the
localized copies of the English-only API and Runner docs, the Docusaurus search
page, and `/1.27/`, `/runner/3/`, `/api/1.27/` which are aliases of the versions
served at the product root. The added ones are the API operation and tag pages
plus the routes the Starlight language fallback serves in English when a
translation is missing — Docusaurus answered those with a 404.

## Workflows

- `checks` builds the site and type checks it on every pull request
- `Build and Publish Docs site` publishes `sites/docs/dist` to S3/CloudFront and
  to Cloudflare Pages, and copies `cloudflare/_headers` and
  `cloudflare/_redirects` into the deployment
- `update swagger files` and `update runner reference` are unchanged, they only
  touch content
- `make cut-version PRODUCT=docs VERSION=1.28` replaces
  `docusaurus docs:version`

## Removed

`docusaurus.config.js`, the swizzled theme under `src/`, and the Docusaurus UI
translations (`i18n/*/code.json`, `i18n/*/docusaurus-theme-classic/`). The
strings those carried — the product names, the footer column titles and the
outdated translation notice — were ported to
`sites/docs/src/config/strings.ts`; Starlight ships the rest of its interface in
both Chinese locales. The documentation content itself is untouched.

## Follow ups

- nine relative links are broken in the sources and reported by every build, the
  same ones Docusaurus warned about; `GITEA_DOCS_STRICT_LINKS=true` turns them
  into an error once they are fixed
- `cloudflare/worker.js` has to be deployed for `/enterprise/` to keep resolving
- the Algolia index has to be created and crawled before the search credentials
  are set

## Testing

```shell
make install
make serve-fast   # english, the version served at the root
make serve        # the whole matrix
make build        # 5751 pages
make check        # 0 errors
make serve-built  # build and serve, the only way to try the search locally
```

## Screenshots

<img width="1371" alt="image.png" src="attachments/69acdd77-cc89-4635-a8bd-1163a34afa86">

<img width="1810" alt="image.png" src="attachments/76212416-4825-4b4d-bf59-3e549900c96f">

<img width="1789" alt="image.png" src="attachments/7be84cda-5abe-48bd-8839-7ae1ee7806e7">

---------

Co-authored-by: bircni <[email protected]>
Reviewed-on: https://gitea.com/gitea/docs/pulls/496
Reviewed-by: bircni <[email protected]>

packages/content-loader/src/segments.ts

@@ -0,0 +1,171 @@
+import {
+  defaultLocale,
+  products,
+  routePrefix,
+  type LocaleId,
+  type ProductDef,
+  type ProductId,
+  type VersionDef,
+} from './products.js';
+
+/**
+ * One content directory of the matrix: a (product, version, language) triple
+ * mapped to its source directory and its route prefix.
+ */
+export interface Segment {
+  product: ProductDef;
+  version: VersionDef;
+  locale: LocaleId;
+  /** Source directory, relative to the repository root. */
+  dir: string;
+  /** Route prefix, without leading or trailing slash. */
+  prefix: string;
+}
+
+export interface ScopeFilter {
+  products?: ProductId[];
+  versions?: string[];
+  locales?: LocaleId[];
+}
+
+/**
+ * Reads the build scope from the environment. Used to build a subset of the
+ * matrix during development, where loading every version is wasteful.
+ *
+ *   GITEA_DOCS_PRODUCTS=docs,runner GITEA_DOCS_VERSIONS=1.27 pnpm dev
+ */
+export function scopeFromEnv(env: NodeJS.ProcessEnv = process.env): ScopeFilter {
+  const list = (value: string | undefined) =>
+    value
+      ?.split(',')
+      .map((item) => item.trim())
+      .filter(Boolean);
+  return {
+    products: list(env.GITEA_DOCS_PRODUCTS) as ProductId[] | undefined,
+    versions: list(env.GITEA_DOCS_VERSIONS),
+    locales: list(env.GITEA_DOCS_LOCALES) as LocaleId[] | undefined,
+  };
+}
+
+/** Expands the product matrix into the content directories to load. */
+export function resolveSegments(filter: ScopeFilter = {}): Segment[] {
+  const segments: Segment[] = [];
+  for (const product of products) {
+    if (product.externalBaseUrl) continue;
+    if (filter.products && !filter.products.includes(product.id)) continue;
+    for (const version of product.versions) {
+      if (filter.versions && !filter.versions.includes(version.id)) continue;
+      for (const locale of product.locales) {
+        if (filter.locales && !filter.locales.includes(locale)) continue;
+        const dir = version.sources[locale];
+        if (!dir) continue;
+        segments.push({
+          product,
+          version,
+          locale,
+          dir,
+          prefix: routePrefix(product, version, locale),
+        });
+      }
+    }
+  }
+  return segments;
+}
+
+/** Finds the segment a repository relative file path belongs to. */
+export function segmentOf(segments: Segment[], file: string): Segment | undefined {
+  let match: Segment | undefined;
+  for (const segment of segments) {
+    if (!file.startsWith(`${segment.dir}/`)) continue;
+    // longest directory wins, `docs` must not swallow `docs-foo`
+    if (!match || segment.dir.length > match.dir.length) match = segment;
+  }
+  return match;
+}
+
+/**
+ * Route id of a source file, mirroring the docusaurus routing rules:
+ * `index` files address their directory, a relative `slug` replaces the last
+ * path segment and `slug: /` addresses the root of the version.
+ */
+export function routeSuffix(segment: Segment, file: string, slug?: string): string[] {
+  const relative = file.slice(segment.dir.length + 1).replace(/\.mdx?$/, '');
+  const parts = relative.split('/');
+  const isIndex = parts.at(-1) === 'index';
+  if (isIndex) parts.pop();
+
+  if (slug === '/') {
+    parts.length = 0;
+  } else if (slug) {
+    if (!isIndex) parts.pop();
+    parts.push(...slug.replace(/^\/+|\/+$/g, '').split('/'));
+  }
+
+  return parts;
+}
+
+/**
+ * Route id of a source file. Translations follow the route of their english
+ * counterpart, a translated `slug` never moves a page, which is how docusaurus
+ * treated the localized trees as well.
+ */
+export function routeIdOf(segment: Segment, suffix: string[]): string {
+  // the data store rejects empty ids, starlight maps `index` to the root
+  return [segment.prefix, ...suffix].filter(Boolean).join('/') || 'index';
+}
+
+/** Path of a source file relative to its segment directory. */
+export function relativeToSegment(segment: Segment, file: string): string {
+  return file.slice(segment.dir.length + 1);
+}
+
+/** Key identifying the (product, version) a segment translates. */
+export function groupKeyOf(segment: Segment): string {
+  return `${segment.product.id}@${segment.version.id}`;
+}
+
+/**
+ * Works out which (product, version, language) a route belongs to. Used for the
+ * pages starlight builds outside of the loader, the api reference in
+ * particular, so they get the same pickers, banner and search facets.
+ */
+export function metaFromRouteId(id: string):
+  | { product: ProductId; version: string; locale: LocaleId; prefix: string }
+  | undefined {
+  const route = id === 'index' ? '' : id;
+  let best: { product: ProductId; version: string; locale: LocaleId; prefix: string } | undefined;
+  for (const product of products) {
+    if (product.externalBaseUrl) continue;
+    for (const version of product.versions) {
+      for (const locale of product.locales) {
+        const prefix = routePrefix(product, version, locale);
+        // the api pages are generated by starlight-openapi, which slugifies the
+        // base path: `api/1.26` is built as `api/126` and renamed afterwards
+        const candidates = [prefix, slugifyPrefix(prefix)];
+        if (prefix && !candidates.some((candidate) => route === candidate || route.startsWith(`${candidate}/`))) {
+          continue;
+        }
+        if (best && best.prefix.length >= prefix.length) continue;
+        best = { product: product.id, version: version.id, locale, prefix };
+      }
+    }
+  }
+  return best;
+}
+
+/** Same slugification github-slugger applies to a starlight-openapi base path. */
+function slugifyPrefix(prefix: string): string {
+  return prefix
+    .split('/')
+    .map((part) => part.toLowerCase().replace(/[^\w-]/g, ''))
+    .join('/');
+}
+
+/** Url of a route id, mirroring the starlight slug to pathname rules. */
+export function routeIdToPath(id: string): string {
+  if (id === '' || id === 'index') return '/';
+  const trimmed = id.endsWith('/index') ? id.slice(0, -'/index'.length) : id;
+  return `/${trimmed}/`;
+}
+
+export { defaultLocale };