fix(search): scope the results to the version being read (#512)

Closes #146.

The published site runs pagefind, not docsearch: `PUBLIC_DOCSEARCH_APP_ID` and `PUBLIC_DOCSEARCH_API_KEY` are not set, so `astro.config.mjs` falls back to `pagefind: true`, and a page of docs.gitea.com ships starlight's `#starlight__search` with the `/pagefind/` bundle. The version scoping we have is only implemented for docsearch (`docsearch:product` / `docsearch:version` meta tags plus `facetFilters` in `src/config/docsearch.ts`), so nothing consumes it today.

Pagefind builds one index for the whole site, partitioned only by the html language, and nothing tags the pages with a version, so a search started in the 1.23 docs answers with every version of the same page and the latest one usually wins. That is what the issue reports; the original cause (the docusaurus search plugin) is gone, this one replaced it.

**What this does**

- `src/components/MarkdownContent.astro` wraps the content in two `data-pagefind-filter` elements, `product` and `version`, so the filters land in the index. One filter per element on purpose: pagefind reads the whole attribute as a single `name:value` pair, a comma separated value ends up as one filter named `product` with the value `docs,version:1.26`.
- The same scope is written to a `gitea:search-filters` meta tag next to the docsearch ones (`src/lib/search.ts`).
- Starlight builds the search ui from a build time configuration and has no option for per page filters, so its `@pagefind/default-ui` import is redirected to a subclass in `src/lib/pagefind-ui.ts` by the `gitea-pagefind-filters` plugin in `astro.config.mjs`, the same approach the `gitea-openapi-overview` plugin already uses. The subclass selects the filters of the page the modal was opened on.
- A "Search all versions" checkbox below the search input drops the filters again and labels each result with the version (and the product for the api and the runner) it comes from, so the duplicates are distinguishable. Translated into 简体中文 and 繁體中文.
- Pagefind's own filter panel is hidden, it is redundant next to the checkbox and starlight does not style it. The checkbox gets its native rendering back, pagefind resets everything it renders with `all: unset`.
- A `gitea-pagefind-filters-check` integration fails the build if the redirect did not run, so a starlight upgrade that moves the import cannot silently bring back an unscoped search.

`@pagefind/default-ui` is added to `sites/docs/package.json`; it was only reachable as a transitive dependency of starlight, and the version is the one already in the lockfile.

**Verification**

Built the whole site and drove the search with a headless browser against `astro preview` (the search does not run in `pnpm dev`):

| Page | Query | Results |
| --- | --- | --- |
| `/1.23/usage/actions/comparison/` | email | only `/1.23/...` |
| `/usage/actions/overview/` | email | only the version served at the root |
| `/runner/registration/` | label | only `/runner/...` |
| `/api/operations/tags/issue/` | label | only `/api/...` |
| `/zh-cn/1.26/usage/actions/overview/` | email | only `/zh-cn/1.26/...` |
| `/1.23/...` with the checkbox ticked | email | every version, results labelled `... · API 1.25`, `... · next` |

`pnpm check` passes. The docsearch path is unchanged: when the credentials are set the plugin takes over, the redirect does not run and the check integration skips.

<!-- cloudflare-preview --> Preview: https://pr-512.docs-gitea-com.pages.dev

Reviewed-on: https://gitea.com/gitea/docs/pulls/512

pnpm-lock.yaml

@@ -31,6 +31,9 @@ importers:
       '@gitea-docs/content-loader':
         specifier: workspace:*
         version: link:../../packages/content-loader
+      '@pagefind/default-ui':
+        specifier: ^1.5.2
+        version: 1.5.2
       astro:
         specifier: 7.2.0
         version: 7.2.0(@astrojs/[email protected]([email protected]))(@emnapi/[email protected])(@emnapi/[email protected])(@types/[email protected])([email protected])

sites/docs/README.md

@@ -101,6 +101,25 @@ into facets and `src/config/docsearch.ts` filters the search on the product,
 version and language being read. `cloudflare/docsearch-crawler.json` holds the
 crawler configuration and only indexes the versions people read.
 
+Pagefind builds one index for the whole site, so the same scope has to be
+applied on its side as well, otherwise a search started in the 1.23 docs answers
+with the pages of the latest release:
+
+- `src/components/MarkdownContent.astro` wraps the content in the
+  `data-pagefind-filter` elements that put `product` and `version` into the
+  index. One filter per element, pagefind reads the whole attribute as a single
+  `name:value` pair.
+- the same scope is written to the `gitea:search-filters` meta tag by
+  `src/lib/search.ts`.
+- starlight builds the search ui with a build time configuration and has no
+  option for per page filters, so its `@pagefind/default-ui` import is
+  redirected to the subclass in `src/lib/pagefind-ui.ts` by the
+  `gitea-pagefind-filters` plugin in `astro.config.mjs`. It selects the filters
+  of the page the modal was opened on and adds the "Search all versions"
+  checkbox, which drops them again and labels the results with their version.
+  A starlight upgrade that moves the import fails the build, the
+  `gitea-pagefind-filters-check` integration verifies that the redirect ran.
+
 ## Deployment
 
 `cloudflare/_headers` and `cloudflare/_redirects` are copied next to the build

sites/docs/astro.config.mjs

@@ -18,6 +18,11 @@ const useDocSearch = Boolean(
   process.env.PUBLIC_DOCSEARCH_APP_ID && process.env.PUBLIC_DOCSEARCH_API_KEY,
 );
 
+// set by the `gitea-pagefind-filters` plugin below, checked after the build so
+// a starlight upgrade that moves the import fails loudly instead of silently
+// serving an unscoped search again
+let pagefindUiRedirected = false;
+
 export default defineConfig({
   site: 'https://docs.gitea.com',
   trailingSlash: 'always',
@@ -44,6 +49,26 @@ export default defineConfig({
           return null;
         },
       },
+      {
+        // Pagefind indexes every version into one index and starlight builds
+        // the search ui with a build time configuration, so a search cannot be
+        // scoped to the version being read. Its `@pagefind/default-ui` import
+        // is redirected to the subclass in src/lib/pagefind-ui.ts, which
+        // selects the filters of the current page.
+        name: 'gitea-pagefind-filters',
+        enforce: 'pre',
+        resolveId(source, importer) {
+          if (
+            !useDocSearch &&
+            source === '@pagefind/default-ui' &&
+            importer?.includes('starlight/components/Search.astro')
+          ) {
+            pagefindUiRedirected = true;
+            return path.join(repoRoot, 'sites/docs/src/lib/pagefind-ui.ts');
+          }
+          return null;
+        },
+      },
     ],
   },
   // languages the sources tag code blocks with that shiki does not know
@@ -62,6 +87,19 @@ export default defineConfig({
   },
   integrations: [
     giteaPostBuild(),
+    {
+      name: 'gitea-pagefind-filters-check',
+      hooks: {
+        'astro:build:done': () => {
+          if (useDocSearch || pagefindUiRedirected) return;
+          throw new Error(
+            'the pagefind search ui was not replaced by src/lib/pagefind-ui.ts, ' +
+              'the search would return results from every version: check the ' +
+              "`gitea-pagefind-filters` plugin against starlight's Search.astro",
+          );
+        },
+      },
+    },
     starlight({
       title: 'Gitea Documentation',
       description: 'Git with a cup of tea',

sites/docs/package.json

@@ -14,6 +14,7 @@
     "@astrojs/starlight": "0.41.7",
     "@astrojs/starlight-docsearch": "0.7.0",
     "@gitea-docs/content-loader": "workspace:*",
+    "@pagefind/default-ui": "^1.5.2",
     "astro": "7.2.0",
     "sharp": "0.34.5",
     "starlight-openapi": "0.26.0"

sites/docs/src/components/MarkdownContent.astro

@@ -1,17 +1,25 @@
 ---
 import Default from '@astrojs/starlight/components/MarkdownContent.astro';
 import { t } from '../config/strings';
+import { searchFilters } from '../lib/search';
 
 /**
  * Translated pages carry the notice the docusaurus site showed: a translation
  * can lag behind the english original, so point at it and at the translation
  * guide. Fallback pages already get starlight's own notice.
+ *
+ * The wrappers carry the pagefind filters of the page. They sit inside the
+ * `data-pagefind-body` element starlight puts on `<main>` and outside the
+ * `.sl-markdown-content` the default component renders, so they are picked up
+ * by the index without taking part in the content styles. One filter per
+ * element: pagefind reads the whole attribute as a single `name:value` pair.
  */
 const route = Astro.locals.starlightRoute;
 const meta = route.entry.data.gitea;
 const translated = Boolean(meta) && meta!.locale !== 'en-us' && !route.isFallback;
 const editUrl = route.editUrl?.href;
 const strings = t(meta?.locale ?? 'en-us');
+const filters = meta ? searchFilters(meta) : undefined;
 ---
 
 {
@@ -22,7 +30,11 @@ const strings = t(meta?.locale ?? 'en-us');
     </div>
   )
 }
-<Default><slot /></Default>
+<div data-pagefind-filter={filters && `product:${filters.product}`}>
+  <div data-pagefind-filter={filters && `version:${filters.version}`}>
+    <Default><slot /></Default>
+  </div>
+</div>
 
 <style>
   .gitea-translation-notice {

sites/docs/src/config/strings.ts

@@ -14,6 +14,8 @@ type Strings = {
   translationNotice: string;
   translationHelp: string;
   dismissAnnouncement: string;
+  /** Checkbox in the search modal, which searches outside the current version. */
+  searchAllVersions: string;
   unreleased: (latest: string) => string;
   outdated: (version: string, latest: string) => string;
 };
@@ -27,6 +29,7 @@ const en: Strings = {
   translationNotice: 'This translation may be behind the english original.',
   translationHelp: 'Help us translate it',
   dismissAnnouncement: 'Dismiss this announcement',
+  searchAllVersions: 'Search all versions',
   unreleased: (latest) =>
     `This is the documentation of the next version, still under development. <a href="${latest}">See the latest release</a>.`,
   outdated: (version, latest) =>
@@ -44,6 +47,7 @@ const strings: Record<string, Strings> = {
     translationNotice: '当前中文文档翻译不是最新版，访问英文版本查看最新内容，或',
     translationHelp: '帮助我们翻译',
     dismissAnnouncement: '关闭此提示',
+    searchAllVersions: '搜索所有版本',
     unreleased: (latest) =>
       `这是下一个版本的文档，仍在开发中。<a href="${latest}">查看最新发布版本</a>。`,
     outdated: (version, latest) =>
@@ -58,6 +62,7 @@ const strings: Record<string, Strings> = {
     translationNotice: '當前中文文檔翻譯不是最新版，訪問英文版本查看最新內容，或',
     translationHelp: '幫助我們翻譯',
     dismissAnnouncement: '關閉此提示',
+    searchAllVersions: '搜尋所有版本',
     unreleased: (latest) =>
       `這是下一個版本的文檔，仍在開發中。<a href="${latest}">查看最新發布版本</a>。`,
     outdated: (version, latest) =>

sites/docs/src/lib/pagefind-ui.ts

@@ -0,0 +1,104 @@
+import { PagefindUI as PagefindDefaultUI } from '@pagefind/default-ui';
+import { t } from '../config/strings';
+import { parseSearchFilters, searchFilterMetaName, type SearchFilters } from './search-filters';
+
+/**
+ * Pagefind builds a single index for the whole site, so a search started in the
+ * 1.23 docs answers with the pages of every other version as well, usually with
+ * the latest release on top. The pages carry `product` and `version` filters
+ * (see `src/components/MarkdownContent.astro`), this subclass selects the ones
+ * of the page the modal was opened on and offers a checkbox to search all
+ * versions instead, in which case the results are labelled with their version.
+ *
+ * Starlight creates the search ui itself and has no option for per page
+ * filters, so its `@pagefind/default-ui` import is redirected here by the
+ * `gitea-pagefind-filters` plugin in `astro.config.mjs`.
+ */
+
+type PagefindResult = {
+  meta?: Record<string, string>;
+  filters?: Record<string, string[]>;
+};
+
+type PagefindUIOptions = Record<string, unknown> & {
+  element?: string | HTMLElement;
+  processResult?: (result: PagefindResult) => void;
+};
+
+export class PagefindUI extends PagefindDefaultUI {
+  constructor(options: PagefindUIOptions) {
+    const filters = pageFilters();
+    // shared with `processResult`, which pagefind calls for every result of
+    // every search, long after the constructor has returned
+    const state = { scoped: Object.keys(filters).length > 0 };
+    const processResult = options.processResult;
+
+    super({
+      ...options,
+      processResult: (result: PagefindResult) => {
+        processResult?.(result);
+        if (!state.scoped) labelResult(result);
+      },
+    });
+
+    if (!state.scoped) return;
+    this.triggerFilters(filters);
+
+    const root = rootElement(options.element);
+    if (!root) return;
+    addScopeToggle(root, (allVersions) => {
+      state.scoped = !allVersions;
+      // the selected filters are a reactive prop of the pagefind ui, setting
+      // them runs the current search again
+      this.triggerFilters(allVersions ? {} : filters);
+    });
+  }
+}
+
+/** The scope of the page the search was opened on, written by `src/lib/search.ts`. */
+function pageFilters(): SearchFilters {
+  const meta = document.querySelector<HTMLMetaElement>(`meta[name="${searchFilterMetaName}"]`);
+  return parseSearchFilters(meta?.content);
+}
+
+function rootElement(element: string | HTMLElement | undefined): HTMLElement | null {
+  if (element instanceof HTMLElement) return element;
+  return document.querySelector<HTMLElement>(element ?? '[data-pagefind-ui]');
+}
+
+function strings() {
+  return t(document.documentElement.lang.toLowerCase());
+}
+
+/** `Search all versions`, below the search input. */
+function addScopeToggle(root: HTMLElement, onChange: (allVersions: boolean) => void): void {
+  const label = document.createElement('label');
+  label.className = 'gitea-search-scope';
+
+  const checkbox = document.createElement('input');
+  checkbox.type = 'checkbox';
+  checkbox.addEventListener('change', () => onChange(checkbox.checked));
+
+  const text = document.createElement('span');
+  text.textContent = strings().searchAllVersions;
+
+  label.append(checkbox, text);
+  const form = root.querySelector('.pagefind-ui__form');
+  if (form) form.insertAdjacentElement('afterend', label);
+  else root.append(label);
+}
+
+/**
+ * While all versions are searched the same page shows up once per version, so
+ * the version (and the product, for the api and the runner) is appended to the
+ * title of a result.
+ */
+function labelResult(result: PagefindResult): void {
+  const version = result.filters?.version?.[0];
+  const product = result.filters?.product?.[0];
+  if (!version || !result.meta?.title) return;
+
+  const products = strings().products;
+  const label = product && product !== 'docs' ? `${products[product] ?? product} ${version}` : version;
+  result.meta.title = `${result.meta.title} · ${label}`;
+}

sites/docs/src/lib/search-filters.ts

@@ -0,0 +1,39 @@
+/**
+ * The scope of a page for the local (pagefind) search: which product and which
+ * version it belongs to. The value is written twice into every page:
+ *
+ * - as `data-pagefind-filter` inside the indexed body, so the filters end up in
+ *   the pagefind index (see `src/components/MarkdownContent.astro`)
+ * - as a meta tag, so the search modal can read the scope of the page it is
+ *   opened on and filter the results down to it (see `src/lib/pagefind-ui.ts`)
+ *
+ * Without them the index is one flat list of every version and searching from
+ * an old version answers with the pages of the latest one.
+ *
+ * This module is imported by the client, keep it free of server side imports.
+ */
+
+/** Name of the meta tag carrying the scope of the page. */
+export const searchFilterMetaName = 'gitea:search-filters';
+
+export type SearchFilters = Record<string, string>;
+
+/** `{ product: 'docs', version: '1.27' }` -> `product:docs,version:1.27`. */
+export function formatSearchFilters(filters: SearchFilters): string {
+  return Object.entries(filters)
+    .map(([name, value]) => `${name}:${value}`)
+    .join(',');
+}
+
+/** Inverse of `formatSearchFilters`, for the client side. */
+export function parseSearchFilters(value: string | null | undefined): SearchFilters {
+  const filters: SearchFilters = {};
+  for (const pair of (value ?? '').split(',')) {
+    const index = pair.indexOf(':');
+    if (index < 1) continue;
+    const name = pair.slice(0, index).trim();
+    const filterValue = pair.slice(index + 1).trim();
+    if (name && filterValue) filters[name] = filterValue;
+  }
+  return filters;
+}

sites/docs/src/lib/search.ts

@@ -1,10 +1,18 @@
 import { locales, type LocaleId } from '@gitea-docs/content-loader';
 import type { GiteaMeta } from '../schema';
+import { formatSearchFilters, searchFilterMetaName } from './search-filters';
+
+/** Product and version of a page, the scope both search backends filter on. */
+export function searchFilters(meta: GiteaMeta): Record<string, string> {
+  return { product: meta.product, version: meta.version };
+}
 
 /**
- * Search facets of a page. The docsearch crawler turns these into the
- * `product`, `version` and `language` facets the search modal filters on, so
- * a search stays inside the product, version and language being read.
+ * Search facets of a page. The docsearch crawler turns the `docsearch:*` tags
+ * into the `product`, `version` and `language` facets the search modal filters
+ * on. `gitea:search-filters` is the same scope for the pagefind fallback, which
+ * is what the published site currently uses. Either way a search stays inside
+ * the product, version and language being read.
  */
 export function searchMetaTags(meta: GiteaMeta): {
   tag: 'meta';
@@ -17,5 +25,9 @@ export function searchMetaTags(meta: GiteaMeta): {
       tag: 'meta',
       attrs: { name: 'docsearch:language', content: locales[meta.locale as LocaleId]?.lang ?? meta.locale },
     },
+    {
+      tag: 'meta',
+      attrs: { name: searchFilterMetaName, content: formatSearchFilters(searchFilters(meta)) },
+    },
   ];
 }

sites/docs/src/pagefind-ui.d.ts

@@ -0,0 +1,12 @@
+/**
+ * `@pagefind/default-ui` ships no types, only the part `src/lib/pagefind-ui.ts`
+ * builds on is declared here.
+ */
+declare module '@pagefind/default-ui' {
+  export class PagefindUI {
+    constructor(options: Record<string, unknown>);
+    triggerSearch(term: string): void;
+    triggerFilters(filters: Record<string, string | string[]>): void;
+    destroy(): void;
+  }
+}

sites/docs/src/styles/custom.css

@@ -153,3 +153,31 @@ starlight-menu-button button {
 .sl-openapi-method-trace {
   --gitea-method: var(--gitea-method-other);
 }
+
+/* Search: the results are scoped to the product and the version being read by
+   src/lib/pagefind-ui.ts, which selects the pagefind filters of the page and
+   adds the checkbox below the search input. The filter panel pagefind renders
+   for the same filters is redundant then, and it is not styled by starlight. */
+#starlight__search .pagefind-ui__filter-panel {
+  display: none;
+}
+
+#starlight__search .gitea-search-scope {
+  display: flex;
+  align-items: center;
+  gap: 0.5rem;
+  padding-block: 0.5rem;
+  color: var(--sl-color-gray-2);
+  font-size: var(--sl-text-sm);
+  cursor: pointer;
+}
+
+#starlight__search .gitea-search-scope input {
+  /* pagefind resets every element it renders with `all: unset`, which also
+     takes the checkbox rendering away */
+  appearance: auto;
+  width: 0.875rem;
+  height: 0.875rem;
+  accent-color: var(--sl-color-accent);
+  cursor: pointer;
+}