[Epic] Fix docs URLs, language navigation and website reporting #773

I want to track the website work in one place, starting with docs URLs and indexing, then language navigation, OSS Health and analytics. Existing issues and PRs are linked below, follow-ups should cover what they leave open.

Checked on production on 2026-10-23: /docs/v1/getting-started/install-cozystack/ returns HTTP 404, /docs/install/talos/ sends readers to v0, and sampled archive pages have conflicting robots tags. /docs/roadmap/ and /roadmap/ already resolve, keep them in regression checks.

Checked again on 2026-10-27, after PR 772 was deployed: /docs/v1/getting-started/install-cozystack/ returns 200 with canonical /docs/getting-started/install-cozystack/, /docs/install/talos/ is a current page, and archive pages have one noindex, follow tag and no canonical. /en/sitemap.xml lists 310 URLs, none of them archive or next/ pages.

Links from /ru/docs/ lead to English version pages. Some links with /ru/ added lead to missing pages, including /ru/demo/. DevStats and OSS Insight share the same GitHub data collector and repeat contributor tables, while their published snapshots still date from August.

Docs and navigation

• A1. Give current docs a permanent indexable URL that survives minor releases. PR 772 serves current docs at /docs/ and /docs/v<major>/, with canonical on /docs/. Numbered versions stay available, archives are noindex, version selection moves to /docs/versions/.

• A2. Map legacy /docs/v1/* and unversioned docs URLs to their current equivalents, including renamed pages. Verify HTTP responses without JavaScript, query strings and fragments. Keep historical version URLs available. PR 775 restores six URLs. The rest depends on the real list of 404s from A9.

• A3. Remove noindex pages from sitemaps, emit one robots tag per content page, and check that latest_version_id matches the published version. Sitemap and canonical URLs must agree with A1. PR 772 did the sitemap and robots parts. The latest_version_id check is left.

• A4. Update links in the main repository README, llms.txt, navigation, footer and blog. Preserve intentional references to historical versions. PR 745 covers some broken links and anchors. PR 776 moves the home button, llms.txt and one blog post to stable URLs. Left: 10 more blog posts, community.md, compliance/gdpr.md and compliance/pci-dss.md that still link to /docs/v1.6/, and the main repository README.

• A5. Add production-build checks for old URLs, canonical, robots, sitemaps and release rollover. Extend issue 496, coordinate with issue 653 for PR builds including next/. PR 777 adds production and preview builds with these checks on every PR.

• A6. Keep the selected language when a translation exists. Show a clear fallback to the original when it does not, without generating missing language URLs. Cover navigation, content links, footer and version switching. Coordinate with PR 767, which changes version navigation.

• A16. Keep mirrored RSS feed links and GUIDs stable across minor releases. IvanHunters' review found numbered URLs in <link>, <guid> and item links inside the stable feed. Update XML link fields in the /docs/ and /docs/v<major>/ copies, preserve numbered archive feeds, and add a rollover test: unchanged entries keep their GUIDs and stable links after a minor release. Follow-up to PR 772, not a merge blocker.

• A17. Add fixtures for refusal paths of hack/mirror_latest.py. 10 of 13 raise ValueError guards can be deleted and hack/test_docs_seo_fixtures.py stays green. One of them, mirror_latest.py:107: without it the script exits 0 and prints that version selection is at /docs/versions/, but the page was never written. Each fixture should check for non-zero exit and no partial output. The directory check in legacy_page from PR 775 is untested too, dropping path.is_dir() keeps all fixtures green. Follow-up to PR 772, not a merge blocker.

• A18. Keep .docs-mirror.json out of the published site. Mirror step writes it to public/ and the Pages workflow uploads hidden files (include-hidden-files: true), so it is probably served at /.docs-mirror.json (not checked on Pages). It has only file paths and hashes. Write it outside public/ or delete it after the mirror step in CI, where the output directory starts empty on every build. Follow-up to PR 772, not a merge blocker.

• A19. Fix the button on the translated home pages. The ru, de, zh-cn and hi homes link it to /<lang>/docs/v1.4/getting-started/, which returns 404 on production (checked for ru and de) because the translated trees have no versioned docs. Link to a page that exists, or to the English page with the fallback from A6. Found while checking PR 776, not caused by it.

• A20. Remove the v0.41 to v1.0 upgrade guide from the current docs. It is operations/upgrades/ in v1.6/ and next/; dropping it from next/ keeps it out of later releases, v1.0-v1.5 keep their copy. Three pages link to its #tenant-isolated-flag-removed section (getting-started/create-tenant.md, guides/tenants/_index.md, operations/stretched/seaweedfs-multidc.md), so move that section to the tenants guide or drop the links. The 1.0 release post links to /docs/v1.6/operations/upgrades/ (content/en/blog/2026-03-16-cozystack-1-0-release/index.md:162), point it at /docs/v1.0/operations/upgrades/. Links from other cozystack repos are not checked.

• A21. Make docs pages readable for AI agents. Checked production on 2026-10-29 with afdocs (npx afdocs check https://cozystack.io/docs/, built on the Agent-Friendly Documentation spec), score is 73/100. None of 50 sampled pages serves markdown on <page>.md (for example /docs/kubernetes/backups-with-velero-addon.md) and Accept: text/markdown is ignored. llms.txt covers 1 of 173 doc pages in the sitemap, links to HTML, and no page points to it in its HTML. About 79% of page HTML is navigation and boilerplate, one sampled page went over 100K characters after conversion to markdown (random sample, the page differs between runs). Add .md output next to every docs page in the Hugo build, generate llms.txt from the sitemap with .md links, and add a pointer to llms.txt in page HTML. GitHub Pages can't negotiate by Accept, so these files are the only way there. PR 776 already moves llms.txt links to stable URLs (A4), the generated file has to use the same paths. Add afdocs to the production build checks from A5 after that, pinned to a version because it is pre-1.0 and its output format changes, and failing only on chosen checks.

OSS Health

• A7. Decide whether DevStats and OSS Insight need separate pages. Compare their metrics, remove repeated sections and name the real data source. Distinguish period totals from values measured at collection time.
• A8. Publish current snapshots and make the collection-to-publication process predictable. PR 770 contains the same snapshot files as PR 674 and PR 730. Choose one publication path, then define who handles monthly updates and how overdue data is reported. Show collection date and measurement period.

Analytics and content

• A9. After deployment, resubmit sitemaps in Search Console, check indexing exclusions and 404s, and request indexing for installation, Talm and platform guides. Compare coverage and organic traffic with a recorded baseline over 4-6 weeks.
• A10. Create separate reporting segments for suspected bots and referral spam, keeping raw data available. Check exclusion rules against several signals before using the filtered figures.
• A11. Compare suspicious analytics hits with available HTTP delivery data to distinguish site requests from hits sent directly to analytics. Choose filtering or delivery changes from that evidence.
• A12. Measure command copying, GitHub and community links, support visits and documentation search. Verify events arrive and agree which ones count as key events.
• A13. Investigate the reported drop in GitHub referrals using repository traffic, outbound links and attribution data.
• A14. Report AI referrals and dashboard-to-docs traffic separately. Dashboard referrals are a usage signal, not a count of unique installations.
• A15. Build a content plan from non-brand Search Console queries: Talm, Talos installation, hairpin proxy, LINSTOR/ZFS, Blockstor and platform comparisons. Reuse PR 667, PR 552 and issue 337 where they fit. PR 640 covers blog publishing tools.

Order of work, most important first. A1 and the indexing part of A3 are done and deployed, they closed the main SEO problems. Next land PR 775, PR 776 and PR 777 (A2, A4, A5), so legacy URLs and the checks stay in place. Then A9, because the 4-6 week baseline starts only when sitemaps are resubmitted, and A10 and A11, because the filtered traffic has to be trusted before it is compared with that baseline. Then A19, A20 and A6, then what is left of A2, A3 and A4, then A8. A7, A12-A15 and A21 come last. Analytics work needs the property owners. Traffic recovery targets stay separate from code acceptance criteria.