Skip to content

chore: improve SEO and GEO#23

Open
henchaves wants to merge 2 commits intochore/starlight-docsfrom
chore/improve-seo-and-geo
Open

chore: improve SEO and GEO#23
henchaves wants to merge 2 commits intochore/starlight-docsfrom
chore/improve-seo-and-geo

Conversation

@henchaves
Copy link
Member

@henchaves henchaves commented Mar 19, 2026

  • removes trailing slashes
  • all page links also have an .md version
  • adds llms.txt

- Set trailingSlash to 'never' in Astro config for consistent URL handling.
- Introduced new llms.txt file containing comprehensive documentation for Giskard products and LLMs.
- Added robots.txt to manage web crawler access and specify sitemap location.
- Implemented middleware for handling trailing slashes in URLs.
- Created Head.astro component for structured data and breadcrumb navigation.
- Updated Header and Sidebar components to remove trailing slashes from links.
- Added dynamic markdown endpoint for documentation pages.
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the documentation site's discoverability and accessibility for both traditional search engines and modern generative AI models. By standardizing URL structures, embedding rich structured data, and providing dedicated content feeds for LLMs, the changes aim to improve how the documentation is indexed, presented in search results, and consumed by AI agents. This strategic update ensures that the Giskard documentation is optimized for a broader range of information retrieval and AI-driven applications.

Highlights

  • SEO Enhancements: Implemented several changes to improve Search Engine Optimization, including adding a custom Head component for structured data (JSON-LD for BreadcrumbList, Organization, and TechArticle), enabling lastUpdated timestamps, and configuring trailingSlash: 'never' in Astro.
  • GEO (Generative Engine Optimization) Support: Introduced a robots.txt file to explicitly allow various AI bots (GPTBot, ChatGPT-User, Google-Extended, anthropic-ai, ClaudeBot, CCBot, PerplexityBot, Bytespider) to crawl the documentation, and added an llms.txt file providing a structured list of documentation pages in raw Markdown format for LLM consumption.
  • URL Standardization: Removed trailing slashes from numerous internal links across Markdown and Jupyter Notebook files to ensure consistent URL formatting, complemented by a new Astro middleware that redirects URLs with trailing slashes to their canonical form.
  • Documentation Metadata: Added description frontmatter to many documentation pages, providing concise summaries that can be used by search engines and LLMs for better content understanding and indexing.
  • API Documentation Update: Updated the TestCase type in the Hub SDK reference documentation to include a new status property, clarifying its possible values as 'active' or 'draft'.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

The pull request introduces a new trailingSlash: 'never' configuration to astro.config.mjs and a new middleware.ts file to enforce canonical URLs by redirecting paths with trailing slashes. Additionally, it adds lastUpdated: true and a custom Head.astro component to astro.config.mjs for improved SEO and metadata generation, including JSON-LD for organization, breadcrumbs, and articles. A new llms.txt file is added, containing a comprehensive markdown overview of Giskard documentation. The PR also updates numerous internal links across various documentation files (MDX and IPYNB) by removing trailing slashes to ensure URL consistency, aligning with the new canonical URL policy.

Comment on lines +786 to 791
"- [Run in pytest](/oss/checks/how-to/run-in-pytest) — integrate batch tests\n",
" into CI with proper failure reporting\n",
"- [Test Suites](/oss/checks/tutorials/test-suites/) — group named scenarios\n",
"- [Test Suites](/oss/checks/tutorials/test-suites) — group named scenarios\n",
" rather than iterate over a data list\n",
"- [Structured Output Testing](/oss/checks/how-to/structured-output/) — validate\n",
"- [Structured Output Testing](/oss/checks/how-to/structured-output) — validate\n",
" Pydantic models or dicts"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- [Run in pytest](/oss/checks/how-to/run-in-pytest) — integrate batch tests\n",
" into CI with proper failure reporting\n",
"- [Test Suites](/oss/checks/tutorials/test-suites/) — group named scenarios\n",
"- [Test Suites](/oss/checks/tutorials/test-suites) — group named scenarios\n",
" rather than iterate over a data list\n",
"- [Structured Output Testing](/oss/checks/how-to/structured-output/) — validate\n",
"- [Structured Output Testing](/oss/checks/how-to/structured-output) — validate\n",
" Pydantic models or dicts"
"- [Run in pytest](/oss/checks/how-to/run-in-pytest) — integrate batch tests\n",
" into CI with proper failure reporting\n",
"- [Test Suites](/oss/checks/tutorials/test-suites) — group named scenarios\n",
" rather than iterate over a data list\n",
"- [Structured Output Testing](/oss/checks/how-to/structured-output) — validate\n"

"\n",
"- Tests are already running locally with pytest (see\n",
" [Run Tests with pytest](/oss/checks/how-to/run-in-pytest/))\n",
" [Run Tests with pytest](/oss/checks/how-to/run-in-pytest))\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The removal of the trailing slash from [Run Tests with pytest](/oss/checks/how-to/run-in-pytest/) to [Run Tests with pytest](/oss/checks/how-to/run-in-pytest) improves URL consistency.

Suggested change
" [Run Tests with pytest](/oss/checks/how-to/run-in-pytest))\n",
" [Run Tests with pytest](/oss/checks/how-to/run-in-pytest))\n"

Comment on lines +337 to 340
"- [Run Tests with pytest](/oss/checks/how-to/run-in-pytest) — full pytest setup\n",
" including parametrize and fixtures\n",
"- [Batch Evaluation](/oss/checks/how-to/batch-evaluation/) — evaluate many\n",
"- [Batch Evaluation](/oss/checks/how-to/batch-evaluation) — evaluate many\n",
" scenarios efficiently in a single run"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- [Run Tests with pytest](/oss/checks/how-to/run-in-pytest) — full pytest setup\n",
" including parametrize and fixtures\n",
"- [Batch Evaluation](/oss/checks/how-to/batch-evaluation/) — evaluate many\n",
"- [Batch Evaluation](/oss/checks/how-to/batch-evaluation) — evaluate many\n",
" scenarios efficiently in a single run"
"- [Run Tests with pytest](/oss/checks/how-to/run-in-pytest) — full pytest setup\n",
" including parametrize and fixtures\n",
"- [Batch Evaluation](/oss/checks/how-to/batch-evaluation) — evaluate many\n"

Comment on lines +736 to +740
"- [API Reference: Checks](/oss/checks/reference/checks) — full list of built-in\n",
" checks and their parameters\n",
"- [Single-Turn Evaluation](/oss/checks/tutorials/single-turn/) — using checks in\n",
"- [Single-Turn Evaluation](/oss/checks/tutorials/single-turn) — using checks in\n",
" a scenario\n",
"- [Stateful Checks](/oss/checks/how-to/stateful-checks/) — checks that\n",
"- [Stateful Checks](/oss/checks/how-to/stateful-checks) — checks that\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- [API Reference: Checks](/oss/checks/reference/checks) — full list of built-in\n",
" checks and their parameters\n",
"- [Single-Turn Evaluation](/oss/checks/tutorials/single-turn/) — using checks in\n",
"- [Single-Turn Evaluation](/oss/checks/tutorials/single-turn) — using checks in\n",
" a scenario\n",
"- [Stateful Checks](/oss/checks/how-to/stateful-checks/) — checks that\n",
"- [Stateful Checks](/oss/checks/how-to/stateful-checks) — checks that\n",
"- [API Reference: Checks](/oss/checks/reference/checks) — full list of built-in\n",
" checks and their parameters\n",
"- [Single-Turn Evaluation](/oss/checks/tutorials/single-turn) — using checks in\n",
" a scenario\n",
"- [Stateful Checks](/oss/checks/how-to/stateful-checks) — checks that\n"

Comment on lines +327 to +331
"- [Async design & pytest](/oss/checks/explanation/async-and-pytest) — why\n",
" `Scenario.run()` is async\n",
"- [Single-turn testing tutorial](/oss/checks/tutorials/single-turn/) — the\n",
"- [Single-turn testing tutorial](/oss/checks/tutorials/single-turn) — the\n",
" scenario patterns used above\n",
"- [Simulate Users](/oss/checks/how-to/simulate-users/) — drive multi-turn tests\n",
"- [Simulate Users](/oss/checks/how-to/simulate-users) — drive multi-turn tests\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- [Async design & pytest](/oss/checks/explanation/async-and-pytest) — why\n",
" `Scenario.run()` is async\n",
"- [Single-turn testing tutorial](/oss/checks/tutorials/single-turn/) — the\n",
"- [Single-turn testing tutorial](/oss/checks/tutorials/single-turn) — the\n",
" scenario patterns used above\n",
"- [Simulate Users](/oss/checks/how-to/simulate-users/) — drive multi-turn tests\n",
"- [Simulate Users](/oss/checks/how-to/simulate-users) — drive multi-turn tests\n",
"- [Async design & pytest](/oss/checks/explanation/async-and-pytest) — why\n",
" `Scenario.run()` is async\n",
"- [Single-turn testing tutorial](/oss/checks/tutorials/single-turn) — the\n",
" scenario patterns used above\n",
"- [Simulate Users](/oss/checks/how-to/simulate-users) — drive multi-turn tests\n"

"tutorial shows you how to configure a generator and test an actual LLM call:\n",
"\n",
"[Your First LLM Call](/oss/checks/tutorials/single-turn/)"
"[Your First LLM Call](/oss/checks/tutorials/single-turn)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The removal of the trailing slash from [Your First LLM Call](/oss/checks/tutorials/single-turn/) to [Your First LLM Call](/oss/checks/tutorials/single-turn) improves URL consistency.

    "[Your First LLM Call](/oss/checks/tutorials/single-turn)"\n

Comment on lines +1169 to 1174
"- See [Content Moderation](/oss/checks/use-cases/content-moderation) for safety\n",
" and filtering\n",
"- Explore [Testing Agents](/oss/checks/use-cases/testing-agents/) for tool-using\n",
"- Explore [Testing Agents](/oss/checks/use-cases/testing-agents) for tool-using\n",
" chatbots\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn/) for complex\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for complex\n",
" flows"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next Steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- See [Content Moderation](/oss/checks/use-cases/content-moderation) for safety\n",
" and filtering\n",
"- Explore [Testing Agents](/oss/checks/use-cases/testing-agents/) for tool-using\n",
"- Explore [Testing Agents](/oss/checks/use-cases/testing-agents) for tool-using\n",
" chatbots\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn/) for complex\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for complex\n",
" flows"
"- See [Content Moderation](/oss/checks/use-cases/content-moderation) for safety\n",
" and filtering\n",
"- Explore [Testing Agents](/oss/checks/use-cases/testing-agents) for tool-using\n",
" chatbots\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for complex\n"

Comment on lines +861 to 866
"- See [Custom Checks](/oss/checks/how-to/custom-checks) for building custom\n",
" safety checks\n",
"- Review [Single-Turn Evaluation](/oss/checks/tutorials/single-turn/) for more\n",
"- Review [Single-Turn Evaluation](/oss/checks/tutorials/single-turn) for more\n",
" guardrail patterns\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing/) for\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational safety testing"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next Steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- See [Custom Checks](/oss/checks/how-to/custom-checks) for building custom\n",
" safety checks\n",
"- Review [Single-Turn Evaluation](/oss/checks/tutorials/single-turn/) for more\n",
"- Review [Single-Turn Evaluation](/oss/checks/tutorials/single-turn) for more\n",
" guardrail patterns\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing/) for\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational safety testing"
"- See [Custom Checks](/oss/checks/how-to/custom-checks) for building custom\n",
" safety checks\n",
"- Review [Single-Turn Evaluation](/oss/checks/tutorials/single-turn) for more\n",
" guardrail patterns\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n"

Comment on lines +1056 to 1061
"- See [Testing Agents](/oss/checks/use-cases/testing-agents) for agent-specific\n",
" testing patterns\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing/) for\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational testing\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn/) for advanced\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for advanced\n",
" scenarios"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next Steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- See [Testing Agents](/oss/checks/use-cases/testing-agents) for agent-specific\n",
" testing patterns\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing/) for\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational testing\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn/) for advanced\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for advanced\n",
" scenarios"
"- See [Testing Agents](/oss/checks/use-cases/testing-agents) for agent-specific\n",
" testing patterns\n",
"- Explore [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational testing\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for advanced\n"

Comment on lines +1193 to 1198
"- See [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational agent patterns\n",
"- Explore [RAG Evaluation](/oss/checks/use-cases/rag-evaluation/) for\n",
"- Explore [RAG Evaluation](/oss/checks/use-cases/rag-evaluation) for\n",
" knowledge-grounded agents\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn/) for complex\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for complex\n",
" workflows"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing trailing slashes from internal links in the 'Next Steps' section ensures URL consistency across the documentation. This aligns with the site's canonical URL policy.

Suggested change
"- See [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational agent patterns\n",
"- Explore [RAG Evaluation](/oss/checks/use-cases/rag-evaluation/) for\n",
"- Explore [RAG Evaluation](/oss/checks/use-cases/rag-evaluation) for\n",
" knowledge-grounded agents\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn/) for complex\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for complex\n",
" workflows"
"- See [Chatbot Testing](/oss/checks/use-cases/chatbot-testing) for\n",
" conversational agent patterns\n",
"- Explore [RAG Evaluation](/oss/checks/use-cases/rag-evaluation) for\n",
" knowledge-grounded agents\n",
"- Review [Multi-Turn Scenarios](/oss/checks/tutorials/multi-turn) for complex\n"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant