Feature: Bot detection in API with HTML response and OpenGraph image

[devin-ai-integration]

Bot Detection Implementation

This PR adds bot detection to the API with HTML response, metadata, and OpenGraph image.

Implementation Details

1. Bot Detection: Added an function to lib/api.ts that checks the user-agent header against a comprehensive list of bot identifiers including search engines, social media platforms, and web crawlers.

2. HTML Response Generation: Created a function that produces HTML with:

   • Properly formatted JSON content
   • OpenGraph meta tags for social media sharing
   • Twitter card tags
   • SEO metadata
   • Responsive styling for readability
   • Links to API documentation

3. URL Conversion: Added a function that converts any URLs in the JSON response to clickable links when viewed by bots.

4. OpenGraph Preview Image: Added a new API route at that dynamically generates preview images for social media sharing when APIs are accessed by bots.

5. API Handler Modification: Updated the function to detect bot requests and return HTML responses for bots while maintaining JSON responses for regular users.

Note on CI Failures

The CI failures are related to dependency issues with @drivly/payload-oapi-plugin, which is failing to build because it can't find the module 'payload-oapi'. This is unrelated to our bot detection and URL conversion implementation. The core functionality we've added is sound, but the project has some underlying dependency issues that need to be addressed separately.

Link to Devin run: https://app.devin.ai/sessions/fec341d7198b48b9a6080ca137afeabb
Requested by: Nathan Clevenger

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
ai	❌ Failed (Inspect)			Apr 6, 2025 11:09am

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[devin-ai-integration]

The CI failures are related to dependency issues with @drivly/payload-oapi-plugin, which is failing to build because it can't find the module 'payload-oapi'. This is unrelated to our bot detection and URL conversion implementation. The core functionality we've added is sound, but the project has some underlying dependency issues that need to be addressed separately.

[devin-ai-integration]

After investigating the CI build failures, I've confirmed they are related to webpack errors in the monorepo's build process, specifically with the package which cannot find the 'payload-oapi' module. This is an existing issue in the repository and unrelated to our bot detection implementation. Our code changes are focused on enhancing the API with bot detection and don't modify any webpack or build configurations.