Skip to content

feat: add Xiaohongshu (RedNote) extractor#292

Open
Afterimages wants to merge 1 commit into
kepano:mainfrom
Afterimages:xiaohongshu-extractor
Open

feat: add Xiaohongshu (RedNote) extractor#292
Afterimages wants to merge 1 commit into
kepano:mainfrom
Afterimages:xiaohongshu-extractor

Conversation

@Afterimages

Copy link
Copy Markdown
Contributor

Extract note content (description, images, video, tags) and comments from xiaohongshu.com note pages via INITIAL_STATE JSON parsing.

Comment extraction uses a three-tier fallback strategy:

  1. Parse from INITIAL_STATE (HTML source / MAIN world bridge)
  2. Fetch from XHS comment API (async fallback)
  3. Extract from rendered DOM elements (works without MAIN world access)

Additional features:

  • Normalize JS-only tokens (undefined/NaN/Infinity) in JSON-like state
  • Remove duplicate hashtag mentions from description text
  • Include comments in contentHtml by default; opt-out via excludeCommentsFromContent option
  • Provide comments as {{comments}} template variable (markdown)
  • Register extractor for xiaohongshu.com URL patterns

Relates #273

@kepano kepano left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add tests/fixtures.

@Afterimages Afterimages requested a review from kepano June 7, 2026 11:11
@Afterimages Afterimages force-pushed the xiaohongshu-extractor branch 3 times, most recently from e55e95c to 6dfb282 Compare July 4, 2026 15:51
- Implement SPA state extraction with robust DOM fallback

- Optimize bundle size by avoiding heavy JSON parsers and TurndownService

- Add comprehensive test fixtures and unit tests
@Afterimages Afterimages force-pushed the xiaohongshu-extractor branch from 6dfb282 to 6e98b0f Compare July 4, 2026 16:54
@Afterimages

Copy link
Copy Markdown
Contributor Author

Please add tests/fixtures.

Force-pushed the branch after squashing the changes into a single commit.

Updates since the previous review:

  • Added Xiaohongshu fixtures and unit tests
  • Improved SPA stale-state handling and DOM fallbacks
  • Fixed DOM comment author extraction
  • Reduced the bundle-size impact
  • Verified the extractor manually in the browser extension against a live Xiaohongshu note

Manual verification confirmed that:

  • The title, author, publication date, description, tags, and images are extracted correctly
  • Comments and nested replies are preserved
  • The extracted content can be saved to Obsidian correctly
  • SPA navigation fallback works as expected

All CI checks are now passing on Node.js 20, 22, and 24.

Ready for re-review. Thank you!

Snipaste_2026-07-05_22-51-47

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants