rag/walkthrough.md.resolved at main · bskthefirst/rag · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
# Naver Blog Scraper Walkthrough

This guide explains how to use the Naver Blog Scraper for [xpfkwh56](https://blog.naver.com/xpfkwh56).

## Features
- **Scrapes all posts**: Starting from the "View All" list.
- **OCR Integration**: Extracts Korean text from images using `EasyOCR`.
- **Incremental Updates**: Tracks the last scraped post to only fetch new content on subsequent runs.
- **Markdown Output**: Saves posts as `YYYY-MM-DD_Title.md` with downloaded images.

## Prerequisite
Ensure dependencies are installed:
```bash
python3 -m pip install -r requirements.txt
```
*(Note: On first run, EasyOCR will download detection models which might take a few minutes)*

## Manual Execution

### Run Full Scrape (or First Run)
To start scraping:
```bash
python3 scraper.py
```

### Run with Limit
To test with just the latest 5 posts:
```bash
python3 scraper.py --limit 5
```

## Daily Automation

A shell script [run_daily.sh](file:///Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/run_daily.sh) is provided for automation. It runs the scraper in `--incremental` mode.

### 1. Test the script
```bash
./run_daily.sh
```

### 2. Schedule with Cron (Mac/Linux)
Open your crontab:
```bash
crontab -e
```
Add the following line to run daily at 9:00 AM (replace `/path/to/...` with your actual path):
```cron
0 9 * * * /Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/run_daily.sh >> /tmp/naver_scraper.log 2>&1
```

## RAG System (Search)

You can search through your scraped posts using the built-in RAG system.

### 1. Install RAG Dependencies
If you haven't already:
```bash
python3 -m pip install -r requirements.txt
```

### 2. Build the Index
Run this command whenever you scrape new posts to update the search index:
```bash
python3 rag_indexer.py
```
*Creates a local vector database in `chroma_db/`.*

### 3. Search
Query your blog archive:
```bash
python3 rag_search.py "your search query"
```

**Options:**
- `--k 5`: Return top 5 results (default 3).
- `--category "Category Name"`: Filter results by category.
  - Example: `python3 rag_search.py "study tips" --category "갓생추구"`

## Output Structure
- `posts/`: Contains Markdown files.
- [images/](file:///Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/scraper.py#127-183): Contains downloaded images.
- [state.json](file:///Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/state.json): Track the last scraped post ID.
- `chroma_db/`: Vector database for RAG.

## 4. Static RAG Deployment (GitHub Pages)

We have implemented a **Serverless RAG** system that runs entirely in the browser using `Transformers.js`.

### Key Components:
-   **[export_for_web.py](file:///Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/export_for_web.py)**: Converts the local ChromaDB index into a lightweight `documents.json` suitable for web use.
-   **[index.html](file:///Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/index.html)**: A simple frontend that loads `documents.json` and runs the embedding model (`all-MiniLM-L6-v2`) in the user's browser to perform semantic search.
-   **[.github/workflows/deploy.yml](file:///Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/.github/workflows/deploy.yml)**: Automates the entire process. It scrapes new posts, updates the index, exports it, and deploys the static site to GitHub Pages daily.

### Enhanced Search & Reader View
We have upgraded the frontend ([index.html](file:///Users/suhyun/.gemini/antigravity/brain/558fbe7a-dd4d-4a21-b204-2d47dcf92244/index.html)) to include:
-   **Contextual Snippets**: Search results now highlight the matching keywords in context, rather than just showing the first 300 characters.
-   **Reader View**: You can now read full posts directly on the site without leaving. Click "Read Saved Version" to open a modal with rendered Markdown.
-   **Browse Mode**: When not searching, the site displays a list of the latest 20 posts for easy browsing.

### How to Deploy:
1.  Push the code to your repository: `git push origin main`.
2.  Go to **GitHub Settings > Pages** and set source to `gh-pages` branch (once the Action runs).
3.  Visit your new RAG-powered blog archive!