[Question]: pipeline

[Kyrie5e]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Describe your problem

I want to customize a parser that converts PDF, TXT, and Word files to Markdown, then performs header-based chunking, and finally embeds the data. Is this workflow possible? Or are there other approaches to automatically convert all files to Markdown before parsing?

[dosubot]

Yes, this workflow is possible in RAGFlow! Here's how you can achieve it:

1. Markdown Conversion
Both PDF and Word parsers support Markdown output via the output_format: "markdown" parameter. TXT files can be processed directly.

2. Header-Based Chunking
Use either:

• book chunk method: Implements hierarchical header detection via hierarchical_merge() for structure-aware chunking
• manual chunk method: Extracts PDF outlines/table of contents for outline-based chunking

3. Configuration via API
When uploading documents via the REST API, you can specify:

• chunk_method: Set to "book" or "manual"
• parser_config: Configure chunk_token_num, delimiter, layout_recognize, etc.

Embedding happens automatically after chunking in the RAGFlow pipeline.

Would you like more details on the specific API parameters or chunking configuration options?

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[Kyrie5e]

@dosu 具体流程如何操作

[dosubot]

以下是具体操作步骤：

方法一：通过 Web UI 操作

1. 进入 Dataset 标签页，点击 Create dataset
2. 选择嵌入模型，并将分块方法（chunking method）设置为 Book 或 Manual（适用于基于标题的分块）
3. 点击 + Add file → Local files 上传 PDF、TXT 或 Word 文件
4. 点击文件旁的 播放按钮 启动解析和嵌入

方法二：通过 REST API 操作

# 步骤 1：创建数据集并配置 book 分块方法
curl --request POST \
     --url http://{address}/api/v1/datasets \
     --header 'Content-Type: application/json' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --data '{
       "name": "my_dataset",
       "chunk_method": "book",
       "parser_config": {
         "chunk_token_num": 256
       }
     }'

# 步骤 2：上传文档
curl --request POST \
     --url http://{address}/api/v1/datasets/{dataset_id}/documents \
     --header 'Content-Type: multipart/form-data' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --form 'file=@./document.pdf'

# 步骤 3：触发解析和嵌入
curl --request POST \
     --url http://{address}/api/v1/datasets/{dataset_id}/chunks \
     --header 'Content-Type: application/json' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --data '{"document_ids": ["doc_id_1"]}'

注意事项：

• PDF 和 Word 文件会自动转换为 Markdown 格式
• 设置较小的 chunk_token_num（如 256）可以更好地保留标题结构
• 嵌入模型在创建数据集时指定，之后无法更改

更多 API 详情可参考 HTTP API 文档。

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[Magicbook1108]

You may not be able to implement them in a single workflow. However, it is quite simple to achieve this by using a document type transformation workflow together with a customized ingestion pipeline, as described by Dosu.

[Kyrie5e]

@Magicbook1108 Can I understand that the parser built into ragflow first converts different file types into Markdown format before splitting the text into blocks? So there's no need to manually convert to Markdown and then parse using General mode?

[Magicbook1108]

No, the parser converts all documents into sections. In RAGFlow, sections are the units we process further.

Check this file:
rag/flow/parser/parser.py