Skip to content

About

PDF to Markdown scraper for Python and the command line: give a domain and get every PDF, Word, Excel and CSV document the site publishes, converted to clean Markdown with headings, tables, lists and links, ready for an LLM or a RAG index, as JSON or CSV. Free and open source; no proxies or headless browser to run.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

PDF to Markdown Scraper

ci python license dependencies

Turn the documents a website publishes into Markdown, from Python or the command line: give a domain and get back every PDF, Word, Excel and CSV file linked from its pages and its sitemap, each converted to clean Markdown (headings, tables, lists, links) with its title, page count, word count and the page that links it. It works by domain, up to 50 documents per run, ready for an LLM, a RAG index or a search over a site's reports and forms. The code is free and open source (MIT). It runs on the QuanticData Data API, so there is nothing to download or parse on your side: sign in once and run a command.

pip install git+https://github.com/quantumproxies/pdf-to-markdown
pdf-to-markdown login
pdf-to-markdown documents bitcoin.org --doc-types pdf -o bitcoin_docs.jsonl

login opens the QuanticData sign-in page in your browser. New accounts get $2 of free usage every month, no card required: create one.

What it scrapes

Command Input Returns Per run
pdf-to-markdown documents domain or site URL Markdown, title, kind, pages, words, where it is linked, 12 fields per row (fields and price) up to 50

--doc-types pdf docx xlsx csv picks the formats (default PDF and Word). Each document is one row with its Markdown in markdown; -o docs.jsonl keeps one document per line, the usual input of a RAG loader.

Sample output

A real row from pdf-to-markdown documents bitcoin.org --doc-types pdf --max-results 2 (2026-10-23):

{
  "url": "https://bitcoin.org/bitcoin.pdf",
  "kind": "pdf",
  "chars": 21711,
  "pages": 9,
  "title": "Bitcoin: A Peer-to-Peer Electronic Cash System",
  "words": 3656,
  "author": null,
  "source": "sitemap",
  "found_on": null,
  "markdown": "# Bitcoin: A Peer-to-Peer Electronic Cash System\n\n**Abstract.** A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending.",
  "truncated": false,
  "scanned_pages": 0
}

Python

from quanticdata_pdf_to_markdown import PdfToMarkdown

pdfs = PdfToMarkdown()  # saved login, or QUANTICDATA_API_KEY

docs = pdfs.documents("bitcoin.org", doc_types=["pdf"], max_results=10)
for d in docs:
    print(d["kind"], d["pages"], d["words"], d["title"], d["url"])

print(docs.rows[0]["markdown"][:500])
docs.to_jsonl("bitcoin_docs.jsonl")

Each call returns a Run: iterate it for the rows (plain dicts), take len(), or save it with to_csv(), to_json(), to_jsonl() or to_dataframe() (pandas). Errors are specific: InsufficientCredit, RateLimited, InvalidInput, JobFailed, AuthError.

Examples

  • A site's documents as .md files: Save every PDF and Word file of a site as a Markdown file named after its URL, with an index.csv of titles, page counts and word counts. See examples/save_markdown.py.
  • Chunks for a RAG index: Split each document's Markdown at its headings into chunks of bounded size, with source URL, title and section, one JSON line per chunk, ready for an embedding job. See examples/rag_chunks.py.
  • Document inventory: List what several sites publish (type, title, pages, words, where each file is linked) in one CSV, with the Markdown kept short. See examples/document_inventory.py.

Command line

Results print as a table; -o file.csv, -o file.json or -o file.jsonl writes a file. List inputs take a b c, a,b,c, @file.txt (one per line) or - (stdin). --no-wait starts a long run in the background and prints its id; pdf-to-markdown run <id> -o out.csv collects it later.

pdf-to-markdown documents

Every PDF and Office document a domain publishes, converted to clean LLM-ready Markdown.

pdf-to-markdown documents DOMAIN [options]
Option Type Default Description
DOMAIN string Domain or site section to collect documents from, e.g. bitcoin.org or www.irs.gov/forms-instructions.
--doc-types pdf / docx / xlsx / csv Which document types to collect: pdf, docx, xlsx, csv (default pdf and docx).
--max-chars-per-doc integer 40000 Cap on the Markdown returned per document, 1,000 to 200,000 characters (default 40,000); truncated says when a document reached it.
--country string Country to run the request from, as an ISO 3166-1 alpha-2 code (us, gb, de...).
--max-results integer 10 How many documents to deliver at most (1–50). You pay only for delivered documents.

Output fields

pdf-to-markdown documents: 12 fields
Field Type Description
url string Document URL.
source string Where the link was found: page | sitemap.
kind string pdf | docx | xlsx | csv.
title string or null Document title (PDF metadata or first heading).
markdown string The document converted to Markdown (headings, tables, lists, links).
chars integer Markdown length in characters.
words integer Word count of the Markdown.
pages integer or null Page count (PDF only).
author string or null Author from document metadata, when present.
scanned_pages integer Image-only pages, without a text layer.
truncated boolean True when the content hit max_chars_per_doc or the page cap.
found_on string or null The page of the site that links the document (null when it came from the sitemap).

Sign-in and API keys

  • pdf-to-markdown login signs you in with OAuth 2.1 and PKCE in the browser. The token is stored in ~/.config/quanticdata/credentials.json (readable only by you) and refreshed automatically. One login serves every QuanticData tool on the machine.
  • No browser on the machine (SSH, Docker, Colab): run pdf-to-markdown login --no-browser, open the link anywhere, then paste back the address the browser ends on.
  • Scripts and CI: set QUANTICDATA_API_KEY to a key from the dashboard, or store one with pdf-to-markdown login --api-key.
  • pdf-to-markdown logout revokes the token. The connection is listed under API keys in the dashboard as "API · QuanticData CLI"; deleting that key disconnects every machine that uses it.

Pricing

The code is free and MIT-licensed. Runs are billed to your QuanticData account per delivered row, at the price shown on each collector page linked above; $2 of usage is free every month and failed runs are never billed. Plans with higher limits: pricing.

Other ways to call it

The same data over plain HTTP:

curl -X POST https://api.quanticdata.io/v1/scraper/collectors/site_documents/run \
  -H "Authorization: Bearer $QUANTICDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": {"domain": "bitcoin.org", "doc_types": ["pdf"], "max_results": 2}}'
  • API reference with Python, Node, PHP and curl examples.
  • MCP server to give Claude, Cursor or any MCP client the same tools.
  • n8n node for no-code workflows.

FAQ

Is this PDF to Markdown scraper free?

The code is free and MIT-licensed. Scraping runs on your QuanticData account: $2 of usage is free every month with no card, then you pay per delivered row at the price shown on each collector page. Failed runs are never billed.

Which file types does it read?

PDF, Word (.docx), Excel (.xlsx) and CSV, chosen with --doc-types; the default is PDF and Word.

How does it find the documents?

From the links on the site's pages and from its sitemap. Each row says which (source) and, for a linked file, the page that links it (found_on).

Are tables and headings kept?

Yes. Headings, tables, lists and links are converted to Markdown, and the row adds the document title, page count, word count and author when the file states one.

How much can one run return?

Up to 50 documents per run and up to 200,000 characters of Markdown per document (--max-chars-per-doc, default 40,000); ask for fewer documents with --max-results and you pay only for what is delivered.

Can I export the Markdown?

Add -o docs.jsonl for one document per line, or -o docs.csv with the Markdown in one column; examples/save_markdown.py writes one .md file per document.

Related scrapers

License

MIT. Built and maintained by QuanticData.

About

PDF to Markdown scraper for Python and the command line: give a domain and get every PDF, Word, Excel and CSV document the site publishes, converted to clean Markdown with headings, tables, lists and links, ready for an LLM or a RAG index, as JSON or CSV. Free and open source; no proxies or headless browser to run.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages