[WIP] Download FineWebPDFs

[praateekmahajan]

Description

Usage

python tutorials/finewebpdfs/download.py --download-dir raw_warcs --output-dir pdfs_output --limit 10 --verbose

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[praateekmahajan]

This is generated using

import json
import time
from datasets import load_dataset

dataset = load_dataset(
	"HuggingFaceFW/finepdfs",
	streaming=True,
	split="train"
).select_columns(["offset", "file_path"])

t0 = time.perf_counter()
with open("finepdfs.jsonl", "a") as f:
    for i, row in enumerate(dataset):
        f.write(json.dumps(row) + "\n")
        if i % 100_000 == 0:
            print(f"{i:,} records done in {(time.perf_counter()-t0):.2f}s")
            t0 = time.perf_counter()

[praateekmahajan]

I'm not sure why bunch of files are index files. So for now I've skipped them. A more thorough approach will include them too