Pi-hole Blocklist Optimizer

A fast, single-binary tool for downloading, optimizing, and organizing blocklists for Pi-hole

Features • Installation • Quick Start • Whitelist • Configuration • Usage

What Does This Tool Do?

1. Downloads blocklists from multiple sources (async concurrent)
2. Validates domains and removes invalid entries
3. Optimizes by removing duplicates across all lists
4. Filters domains using your whitelist (exact, wildcard, regex)
5. Organizes into categories (advertising, tracking, malicious, etc.)
6. Combines into production-ready lists

Features

• Async concurrent downloads — configurable 1-16 concurrent connections
• Whitelist support — filter domains with exact matches, wildcards, or regex patterns
• Incremental updates — only re-download changed lists (ETag/Last-Modified support)
• Multi-format support — handles hosts files, AdBlock, and plain domain formats
• Progress tracking — resume interrupted downloads
• Detailed reporting — statistics and whitelist match reports
• Error recovery — automatic retry with exponential backoff
• Single binary — no runtime dependencies, statically linked TLS

Installation

From GitHub Releases (Recommended)

Download the latest binary for your platform from Releases:

# Linux x86_64
curl -LO https://github.com/zachlagden/Pi-hole-Blocklist-Optimizer/releases/latest/download/pihole-optimizer-linux-x86_64
chmod +x pihole-optimizer-linux-x86_64
sudo mv pihole-optimizer-linux-x86_64 /usr/local/bin/pihole-optimizer

# Linux ARM (Raspberry Pi)
curl -LO https://github.com/zachlagden/Pi-hole-Blocklist-Optimizer/releases/latest/download/pihole-optimizer-linux-aarch64
chmod +x pihole-optimizer-linux-aarch64
sudo mv pihole-optimizer-linux-aarch64 /usr/local/bin/pihole-optimizer

From Source

Requires Rust (stable).

git clone https://github.com/zachlagden/Pi-hole-Blocklist-Optimizer
cd Pi-hole-Blocklist-Optimizer
cargo build --release
# Binary at target/release/pihole-optimizer

Quick Start

pihole-optimizer

That's it! The tool will:

1. Download blocklists from blocklists.conf
2. Validate and optimize domains
3. Apply whitelist filtering
4. Create production lists in pihole_blocklists_prod/

Whitelist Support

Create a whitelist.txt file to exclude domains from the final output. Three matching types are supported:

Exact Domains

example.com           # Matches example.com and *.example.com (subdomains)
google.com

Wildcard Patterns

*.tracking.com        # Matches any.tracking.com
ads.*                 # Matches ads.example.com, ads.site.net
*analytics*           # Matches myanalytics.com, analytics.site.net

Regex Patterns

/^track.*\.com$/      # Matches tracker.com, tracking.com
/.*\.ads\..*$/        # Matches sub.ads.example.com

Example whitelist.txt

# Exact domains (with subdomain matching)
github.com
googleapis.com

# Wildcards
*.cdn.example.com
*cloudfront*

# Regex patterns
/^api\..*\.com$/

Run with --whitelist-report to see which domains were filtered and by which patterns.

Whitelist entries match ABP rules by their domain, so example.com removes ||example.com^ and, with subdomain matching, ||ads.example.com^. A whitelist entry for a subdomain does not remove a rule for its parent. For example, ads.example.com leaves ||example.com^ in place, and that rule still blocks ads.example.com. Use Pi-hole's own allowlist for that case.

Configuration

blocklists.conf

Define blocklist sources in blocklists.conf:

url|name|category[|flags]

Example:

https://adaway.org/hosts.txt|adaway|advertising
https://someonewhocares.org/hosts/hosts|someonewhocares|comprehensive
https://raw.githubusercontent.com/you/lists/main/custom.txt|custom|malicious|abp

Categories: advertising, tracking, malicious, suspicious, nsfw, comprehensive

Optional flags (4th field): abp enables ABP-style wildcard entries for that source. On an abp source, the lines ||domain^ and *.domain block the domain and all its subdomains (emitted as ||domain^ in the output); without the flag, those forms flatten to an exact domain. Use only on trusted, curated sources. ABP-style entries require Pi-hole Core ≥ 5.16 / FTL ≥ 5.22 (released 2023).

Production lists leave out any entry that a ||domain^ rule in the same list already covers. If a list has ||example.com^, the entries example.com, ads.example.com and ||cdn.example.com^ are dropped, because the rule already blocks them.

After adding or removing the abp flag on a source, run once with --no-incremental. Otherwise an unchanged source is rebuilt from the cached file that was written under the old setting.

Lines starting with # are ignored.

Usage

Basic

pihole-optimizer

With Options

# Fast download with 8 threads and whitelist report
pihole-optimizer -t 8 --whitelist-report

# Custom config and output directory
pihole-optimizer -c myconfig.conf -p /var/blocklists

# Verbose logging
pihole-optimizer -v

# Dry run
pihole-optimizer --dry-run

All Options

Usage: pihole-optimizer [OPTIONS]

Options:
  -c, --config <CONFIG>         Configuration file path [default: blocklists.conf]
  -w, --whitelist <WHITELIST>   Whitelist file path [default: whitelist.txt]
  -b, --base-dir <BASE_DIR>    Base output directory [default: pihole_blocklists]
  -p, --prod-dir <PROD_DIR>    Production output directory [default: pihole_blocklists_prod]
  -t, --threads <THREADS>      Concurrent downloads 1-16 [default: 4]
      --timeout <TIMEOUT>      HTTP timeout in seconds [default: 30]
      --skip-download          Use existing local files
      --skip-optimize          Skip creating production lists
      --no-incremental         Force re-download all lists
      --dry-run                Show what would happen without doing it
      --no-whitelist-subdomain Disable subdomain matching in whitelist
      --whitelist-report       Generate detailed whitelist match report
      --abp-lists <CATEGORIES> Also emit ABP-style variants (e.g. nsfw) that block subdomains
  -v, --verbose                Debug logging
  -q, --quiet                  Errors only
  -h, --help                   Print help
  -V, --version                Print version

--abp-lists takes a comma-separated list of categories (e.g. --abp-lists nsfw). For each, the optimizer writes an additional <category>_abp.txt in the production directory where every entry is in ABP form (||domain^) so it blocks the domain and all its subdomains. Subdomains of a domain already in the file are left out, so the ABP file is usually much shorter than the normal one. The normal <category>.txt is still written.

In Pi-hole, add <category>_abp.txt instead of <category>.txt, not alongside it. The ABP file blocks everything the normal file blocks, so enabling both stores every domain twice and blocks nothing extra.

ABP entries require Pi-hole Core ≥ 5.16 / FTL ≥ 5.22, and are more aggressive. Use them for categories of dedicated domains (like NSFW) rather than lists that contain shared, multi-tenant hosts.

--skip-download rebuilds the production lists from the per-source files already in the base directory, without using the network. Use it to try different flags. Scheduled runs don't need it, because a normal run only downloads sources that have changed.

Output Structure

pihole_blocklists/              # Individual optimized lists
├── advertising/
├── tracking/
├── malicious/
├── suspicious/
├── nsfw/
└── comprehensive/

pihole_blocklists_prod/         # Combined production lists
├── all_domains.txt             # All unique domains (excludes NSFW)
├── advertising.txt
├── tracking.txt
├── malicious.txt
├── suspicious.txt
├── nsfw.txt                    # Separate — not included in all_domains.txt
├── comprehensive.txt
└── whitelist_report.txt        # (if --whitelist-report used)

Using with Pi-hole

Option 1: Use Pre-built Lists (Recommended)

Use the companion repository Pi-hole-Optimized-Blocklists which runs this optimizer weekly and hosts the results.

Add these URLs to Pi-hole's Adlists:

https://media.githubusercontent.com/media/zachlagden/Pi-hole-Optimized-Blocklists/main/lists/all_domains.txt

Option 2: Self-Host

Run the optimizer and host the files on your own server, then add the URLs to Pi-hole.

Option 3: Local Files

sudo cp pihole_blocklists_prod/*.txt /etc/pihole/
pihole -g

Performance

• 30+ blocklists, 1.6M+ unique domains processed in ~20 seconds
• Async I/O with configurable concurrency
• ~5MB self-contained binary, no runtime dependencies

Troubleshooting

Issue	Solution
Connection errors	Check internet, try fewer threads (-t 2)
Slow downloads	Increase threads (-t 8)
Missing domains	Check whitelist isn't too broad

Contributing

Contributions welcome! Open an issue or submit a PR.

git clone https://github.com/zachlagden/Pi-hole-Blocklist-Optimizer
cd Pi-hole-Blocklist-Optimizer
cargo build
cargo test
cargo fmt --check

# Enable pre-commit hooks
git config core.hooksPath .githooks

License

This project is licensed under the MIT License — see the LICENCE file for details.

Acknowledgements

• Blocklist maintainers listed in the configuration file
• Pi-hole team