0.5.2

• Filter out revisits

Full Changelog: 0.5.1...0.5.2

0.5.1

• Fix mismatch between crawling from local storage vs. S3

Full Changelog: 0.5.0...0.5.1

0.5.0

• Add archive_regexp, archive_blacklist_regexp; remove archive_disallow_regexp (#39) - possibly breaking change
• Ignore unrecognized index entries (#38)
• Fix reading compressed index files (#42)

Full Changelog: 0.4.1...0.5.0

0.4.1

What's Changed

• Fix for getting spider name in different scrapy versions

Full Changelog: 0.4.0...0.4.1

0.4.0

What's Changed

• Update WARC version by @leewesleyv in #28
• Crawl source information per item/page by @leewesleyv in #31
• Add settings to run a spider against previously generated archives by @leewesleyv in #32
• Seperate WARC and CDXJ module by @leewesleyv in #33

Full Changelog: 0.3.0...0.4.0

0.3.0

• Change _check_configuration_prerequisites logic in WaczExporter

Full Changelog: 0.2.0...0.3.0

v0.2.0 - Hotfixes

• Removed the unused smart-open dependency
• Adjusted logic to always use downloader middleware when SW_WACZ_SOURCE_URI is configured

Full Changelog: 0.1.0...0.2.0

Initial Release

• Save web crawls in WACZ format (multiple storages supported; local and cloud).
• Crawl against WACZ format archives.