Add WARC datatype (warc, warc.gz)#23033
Draft
jnussbaum wants to merge 1 commit into
Draft
Conversation
WARC (Web ARChive) files store archived web content together with the metadata describing how it was captured, and are commonly distributed gzip-compressed (each record as a separate gzip member). The gzipped form must be kept compressed on upload: decompressing it and applying line-ending normalization corrupts the per-record byte offsets and the exact Content-Length values that WARC tooling relies on. WarcGz therefore sets compressed=True so the upload pipeline preserves the file verbatim. Adds: - Warc (uncompressed) and WarcGz (gzip, kept compressed) datatypes with sniffers - registration in datatypes_conf.xml.sample - test fixtures (example.warc, example.warc.gz) and doctests/unit tests Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What did you do?
During the GCC Cofest at Clermont-Ferrand, @eamonnbell-dur and me started writing a wrapper for the warcio tool (see galaxyproject/tools-iuc#8143). It failed because of a missing datatype for WARC files. @wm75 helped us to identify the problem and suggested that we open this PR to add the missing datatype.
Add datatypes for the WARC (Web ARChive) format:
warc— uncompressed WARC (galaxy.datatypes.warc:Warc)warc.gz— gzip-compressed WARC (galaxy.datatypes.warc:WarcGz)Both register sniffers (the format signature is the
WARC/record prefix), with test fixtures, doctests, and unit tests.Why is this needed?
WARC files store archived web content together with capture metadata, and are most commonly distributed gzip-compressed (often with each record as a separate gzip member).
The gzipped form must be kept compressed on upload. If Galaxy decompresses it and then applies POSIX line-ending normalization (because the decompressed content sniffs as text), the per-record byte offsets shift and the exact
Content-Lengthbyte counts no longer match — which breaks WARC parsers such aswarcio.WarcGztherefore setscompressed = True(modeled on the existingAthenaProjectgzip datatype) so the upload pipeline preserves the bytes verbatim.This unblocks a Galaxy tool wrapper for
warcio index(intended for tools-iuc), which needs the.warc.gzdelivered untouched to report record offsets.How was it tested?
pytest --doctest-modules lib/galaxy/datatypes/warc.py— sniffers passpytest test/unit/data/datatypes/test_warc.py— positive/negative sniff tests passruff check,ruff format --check,isort --check— cleanplanemo test --galaxy_rootagainst thewarcio indexwrapper: the.warc.gzis delivered byte-identical and the tool test passes.License