3gb duplicate fails on decompression with "Failed to decompress buffer - lzmaerr=6"

[terje2001]

I have several large lzr files that I am unable to decompress. Every time I get the same error message:
Failed to decompress buffer - lzmaerr=6

I have been able to reproduce with a simple script that uses zero byte files and large duplications. Please note that my failing lzr archives do not have zero filled files but they do have very large files that are duplicated (for various reasons not in my control). I was of course using lrzip since this gives wildly good compression for my use case.

Script to reproduce:

#!/bin/bash

fallocate -l $((1*1024*1024)) test1a.dat

# WORKS if slightly less than 3gb:
#fallocate -l $((3*1024*1024*1024 - 1024)) test2a.dat

# Fails if 3gb or more:
fallocate -l $((3*1024*1024*1024)) test2a.dat

# Duplicate 2nd file exactly
cp --reflink=always test2a.dat test2b.dat

# Create zip sorted so duplicates are close:
tar cvf test.tar --sort=name test*.dat

# Compress never fails, however decompress below fails if we include '-p 1'
lrzip -p 1 test.tar

# Fails with "Failed to decompress buffer - lzmaerr=6"
lrzip -t test.tar.lrz

The same failure can be achieved if the file is filled with /dev/urandom too, but takes considerably more time. Also, unsure if the -p 1 is always required or just in this reproducable case, since I have seen my real lrz files fail with cases where there were no 3gb duplications and compression was done without that flag.

Output when I run the scripts above:

+ fallocate -l 1048576 test1a.dat
+ fallocate -l 3221225472 test2a.dat
+ cp --reflink=always test2a.dat test2b.dat
+ tar cvf test.tar --sort=name test1a.dat test2a.dat test2b.dat
test1a.dat
test2a.dat
test2b.dat
+ lrzip -p 1 test.tar
Output filename is: test.tar.lrz
test.tar - Compression Ratio: 10668.734. Average Compression Speed: 46.553MB/s.
Total time: 00:02:11.92
+ lrzip -t test.tar.lrz
Decompressing...
Failed to decompress buffer - lzmaerr=6
Failed to decompress buffer - lzmaerr=6
Failed to decompress in ucompthread
Fatal error - exiting

Machine used:

• Up to date Arch Linux
• 64gb RAM, AMD Ryzen 7 5825U
• Kernel: 6.13.7-arch1-1
• lrzip 0.651-3 (from Arch Extra repo)

test.tar.lrz.gz

[ckolivas]

Thanks for this report. This is a serious issue indeed. Unfortunately I do not know when I'll have the time and energy to investigate, and no one else appears to understand the internal workings of lrzip well enough to debug it for the time being.

[ckolivas]

Oddly enough I have tried to reproduce this locally with your commands (minus the reflink which ext4 doesn't support) and the archive I create decompresses fine, but I can see yours fails to decompress. Additionally, yours is 603959 bytes where mine turns out 454609 bytes. It varies slightly from run to run since lrzip is not deterministic but not as much as yours. Run from latest git.

../lrzip -vvp 1 test.tar
The following options are in effect for this COMPRESSION.
Threading is DISABLED. Number of CPUs available detected: 1
Detected 134723981312 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Max Verbose
Temporary Directory set as: ./
Compression mode is: LZMA. LZ4 Compressibility testing enabled
Heuristically Computed Compression Window: 856 = 85600MB
Storage time in seconds 1413104173
Output filename is: test.tar.lrz
File size: 6443509760
Succeeded in testing 6443509760 sized mmap for rzip pre-processing
Will take 1 pass
Chunk size: 6443509760
Byte width: 5
Succeeded in testing 6642755584 sized malloc for back end compression
Using only 1 thread to compress up to 6443509760 bytes
Beginning rzip pre-processing phase
hashsize = 4194304. bits = 22. 64MB
73 total hashes -- 60 in primary bucket (82.192%)
Malloced 44907991040 for checksum ckbuf
Starting thread 0 to compress 541058 bytes from stream 0
lz4 testing OK for chunk 541058. Compressed size = 0.58% of chunk, 1 Passes
Starting lzma back end compression thread...
Writing initial chunk bytes value 5 at 24
Writing EOF flag as 1
Writing initial header at 31
Compthread 0 seeking to 11 to store length 5
Compthread 0 seeking to 32 to write header
Thread 0 writing 240 compressed bytes from stream 0
Compthread 0 writing data at 48
Starting thread 0 to compress 3219137006 bytes from stream 1
lz4 testing OK for chunk 3219137006. Compressed size = 0.68% of chunk, 1 Passes
Starting lzma back end compression thread...
Compthread 0 seeking to 27 to store length 5
Compthread 0 seeking to 288 to write header
Thread 0 writing 454258 compressed bytes from stream 1
Compthread 0 writing data at 304
MD5: 861540c307aaf88900efbec190d911fc
matches=49208 match_bytes=3224372754
literals=49130 literal_bytes=3219137006
true_tag_positives=9 false_tag_positives=61
inserts=128 match 1.002
test.tar - Compression Ratio: 14173.740. Average Compression Speed: 19.887MB/s.
Total time: 00:05:09.61

../lrzip -vvt test.tar.lrz
Threading is ENABLED. Number of CPUs available detected: 32
Detected 134723981312 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Max Verbose
Test file integrity
Temporary Directory set as: ./
Created temporary outfile ./lrzipout.Rv5DbO
Detected lrzip version 0.6 file.
MD5 being used for integrity testing.
Decompressing...
Reading chunk_bytes at 24
Expected size: 6443509760
Chunk byte width: 5
Reading eof flag at 25
EOF: 1
Reading expected chunksize at 26
Chunk size: 6443509760
Reading stream 0 header at 32
Reading stream 1 header at 48
Reading ucomp header at 64
Fill_buffer stream 0 c_len 240 u_len 541058 last_head 0
Starting thread 0 to decompress 240 bytes from stream 0
Thread 0 decompressed 541058 bytes from stream 0
Taking decompressed data from thread 0
Reading ucomp header at 320
Fill_buffer stream 1 c_len 454258 u_len 3219137006 last_head 0
Starting thread 1 to decompress 454258 bytes from stream 1
Thread 1 decompressed 3219137006 bytes from stream 1
Taking decompressed data from thread 1
Closing stream at 454593, want to seek to 454593

Average DeCompression Speed: 227.593MB/s
MD5: 861540c307aaf88900efbec190d911fc
[OK] - 6443509760 bytes
Total time: 00:00:27.86