Fix memory leak on messages consuming

[abc94e4f31]

when best compression enabled it uses about 500Mb memory (Maximum resident set size (kbytes): 497300)

$ time -vvv rabbitio out -b 500 -u amqp://rabbitmq:rabbitmq@localhost/rabbitmq -e rabbitmq -r rabbtimq -q rabbtimq -d ./data
2029/11/11 11:22:31 RabbitMQ connected: amqp://rabbitmq:rabbitmq@localhost/rabbitmq
2029/11/11 11:22:31 Bind to Exchange: "rabbtimq" and Queue: "rabbtimq", Messaging waiting: 217093        
2029/11/11 11:22:32 Wrote 70010 bytes to data/1_messages_500.tgz 
2029/11/11 11:22:32 Wrote 70157 bytes to data/2_messages_500.tgz 
...
2029/11/11 11:22:56 Wrote 70034 bytes to data/433_messages_500.tgz 
2029/11/11 11:22:57 Wrote 70139 bytes to data/434_messages_500.tgz 
^CInterruption, saving last memory bits..
2029/11/11 11:22:25 All messages consumed
2029/11/11 11:22:27 Wrote 14038 bytes to data/435_messages_93.tgz 
2029/11/11 11:22:27 tarball writer closing
        Command being timed: "./rabbitio out -b 500 -u amqp://rabbitmq:rabbitmq@localhost/rabbitmq -e rabbtimq -r rabbtimq -q rabbtimq -d ./data/"                                        
        User time (seconds): 31.26
        System time (seconds): 3.09
        Percent of CPU this job got: 61%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:55.49
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 497300
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 146700
        Voluntary context switches: 743627
        Involuntary context switches: 3452
        Swaps: 0
        File system inputs: 0
        File system outputs: 62912
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

with fix:

$ time -vvv ./rabbitio-default-compression out -b 500 -u amqp://rabbitmq:rabbitmq@localhost/rabbitmq -e rabbtimq -r rabbtimq -q rabbtimq -d ./data/ 
2029/11/11 11:22:42 RabbitMQ connected: amqp://rabbitmq:rabbitmq@localhost/rabbitmq
2029/11/11 11:22:42 Bind to Exchange: "rabbtimq" and Queue: "rabbtimq", Messaging waiting: 180054
2029/11/11 11:22:42 Wrote 72686 bytes to data/1_messages_500.tgz
2029/11/11 11:22:42 Wrote 72893 bytes to data/2_messages_500.tgz
...
2029/11/11 11:22:53 Wrote 72687 bytes to data/359_messages_500.tgz
2029/11/11 11:22:53 Wrote 72736 bytes to data/360_messages_500.tgz
^CInterruption, saving last memory bits..
2029/11/11 11:22:56 All messages consumed
2029/11/11 11:22:58 Wrote 8891 bytes to data/361_messages_54.tgz
2029/11/11 11:22:58 tarball writer closing
        Command being timed: "./rabbitio-default-compression out -b 500 -u amqp://rabbitmq:rabbitmq@localhost/rabbitmq -e rabbtimq -r rabbtimq -q rabbtimq -d ./data/"
        User time (seconds): 13.66
        System time (seconds): 2.43
        Percent of CPU this job got: 98%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:16.42
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 22852
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 5680
        Voluntary context switches: 364269
        Involuntary context switches: 1544
        Swaps: 0
        File system inputs: 0
        File system outputs: 52216
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[vorce]

@abc94e4f31 thanks for the PRs!

@stiangrindvoll you wanna have a look at this and #38

[vorce]

@abc94e4f31 I can't find any bug reports on gzip.BestCompression causing memory leaks. Not sure this is a memory leak, the better compression just requires more memory. Or am I missing something? :)

[abc94e4f31]

@abc94e4f31 I can't find any bug reports on gzip.BestCompression causing memory leaks. Not sure this is a memory leak, the better compression just requires more memory. Or am I missing something? :)

I have made a simple gzip-like program to check is there any memory leak in 'compress/gzip/'. The program works correctly with constant memory consumption. It looks like the lower compression level has a side effect on memory usage, I guess, there is an issue somewhere in the getWriters function, perhaps in the tarball buffer. Also, the memory consumption of rabbtio out sometimes is up to 1.5 Gb