Reading in chunks example, memory usage #1061
Description
I'm using the example chunks examples (11 and 12), to read a large file (1M+ rows) in chunks, I've noticed that after enough rows have been read, I'll run out of memory. In my test case somewhere between 20k and 30k rows the crash will occur, but that would vary with the columns and data I'm sure.
I thought maybe the load step was loading all of the rows until the last row needed, so I adjusted my start point to 30k assuming it would crash immediately. However, it did not run out of memory until somewhere between 50k and 60k rows had been loaded, roughly the same number of rows that were loaded before.
I need to be able to iterate through entire large files, but I'm not sure where the memory is going. I'm not sure if this is a memory leak or the example isn't meant for what I'm trying to do.