highWaterMark not considered during read

[jjmartin]

I'd like to use this in conjunction with the pg-promise library and one of the things i'm trying to do is stream data from a redshift table to a dynamoDb table in AWS, but one of the issues i'm having is when i am using the promises is that the buffer of the readable stream is filling up more than i want at a time. You are implementing the readable._read(n) which (n) is supposed to take the highWatermark.

// abstract method.  to be overridden in specific implementation classes.
// call cb(er, data) where data is <= n in length.
// for virtual (non-string, non-buffer) streams, "length" is somewhat
// arbitrary, and perhaps not very meaningful.
Readable.prototype._read = function (n) {
  this.emit('error', new Error('not implemented'));
};

It seems like you should look at n and the current amount of recorrds (state.length? ) before doing the self.push..

[aheinz-fe]

Instantiating QueryStream(text, values, { batchSize: 16 }) seems to have resolved my own memory consumption issues. The batchSize normally defaults to 100; quite a bit higher than the highWaterMark default of 16.