Skip to content

createRowStream is suboptimal #12

Description

@vweevers

Because it sets highWaterMark to 0 (though documented as 16):

lento/lib/client.js

Lines 176 to 180 in 8670652

createRowStream (sql, opts) {
const pageOpts = Object.assign({}, opts, {
paging: false,
highWaterMark: 0,
})

This makes sense for a page stream, because it means it won't make preemptive HTTP requests. But for
a row stream it means it'll push only one row (or "chunk") at a time, possibly hurting throughput:

// Push single rows
while (this._buffer !== undefined && this._bufferIndex < this._buffer.length) {
const chunk = this._buffer[this._bufferIndex++]
if (!this.push(chunk)) {
this._reading = false
return
}
}

I.e. the while loop above never loops. It might be easier to push everything at once (ignoring the return value of push). It's already occupying memory anyway.

Also, when the next readable-stream is out (with nodejs/node@1e0f331), I can maybe get rid of:

// See https://github.com/nodejs/node/issues/3203
if (this._reading) return
else this._reading = true

For now, I recommend to use createPageStream instead.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions