You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Yes, this is the intended behaviour. batchSize and highWaterMark control the behaviour in the underlying db queries, so tuning the batch size and high water mark can improve throughput/performance, but the "chunks" returned in the stream are still individual records. If you need to batch them, you can write a separate utility that pulls a set number of items from the stream and emits that array as a chunk. Something like:
P.S. the batchSize is how many records to fetch at a time, the highWaterMark is how many records to allow the stream to buffer before applying back pressure (if you are consuming items slower than they are being returned form the database). 16 * 1024 would be a very large number of records to have as your high water mark if your batch size is only 10.
In:
I expect
chunk
to be an array of 10 items, but instead it is just one item as a plain object instead of arrayThe text was updated successfully, but these errors were encountered: