-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
readFile will not read files larger than 2 GiB even if buffers can be larger #55864
Comments
This is a documentation issue. The 2GB limit is not for the Buffer, but rather an I/O limit. |
@RedYetiDev @davazp can I update de documentation? |
It would probably be good to always include the current limit in the error message. |
The reason is explained here: libuv/libuv#1501 |
I imagined it was something like that. On Linux it also seems that the read syscall is limited to 2GiB. However that is pretty low level. Wouldn’t be better if Even if we do not want to allow arbitrarily large files, they would allow us to increase the limit to something a bit more forgiving. |
handling multiple calls from the Node.js side SGTM. |
To read large files in chunks (greater than 2 GiB) using Node.js, you can use fs.createReadStream() to handle the file in smaller, manageable chunks rather than loading the entire file into memory. This avoids hitting the 2 GiB limit that occurs when using fs.readFile(), as it loads the entire file into memory. const fs = require('fs'); const filePath = 'path/to/large/file'; // Specify the path to your large file stream.on('data', (chunk) => { stream.on('end', () => { stream.on('error', (err) => { |
Version
v22.11.0
Platform
Subsystem
No response
What steps will reproduce the bug?
How often does it reproduce? Is there a required condition?
It is deterministic.
What is the expected behavior? Why is that the expected behavior?
readFile should allow for files as large as the max buffer size, as according to the documentation:
https://nodejs.org/api/errors.html#err_fs_file_too_large
In newer node versions, the maximum buffer has increased but the maximum file size is still capped at 2 GiB
In older versions (v18), the max buffer size on 64bit platforms was 4GB, but files cannot be that large either.
What do you see instead?
readFile
will throw the errorAdditional information
No response
The text was updated successfully, but these errors were encountered: