Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs cleanup #480

Merged
merged 6 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 85 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Summary

KvikIO is a Python and C++ library for high performance file IO. It provides C++ and Python
KvikIO (pronounced "kuh-VICK-eye-oh", see [here](https://ordnet.dk/ddo_en/dict?query=kvik) for pronunciation of kvik) is a Python and C++ library for high performance file IO. It provides C++ and Python
bindings to [cuFile](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html),
which enables [GPUDirect Storage (GDS)](https://developer.nvidia.com/blog/gpudirect-storage/).
KvikIO also works efficiently when GDS isn't available and can read/write both host and device data seamlessly.
Expand All @@ -15,9 +15,92 @@ The C++ library is header-only making it easy to include in [existing projects](
* A Python [Zarr](https://zarr.readthedocs.io/en/stable/) backend for reading and writing GPU data to file seamlessly.
* Concurrent reads and writes using an internal thread pool.
* Non-blocking API.
* Handle both host and device IO seamlessly.
* Transparently handles reads and writes to/from memory on both host and device.
* Provides Python bindings to [nvCOMP](https://github.com/NVIDIA/nvcomp).


### Documentation
* Python: <https://docs.rapids.ai/api/kvikio/nightly/>
* C++: <https://docs.rapids.ai/api/libkvikio/nightly/>


### Examples

#### Python
```python
import cupy
import kvikio

def main(path):
a = cupy.arange(100)
f = kvikio.CuFile(path, "w")
# Write whole array to file
f.write(a)
f.close()

b = cupy.empty_like(a)
f = kvikio.CuFile(path, "r")
# Read whole array from file
f.read(b)
assert all(a == b)
f.close()

wence- marked this conversation as resolved.
Show resolved Hide resolved
# Use contexmanager
c = cupy.empty_like(a)
with kvikio.CuFile(path, "r") as f:
f.read(c)
assert all(a == c)

# Non-blocking read
d = cupy.empty_like(a)
with kvikio.CuFile(path, "r") as f:
future1 = f.pread(d[:50])
future2 = f.pread(d[50:], file_offset=d[:50].nbytes)
# Note: must wait for futures before exiting block
# at which point the file is closed.
future1.get() # Wait for first read
future2.get() # Wait for second read
wence- marked this conversation as resolved.
Show resolved Hide resolved
assert all(a == d)


if __name__ == "__main__":
main("/tmp/kvikio-hello-world-file")
```

#### C++
```c++
#include <cstddef>
#include <cuda_runtime.h>
#include <kvikio/file_handle.hpp>
using namespace std;

int main()
{
// Create two arrays `a` and `b`
constexpr std::size_t size = 100;
void *a = nullptr;
void *b = nullptr;
cudaMalloc(&a, size);
cudaMalloc(&b, size);

// Write `a` to file
kvikio::FileHandle fw("test-file", "w");
size_t written = fw.write(a, size);
fw.close();

// Read file into `b`
kvikio::FileHandle fr("test-file", "r");
size_t read = fr.read(b, size);
fr.close();

// Read file into `b` in parallel using 16 threads
kvikio::default_thread_pool::reset(16);
{
// FileHandles have RAII semantics
kvikio::FileHandle f("test-file", "r");
wence- marked this conversation as resolved.
Show resolved Hide resolved
future<size_t> future = f.pread(b_dev, sizeof(a), 0); // Non-blocking
size_t read = future.get(); // Blocking
// Notice, `f` closes automatically on destruction.
}
}
```
7 changes: 7 additions & 0 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,13 @@
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None

autodoc_default_options = {
'members': True,
'member-order': 'bysource',
'special-members': '__init__',
'undoc-members': True,
'exclude-members': '__weakref__'
}

# -- Options for HTML output -------------------------------------------------

Expand Down