Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Legate Zarr #198

Merged
merged 13 commits into from
Apr 25, 2023
Merged

Legate Zarr #198

merged 13 commits into from
Apr 25, 2023

Conversation

madsbk
Copy link
Member

@madsbk madsbk commented Apr 18, 2023

Read and write Zarr arrays to/from cuNumeric arrays.

Example

import cunumeric as num
import legate.core
import legate_kvikio.zarr

a = num.arange(10000).reshape(100, 100)

# Write array to a Zarr file by chunks of 10x10.
legate_kvikio.zarr.write_array(a, "/tmp/myfile.zarr", chunks=(10, 10))

# Block until done writing.
legate.core.get_legate_runtime().issue_execution_fence(block=True)

# Read array from a Zarr file.
b = legate_kvikio.zarr.read_array("/tmp/myfile.zarr")

# They should be equal
assert (a == b).all()

Limitations

  • Groups not supported -- only access to individual Zarr arrays are implemented.
  • Can only read and write whole arrays -- no partial access.
  • No compression -- only Zarr arrays written with compressor=None is supported.
  • Writing an array with shape that isn't divisible by the shape of the Zarr chunks on disk involves an array copy, which can be expensive both in terms of performance and memory usage.

@madsbk madsbk added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Apr 18, 2023
@madsbk madsbk marked this pull request as ready for review April 18, 2023 14:16
@quasiben
Copy link
Member

/merge

@rapids-bot rapids-bot bot merged commit 893b434 into rapidsai:branch-23.06 Apr 25, 2023
@madsbk madsbk deleted the legate_zarr branch April 26, 2023 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improves an existing functionality non-breaking Introduces a non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants