Open
Description
We could add some APIs to allow access to the files in a BlobTree without the user having to actually download the whole dataset. That metadata is available.
Just a sketch, but:
JuliaHub.dataset_filetree(dataset) -> FileTree
: returns a data structure that can be used to iterate over the file tree (maybe based on https://github.com/shashi/FileTrees.jl).JuliaHub.download_dataset_file(dataset, filepath, dstfile)
: downloads an individual file from a BlobTree, without having to download the whole BlobTree in one go.
Activity