Simplify bench/ann scripts to Python based module
#1642
Conversation
divyegala
added
improvement
cpp/bench/ann/scripts/get_dataset.py
Outdated
| def convert_hdf5_to_fbin(path, normalize): | ||
| if normalize and "angular" in path: | ||
| p = subprocess.Popen(["python", "scripts/hdf5_to_fbin.py", "-n", | ||
| "%s" % path]) | ||
| else: | ||
| p = subprocess.Popen(["python", "scripts/hdf5_to_fbin.py", | ||
| "%s" % path]) | ||
| p.wait() |
There was a problem hiding this comment.
Instead of invoking via a subprocess - can we just call the python function directly?
There was a problem hiding this comment.
Unfortunately, that script doesn't have a callable function. Would you prefer I refactor the script to make it work or we can do it later?
Co-authored-by: Ben Frederickson <[email protected]>
| # (1) prepare dataset | ||
| # download manually "Ground Truth" file of "Yandex DEEP" | ||
| # suppose the file name is deep_new_groundtruth.public.10K.bin | ||
| ../../scripts/split_groundtruth.pl deep_new_groundtruth.public.10K.bin groundtruth |
There was a problem hiding this comment.
We should wrap this in python for consistency. It's just confusing seeing a bunch of python scripts and then seeing pearl.
There was a problem hiding this comment.
I think we should slip this task to a follow up PR
There was a problem hiding this comment.
I think that can work as an immediate follow-up. I'd prefer for it to still be worked into 23.08. We don't necessarily need to support every unique parameter combination initially- so long as we support what's needed to run a basic benchmark end-to-end for billion-scale.
| mamba env create --name raft_ann_benchmarks -f conda/environments/bench_ann_cuda-118_arch-x86_64.yaml | ||
| conda activate raft_ann_benchmarks | ||
|
|
||
| mamba install -c rapidsai libraft-ann-bench |
There was a problem hiding this comment.
Would it be realistic to setup a conda environment that does not depend on cuda conda packages and uses system cuda installation instead? I'd love to be able to use these scripts using docker containers with latest cuda drivers. In fact, I think this would be the main use case on devtech side: to test and adjust raft implementation for the upcoming hardware.
There was a problem hiding this comment.
Sure, with some testing we can come up with a more minimal environment that does not include any cuda conda packages. Let me know if we can work on this together.
|
/merge |
Closes #1633