I'm not immediately sure what's going wrong here unfortunately. It seems like the version string from pp-sketchlib isn't as expected.

Can you run poppunk_sketch in your installation?
If you start a python session and run import pp_sketchlib what happens?

Copying out some relevant parts for reference:

'/bin/sh: 1: poppunk_sketch: not found\nTraceback (most recent call last):\n  File "/opt/conda/bin/poppunk_assign", line 10, in <module>\n    sys.exit(main())\n  File "/opt
/conda/lib/python3.9/site-packages/PopPUNK/assign.py", line 389, in main\n    dbFuncs = setupDBFuncs(args, args.min_kmer_count, qc_dict)\n  File "/opt/conda/lib/python3.9/si
te-packages/PopPUNK/utils.py", line 58, in setupDBFuncs\n    version = checkSketchlibVersion()\n  File "/opt/conda/lib/python3.9/site-packages/PopPUNK/sketchlib.py", line 49
, in checkSketchlibVersion\n    version = line.rstrip().decode().split(" ")[1]\nIndexError: list index out of range\n'

Looks like both sketchlib 2.0.0 and poppunk 2.4.0 are installed

Note: a similar error appears in #210

Thanks for the quick answer. poppunk_sketch seems not be installed. In the /conda/bin/ there are different executables (poppunk, poppunk_assign, poppunk_prune....) and *.py files (poppunk_add_weights.py, poppunk_batch_mst.py...) but not poppunk_sketch. I am able to import pp_sketchlib without problems though.

And conda list in that environment shows pp-sketchlib >=2.0.0?

Yes, that's the case (see list of packages attached).

conda_list.txt

Could you try making a fresh environment with conda create -n pp_retry poppunk==2.4.0 pp-sketchlib==2.0.0 and see if you have any luck there?

A new installation still gives me an error (see attached file). In this env, poppunk is installed but not poppunk_sketch still.

pp_fresh_install.txt

Ah, apologies, I now see the problem. From pp-sketchlib 2.0.0 poppunk_sketch was renamed to sketchlib. If you instead install pp-sketchlib v1.7.4 it should work ok.

From v2.5.0 this will be updated and fixed so they work together.

The only thing I don't understand is why you aren't getting the version from the library file. Could you try running, in a python session:

import pp_sketchlib
pp_sketchlib.version
dir(pp_sketchlib)

Indeed, installing pp-sketchlib v1.7.4 makes poppunk_sketch available but still I get an error when I run poppunk_assign (see attached file). I also attach another file to show you what I get from running the import and the other commands in my python session. How do you run poppunk_assign in poppunk version 2.4 then?

pp_assign_error.txt
python_import.txt

That's a different error now, which appears to be caused by scikit-learn changing their API. Can you try downgrading to v0.24? I'll need to put in a fix for this in future versions

Would you be able to attach the fit.pkl file you are using here?

Thanks! Downgrading to v0.24 solved the issue indeed. I attach here the pkl I'm using for this run.
vanAB_dataset_updated.dists.pkl.zip

Ok, glad to hear this sorted the issue!

The above pickle I think is the sample labels/dists pickle, do you also have a _fit.pkl you could share so I can look into the error?

Do you mean this one?
vanAB_dataset_updated_fit.pkl.zip

Ok that's great thank you, I can now replicate

Just to state the problem and resolution here:
sklearn changed it's API from v0.24 -> v1.0 so that loading a HDBSCAN model created with a different version won't work. So loading an older HDBSCAN fit with a newer sklearn installation throws an error as reported above:

ModuleNotFoundError: No module named 'sklearn.neighbors._dist_metrics'

Most distributed models don't use this mode, so I don't forsee this being a big problem. I will add a note to the documentation that to use such a model the sklearn version needs to be downgraded, or that you may generally want to run refine model to give a simpler & faster model in the first place.
We could write a script to convert HDBSCAN models to the new version of the API, but this would require some digging into the pickle, and it's not immediately clear to me from the sklearn docs what they changed and how to update the dist_metrics part. Will do this only if it keeps cropping up and downgrading sklearn is no longer viable.