Do we have a way to avoid thread over-subscription with BLAS and joblib while doing this?

Absolutely ! not on master however :/
Currently I put it in #11950 for now because there was no use elsewhere yet. But I can make a separate PR for that.

It's adapted from some code of loky. Essentially loop over dynamically shared libs to check which blas / openmp is loaded and then use the appropriate method to dynamically set the max number of threads they are allowed to use.

Nice! Yes, maybe splitting it out of #11950 would make review of that one easier as well.

Does the user need a global way to control how many processors are available beyond explicit use of n_jobs?

I'm not sure to understand. Here's my thought.

When inside code parallelized using joblib, there's no need to add a second level of parallelism with prange. At least for now, while cython only supports OpenMP. Maybe when cython supports TBB it could be worth.

Inside code not parallelized, there may be some places where we could use a prange. However, there might be some BLAS calls inside these loops. In that case, we need to control the number of threads of the BLAS to avoid oversubscription. I made a contextmanager for that, to locally control BLAS (it can be used to control openmp also but no need for now).

By default, joblib's 'loky' backend forces workers to use only one thread: https://joblib.readthedocs.io/en/latest/parallel.html#avoiding-over-subscription-of-cpu-ressources

To cope with this problem, joblib forces by default supported third-party libraries to use only one thread in workers with the 'loky' backend.

FYI, tested switching Joblib for prange in nearest neighbors tree-based queries: it did not improve performance but deactivating some debugging stats did

See #13330 and #13331