Place data on CPU memory for `nn_descent` option in HDBSCAN

[jinsolp]

Current implementation of HDBSCAN puts data on GPU memory at the python layer.

cuml/python/cuml/cuml/cluster/hdbscan/hdbscan.pyx

Lines 916 to 928 in e05c9a5

	def fit(self, X, y=None, *, convert_dtype=True) -> "HDBSCAN":
	"""
	Fit HDBSCAN model from features.
	"""
	kwds = self.build_kwds or {}
	if kwds.get("knn_n_clusters", 1) > 1:
	logger.warn("Using data on host memory because knn_n_clusters > 1.")
	convert_to_mem_type = MemoryType.host
	else:
	logger.warn("Using data on device memory because knn_n_clusters = 1.")
	convert_to_mem_type = MemoryType.device

NN-Descent always copies the input data to GPU memory, even if the user originally provides it on the GPU. This is required because the algorithm internally converts the data to FP16. As a result, providing GPU-resident data leads to duplicate GPU allocations and unnecessary memory usage.

Therefore, when using the `nn_descent option, the input data should be put on CPU memory to avoid this extra GPU-side copy.

[csadorf]

This is required because the algorithm internally converts the data to FP16

Why would it not be be possible for a user to provide data using half types at the Python level to avoid this?

[jinsolp]

Why would it not be be possible for a user to provide data using half types at the Python level to avoid this?

I think it's better for us to handle this rather than asking the user to provide fp16 data types if they want to use with nn descent

Other reasons why this is needed

1. Given fp32 data, nn descent can internally run with fp16 or fp32. If it runs in fp16 it needs to allocate space on device after conversion anyway, so if the original fp32 data is on device we occupy unnecessary space on device (fp32 and fp16 versions of the data). More details to come after this PR: Add fp32 distance computation option in NN Descent cuvs#1415
2. if we place data on gpu at the python level, this occupies the device memory throughout the entire hdbscan run (even after steps where the data is necessary). it is better to handle this in cpp for freeing up memory early

[csadorf]

if we place data on gpu at the python level, this occupies the device memory throughout the entire hdbscan run (even after steps where the data is necessary). it is better to handle this in cpp for freeing up memory early

Maybe we should start experimenting with weakrefs.

[jcrist]

How would a weakref resolve this? In general you can't release memory of an object passed in to a python method or function from within the function - the caller will always still hold a reference.

A few assorted thoughts:

• We already do this for UMAP when build_algo="nn_descent", so doing it for HDBSCAN as well has precedent
• We might want to add a path for both where if the data is already on GPU we don't do a GPU->CPU->GPU transfer, but instead leave it on GPU. As I said above - moving it to host doesn't release the GPU memory, it's still occupied since the caller holds a reference. In the scheme of the fit the extra transfers are probably a wash, but they are unnecessary since they don't reduce memory usage.
• Breaking HDBSCAN and UMAP fits into smaller functions on the python side would let us better compose them to get the memory behavior we're seeking without trying to cram all codepaths into a single C++ fit method. I suspect this would simplify our existing code, and would let sufficiently dedicated users combine the function primitives to get the memory behavior they want if our high-level fit method doesn't suffice. If nothing else, right now HDBSCAN and UMAP both call nn_descent at the C++ level (with duplicated args handling), having a python nn_descent function may lead to less duplication and simpler code.