Question: Custom clustering cutoff?

[shiraz-shah]

When we used VAMB5 on a large data set of 1500 samples, we found that VAMB clusters are at the strain-level. Most species are often split into a dozen clusters.

When later we profile the clusters using MAGinator, we find that marker-gene mappings are unspecific and can't distinguish between the clusters, so cluster-level abundances end up being highly correlated within the same species. Also, clusters are so narrow that they become quite sample-specific, so we end up having to agglomerate our data to the species-level anyway, in order to have enough statistical power to associate against metadata.

I get the impression that you spent a lot of effort tuning the clustering cutoff, and also like the idea of the operational taxonomic unit being so data-driven. But we have collaborators who merge their VAMB bins into species-level clusters using something as crude as dRep, and they're getting much more meaningful statistics against metadata than us.

Could it be 1) a bug or 2) that larger data sets somehow push VAMB clustering towards being too fine-grained? And 3) could it make sense with customizable clustering cutoff?

We are happy to benchmark.

[jakobnissen]

That's interesting. I've heard something similar from Joseph Nesme - that Vamb tends to cluster too finely, and sometimes even split individual genomes. I also have a vague feeling you're right it seems to be worse with large datasets, though I'm not sure why - perhaps the co-abundance signal just becomes more and more powerful with more samples, allowing finer and finer distinctions between contigs.
You're right that I think it would be best if clustering was adjustable or hierarchical, such that users could trade off precision vs recall.

Unfortunately, I'm not clear what to do about it. The problem doesn't seem to be in any parameters of the clustering, but rather in the combination of how the embedding looks, and how clustering works in general. Using Vamb's iterative medoid clustering algorithm with Vamb's embedding, you don't have that much control of the cluster size, they are mostly determined by the embedding. The fiddling with the clustering hyperparameters that I've done is more about making the clustering algorithm work as expected (i.e. segment a cluster when it appears instead of cutting it in half or failing to detect it).

Ultimately, Vamb probably suffers from its development being driven by the metric of counting "near-complete genomes" (<= 5% contamination, >= 90% completion). This, combined with synthetic datasets with closely related strains means the binner becomes optimised for very high precision. At this point, it's deeply baked into Vamb and it would take months to re-adjust the network architechture, training hyperparameters, and then clustering to combat it. You're probably better off running some summary statistics on the output clusters to detect similar clusters, then merging them in a post-processing step.

On the longer term, I'm eyeing the possibility of entirely ditching the embed-cluster paradigm that dominates modern binners. Instead, I think an old fashioned supervised DL model combined with some clever clustering could work much better. However, that's a whole new project, and I would need to get some grants to fund that investigation.

[shiraz-shah]

Jakob, just thinking out loud here. I could be completely off. But could one consider training for fewer epochs on larger data sets? Could one adapt the number of epochs to the data set size, such the total number of parametre update steps is kept constant?

When I look at the loss, it hardly develops beyond the first couple of epochs. What if VAMB has learnt all it needs to after just a few epochs when the data set is large enough? Any learning beyond that is going to be "overfitting" as far as clustering is concerned, even if the loss is still declining.

For your benchmarking data, do you remember what ended up being the final number of update steps correponding to those default 300 epochs? Because then I can try to benchmark it.

[jakobnissen]

Our experiments with training has shown us that:

1. Even as loss barely changes, the latent space still changes. So, even if you see almost no loss change from e.g. epoch 50 to 300, at least some of these epochs are probably necessary for good clustering.
2. Vamb tends not to overfit due to too long training. The regularization prevents that. When training even for several thousand epochs, Vamb's performance just levels off at its maximal level and stays there. That seems to align with how VAEs work in other contexts, as well.

Of course since these optimisers are black boxes, we're basing this based on only a few experiments on a few datasets.

If you are interested, it could be valuable to try to run an experiment where you train for e.g. 1000 epochs, and then dump the encoding every 10 epochs (and repeat the experiment a handful of times). Then, it would be clear if the network began to overfit.
Of course, that's going to be a computationally very expensive plot!

[shiraz-shah]

That's a great idea, Jakob. I guess we'd want to cluster after each dump, right? I'm going to see if this is computationally feasible on our end.