Hugging Face's Tom Aarsen has published a guide to training multi-vector embedding models so they keep their retrieval quality when document vectors are compressed. Multi-vector models such as ColBERT store many vectors per document, which lets each part of a query find its own match. A corpus of millions of documents can contain hundreds of millions of vectors, and storage costs rise accordingly. Token pooling reduces that count by merging similar vectors. The Sentence Transformers library implements it as HierarchicalTokenPooling, which uses Ward hierarchical clustering to group the closest vectors and replaces each group with its mean.
The post's central finding is that the more aggressively vectors are pooled, the more retrieval quality is lost. A conventionally trained model keeps 94.6% of its unpooled NanoBEIR nDCG@10 score at pool_factor=4, a fourfold reduction in document vectors. Training the encoder with a pooling-aware loss, which teaches it to preserve information that survives merging, raises retention to 97.8% at the same storage budget. The article gives a worked example: 200 vectors of 128 dimensions at 4-bit residual quantization take 12.8 KB per document, or 12.8 GB for a million documents, falling to 3.2 GB when each document is pooled to 50 vectors. The author notes that the method builds on earlier pooling research by Stefan Josef and LightOn, and that adaptive pooling can be calibrated to a target average vector count, even for models not trained with the pooling-aware objective.
For Australian businesses and professionals, this is a technical optimisation for teams that run their own multi-vector search and face rising storage bills. Organisations that use hosted search products will see no change. Teams building retrieval over their own document collections could reduce storage spending, but they should test retention on their own data rather than rely on benchmark figures. The Privacy Act 1988 applies to any personal information held in indexed documents however compactly the vectors are stored, so compression does not reduce those obligations.
Key Points
Multi-vector models like ColBERT store many vectors per document, so large corpora can require very large storage.
Token pooling merges similar vectors using Ward hierarchical clustering in Sentence Transformers via HierarchicalTokenPooling.
At pool_factor=4, a conventionally trained model keeps 94.6% of unpooled NanoBEIR nDCG@10.
Pooling-aware training raises retention to 97.8% at the same storage budget, according to the post.
For most Australian businesses the change matters only if they run their own multi-vector retrieval, and compression does not alter Privacy Act duties.