Hugging Face has released NeoMME, a new family of multimodal encoders that consolidates text and image processing into a single efficient architecture. Rather than relying on separate pretrained vision towers and language models—a common approach in current visual AI systems—NeoMME processes both modalities through one bidirectional Transformer, reducing computational overhead while maintaining competitive performance.
Available in two sizes (260M and 800M parameters), NeoMME was trained from scratch using a masked discrete-diffusion objective and fine-tuned for visual document retrieval. The smaller variant achieves approximately 51 pages per second on NVIDIA L40S GPUs, roughly double the throughput of comparable systems. The model supports 16,384 token context length and multilingual text, trained on a diverse dataset including natural images, document images, code, and mathematics.
The most striking advantage lies in storage efficiency: through hierarchical token pooling and asymmetric quantization, NeoMME reduces per-page index storage from 1.5 MB to 6 kilobytes—a 255-fold reduction—while retaining 95% of baseline retrieval performance. Both models are available under the Apache 2.0 open-source license and fully integrated into Hugging Face Transformers, making the technology immediately available for research and commercial applications.
Key Points
NeoMME is a unified multimodal Transformer that processes text and images in a single architecture, eliminating the need for separate pretrained vision and language towers
The 260M model achieves approximately 51 pages per second encoding speed—roughly 2x faster than comparable systems—while maintaining strong Pareto frontier performance
Hierarchical token pooling and quantization deliver 255x storage reduction per document page while retaining 95% retrieval accuracy
Supports 16,384 token context, dynamic image resolution, and multilingual text with 131k-token vocabulary; available open-source under Apache 2.0 license