MongoDB Research introduces the Lightweight Embedding Alignment Framework (LEAF) to compress large text embedding models into 23M parameter versions suitable for CPU-only or edge environments. The framework utilizes knowledge distillation to maintain performance, producing models that support quantization and can be fine-tuned for specific domains. These models are intended to lower infrastructure costs and latency for retrieval-augmented generation and other NLP tasks.
Key points
LEAF utilizes knowledge distillation to compress large text embedding models into highly efficient 23M parameter variants.
The distilled models achieve high performance on CPU-only infrastructure, enabling deployment on IoT or mobile devices.
Embedding models trained via LEAF are compatible with output quantization and domain-specific fine-tuning.
Decoupling model requirements for document indexing and query encoding allows for better performance optimization in retrieval systems.