Instacart improved its product search engine by integrating Large Language Models into its query understanding process. By focusing on query categorization, entity extraction, and intent identification, the team successfully optimized search relevance for tail queries while utilizing caching and GPU autoscaling to maintain cost-efficiency and latency targets.
Key points
Query understanding acts as the foundational layer that directs both retrieval and ranking phases in e-commerce search.
Real-time inference costs can be significantly mitigated by leveraging caching for frequent queries, limiting live LLM calls to a small fraction of total traffic.
Small, specialized models can provide high-quality classification and intent extraction while meeting strict latency requirements.
System-wide performance improvements for tail queries were achieved by combining generative intent classification with intelligent query rewriting and tag extraction.