Handling 11 million embeddings in a large-scale knowledge graph project presented significant challenges in terms of storage, cost, and performance. The Gemini-embeddings-001 model was chosen for its strong semantic representations, but its high dimensionality led to substantial storage requirements. Storing these embeddings in Neo4j resulted in a prohibitive monthly cost of $32,500 due to the high memory footprint. To address this, Product Quantization (PQ), specifically PQ64, was implemented, reducing storage needs by approximately 192 times, bringing the total storage requirement to just 0.704 GB. While there are concerns about retrieval accuracy with such compression, PQ64 maintained a recall@10 of 0.92, with options like PQ128 available for even higher accuracy. This matters because it demonstrates a scalable and cost-effective approach to managing large-scale vector data without significantly compromising performance.
