embedded systems
-
Accelerating LLM and VLM Inference with TensorRT Edge-LLM
Read Full Article: Accelerating LLM and VLM Inference with TensorRT Edge-LLM
NVIDIA TensorRT Edge-LLM is a new open-source C++ framework designed to accelerate large language model (LLM) and vision language model (VLM) inference for real-time applications in automotive and robotics. It addresses the need for low-latency, reliable, and offline operations directly on embedded platforms like NVIDIA DRIVE AGX Thor and NVIDIA Jetson Thor. The framework is optimized for minimal resource use and includes advanced features such as EAGLE-3 speculative decoding and NVFP4 quantization support, making it suitable for demanding edge use cases. Companies like Bosch, ThunderSoft, and MediaTek are already integrating TensorRT Edge-LLM into their AI solutions, showcasing its potential in enhancing on-device AI capabilities. This matters because it enables more efficient and capable AI systems in vehicles and robots, paving the way for smarter, real-time interactions without relying on cloud-based processing.
-
GraphQLite: Embedded Graph Database with SQLite
Read Full Article: GraphQLite: Embedded Graph Database with SQLite
GraphQLite is an SQLite extension designed for those building GraphRAG systems who prefer not to use Neo4j for storing knowledge graphs. It introduces Cypher query support, allowing users to store entities and relationships in a graph structure and utilize Cypher for context expansion during data retrieval. By integrating with sqlite-vec for vector search, GraphQLite provides a comprehensive embedded RAG stack within a single database file. It also includes graph algorithms like PageRank and community detection, which help identify key entities and cluster related concepts. This extension is particularly useful for developers looking for a streamlined solution to manage graph data efficiently. This matters because it offers a lightweight, integrated alternative for handling complex graph data without the overhead of additional database systems.