real-time applications
-
Accelerating LLM and VLM Inference with TensorRT Edge-LLM
Read Full Article: Accelerating LLM and VLM Inference with TensorRT Edge-LLM
NVIDIA TensorRT Edge-LLM is a new open-source C++ framework designed to accelerate large language model (LLM) and vision language model (VLM) inference for real-time applications in automotive and robotics. It addresses the need for low-latency, reliable, and offline operations directly on embedded platforms like NVIDIA DRIVE AGX Thor and NVIDIA Jetson Thor. The framework is optimized for minimal resource use and includes advanced features such as EAGLE-3 speculative decoding and NVFP4 quantization support, making it suitable for demanding edge use cases. Companies like Bosch, ThunderSoft, and MediaTek are already integrating TensorRT Edge-LLM into their AI solutions, showcasing its potential in enhancing on-device AI capabilities. This matters because it enables more efficient and capable AI systems in vehicles and robots, paving the way for smarter, real-time interactions without relying on cloud-based processing.
-
Sonya TTS: Fast, Expressive Neural Voice Anywhere
Read Full Article: Sonya TTS: Fast, Expressive Neural Voice Anywhere
Sonya TTS is a newly released, small, and fast text-to-speech model that offers an expressive single speaker English voice, built on the VITS framework and trained with an expressive voice dataset. It is designed to run efficiently on various devices, including GPUs, CPUs, laptops, and edge devices, delivering natural-sounding speech with emotion, rhythm, and prosody. The model provides instant generation with low latency, suitable for real-time applications, and includes an audiobook mode for handling long-form text with natural pauses. Users can adjust emotion, rhythm, and speed during inference, making it versatile and adaptable for different use cases. This matters because it democratizes access to high-quality, expressive TTS technology across a wide range of devices without requiring specialized hardware.