VidaiMock: Local Mock Server for LLM APIs

VidaiMock is a newly open-sourced local-first mock server designed to emulate the precise wire-format and latency of major LLM API providers, allowing developers to test streaming UIs and SDK resilience without incurring API costs. Unlike traditional mock servers that return static JSON, VidaiMock provides physics-accurate streaming by simulating the exact network protocols and per-token timing of providers like OpenAI and Anthropic. With features like chaos engineering for testing retry logic and dynamic response generation through Tera templates, VidaiMock offers a versatile and high-performance solution for developers needing realistic mock infrastructure. Built in Rust, it is easy to deploy with no external dependencies, making it accessible for developers to catch streaming bugs before they reach production. Why this matters: VidaiMock provides a cost-effective and realistic testing environment for developers working with LLM APIs, helping to ensure robust and reliable application performance in production.

VidaiMock presents a significant advancement for developers working with large language model (LLM) APIs. It addresses a common challenge faced by developers: the cost and complexity of testing streaming user interfaces (UIs) and software development kits (SDKs) against real-world APIs. Traditional mock servers often fall short as they typically return static JSON data, which fails to capture the nuances of real-world streaming, such as network protocols and latency. VidaiMock, however, offers a local-first mock server that accurately emulates the wire-format and latency of major providers, enabling developers to test their applications more effectively and without incurring additional costs.

One of the standout features of VidaiMock is its ability to simulate physics-accurate streaming. This means it doesn’t just output text but replicates the exact wire-format and per-token timing of major LLM providers like OpenAI and Anthropic. This capability allows developers to test their loading states and streaming UI/UX in a manner that closely mirrors production environments. By providing a more realistic testing environment, VidaiMock helps developers identify and resolve potential issues before they impact end-users, thereby improving the overall quality and reliability of their applications.

In addition to its realistic streaming simulation, VidaiMock is designed for ease of use and flexibility. It requires no configuration or external fixtures, making it accessible even to those with limited technical expertise. The tool is a single Rust binary that can be run without the need for Docker, databases, or API keys. Furthermore, VidaiMock supports dynamic responses through Tera templates, allowing developers to create more complex and interactive mock scenarios. This extensibility means that developers can customize their testing environments to suit their specific needs, whether they are working with existing providers or developing internal APIs.

By open-sourcing VidaiMock under the Apache 2.0 license, the Vidai team aims to address the broader community’s need for robust and realistic mock infrastructure. The tool’s high performance, capable of handling over 50,000 requests per second, ensures it can meet the demands of large-scale testing. This initiative not only benefits individual developers but also contributes to the overall improvement of LLM integrations by helping to catch streaming bugs before they reach production. In an industry where reliability and user experience are paramount, VidaiMock offers a valuable resource for enhancing the testing and development of LLM-based applications.

Read the original article here