A new open-source tool called Flakestorm has been developed to stress-test AI agents running on local models like Ollama, Qwen, and Gemma. The tool addresses the issue of AI agents performing well with clean prompts but exhibiting unpredictable behavior when faced with adversarial inputs such as typos, tone shifts, and prompt injections. Flakestorm generates adversarial mutations from a "golden prompt" and evaluates the AI's robustness, providing a score and a detailed HTML report of failures. The tool is designed for local use, requiring no cloud services or API keys, and aims to improve the reliability of local AI agents by identifying potential weaknesses. This matters because ensuring the robustness of AI systems against varied inputs is crucial for their reliable deployment in real-world applications.
