An innovative agent has been developed to streamline the process of benchmarking multiple open and closed source Large Language Models (LLMs) on specific problems or datasets. By simply loading a dataset and defining the problem, the agent can prompt various LLMs to evaluate their performance, as demonstrated with the TweetEval tweet emoji prediction task. The agent facilitates dataset curation, model inference, and analysis of predictions, while also enabling benchmarking of additional models to compare their relative performance. Notably, in a particular task, the open-source Llama-3-70b model outperformed closed-source models like GPT-4o and Claude-3.5, highlighting the potential of open-source solutions. This matters because it simplifies the evaluation of LLMs, enabling more efficient selection of the best model for specific tasks.
A new Text-to-SQL agent has been developed to assist non-technical railway managers in querying fault detection logs without needing to write SQL. Utilizing the Llama-3-70B model via Groq for fast processing, the system achieves sub-1.2 second latency and 96% accuracy by implementing strict schema binding and a custom 'Bouncer' guardrail. This approach prevents hallucinations and dangerous queries by injecting a specific SQLite schema into the system prompt and using a pre-execution Python layer to block destructive commands. This matters because it enhances the accessibility and safety of data querying for non-technical users in the railway industry.