The FACTS Benchmark Suite aims to enhance the evaluation of large language models (LLMs) by measuring their factual accuracy across various scenarios. It introduces three new benchmarks: the Parametric Benchmark, which tests models' internal knowledge through trivia-style questions; the Search Benchmark, which evaluates the ability to retrieve and synthesize information using search tools; and the Multimodal Benchmark, which assesses models' capability to answer questions related to images accurately. Additionally, the original FACTS Grounding Benchmark has been updated to version 2, focusing on context-based answer grounding. The suite comprises 3,513 examples, with a FACTS Score calculated from both public and private sets. Kaggle will manage the suite, including the private sets and public leaderboard. This initiative is crucial for advancing the factual reliability of LLMs in diverse applications.
