Researchers from FAIR Meta, HuggingFace, AutoGPT, and GenAI Meta address the problem of testing the capabilities of general AI assistants in handling real-world questions that require fundamental skills such as reasoning and multi-modality handling, which proves to be challenging for advanced AIs with human-like responses. The development of GAIA aims to achieve Artificial General Intelligence by targeting human-level robustness.
Focusing on real-world questions necessitating reasoning and multi-modality skills, GAIA diverges from current trends by emphasizing tasks challenging for both humans and advanced AIs. Unlike closed systems, GAIA mirrors realistic AI assistant use cases. GAIA features carefully curated non-gameable questions, prioritizing quality and showcasing human superiority over GPT-4 with plugins. It aims to guide question design, ensuring multi-step completion and preventing data contamination.
As LLMs surpass current benchmarks, evaluating their ability becomes increasingly challenging. Despite the emphasis on complex tasks, researchers argue that difficulty levels for humans do not necessarily challenge LLMs. To address this challenge, a new model called GAIA has been introduced. It is a General AI Assistant that focuses on real-world questions, avoiding LLM evaluation pitfalls. With human-crafted questions that reflect AI assistant use cases, GAIA ensures practicality. By targeting open-ended generation in NLP, GAIA aims to redefine evaluation benchmarks and advance the next generation of AI systems.
A proposed research method involves utilizing a benchmark created by GAIA for testing general AI assistants. This benchmark consists of real-world questions prioritizing reasoning and practical skills, which humans have designed to prevent data contamination and allow for efficient and factual evaluation. The evaluation process employs a quasi-exact match to align model answers with ground truth through a system prompt. A developer set and 300 questions have been released to establish a leaderboard. The methodology behind GAIA’s benchmark aims to evaluate open-ended generation in NLP and provide insights to advance the next generation of AI systems.
The benchmark conducted by GAIA revealed a significant performance gap between humans and GPT-4 when answering real-world questions. While humans achieved a success rate of 92%, GPT-4 only scored 15%. However, GAIA’s evaluation also showed that LLMs’ accuracy and use cases can be enhanced by augmenting them with tool APIs or web access. It presents an opportunity for collaborative human-AI models and advancements in next-gen AI systems. Overall, the benchmark provides a clear ranking of AI assistants and highlights the need for further improvements in the performance of General AI Assistants.
In conclusion, Gaia’s benchmark for evaluating General AI Assistants on real-world questions has shown that humans outperform GPT-4 with plugins. It highlights the need for AI systems to exhibit robustness similar to humans on conceptually simple yet complex questions. The benchmark methodology’s simplicity, non-gameability, and interpretability make it an efficient tool for achieving Artificial General Intelligence. Furthermore, the release of annotated questions and a leaderboard aims to address open-ended generation evaluation challenges in NLP and beyond.
Check out the Paper and Code. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 33k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.
If you like our work, you will love our newsletter..
Hello, My name is Adnan Hassan. I am a consulting intern at Marktechpost and soon to be a management trainee at American Express. I am currently pursuing a dual degree at the Indian Institute of Technology, Kharagpur. I am passionate about technology and want to create new products that make a difference.
