"MMLU is the SAT score for LLMs. While it doesn't measure agentic loops, it measures the raw intelligence of the brain powering your agent."

"SWE-bench provides an agent with a codebase and a GitHub issue description. The agent is successful only if it writes a patch that passes the repo existing unit tests."

"AgentBench is a comprehensive framework that evaluates agents across diverse environments like OS, database, knowledge graph, and web."

"HumanEval is a dataset of 164 handwritten Python programming problems used to test code generation logic."

"Leaderboard contamination risk: when benchmark questions are accidentally included in the model's training data, leading to fake high scores."