Model evaluation sources

Check date: 15 November 2026. These notes back guide/any-agent/06-choose-a-model-for-your-job.md. Every URL below was opened on that date, and each entry records what the page itself said. Where a page did not say something, the entry says not published or UNVERIFIED. Scores printed on model cards are each publisher's claim, and none was re-run for this guide.

Benchmarks the guide already cites

Terminal-Bench

• https://www.tbench.ai/ and https://www.tbench.ai/leaderboard: tagline "A benchmark to measure and evolve with the frontier of agent work." Hosted by Stanford, Harbor, and the Laude Institute. The default leaderboard is now labeled Terminal-Bench 4.1, captioned "Resolution rate of Terminal-Bench 4.1 tasks," with 95% confidence whiskers. Columns: rank, model, agent, resolution rate, cost, tokens. The rows render in the browser and were not in the fetched text. Whether entries are verified or self-reported is not published on that page.
• https://www.tbench.ai/leaderboard/terminal-bench/2.1: the fetched text showed the same 4.1 caption and no rows, so the 2.1 table itself was not read.
• https://tbench.ai/news/terminal-bench-2-1: Terminal-Bench 2.1 is "A revision of Terminal-Bench 2.0 that fixes 28 tasks" out of 89. Fix groups: external dependencies (9), resource mismatches (8), and misspecified instructions. The page says "no task is unsolved in Terminal-Bench 2.1." Its results show gains that depend on the harness: one model gained +12.1% under Claude Code but +0.9% under Terminus 2. Example task names include fix-git, configure-git-webserver, build-pmars, and train-fasttext. Release date not printed on the fetched text (search listing shows 6 May 2026, UNVERIFIED on the page body).
• https://www.tbench.ai/news/terminal-bench-4-0: 4.0 calibrated task resources, removed 8 tasks, fixed 19, set a flat 8-hour agent timeout, and moved to semantic versioning. It calls these "breaking changes that require re-running trials." Release date not published on the page. A 4.1 announcement on tbench.ai was not found; a third-party Snorkel post (9 Nov 2026) describes 4.1 but is not used as a fact here.
• https://github.com/laude-institute/terminal-bench (README): Terminal-Bench "is the benchmark for testing AI agents in real terminal environments" and "evaluates how well agents can handle real-world, end-to-end tasks - autonomously." Each task has an English instruction, a test script that checks whether the agent finished, and a reference solution.
• Consequence for the guide: model cards that print "Terminal-Bench 2.1" scores (Qwen3.8-27B "Terminus", Ripple-65B "self-reported" and "Terminus 2") are on an older version than the current tbench.ai default board, and 4.x results are not comparable with 2.1 results by the project's own wording.

SWE-bench Verified

• https://www.swebench.com/: "SWE-bench Leaderboards," with tabs for Verified, Multimodal, Multilingual, Lite, and Full, and a Bash Only label. The fetched text was truncated; a note on scaffold comparability was not found in the visible text.
• https://www.swebench.com/SWE-bench/: "Given a codebase and an issue," a model must generate a patch that "resolves the described problem." The issues come from GitHub. Verified is "500 engineer-confirmed solvable problems" (release note dated 13 August 2024). It uses a Docker-based evaluation harness.

SWE-bench Pro

• https://scale.com/blog/swe-bench-pro (Scale AI, 19 September 2025): a "contamination-resistant, human-augmented benchmark" that "picks up where SWE-Bench Verified leaves off." 1,865 tasks across 41 repositories: public 731, held-out 858, commercial 276. Tasks average 107.4 changed lines over 4.1 files. Scale says models above 70% on Verified scored about 23% on Pro at launch. Leaderboards at scale.com/leaderboard (public and commercial).
• https://epoch.ai/benchmarks/swe-bench-pro/review (Epoch AI, 1 September 2026): verdict Flawed. "Audits of SWE-Bench Pro have estimated that 30% or more of tasks are broken."

LiveCodeBench v6

• https://livecodebench.github.io/: "a holistic and contamination-free evaluation benchmark of LLMs for code," with problems from LeetCode, AtCoder, and Codeforces contests, each tagged with its release date so a model can be tested on problems after its training cutoff. It also covers self-repair, code execution, and test output prediction.
• https://github.com/LiveCodeBench/LiveCodeBench (README): release_v6 covers problems from May 2023 to April 2025, 1,055 problems in total.

TAU2-Bench (τ²-bench)

• https://taubench.com/: τ-bench "measures how well agents converse with users, call tools, retrieve knowledge, and follow policy across enterprise domains — in text and voice." τ²-bench added "Dual control," where the user must do some steps, and a Telecom domain beside Retail and Airline. The leaderboard reports Pass^1. Verification of submissions is not published on the page.
• https://arxiv.org/abs/2506.07982 (9 June 2025): "τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment." Agent and simulated user both use tools on shared state. The authors report that performance drops when agents move from no-user settings to dual control.
• https://github.com/sierra-research/tau2-bench (README): "a simulation framework for evaluating customer service agents across multiple domains." The README now calls the current release τ-bench 2.0, with τ³-bench (v1.0) as the previous release.

BFCL (Berkeley Function Calling Leaderboard)

• https://gorilla.cs.berkeley.edu/leaderboard.html: BFCL V4 "evaluates the LLM's ability to call functions (aka tools) accurately," and V4 introduces "holistic agentic evaluation." Last updated 12 April 2026. Columns include Overall Accuracy (an unweighted average of sub-categories), Cost, and Latency. "FC" models use native function calling; "Prompt" models imitate it in text. Coverage of every open model is not published.

DeepSWE 1.1

• https://deepswe.datacurve.ai/: Datacurve's benchmark for "frontier coding agents on original, long-horizon engineering tasks." 113 tasks across 91 repositories in 5 languages, written from scratch, with hand-written verifiers. All official runs use mini-swe-agent "for consistency." Leaderboard v1.1 "updated September 22, 2026." Whether any entries are self-reported is not published.
• https://epoch.ai/benchmarks/deepswe/review (Epoch AI, 7 September 2026): verdict Flawed, with problems in "at least 23 out of 113 tasks," mostly correct work that failed grading because models edited existing tests.

Independent places to compare models

• https://epoch.ai/benchmarks (Epoch AI Benchmarking Hub, "Updated Nov. 15, 2026"): mixes results Epoch ran itself with results from external sources such as official leaderboards. It warns that "small changes in prompts or evaluation settings can sometimes lead to significant differences in results." It lists 99 benchmarks.
• https://arena.ai/about (redirected from https://lmarena.ai/about): "Arena (formerly LMArena)," previously Chatbot Arena. Leaderboards include Agent, Text, WebDev, Vision, Document, and Search.
• https://arena.ai/faq: models stay anonymous while you vote, and names are revealed after the vote. Votes feed a Bradley-Terry rating, which the FAQ compares to chess Elo. "Your conversations may be used and shared to support our community," and Arena shares de-identified conversations in public research datasets. It recommends "avoiding including any sensitive information" in conversations.
• https://arena.ai/blog/agent-arena-methodology (published 4 June 2026, updated 15 October 2026): the Agent leaderboard comes from real Agent Mode sessions, not pairwise votes, using what Arena calls causal tracing. It ranks only the orchestrator model for now. Signals: confirmed success, praise or complaint, steerability, bash recovery, and tool hallucination.
• https://artificialanalysis.ai/methodology: benchmarks "proprietary and open weights models" on intelligence, price (including a blended price and cost per task), output speed in tokens per second, and latency.
• https://artificialanalysis.ai/methodology/intelligence-benchmarking: Intelligence Index v5.0.1, four 25% categories (knowledge work, coding, science, general). Coding includes Terminal-Bench 4.0 (10%). "All models are evaluated under identical conditions." Publication date of v5.0.1 not published on the page.
• https://localagentbench.com/results/: operator not named on the page. Not a paper. It tests local models through Ollama with the Hermes Agent harness and a mock MCP tool server, scoring JSON, schema, tool choice, multi-step tasks, and injection. Mostly RTX 3090 (24 GB) hosts, tiers set by measured memory at a 32K context. "Tested October 2026," data updated 13 November 2026, suite v0.1.5. qwen3.8:27b is #2 of 32 models in the 24 GB tier at 17 GB with 32K, with a marker saying its score sits above the interval from reruns. Scores from older scoring versions are not comparable, and speed is comparable only within one host. Models without native tool calling are unranked.

Checking what a model can do

• https://ollama.com/library/qwen3.8: badges vision, tools, thinking; qwen3.8:27b prints 256K and Text and Image input.
• https://ollama.com/search?c=tools: filtered list titled "Tools models." Capability filters: Tools, Thinking, Vision, Embedding, Decision. Local or cloud is a separate "Where" filter.
• https://aistudio.google.com/docs/models/gemini-4-hydrogen (redirected from ai.google.dev): Function calling, Computer use, Grounding with Google Search, and Structured outputs all "Supported"; context window 1,048,576 tokens; inputs Text, Image, Video, Audio, PDF. The free-tier limits (Computer use "Not available" on the free tier) are on the pricing page recorded in research/02-official-free-and-cheap-apis.md.
• https://docs.ollama.com/context-length: "< 24 GiB VRAM: 4k context," "24-48 GiB VRAM: 32k context," ">= 48 GiB VRAM: 256k context," and agents, coding tools, and web search "should be set to at least 64000 tokens." ollama ps shows PROCESSOR and CONTEXT columns.
• https://docs.ollama.com/cli and https://github.com/ollama/ollama/blob/main/docs/cli.mdx: ollama ps is "List running models." Neither page has an ollama show section, so the guide page does not use ollama show in a command block (UNVERIFIED as a documented CLI command on this date). https://docs.ollama.com/api-reference/show-model-details.md documents /api/show with a "capabilities" list.