# Agent evaluations

Results: https://huggingbay.xyz/agents/evals.json
Page: https://huggingbay.xyz/agents/evals

The fixed catalog contains twenty discovery tasks for Claude Messages API with its MCP connector, Codex CLI, and a plain JSON-RPC client. JSON-RPC is a transport check, not an autonomous agent. No tool-choice router supplies the agent's answer. The harness does not retry a task.

First-call success: first observed tool result is neither a protocol nor execution error. Tool-choice correctness: first observed tool is in the task's expected set. Task success: a completed client made at most three calls and an expected call has matching arguments and required returned fields, or the expected terminal not-found result. Final-answer prose is not graded. A task with no tool call fails. Missing clients are unavailable, never passed. All attempted tasks remain in denominators. Tokens are provider-reported input plus output tokens, null if unavailable. Latency is whole task wall time; medians exclude unavailable clients, not failed attempts.

The prepared Mac runner selects six representative fixed tasks per client and can publish redacted JSON plus a dated copy to public GCS. Scheduling and publication are managed separately; this page reports only the observed run and does not imply an active nightly job. The page and JSON routes fetch that fixed public object with a two-second deadline and a sixty-second cache; unavailable public reports fall back to an explicitly labeled dated release snapshot. This does not deploy the site. Eighteen client-task invocations include at most six Claude API requests. Actual MCP calls are recorded; provider-internal HTTP retries and tool loops are not hard-capped by the task budget. Target URL, source SHA, client/model version and task-set digest accompany every run. Live observations, local fixtures and deterministic routing diagnostics are separate.