---
title: Harness Leaderboard: 28,849 trials show coding-agent harness choice can swing open-model scores by tens of points on Terminal-Bench | ctxsec
url: https://ctxsec.com/items/joel-niklaus-harness-leaderboard-28849-trials
description: "Jo\u00ebl Niklaus released a public Harness Leaderboard that holds the model, provider and reasoning effort fixed and varies only the coding-agent harness across Claude Code, Codex, Pi, OpenCode, Hermes Agent, mini-swe-agent, Kimi Code, Qwen Code and OpenClaw. Benchmarks are Terminal-Bench 2.1 and DeepSWE via Harbor on Hugging Face. Community summaries of the board put mini-swe-agent near the top (~51% macro resolve in one cut) and OpenClaw much lower (~27%); the README stresses per-model winners differ and publishes every trajectory. Cheating (looking up published solutions, special-casing graders, calling another model) counts as a fail."
status: "confirmed research release (Joël Niklaus Harness Leaderboard + HF Space README, Nov 5); open-model coding-agent harness comparison"
published: 2026-11-05
---

> Joël Niklaus released a public Harness Leaderboard that holds the model, provider and reasoning effort fixed and varies only the coding-agent harness across Claude Code, Codex, Pi, OpenCode, Hermes Agent, mini-swe-agent, Kimi Code, Qwen Code and OpenClaw. Benchmarks are Terminal-Bench 2.1 and DeepSWE via Harbor on Hugging Face. Community summaries of the board put mini-swe-agent near the top (~51% macro resolve in one cut) and OpenClaw much lower (~27%); the README stresses per-model winners differ and publishes every trajectory. Cheating (looking up published solutions, special-casing graders, calling another model) counts as a fail.

[← Context & Harness Attacks](/context) · [markdown](/items/joel-niklaus-harness-leaderboard-28849-trials.md)

confirmed research release (Joël Niklaus Harness Leaderboard + HF Space README, Nov 5); open-model coding-agent harness comparison

2026-11-05

### Harness Leaderboard: 28,849 trials show coding-agent harness choice can swing open-model scores by tens of points on Terminal-Bench

- **Why it belongs in Context & Harness Attacks:** the wrapper around the model — tools, prompts, compaction, permissions — is a first-class security and reliability surface, not a detail.
- Method highlights: three attempts with CIs, infra failures excluded, DeepSWE offline since Oct 23, cost/time metered outside the harness, secrets redacted.
- Companion: Harness Lens documents what each harness actually does, with sources.

Sources: [Hugging Face Space: joelniklaus/harness-leaderboard](https://huggingface.co/spaces/joelniklaus/harness-leaderboard) · [Harness Leaderboard README](https://huggingface.co/spaces/joelniklaus/harness-leaderboard/raw/main/README.md) · [Joël Niklaus on X (Nov 5)](https://x.com/joelniklaus/status/2118373808692437403)