← Back to feed

A submission from PCL-RSI-lab called Seshat-agent appeared at the top of HuggingFace’s SWE-bench Pro leaderboard on October 31, posting a resolve rate of 694/731, or 94.94%. The next entry sits at 68.74%. That 26-point gap is the first number everyone notices, and the second number to check.

What the score actually covers

SWE-bench Pro exists in two versions. V1, published originally, has 731 tasks. V2, released September 22, has 642 tasks: 89 were dropped after review, and the task set was hardened with rewritten problem statements, revised test patches, and a protocol that locks down network access during evaluation and re-grades every diff in a clean container.

The HuggingFace leaderboard’s official task identifier points at V2. Seshat-agent’s entry uses V1. The submission notes say “SWE-Bench Pro v1 (731), resolve rate 694/731=94.94%”. This is a V1 score on a V2 board, and the 26.2-point cliff reflects three separate things: model quality, harness choice, and benchmark version. Those three things cannot be separated from a leaderboard row.

For reference, the V2 paper’s abstract reports that frontier agents resolve up to 99.4% of the 642 V2 tasks under the hardened protocol, so a 94.94% on the easier V1 set is in range for a strong agent — it is not inherently suspicious.

What the evidence files show

Seshat-agent linked an evidence dataset, which is more than most entries do. An independent analysis of the files on October 31 found:

• All 731 task IDs match the official V1 list exactly.
• 694 are marked resolved in every location: the README, the summary file, and the results file.
• Resolved rate on the 642 IDs that V2 kept: 610/642, or 95.02%.
• Resolved rate on the 89 IDs V2 dropped: 84/89, or 94.38%.
• The hard config (51 tasks): 41/51, or 80.39%.

The files are internally consistent. The files do not show how many attempts each task received, whether the network was locked during evaluation, or whether the self-improvement training rounds used the same 731 tasks. The model card describes a self-improving agent whose successful traces feed back into the meta-knowledge base each round and does not address contamination.

The agent architecture

Seshat-agent is from PCL-RSI-lab, a group at PengCheng Laboratory (PengCheng Laboratory Recursive Self-Improvement Lab). The system runs on Kimi K3 as its base agent model, disclosed in the model card. The routing architecture is called MetaEvo: at inference time, the system retrieves and composes meta-knowledge vectors into cognitive paths, rather than retraining weights between runs. Training infrastructure and compute budget are not disclosed.

How to read this

The 94.94% result is documented and internally consistent. The open questions are number of attempts, network policy during grading, and whether the self-improvement rounds saw the test tasks. None of those can be answered from the public submission.

The score is not a claim about V2, which has both harder and fewer tasks. Any comparison to models benchmarked on V2 conflates two different evaluations.

What is unambiguously true: the evidence files check out, the HF board has 47 entries and zero verified entries, and the top-to-second gap is partly a version artifact, not purely a model capability gap.