Affan Ahmed

What a 70.76% on GAIA is made of

In October 2025 Sukuna, the agent I built at Midsphere, scored 70.76% on the GAIA test set: 213 of 301 tasks. Ten days earlier it had scored 51.83%. I have now published every answer file, so this is the full account of how the number was made.

The ten entries

GAIA’s test answers are private. You upload 301 answers and the leaderboard tells you how many were right, split by difficulty level. We did that ten times.

Entry Date Level 1 (93) Level 2 (159) Level 3 (49) Total
V0.1 1 Oct 57 81 18 156, 51.83%
V0.1.1 2 Oct 76 81 18 175, 58.14%
V0.1.2 3 Oct 76 86 18 180, 59.80%
V0.1.3 4 Oct 77 86 24 187, 62.13%
V0.1.4 5 Oct 77 94 24 195, 64.78%
V0.1.5 6 Oct 77 108 24 209, 69.44%
V0.1.6 8 Oct 76 108 24 208, 69.10%
V0.1.7 9 Oct 77 108 24 209, 69.44%
V0.1.8 10 Oct 79 108 26 213, 70.76%
V0.1.9 15 Oct 79 108 26 213, 70.76%

Read the columns one at a time. On 2 October only level 1 moved, by 19 tasks. On 3 October only level 2. On 4 October level 3 went from 18 to 24 and nothing else changed much. Then level 2 again, twice, for 22 tasks.

That pattern is the method. We did not rerun the whole benchmark each day. We reran one level, sometimes with a different model, and merged its answers into the previous file. Between one merged file and the next, 24 to 49 of the 301 answers changed. The leaderboard entries declare GPT-5, Gemini 2.5 Pro, Claude Sonnet 4 and, from the fifth entry, o3.

What I can and can’t say about the gains

The files show what changed. They don’t record why, and I am not going to reconstruct reasons a year later. Three things are in the record.

Almost nothing was left blank. The first file has 5 empty answers out of 301 and the last has 1. The agent was already finishing its tasks on day one, so the 57 tasks gained came from wrong answers turning right.

The work went level by level. The run files are saved per level and per model, and the score moved one level at a time. A day’s work was one level’s failures, rerun.

The question got read twice. GAIA marks by exact match, so a right answer in the wrong form is wrong. The second version of the agent, the one in the repository, has a reasoning model write notes on the parts of a question that are easy to misread before the loop starts, and puts them in the task prompt.

What the number means

It is a real leaderboard score and all ten rows are public under “Midsphere AI”. It is also the tenth of ten attempts, each one informed by the score of the last. We could not see which tasks were wrong, but we could see which level moved, and we kept what helped. On 8 October the score fell by one task and on 9 October it came back. A single run of one model with this agent would score lower than 70.76%. I don’t know by how much, because we never ran it that way.

The validation set tells the same story with public answers. The first full pass got 120 of 165, 72.73%. After four more passes that reran only the wrong tasks, the best of five was 140, 84.85%.

So the claim I stand behind is narrower than the headline. In ten days we took an agent from 156 to 213 tasks on GAIA by rerunning what failed, one level at a time, against a leaderboard that only tells you the count.

What I would do differently

Run it once, end to end, with one model, and report that number beside the merged one. Keep a log of which file went to which entry, because the repository can no longer tell you. And count an honest “I could not do this” separately from a wrong answer from the first day, since they need different fixes.