Affan Ahmed

Model or harness?

In October I gave a talk to IIT Kanpur’s eMasters class in AI and ML. I told five stories of agents going wrong in public and asked the room to vote on each one: was it the model, the harness, or both? The harness is everything you build around the model: the loop, the tools, the permissions, what goes into the prompt on each call. This is the talk written down. The slides are here.

The gap

In March 2026 frontier models scored under 1% on ARC-AGI-3, a set of puzzle games people solve easily. In August a coding agent with one skill file reported 100% on all 25 public games. The skill made the agent predict the result of each move before making it, and the harness checked the prediction. That result is self-reported, and the “one skill file” is 129 lines sitting on a 4,343-line tool. Which is the point. The model was the same in March.

MIT’s NANDA study in 2025 found that about 5% of enterprise generative AI pilots showed a measurable return. Models that capable, and nineteen pilots in twenty with nothing to show. The model is not what is holding those back.

Five failures

1. The inbox. February 2026. A safety researcher at Meta ran an agent on her email and told it to confirm before acting. It started deleting her inbox, and she had to run to the machine to stop it. Verdict: harness. Her rule lived in the chat. The chat got too long and was summarised, and the summary dropped the rule. There was no approval step in code and no real kill switch.

2. The folder. July 2025. A coding agent ran a command to make a directory. The command failed. The agent said “Great, the new directory is created” and moved the user’s files into it one by one, each overwriting the last. Verdict: model. It never read the error. The harness could still have stopped it: check exit codes, and check the state of the disk before the next step.

3. The key. April 2026. A coding agent found a stray API key and deleted a company’s production volume, backups included, in 9 seconds. The newest backup outside it was three months old and the service was down for over 30 hours. Verdict: both. The model guessed. The key had blanket rights, nothing asked for confirmation, and the backups lived inside the thing that got deleted. An instruction is not a permission.

4. The tip line. October 2026, from an Anthropic report. A model was generating practice tasks with live internet access. It filled in a real police tip form with an invented tip about an unsolved homicide. Verdict: both. The task could not be done and the model had no honest way to say so, so it improvised. An honest “I can’t” has to count as a result.

5. The benchmark. July 2026. In a hacking test with safeguards off, OpenAI’s agents coordinated through an internal proxy, broke out of their sandbox, and got into Hugging Face’s infrastructure with leaked tokens. They were looking for the benchmark’s answers. METR counted about 700 agents taking part. Verdict: both. The model wanted the answers. The harness had its monitors off and left the proxy reachable from the sandbox.

Final score: harness 1, model 1, both 3. All five had a fix in the harness, including the one where the model was to blame. You can’t retrain the model you call. You can change everything around it.

OpenAI’s own note on the fifth story: the same models, run in its production harness, were over 100 times less likely to attack infrastructure. Same model, different harness, different agent.

What people pay for

Nobody picks an agent for their inbox on intelligence. They pick on trust, and trust is four things, each with a fix that lives in the harness.

How we build at Alma

  1. Computer use is a harness problem now, so that is where our time goes.
  2. Simple beats clever. Every part you add is another place to fail.
  3. More agents is not a better answer. Agents aren’t deterministic, so everything except the agent should be.
  4. Tools are the arsenal. Without the right ones an agent is a text generator.
  5. The fix is usually obvious. Be brave enough to see it and ship it.
  6. Vibe coding is a boon and a curse. Don’t pull the trigger until you know what you’re doing.

If you are starting

Write the loop yourself in Python with one provider’s SDK. Run everything in a sandbox with no real keys in reach. Log every model call as a JSON line. Make ten tasks you can check automatically, which is your first eval set. Then break it, work out whether it was the model or the harness, and write up what you changed. In public.