The Karpathy Loop: 700 experiments in 2 days, then 5x better with a loop on the loop

Karpathy left an agent running for 2 days on a model he had tuned by hand for two decades. The agent ran 700 experiments. It found 20 improvements he had missed. The "Time to GPT-2" benchmark on the leaderboard dropped from 2.02 hours to 1.80 hours, an 11% gain stacked on top of a project he had already over-tuned. The repo hit 91,000 GitHub stars in four months. Fortune gave the workflow a name: The Karpathy Loop.

The shift is not "AI is getting better." The shift is that one person with a loop now does the work of a research team. The pattern is moving out of ML labs and into every team that has a measurable objective and a budget for tokens. We have been watching this land across our own client work for the last 90 days. The teams that adopt loops early compound. The teams that do not will run campaigns their competitors already ran 200 times overnight.

What a loop actually is

A prompt is one instruction. You ask, the model answers, you decide what to do next. A loop is a goal the agent keeps working toward until it gets there, without you sitting in the chair prompting every step. The agent discovers what needs doing, plans how to do it, does the work, checks the result, and if it is not there yet, feeds the result back in and goes again.

Three parts make or break a loop:

• A verifier turns repetition into progress. Without a real check on the output, you do not have a loop. You have the agent agreeing with itself on repeat. The check can be a test that passes or fails, a metric that goes up or down, a build that compiles or crashes. No gate means the agent grades its own homework.
• State makes the loop learn. Each pass, the agent has to know what it already tried. Without that, it repeats the same mistake every cycle. A small log file on the side records what is done, what failed, what is next. Tomorrow's run resumes instead of starting from zero.
• A stop condition keeps it sane. A loop with no exit runs until it succeeds, breaks, or drains your account. Every working loop has two ways to stop: the goal is met, or a hard limit says "after N tries, stop and report."

If any of these three is missing, you do not have a loop. You have a cron job, a chatbot, or a cost line.

The chart above is the real thing. The repo is 91.1k stars, MIT licensed, and the progress.png sits at the top of the README. The dark cluster on the right is the agent's overnight run, the lighter earlier cluster is Karpathy's hand-tuned baseline. The two do not overlap.

The Karpathy setup

The repo is three files. About 630 lines of code.

• prepare.py is the evaluator. It downloads data, trains a tokenizer, computes val_bpb. The agent cannot touch it. If it could, it would just make the test easier instead of making the model better.
• train.py is the training script. The only file the agent is allowed to edit. Model, optimizer, hyperparameters, attention pattern, all fair game.
• program.md is the instructions. The human writes this. The agent reads it.

The agent runs in a loop. Read the code. Propose a change. Train for 5 minutes wall clock. Check if val_bpb improved. Commit if it did, roll back if it did not. Repeat. You go to sleep. You wake up to a log of experiments and, usually, a better model.

The 5-minute time budget is deliberate. It makes experiments directly comparable across hardware. It also means the agent finds the best model for your platform, not the best model in absolute terms. A smaller model that runs 200 experiments in your window will beat a larger model that only fits 20.

Karpathy's actual result is the strongest version of the claim. He pointed the agent at a project he had over-tuned. Two decades of manual research. The agent found improvements like a missing scalar multiplier in QKnorm that made attention too diffuse, the absence of weight decay on value embeddings, AdamW betas that were all set incorrectly, and an undertuned banded attention pattern. He tested the changes on a depth=24 model. All 20 transferred. The leaderboard entry moved.

Shopify CEO Tobi Lutke ran the same pattern overnight on an internal model. He woke up to a 19% quality improvement. The optimized model was half the size of his previous one. A smaller model beat a larger one because the agent optimized for the hardware instead of defaulting to "bigger is better."

Karpathy's line on the whole thing: if you have an objective metric, you should not be the one running the experiments. You are the bottleneck. Remove yourself from the loop and let it run.

The 5 building blocks

Every working loop, whether you build it in Claude Code, Codex, or bash, is assembled from five pieces. Both Claude Code and Codex ship all five now.

• Automation is the heartbeat. Something that fires the loop on a schedule, on an event, or on a trigger. In Claude Code: /loop for cadence, /goal for running until a condition holds. In Codex: the Automations tab. Without the heartbeat, you ran a script once and forgot about it. That is not a loop.
• A skill stores project knowledge so the agent stops guessing every session. Your conventions, your build steps, the rule you do not break because of that one incident three months ago. Written once in a markdown file, read by every run. Without skills, the loop re-derives your entire project context from zero every cycle.
• Sub-agents split the maker from the checker. The model that wrote the code is too generous grading its own homework. A second agent with different instructions catches what the first one talked itself into. Your writer can be fast and cheap. Your reviewer slow and strict. That separation is most of the quality.
• Connectors let the loop act inside your real environment. Read your issue tracker, open a PR, ping Slack, update a Linear ticket. The difference between an agent that proposes the fix in chat and a loop that ships the fix and reports it in the morning.
• A verifier is the gate. The test, type check, lint, or build that automatically rejects bad work. Everything else is plumbing. This is the part that makes the loop real. Without it you are paying for an agent to agree with itself all night.

If a loop is missing any of the five, it works once. Then it stops working the day the assumptions change.

The loop on the loop

A March 2026 arxiv paper called "Bilevel Autoresearch: Meta-Autoresearching Itself" took the Karpathy setup and asked a simple question. If autoresearch is itself a form of research, can autoresearch research autoresearch?

The answer was 5x.

The architecture is two loops stacked vertically:

• The inner loop does what Karpathy's original does. Propose a change. Train. Evaluate. Keep or discard.
• The outer loop watches the inner loop work. It reads the inner loop's code and run traces, identifies where the search process itself is getting stuck, and generates new Python that changes how the inner loop searches. Then it injects that code and lets the inner loop run again.

On Karpathy's GPT pretraining benchmark, the bilevel setup reached val_bpb of -0.045 against the single-loop baseline of -0.009. Five times better. Both levels used the same LLM. You do not need a smarter model for the meta level. The improvement comes from the architecture, not from raw intelligence.

What the outer loop actually found is the part worth sitting with. The inner loop kept falling into the same search patterns. The LLM has priors about which optimizations to try, and it keeps returning to those priors even after they stopped working. The outer loop broke those patterns by forcing exploration in directions the model's instincts avoided.

The paper's closing line: "If autoresearch can meta-autoresearch itself, it can in principle meta-autoresearch anything with a measurable objective."

The honest part

Loops change the work. They do not delete you from it. Two problems get sharper as the loop gets better, not easier.

Comprehension debt. The faster the loop ships code you did not write, the larger the gap between what exists in your repo and what you actually understand. A smooth running loop charges compound interest on that gap. The day you have to debug a system nobody on the team has read will cost more than the tokens ever did.

Cognitive surrender. When the loop runs itself, it is tempting to stop forming an opinion and just accept whatever comes back. Designing the loop is the cure when you do it with judgment. It is the accelerant when you do it to avoid thinking. Same action, opposite result.

Two people can build the exact same loop and get completely opposite outcomes. One uses it to move faster on work they understand deeply. The other uses it to avoid understanding the work at all. The loop does not know the difference. You do.

Karpathy stopped writing code in train.py. He did not stop thinking about what the agent should try. The discipline is the same one you bring to any system you operate: design the loop, watch it run, audit the output, and never let the loop outpace your ability to question it.

Source