Karpathy Autoresearch: Single-GPU Setup and Experiment Review

Editorial update: November 2, 2026. This article now follows Karpathy's autoresearch repository. The commands below are an upstream setup sequence, not a report of a training experiment performed by CCAPI.

What autoresearch actually runs

Karpathy's autoresearch gives a coding agent a small training experiment derived from nanochat. The agent changes training code, runs an experiment and decides whether to retain the change. The reviewed README specifies a single NVIDIA GPU, Python 3.10 or later and uv; it reports testing on H100 hardware.

The documented training budget is five minutes per experiment, excluding startup and compilation. The comparison metric is validation bits per byte, val_bpb, with smaller values preferred. This budget supports comparison on the same setup; it does not make results across different GPUs directly equivalent.

Three files define the basic boundary: prepare.py handles fixed preparation and evaluation support, the agent edits train.py, and the human maintains the research instructions in program.md. This is a specific training-code experiment, rather than a generic multi-agent conversation framework.

Establish a baseline before automating experiments

After checking the upstream prerequisites, you can inspect a fixed revision and follow its setup sequence:

git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
git checkout --detach 228791fb499afffb54b46200aca536f79142f117
uv sync
uv run prepare.py
uv run train.py

The final two commands prepare data and run training. They require the documented environment and consume local resources; we did not execute them for this article. A successful dependency installation alone would not demonstrate a successful training run.

Before starting an automated loop, save the baseline commit, environment, training output, final validation metric and any failure log. Read the upstream research instructions at the same revision before deciding what your agent may edit and execute.

Keep each experiment reviewable

The following record format is our suggested operational checklist:

Field Why save it?
Starting commit and patch Reconstruct the exact candidate
Hypothesis Explain what the change is intended to improve
Environment and hardware Recognize changes that affect comparability
Training and evaluation output Separate a completed measurement from a partial or failed run
Keep or discard decision Connect the decision to the observed metric

Test the stop and recovery path before leaving a loop unattended. A crash, missing evaluation or exhausted resource budget should remain a failed experiment in the record, rather than becoming a claimed improvement. Review retained changes together to spot accidental changes to the comparison conditions.

Separate the coding agent from the training workload

The coding agent may use a model service to propose edits; the GPU executes the training experiment. Access to an inference API does not supply that GPU or prove a training gain. If you evaluate CCAPI for an agent's model requests, verify the client's endpoint and model against the CCAPI documentation independently of the training setup.

This revision removes the earlier generic AutoGen discussion, unverified configuration examples and claimed training improvements. It provides source-backed setup context and a measurement checklist, without presenting unrun experiments as results.