One Bug, Three Clouds: Building Repo Rodeo

October 11, 2026 · Andrew Arrow

Move “Join the waitlist” to the top of the editor, hit Save, and everything looks fine. The button is still there. The editor says “Saved.” Reload the page and the button disappears. Three blocks went in; two came back.

You can try the broken editor without an account. We planted this bug deliberately. It gives Repo Rodeo a small, repeatable failure while we build the machinery for handling somebody else’s GitHub issue.

The product we want takes an issue and a reproduction recipe, prepares the application, then gives independent coding agents their own copies to repair. A maintainer should be able to open the broken app, open a proposed repair, repeat the same actions, and inspect the diff. Getting both apps into the browser would be useful even when every agent fails.

The three clouds in the title are E2B, Daytona, and Mainbrella. So far, the deterministic controls work in a manual local Mainbrella run and an automated E2B cloud run. Daytona reproduced the baseline bug, then ran into snapshot and cleanup problems. A completed three-provider comparison and AI repair attempts are still ahead of us. Before adding agents, we need to know that the experiment can reject a bad repair.

The editor holds the current document in memory and saves a serialized copy to localStorage. Its serializer assigns positions from the array index, then filters them:

blocks
  .map((block, position) => ({ ...block, position }))
  .filter(block => block.position);

Zero is false, so the first block gets thrown away. The storage write succeeds, which earns the “Saved” message, while the editor continues displaying its untouched in-memory draft. Moving the button makes the loss easy to see. Saving in the original order loses the heading instead.

These captures show the preview in the broken editor immediately after saving and after a browser reload. The same heading and paragraph survive; the button doesn’t.

After Save

After reload

The correct patch removes the filter. The deliberately misleading patch adds render() after saving, leaving the serializer alone. It redraws the document that’s already in memory, so a demonstration that ends at Save looks perfectly healthy. Reload still loses the first block.

That bad patch is useful. If our verifier accepts it, adding a cleverer agent just gives us a more persuasive wrong answer. The browser test therefore compares ordered block IDs and text before Save, after Save, and after reload. Looking for “Join the waitlist” somewhere on the page would be too weak; help text could contain the words while the document had lost the button.

The suite also saves without reordering, edits the text, checks that Reset leaves unrelated browser storage alone, and generates fresh documents from a recorded random seed. The seed makes the randomized case repeatable. These are checks of this editor’s persistence behavior, with known answers, rather than a score for an agent’s ability to repair unfamiliar code.

The retained local result has the expected pattern:

Five failures count as reproduction only when they show the specified loss. The verifier checks that the document was intact before reload and that afterward it contains exactly the expected document minus its first block. A timeout, HTTP error, missing test, or unrelated assertion produces an unverified result. An unverified baseline stops the experiment before either candidate runs.

This mattered in the first Mainbrella run. We archived a pinned checkout of this website, uploaded it through the file API, installed its dependencies in a container, and started Vite. The browser reached the protected preview and got HTTP 403. The experiment hadn’t reproduced the bug. It hadn’t even reached the editor.

Mainbrella forwarded the guest Host header as container, which Vite didn’t allow. Adding that exact host to the run’s launcher got us to the application. The launcher also set the local API address explicitly so the copied website wouldn’t default to the production API. Those were changes to the experimental launch setup; the host checkout stayed untouched. The run notes retain the failed attempt along with the working one.

Once the baseline reproduced the loss, we wanted three independent copies of that prepared application. Installing dependencies belongs in the common starting point. A patch belongs in one branch. The deployment advertised saved workspaces but disabled native filesystem forking and memory preservation, so we tested saving the files and restoring them into fresh containers.

The first capture request hit another obstacle: HTTP 429, workspace_quota_exceeded. The account reported 100 saved workspaces against a limit of 20. We left those workspaces alone. The manual experiment continued with a disposable account and a synthetic trial entitlement in local D1. That let us test capture and restore through the same API, but it tells us nothing about signup or payment.

One saved workspace then supplied the baseline, correct-patch, and misleading-patch branches. All three retained the checkout, lockfile, and installed dependencies. None needed another npm ci. A different marker file in each branch stayed private to that branch, and the source had no marker file.

Keeping the files was only half the question. The running server exposed a random nonce held solely in process memory. Its nonce stayed the same across capture. Each restored container initially refused connections; explicitly starting Vite produced a fresh nonce in each one. The files survived. The running servers had to be started again.

For this bug, a fresh process works fine. The interesting state is code on disk and a document the external browser can reconstruct. A bug involving a half-processed in-memory queue would need a different experiment. Calling both operations “copy the environment” would hide precisely the difference we’re trying to measure.

The same six browser checks ran outside all the containers, through each branch’s protected preview, with a fresh browser context per test. The restored baseline and misleading patch reproduced the loss. The correct patch passed, including all twelve reloads in the seeded holdout. The public observation bundle lets you inspect the recorded documents. Raw traces and screenshots remain private because the preview URLs carry bearer credentials; the public bundle alone can’t rerun the verifier’s full decision.

A repair attempt will also need to outlive the page watching it. In the manual run, we submitted an eight-second managed job, attached to its output, and closed the browser after the first message. A new browser recovered the same execution ID, both messages, and exit status zero. Reattaching didn’t restart the job. That’s a useful result for the eventual agent workflow, though the job itself was a deterministic command.

Cleanup gave us a less tidy result. All four fixture previews returned 404 after revocation. Deleting their exact container generations and the two saved workspaces left the fixture account with no containers, no workspaces, and no live reserved workspace bytes. Two Docker snapshot images still existed, with no attached containers. An API receipt saying “deleted” hadn’t reclaimed those images. The original account also retained a separate baseline container for inspection.

The cleanup record keeps those facts separate. We still need to measure physical snapshot retention. This is also why there’s no price or speed ranking here: the run used local OrbStack, and Docker inspection showed no matching per-container CPU or memory limits for the requested Small configuration. Those timings wouldn’t make a fair cloud comparison.

A subsequent commit turns the manual recipe into a tracked Mainbrella adapter. Its full capture-and-restore path still needs a live run. Its first recorded preflight encountered the original account’s quota problem and stopped before creating anything. The earlier manual result doesn’t automatically qualify the new implementation.

E2B gave us another way to reach a running machine without reaching a reproduced bug. Its console showed a running sandbox and process start and exit messages during setup. These are the Level and Message columns from that console capture:

The initial E2B run notes explain the setup failures. The template’s Node version was too old for the pinned Vite build. Installing Node 22.22.2 and setting the command’s PATH explicitly got the server running. The filesystem view makes the shell startup files inspectable: this capture shows .profile sourcing .bashrc when Bash is running. Installing a binary doesn’t by itself settle which one a login shell selects.

Authenticated HTTP readiness eventually returned 200, but Chromium followed a redirect and received 403. Our browser helper had authenticated the first request without handling the redirect chain correctly. The first retained cloud result stopped before capture or repair attempts.

After fixing authenticated redirects, the follow-up run got through the baseline and made one native fork request for three branches. Each branch inherited a listening server, the source’s process nonce, and its PID. We stopped those inherited servers and started fresh ones for patch verification. That lets us test E2B’s memory-preserving fork while keeping the repair checks repeatable.

The terminal capture belongs to that run’s source sandbox. Its header shows four concurrent sandboxes while htop is open: the prepared source and the three branches were alive together.

The verifier then got the same answers as the local controls: the baseline and misleading patch lost the first block, and the correct patch passed all six checks, including twelve holdout reloads. Each branch kept its own marker and expected source hashes after the others finished. A new SDK client also recovered the same background command and its output after a stream disconnect. Cleanup confirmed all four sandbox IDs absent from the API and the applications unavailable through their previews. This run used the deterministic patches throughout; no repair agent participated.

Daytona’s setup makes the sandbox class worth checking before asking it to fork. The details panel in this capture identifies a Container, with one vCPU, 1 GiB of memory, 3 GiB of disk, and private preview access. Its lifecycle settings are visible alongside those resources.

Daytona’s audit log records a successful create, a fork rejected with HTTP 422, and a successful delete for the same sandbox. Its action and outcome fields read:

create   200   Success
fork     422   Error
delete   200   Success

That rejection gives us a reason to keep filesystem snapshots and memory forks separate in the adapter. Daytona documents filesystem captures for containers and memory captures for VMs. The separate snapshot run reproduced the baseline bug, but capture exceeded its original three-minute deadline. A later inventory showed the snapshot active. Deleting it returned 403 because the key lacked snapshot deletion permission. The failed result remains failed: none of the restored controls ran, and cleanup is still unconfirmed.

You can run the standalone controls now from the repository, using its pinned Node version:

npm ci
npx playwright install chromium
npm run experiment -- --seed 12345

The command writes a new evidence directory with the inputs, patches, document observations, traces, screenshots, and cleanup record. Its temporary working copies are a local test arrangement, not security sandboxes for untrusted repositories.

Daytona still needs the three restores and their control results. Before another run, its preflight needs to check that the credential can delete the persistent snapshot as well as create and delete the source container.