What a 100-machine burst measures
We have an old result reporting 100 successful concurrent machine starts with a P95 of 680 ms. A separate creation test reports 30 successes, a P50 of 381 ms, and a P95 of 528 ms. Those are appealing numbers, but the repository is missing the raw samples, original harness, test date, region, image, and even the exact points where the timer started and stopped.
We can report those prototype observations. We can’t honestly use them to claim a cold-start time, describe today’s API latency, or beat somebody else’s benchmark. Without the conditions, lining up two P95s is just comparing numbers with the same unit. The benchmark record keeps the old results and their missing context together.
Start the clock where the client starts waiting
The current HTTP benchmark follows the same workflow an agent uses to check its setup. Each sample requests a machine with an idempotency key, waits until the creation record matches a running instance, and executes a hello command. It checks the exit code and output, writes and reads a six-byte binary probe, then attempts cleanup.
We record two times. Creation time starts with the first launch request and ends with the returned running state, including retries along the way. Time to the first command response starts at that same request and continues through execution. A machine can report that it’s running before the client has done anything useful with it, so that gap deserves to be visible.
File verification and cleanup happen outside both timing intervals, but a sample has to pass them too. The JSON report also keeps the creation request count, selected catalog ID, account limits, client metadata, and individual outcomes.
Don’t retry the bad samples out of the results
The runner launches a batch up to the requested concurrency and waits for that batch’s cleanup. If any sample fails, it stops after the batch. Launching replacement machines until the success count looks right would hide the failures we need to see.
P50 and P95 use the nearest-rank method over successful samples. Attempted and successful counts, along with individual failures, remain in the JSON. Read them together: a fast percentile says little if half the attempts failed.
The report marks image cache state as uncontrolled and compute region as unreported. A new creation identity doesn’t establish that the host or image cache was cold. Keep the runner version beside the raw report so someone else can check the timing intervals and percentile calculation.
Choose the run size before launching
The read-only preflight checks available starts, slots, and the selected runtime image. Each sample consumes a start. If an agent runs the benchmark for you, approve the sample count before it starts; a 100-machine burst needs an account with enough concurrency and allowance.
We haven’t run a new benchmark for this article. To collect your own results, use the public HTTP runner, its verification workflow, and the run instructions. Save the full JSON, including failures, alongside whatever number you publish.