Fifteen companies, 480 containers, one awkward profit question
We provisioned 480 Linux containers on a Mac, split across fifteen private service networks. Docker ran out of bridge addresses after 253. That was the first problem to fix before the machines could do anything useful.
The intended job is a reporting demo for fifteen fictional companies. Take Margin of Error Ltd: sales are up, profit is down, and someone wants to know where the money went. A collection of workers should analyze its invoices, returns, inventory, and discounts, then produce an HTML report with CSV files behind the numbers.
The provisioning and a hundred save-and-stop operations are verified. The reporting application is still a design, so the queue, analysis, validation, and report examples below describe what remains to be built. Booting 480 containers is useful evidence about the platform. It doesn’t answer the profit question.
Designing one company
Each company gets one private service network. The application design assigns four of its thirty-two containers to shared services and the other twenty-eight to analysis. Node code and small JSON shards should keep individual jobs within a Lite machine’s 256 MiB of memory. The illustration shows three of the fifteen planned companies.
| Role | Count | Responsibility |
|---|---|---|
| Coordinator | 1 | Task leases, attempt IDs, progress, and completion records. |
| Data service | 1 | Immutable input shards, schemas, and content hashes. |
| Artifact service | 1 | Accepted results, CSV exports, and the rendered report. |
| Validator | 1 | Independent checks on totals, coverage, and provenance. |
| Analysts | 28 | Partitioned calculations for revenue, returns, costs, and margin changes. |
A task might compare two months of returns in one region, calculate a discount’s effect, or reconcile a product category. Those calculations can be partitioned without making every worker load the company’s entire history. The coordinator needs to track those partitions so a missing shard can’t quietly become a lower total.
The controller talking to the LLM belongs outside these company containers. It can turn a question into an analysis plan and generate code when needed, while retaining the model credentials and Mainbrella API key itself. Workers get code and task data. They don’t need permission to create more machines.
Record the launch before sending it
Before launching the batch, we checked GET /capabilities and authenticated GET /containers. The deployment needed private services, the selected image, and execution and file operations. The account also needed enough starts, slots, concurrent compute units, and runtime allowance. A launch loop should discover those limits before it fills half a network.
The Scale account used for this run allowed 500 simultaneous containers and 640 concurrent compute units. Our 480 Lite machines consumed one concurrent unit each. Private Services allowed sixteen networks per account and thirty-two members per network, so fifteen groups of thirty-two fit without changing those limits.
We recorded each intended creation before sending its request and bounded the launch concurrency. This server-side Node example shows the record a batch runner needs to keep:
// Server-side Node. Keep this key out of company containers.
import { randomUUID } from 'node:crypto';
import { mkdir, writeFile, rename } from 'node:fs/promises';
const origin = process.env.MAINBRELLA_API_URL
?? 'http://localhost:8787';
const key = process.env.MAINBRELLA_LOCAL_API_KEY
?? process.env.MAINBRELLA_API_KEY;
if (!key) throw new Error('Provision a server-side API key');
async function api(path, { method = 'GET', body, headers = {} } = {}) {
const response = await fetch(new URL(path, origin), {
method,
headers: {
Authorization: `Bearer ${key}`,
...(body === undefined ? {} : { 'Content-Type': 'application/json' }),
...headers,
},
body: body === undefined ? undefined : JSON.stringify(body),
redirect: 'error',
signal: AbortSignal.timeout(90_000),
});
if (!response.ok) throw new Error(`Mainbrella HTTP ${response.status}`);
return response.json();
}
async function journal(name, value) {
await mkdir('.mainbrella', { recursive: true, mode: 0o700 });
const path = `.mainbrella/${name}.json`;
await writeFile(`${path}.tmp`, JSON.stringify(value), { mode: 0o600 });
await rename(`${path}.tmp`, path);
}
const operation = {
key: randomUUID(),
issuedAt: new Date().toISOString(),
body: { catalogId: 'node', size: 'lite' },
};
await journal(operation.key, operation); // Before the network request.
const started = await api('/containers', {
method: 'POST', body: operation.body,
headers: { 'Idempotency-Key': operation.key },
});
const { creation } = started;
if (creation?.status !== 'running') {
throw new Error('Reconcile the saved creation key and body');
}
const machine = started.containers.find(c =>
c.id === creation.containerId && c.createdAt === creation.createdAt
&& c.status === 'running');
if (!machine) throw new Error('Running generation missing');
const identity = { id: machine.id, createdAt: machine.createdAt };
await journal(operation.key, { ...operation, identity });
If the response disappears, that journal lets the controller retry the same body and Idempotency-Key within the twenty-four-hour recovery window. The machine may already be running. Generating a fresh UUID would ask the backend to launch another one.
The account coordinator reserves the slot and quota and saves the keyed creation record before dispatching the boot. Booting happens outside the reservation lock. Pending slots count toward capacity, so concurrent requests can’t all claim the same last slot.
We kept both id and createdAt for every running container. The ID names a reusable slot; the timestamp identifies the machine currently occupying it. If a worker stops and its slot is reused, a delayed cleanup request needs both values to avoid killing the replacement.
Calling coordinator.internal
The reporting design reuses four service names in every network: coordinator, data, artifacts, and validator. A service registers the port that accepts requests. A worker joins as a caller-only member, with no exposed port. For example:
// coordinator and worker are previously recorded { id, createdAt } pairs.
const network = 'margin-of-error';
await api('/private-services/networks', {
method: 'POST', body: { name: network },
});
const membership = '/private-services/members?'
+ new URLSearchParams({ network });
await api(membership, {
method: 'PUT',
body: { ...coordinator, name: 'coordinator', port: 8080 },
});
await api(membership, {
method: 'PUT',
body: { ...worker, name: 'analyst-01' }, // Caller-only member.
});
// Inside that worker: port 80 is the private HTTP routing entry point.
const response = await fetch('http://coordinator.internal/tasks/next');
if (!response.ok) throw new Error(`Queue HTTP ${response.status}`);
const task = await response.json();
Inside this worker, coordinator.internal uses plain HTTP on port 80. The coordinator application listens on port 8080. Mainbrella maps between them using the source’s account, network membership, and exact container generation.
The outbound interceptor in private-services-runtime.js handles *.internal. Its relay strips reserved identity headers and supplies trusted source identity through the runtime entrypoint. A worker can’t select another account by inventing an x-mainbrella-user header.
PrivateServicesController.route() finds the source network, then a destination with the requested name and a registered port. Before opening that port, the destination checks its generation and registration under its lifecycle lock. Membership is checked again before releasing the response, since a detach or replacement can happen while the request is in flight.
The prototype supports private HTTP routing, with one-MiB request and response bodies and a ten-second timeout. It doesn’t provide general TCP networking, PostgreSQL connections, or a different outbound internet policy. Small shards and paginated records fit that contract. Each company can resolve its own data.internal; it can’t resolve a service registered only in another company’s network.
Private Services requires networking.privateServices: true and deployment enablement. This run used the local prototype. Its routing checks alone don’t establish enterprise tenancy or compliance.
Assigning work without accepting it twice
Mainbrella can keep a command running and retain its result. The reporting app still needs to decide who owns each piece of analysis. Its coordinator needs a task queue that leases a shard to one attempt and accepts one valid completion. Here’s the proposed task record:
// Application protocol; Mainbrella does not supply this task queue.
{
"runId": "profit-2026-09",
"taskId": "returns-west",
"attemptId": "returns-west-02",
"leaseToken": "<opaque token issued by the coordinator>",
"inputSha256": "<hash of the assigned input shard>",
"leaseExpiresAt": "<application task deadline>",
"inputUrl": "http://data.internal/shards/returns-west.json"
}
Suppose a worker calculates a result and dies before acknowledging completion. After the task lease expires, the coordinator assigns it again. The first attempt might wake up later with an answer, so accepting any result with the right task ID would be a bug. Check the attempt ID, lease token, deadline, and input hash before accepting it.
The coordinator needs durable completion records keyed by (runId, taskId), with at most one accepted result per key. Immutable input shards and results carrying the shard hash and analysis version make that result inspectable. Otherwise “September returns” might mean whichever copy of the dataset a worker happened to fetch.
A task lease determines which attempt owns the analysis. The container lease determines how long its compute can run. Extending a task lease won’t buy the worker more runtime; the scheduler has to account for both.
Keeping track of longer jobs
A quick probe fits foreground POST /containers/exec, which has a sixty-second maximum timeout. For an analyst processing a batch, use managed execution and save the job ID alongside the container’s generation:
// worker.mjs has already been uploaded into /workspace/demo.
const query = new URLSearchParams(worker);
const execution = {
key: randomUUID(),
body: {
argv: ['node', '/workspace/demo/worker.mjs'],
timeoutMs: 120_000,
},
};
await journal(execution.key, { ...execution, container: worker });
const job = await api(`/containers/executions?${query}`, {
method: 'POST', body: execution.body,
headers: { 'Idempotency-Key': execution.key },
});
await journal(execution.key, { ...execution, container: worker, jobId: job.id });
// SSE: /containers/executions/{job.id}/events?...&cursor=0
// Poll: /containers/executions/{job.id}?...
// Cancel: DELETE /containers/executions/{job.id}?...
// Closing the SSE connection only detaches the viewer.
argv passes literal arguments without assembling shell text. This example’s two-minute timeout fits under the fifteen-minute maximum, and the container’s remaining lease can shorten it further. Shared services also need an explicit lifecycle. Running them as managed jobs means deliberately restarting them when those jobs end.
executions.js saves the execution identity before starting the process. A matching keyed retry recovers the same job within its one-hour retention window. Output events have increasing sequence numbers; on reconnect, send the last processed sequence as cursor and ignore duplicates.
Closing a browser tab only detaches its stream. To cancel, delete the retained execution and check its terminal status. Check exitCode, timedOut, and outputTruncated before parsing the result too. Half a JSON document doesn’t become usable because the process printed a plausible first line.
Each container permits four concurrent command or file operations, with bounded output, and retains up to thirty-two managed jobs during the retention window. That favors one analyst execution processing a finite batch of task leases. Starting a retained execution for every invoice would spend the limit on bookkeeping.
Check the arithmetic independently
The report can use an LLM to explain results, but financial totals need deterministic code. The proposed calculations use integer cents, schema validation, row counts, and input hashes. Larger totals may need decimal arithmetic or BigInt with an explicit serialization format. There’s no upside to letting a language model improvise the arithmetic.
// Money stays in integer cents. Inputs are schema-checked first.
const profitCents = revenueCents - cogsCents - operatingExpenseCents;
const explainedChangeCents = contributions.reduce(
(sum, item) => sum + item.deltaCents, 0,
);
assert.equal(explainedChangeCents, currentProfitCents - previousProfitCents);
assert.equal(result.inputSha256, assignedTask.inputSha256);
assert.equal(result.attemptId, assignedTask.attemptId);
assert.equal(result.leaseToken, assignedTask.leaseToken);
assert.ok(Date.now() < Date.parse(assignedTask.leaseExpiresAt));
// Atomically accept at most one valid completion for this runId/taskId.
The validator needs to recompute the aggregates independently and check that the contributions add up to the profit change. Missing shards, duplicate completions, mixed analysis versions, and expired attempts should fail validation. An exit code of zero only says the worker finished successfully. Subtracting the wrong column can finish successfully all day.
Each company’s artifact service should produce report.html, metrics.csv, exceptions.csv, and a manifest of input hashes, analysis versions, and accepted task IDs. The narrative should link its claims to those calculations. Keep exceptions visible in their own export instead of quietly dropping them from the totals.
Report rendering also needs to escape data and keep generated scripts out of the HTML. Treat a company name as a string, even if it contains markup. Executing worker-supplied HTML would give analysis code another place to run when all we wanted was a report.
Download the artifacts through GET /containers/files with encoded paths and the exact container generation. Writes use raw bytes through PUT, require an existing parent directory, and replace regular files atomically. files.js rejects symlink write targets; reads can follow symlinks inside the owned guest. Larger exports need chunking or a portable-export workflow. Sending them through stdout just adds another truncation problem.
Running out of Docker addresses
Provisioning reached a limit before any of that application work could begin. After 253 task containers attached, three launches returned HTTP 503. The account had enough allowance, but Docker’s local default bridge was a /24. All 253 usable container addresses were occupied.
This was a Docker networking limit, separate from the fifteen logical company networks. We paused launches, kept the ambiguous creation keys, and moved only task-owned Docker proxies to a larger dedicated bridge. To identify each proxy, we ran hostname through its recorded container generation. All 253 migrated proxies passed a command-execution check afterward.
We then confirmed through the original creation keys that the three failed launches had stopped, and requested replacements with new keys. Provisioning resumed. The final check matched all 480 running generations against fifteen membership lists of thirty-two.
The saved creation records let us recover without guessing how many machines existed. Blindly replacing the three failed HTTP requests could have left extra machines running, which is exactly the kind of mess the launch journal is supposed to prevent.
Saving before stopping
We also exercised the save-and-stop path. Stopping normally discards the guest filesystem, so saving has to finish first. The controller should record the returned workspace ID before requesting teardown:
const save = {
key: randomUUID(),
body: { ...worker, name: 'margin-of-error-analyst-01', stop: false },
};
await journal(save.key, save);
const workspace = await api('/workspaces', {
method: 'POST', body: save.body,
headers: { 'Idempotency-Key': save.key },
});
if (workspace.status !== 'ready'
|| workspace.source.id !== worker.id
|| workspace.source.createdAt !== worker.createdAt) {
throw new Error('Save unresolved; preserve the source container');
}
await journal(save.key, { ...save, workspaceId: workspace.id });
const after = await api('/containers?' + new URLSearchParams(worker), {
method: 'DELETE',
});
if (after.containers.some(c =>
c.id === worker.id && c.createdAt === worker.createdAt)) {
throw new Error('Stop unresolved; reconcile this exact generation');
}
workspaces.js reserves save capacity before capture and persists the provider handle before optionally stopping the source. This example uses separate save and stop requests so the controller can record the workspace ID between them. The API also accepts stop: true in the save body.
Our account could retain one hundred saved workspaces. We saved the first hundred containers in manifest order, confirmed each workspace was ready for its exact source generation, and checked that those generations no longer appeared in the running list. The other 380 containers were still running at that check. The account’s save limit prevented us from keeping a workspace for every machine.
To restore, send POST /containers with { workspaceId } and a fresh, saved creation key. A restore consumes a start and returns a new generation. Files survive; RAM, processes, previews, and network memberships don’t. Restored services have to start again and register their new identities.
The reporting app will therefore need progress in files or durable coordinator records, with writers quiesced before a checkpoint. An in-memory queue would disappear on restore. Our hundred infrastructure saves exercise capture and teardown; they don’t show that a report can resume halfway through rendering.
What remains to build
The finished demo should let a viewer choose a company, ask the profit question, and follow a task through assignment, calculation, and validation into the report. Selecting a failed attempt should show its container generation, execution ID, logs, and input hash. That gives someone enough information to investigate the failure instead of watching a spinner.
We still need to measure completed, validated tasks per second, error rate, command latency, time to a finished report, and artifact correctness. Provisioning time and dispatch into an already-running worker need separate measurements. A large process count tells us how much compute we started; it doesn’t tell us whether parallelizing the analysis was worth it.
The controller also has to finish within the platform’s deadlines. Status polling doesn’t renew idle time, and a managed execution can’t extend the hard deadline. Saved workspaces have expiration dates and finite retention. Export the outputs, then stop the exact generations the controller owns.
The Mainbrella backend, HTTP contract, and agent setup workflow provide the infrastructure for this design. The next useful result is a metrics.csv whose profit calculation matches the input records, with enough provenance to explain every dollar.