Research / report

Stem-Cell Micro-Harnesses: Model Choice, Selection Machinery and Code Diversity in Self-Improving LLM Agents

1. Introduction and research questions

Agent harnesses usually come pre-built: a fixed toolset, a fixed loop and a model behind it. We asked the opposite question. What happens when a harness starts from nothing, builds only the tooling its task demands, and improves through a biological loop: vary, compete, keep the winner, record the lesson? The program is open-ended by design. The project lead handed full agency over its direction to the investigating agent, and asked for constant iteration, the way the cells themselves are meant to work.

2. The system

2.1 The cell

A cell is a folder. It holds a read-only NEED, a self-written genome.md (memory, capped at 4,000 chars), any files it creates, and an append-only log.jsonl. On each step the cell's brain (one model call through Exo's Router, with a per-cell session_id) chooses exactly one action. Implementation: code/cell.py in the record (486 lines, stdlib plus Exo's router). The action set is in Appendix B.

2.2 Sandbox and safety

Every shell action runs under sandbox-exec: no network, no writes outside the cell, no reads of $HOME, 30 s limit. Each rule was verified on its own (outside write denied, curl exit 6, ~/.zshrc read denied). Model access goes only through the router in the parent process, so cells never see credentials.

2.3 Selection machinery

• Assay validation. The harness runs the cell's fitness command 5 times on the current body. It is accepted only if it exits 0, prints a number above 0, and has spread (max−min)/median of 15% or less.
• Win rule. A challenger wins only if median(challenger) beats median(incumbent) by more than max(3%, measured noise), and the challenger's worst run is no worse than the incumbent's median. Ties keep the incumbent.
• Bench lock. One measurement at a time, machine-wide, so timing assays do not disturb each other.
• variants: the model writes 2 to 6 full versions of one file in one reply, and the CPU picks. This is cheap mutation with strict selection.
• try (asymmetric mitosis): one daughter cell explores one idea for half the parent's remaining steps (depth 1). The parent stays the control. If the daughter loses, an "autopsy" model call writes a one-line lesson into LIBRARY. This is the first form of loser-informed mitosis.
• Germline. Every win and every loss lesson is appended to a colony-wide LIBRARY.md (max 400 chars per line, never code), which all cells read.
• Apoptosis. Three failed steps in a row end the cell. Rate limits and timeouts back off instead (at most 6 waits, added after b6).

2.4 Colony runner

colony.py runs N cells as separate processes. It samples RSS and CPU of every process tree every 2 s and writes summary.json. Each cell can be pinned to one model. A pinned cell has no hidden fallback, and the route of every call is logged.

2.5 Version history

3. Scientific method

3.1 Design

The program runs in two phases, and every result is labelled with the phase it came from.

• Exploratory phase (done: c1 to b11). Iterative engineering and observation. The harness changed between runs on purpose. Results here generate hypotheses. They do not confirm them.
• Confirmatory phase (from now). The harness is frozen at a tagged version for each experiment. Hypotheses, predictions and decision rules are written down first (section 9). Each condition gets at least 3 replicate runs, and conditions are interleaved in time to spread out provider and machine drift.

3.2 Units of analysis

3.3 Procedure for one run

1. Freeze code. Record the cell.py and colony.py versions and the exo-live commit (currently 6054da56).
2. Remove the bench lock and clear the run folder.
3. Launch colony.py RUN needs8v3.json --par 8 --calls 40 --steps 16 [--model REF] as a harness background job.
4. Do not touch the run. No polling, no restarts, no edits.
5. On exit, read summary.json and check from routes that every call was served by the intended model.
6. Extract metrics (section 4.2). Embed variants (b8 onward) with all three embedders.
7. Append a dated entry to TESTS.md before interpreting anything.

3.4 Instrument validity checks

• The sandbox was verified with negative tests (2.2).
• Assay acceptance rejects unstable or wrong-output measurements before any comparison.
• The win rule requires clearing measured noise. We have observed fake-win protection: in g2 a 0 vs 0 split was decided, which v3 now blocks.
• Route logging proves model identity per call. Pinned cells cannot silently fall back.
• Only primes (348,513 below 5,000,000) and fibmod (fib(10) = 55) state a correct answer in the NEED. The other 6 tasks rely on the cell's own correctness check inside its assay. This is a known weakness: a cell could reward a fast wrong answer it fails to detect. Adversary cells (R6) and harness-held reference outputs are the planned fix.

4. Variables

4.1 Independent variables (manipulated)

4.2 Dependent variables (measured)

4.3 Controlled variables (held constant)

4.4 Uncontrolled and confounding variables

5. Analysis plan and decision rules

• Primary comparison: kept improvements per 100 calls, one value per run. Two conditions: Welch t-test and Mann-Whitney U, two-sided, α = 0.05. More than two conditions: Kruskal-Wallis, then pairwise tests with Holm correction.
• Effect size: report the difference in means with a bootstrap 95% CI (10,000 resamples of runs) and Cohen's d. A difference under 2 per 100 calls is treated as practically equal.
• Replication: at least 3 runs per condition before any ranking. 5 runs for headline claims. Our observed SD (1.5 to 2.6) means n = 5 per arm detects a gap of about 3.5 per 100 at 80% power.
• Embeddings: permutation tests shuffle whole runs between labels, not single variants, because variants share lineage. A model-label effect counts only if the cross-model gap exceeds the same-model gap.
• Correlations: report r with n and a note when the measures are linked by construction (variants and improvements).
• Scaling gate: add cells or colonies only when the rate rises in two runs in a row. Scale for throughput is allowed when the rate per 100 calls holds steady.
• Stopping: a condition stops early if 2 runs in a row produce 0 kept improvements and a log check shows no harness fault.
• Exclusions: a run is excluded only for a verified harness fault (example: b6-failed-400, where every call returned HTTP 400). Exclusions and their reasons are kept in TESTS.md, and the data is kept.

6. Results to date (exploratory phase)

6.1 Feasibility (pilots)

c1 (word frequency) built a working, tested tool in 10 calls and 50.7 s. c2 (primes) ran the first mitosis: daughter A (odd-only half sieve) scored 34.6 and daughter B (full bytearray sieve) 27.9 on 1/s. The winner re-verified at about 30 ms for N = 5,000,000 with the correct count, in 50 calls and 474 s. RQ1: yes.

6.2 Harness design (g1 to g4)

v1 and v2 did not log kept-improvement events, so no rate is available for them.

• Membrane failure (g2): 46 replies were lost (22 written as tool markup in text, 24 empty). After v3, malformed replies ran at 0 or 1 per run.
• Untrusted fitness (g2): a split was decided with both daughters scoring 0. The validated assay now blocks this.
• Symmetric mitosis cost (g2): split twins used 14 to 24 calls against 14 for plain cells, and 2 split cells did not finish.
• Scaling gate failed (g3 to g4): the rate fell from 3.12 to 1.57, so colonies stayed at 8 cells.

6.3 Model comparison

6.4 Protocol adherence

• The variants share correlates with the rate at r = 0.991 (n = 8 runs; 0.938 for frontier runs only). The two are partly linked by construction, since most kept wins come through variants.
• Only Astra used mitosis (try). 8 tries, 2 won: a reversible word-token transform for compress (+37%) and a native C fast path for csvsum (+826%). The 6 losers each wrote an autopsy lesson to LIBRARY. No other model ever called try. So loser-informed mitosis exists in the code but has barely been exercised.
• Only Fable used remember (its genome), without being prompted.
• hy4 case: 0 variants and 0 kept, but it benchmarked 12 encoders inside one shell script and reached a 14.3x compression ratio by hand. It optimized outside the measured channel.

6.5 Task structure

• fibmod gave the largest compounded gain in 5 of 7 frontier runs: Argon 251.9x and 47.0x, Fable 43.4x and 42.9x, Astra 27.9x. The exceptions were Fable b9 (1.24x) and hy4 (no variants). The task has an algorithmic jump (naive to fast doubling), and selection finds it.
• compress was flat in 5 of the 6 frontier runs that finished it (the exception was Argon b8 at 1.35x). The first stdlib choice is already near the ceiling.
• csvsum deaths depend on the model (hy4 and Argon b8 died, Fable never did). Our first reading, that the task was badly written, was wrong.

6.6 Embeddings

Embedder A/B (synthetic set): all three embedders gave the same ordering. Spreads: nomic 0.20, 3-small 0.38, 3-large 0.31 to 0.35.

E1 split width vs gain (b8, 5 batches): the only adopted batch (maze, +10.5) was the widest. With n = 1 adoption, we make no claim.

The same-model control breaks the fingerprint reading. Two Fable runs differ about as often as Fable and Argon do. What survives: Argon converges on similar code from run to run, while Fable's code varies widely, and that extra variety does not show up as a higher rate.

6.7 Resources

8-cell colonies peaked at 0.84 to 1.38 GB RSS. Peak CPU was 1.1 to 3.2 cores, and mean CPU was 7 to 72% of one core. Cells are I/O bound (they wait on the model). The serialized bench is the only real CPU consumer. A 64 GB M1 has headroom for many more colonies.

7. Discussion

1. Improvement rate behaves like capability × protocol adherence × harness quality. Each factor alone changed outcomes by large amounts, and adherence can zero out capability (hy4).
2. "The model is key" holds between tiers but not within the frontier. At the frontier, run-to-run variance dominates.
3. The selection layer is the instrument. It blocks fake wins, but it also cannot see gains made outside it. That trade-off is the reason for the adherence research thread.
4. The mitosis action is under-used. Models prefer cheap variants to an expensive daughter cell. Loser-informed mitosis can only be studied once cells actually divide, by prompting it or by making the harness divide on schedule.
5. Embeddings need lineage-aware statistics. Naive tests produce false fingerprints.

8. Threats to validity

9. Hypothesis register

9.1 Hypotheses already tested (exploratory)

9.2 Registered hypotheses (predictions written before testing)

Each one gives a prediction, what would falsify it, and how confident we are now (a subjective prior). Results will be filled into this table without editing the predictions.

10. Areas of exploration and experiment roadmap

10.1 Map of areas

10.2 Planned experiments in detail

10.3 Order of work

1. Instrument first: log cost per call, add harness-fixed reference workloads, add the Responses API transport for gpt-6, and tag a frozen harness version.
2. Cheap, high-information: H1 (hy4 adherence), E3 offline on existing archives, S1 microbenchmark.
3. Model gradient: B2 to B4 and B7, then reps of Fable and Argon to 5. C1 frontier plot.
4. Mitosis design: M1 forced and loser-informed mitosis, then M2.
5. Mixtures: D1, E2.
6. Formal: R2, then R1, then R3/F3.
7. Agency: R5, R6, then R4.
8. Scale: S3, only after the gate in section 5 allows it.

Volume estimate: one run is about 120 model calls. The plan above is roughly 100 runs, or about 12,000 calls. Cost per call has not been measured yet, so no dollar estimate is given.

11. Safety, ethics and governance

• Cells hold no credentials and have no network access. All model traffic passes through the parent's router, which logs every call's route.
• Cells cannot read the user's home folder or write outside their own folder. This was verified with negative tests, not assumed.
• Meta-cells (R4) will never get the assay, the sandbox profile or the selection rule inside their editable body. Without that rule, a cell could improve its score by editing the judge.
• Human oversight: the project lead sets budget and direction. Outward actions (publishing, posting) need the project lead's approval.
• Embedding calls to OpenAI send cell-written code only. That code contains no personal data.
• Budget: the project lead granted an uncapped budget for this program, with model choice limited to configured or previously used models.

12. Harness findings filed during the program

13. Reproducibility, data dictionary and timeline

13.1 Files

• Code and data: published as the E07 record in the Weft repository, under docs/program/experiments/E07/razi-cells-2026-11/ (code/: cell.py, colony.py, embed.py, perm.py; data/: run summaries for g3, g4 and b5 to b11). Record, under review.
• Logs: exp/TESTS.md (dated results), exp/IDEAS.md (idea log A, R, E, F), this report (REPORT.html; version 1 kept as REPORT_v1.html).
• Toolchain: Lean 4.35, Bend 0.2.38, HVM 2.0.22 (Bend smoke test returned 499,500 on both runtimes), Ollama with nomic-embed-text.

13.2 Data dictionary

13.3 Timeline

14. Glossary

15. Appendices

A. System prompt (verbatim)

You are a cell: a tiny agent that starts with nothing but a NEED and an empty folder (your body).
Grow only the tools your need demands, one step at a time. Call exactly one tool per reply.
Path to fitness: 1) build a correct version and check it. 2) register an `assay`: a shell command that
prints one number, higher is better, 0 if the output is wrong. Make its workload heavy enough (about
0.3 to 3 s) that process startup does not dominate. 3) improve with `variants` (cheap: several versions
of one file in one reply, all scored) or `try` (expensive: a daughter cell explores one bigger idea).
Only scored wins are kept. Read LIBRARY: other cells' results and lessons, often useful to you.

B. Tool specification (v3)

C. Tasks (needs8v3.json)

Fitness is 0 when output is wrong. The assay command and workload are written by each cell and validated by the harness.