A multi-agent harness with rules in code
context
My work runs through a multi-agent system I built. Client work, job search, interview prep, my notes. I decide. Agents execute, in parallel, in the background.
The core is agent-agnostic. Hooks only call it, so the same rules work with any coding agent.
problem
A rule written in a prompt is a suggestion. An agent once created a ticket in a team tracker without asking. The rule was written down. Nothing enforced it.
An agent's report is a summary with a bias. Two agents in one checkout made a test count lie: 204 tests on the dirty tree, 197 on a clean checkout.
And a gate that cannot run looks exactly like a gate that found nothing. A judge that ran out of output budget returned empty content. The code read that as zero findings.
decisions
Chose: rules that matter are code. A hook denies the edit at write time. A written rule with no mechanical check goes on the gap list.
Rejected: longer instructions. More prose is more context to skip.
Chose: human gates only where the action is public or hard to undo. A write to a team tracker. An email to another person, which only ever goes out as a draft. Promoting a note into durable memory. A merge. Everything else runs on its own.
Chose: fail-open or fail-closed by the cost of the error. A broken probe warns and continues. A probe that worked and proved a violation blocks.
mechanism
orchestration
- Roles with a model routed by risk. Mechanical work on the small model. Building on the mid tier. Review and architecture on the top tier.
- One git worktree per front. Every brief is self-contained, with a done condition.
- Review reads the diff, not the report. It runs in a clean checkout and attacks the tests by mutation. A hook denies the merge without a review of the current commit.
hooks
- Deny a multi-line comment block at write time.
- Deny
git stash, which is shared across worktrees. - Send a broad scan of another repository to a cheap subagent.
- Inject an output contract on every turn.
retrieval over my notes
Local embeddings on CPU, no API call. About 10 thousand chunks, re-indexed by content hash when a session starts. BM25 and vector search, fused by best rank: a variant of reciprocal rank fusion, because my two retrievers hit different questions. A capped weight for source authority breaks ties.
Measured against a gold set of 22 questions, hit@k and MRR per leg. Declared bias: I wrote the questions knowing the answers.
judges
LLM-as-judge (a model scoring another model's output) for CV claims against my evidence files, memory coherence and prose. Strict schema. A truncated or empty answer fails loud. Calibrated with a planted fixture and a clean one. The stable prefix is prompt-cached.
the rest
- Memory: the agent proposes a delta, I approve by exception, a command stamps it.
- Meetings: a recorded call becomes decisions on top of the transcript. Tasks only with my ok.
- Job bot: a deterministic workflow. The model only summarises what passed the filter.
- At Celcoin, the harness went into the team's repository, with a portable kit. The production watcher was a deterministic script every 15 minutes, not an agent in a loop.
what changed
| before | after | |
|---|---|---|
| retrieval MRR, ranking change | 0.57 | 0.78 |
| MRR by leg | vector 0.48, keyword 0.66 | fusion 0.78 |
| hit@1, fusion and weight together | 0.41 | 0.73 |
| judge run cost | $0.19 | $0.06 with prompt caching |
Cortex: an experiment, now paused
In 2026 I built Cortex, an assistant that brought the day's plan to a person at the moment they needed to act. TypeScript on Cloudflare Workers, Postgres, models through OpenRouter. I paused it in October 2026. The code and the measurements stay.
- AI stack: OpenRouter through the OpenAI SDK, so the model is a parameter, not code. Zod schemas sent as strict JSON schema. kimi-k2.6 on the conversation. gpt-4o-mini as the judge on the memory write path. garak for red teaming.
- Structured output with a strict schema and a closed set of 10 actions. The model proposes. Deterministic code writes. Every write waits for the person to confirm.
- Model chosen by measurement on 60 real calls: kimi-k2.6 at 5.3 s median and 12.2 s worst case, gpt-4o-mini at 2.3 s.
- A prompt lock: the prompt's hash is pinned to a measured score. The same code scored 0.47, 0.74, 0.84 and 0.89 in four runs. So the lock is a median of 6 runs (0.8684) with a 0.1 tolerance. About $0.028 per run.
- Red team on the write path. 6 hand-written injection cases, 15 attempts, no prompt leak. garak, NVIDIA's attack battery: promptinject 0 of 4, latent injection 0 of 8 on the reply and 1 of 8 on the write path (a fixture artifact; dedup absorbs it on a real person). $0.064 for 55 calls. It runs by hand, not in CI.