Skip to content

DR-007 — Three harnesses, Gemini in real use, on-demand options

Converted from docs/decisions/DR-007-harness-streams-and-on-demand.md to MADR (backlog B25). The text is unchanged; its sections are grouped under the MADR headings below, in their original order within each group.

Context

1. What Atul asked (17 Oct)

  1. When does the Gemini API get tested in actual use?
  2. Split the harness into three streams, Design, Code and Test, instead of every project agent doing everything. Make reuse and cross-project dependencies explicit.
  3. Design works on three phases: existing, immediate goal, long-term goal.
  4. Code works on current-phase issues and bugs reported by Test.
  5. Test designs tests and runs them periodically and on every release (unit, UAT, performance and others).
  6. Later, use on-demand options for agents (Claude or ChatGPT APIs; hosted containers for short bursts) with measured usage, chosen per task. Compare their cost with monthly subscriptions or upgrading the current set-up.

2. What exists and what conflicts

Topic Exists today Conflict or gap
Roles One ChatGPT agent per product does plan, design, code, tests and docs Atul's split is new. It fits spec 0.11 roles (Architect, Developer, Reviewer) and the "no LLD, no sprint" rule (owner note 30)
Three phases Owner note 6: VISION.md, CURRENT.md, MILESTONES.md per product Same idea, different names. Proposal: Design owns all three files, with existing = CURRENT, immediate goal = MILESTONES plus LLDs, long term = VISION
Tests Per-project CI (GitLab laptop runner, GitHub Actions); MWMC route crawler; PE has 1,558 unit tests No UAT or performance tests anywhere; no test catalogue; no release test gate shared across projects
Reuse Ad hoc. PE's API normaliser lists every known API defect; MWMC's crawler; collector probes No shared-components register; each agent rebuilds
Gemini Controller designed (DR-006 phase 3); free-tier key on the laptop; planned for sprint 2 (D2, 24–31 Oct forecast) Not yet used for real work
Unattended work ChatGPT scheduled runs can read but not write (rules §7) Subscriptions alone cannot run unattended work. Metered APIs plus a runner can

3. Three harnesses

Programme-wide Design and Test harnesses serve all products. Code stays per product (the current project agents, with a narrower brief). This makes reuse and dependencies one team's job instead of six.

Design harness Code harness Test harness
Scope All products One per product (API/MM-Cron, MWMC, MCRM, PE, WPDev, MMDEP) All products
Owns HLD, LLDs, the three views per product, interface contracts, the shared-components register, the dependency map Implementation of LLD items and bugs, in the current phase only Test strategy, test catalogue per product, test code, schedules, release gate, bug reports
Inputs Owner notes, Atul's decisions, Test findings that imply design change Issues labelled lld-ready or bug Merged LLDs (to design tests), MRs (ready-for-test), releases
Outputs lld-ready issues, updated VISION/CURRENT/MILESTONES, contract changes MRs with unit tests, labelled ready-for-test Test results, bug issues (with found-by:test), release verdict (pass, fail, waived by Atul)
May not Write product code Change scope or contracts without an LLD update; mark its own work as tested Fix product code (only test code)
Runs on ChatGPT project (long-context reasoning, already paid) or Claude for HLD work Existing ChatGPT project agents; Claude codes MMDEP Deterministic tests on runners (no AI needed). Gemini for test design drafts and failure triage

Operating modes (Atul, 17 Oct 18:00):

Harness Mode Where it runs How it triggers other work
Design Attended, with Atul in the browser ChatGPT or Claude chat (browser or desktop app) Files issues for the other harnesses: research (study or comparison), test-request (a test to design or run), lld-ready (to Code), each with a "by" date. A scheduled task can be attached when the work must recur, for example a monthly drift check. Design never waits: it records what it asked for and moves on
Test Unattended on the laptop, within available capacity Laptop runner (GitLab and GitHub, laptop tag), triggered by schedule, MR and release; collector heartbeat shows availability Files bug issues to Code and test-pass or release verdicts. Capacity rule: test jobs yield to CI on open MRs; heavy runs (performance, full crawl) only off-hours or on request. Load is low now (legacy apps only) and grows with releases; the capacity register tracks runner minutes so we know when the laptop is no longer enough
Code All three modes See below Opens MRs labelled ready-for-test

Code modes:

Code mode Use for Runs on Cost
Attended Design-heavy or risky changes, anything needing Atul's judgement ChatGPT project agents or Claude in chat Subscriptions (flat)
Scheduled (unattended) Queued lld-ready and bug issues with clear acceptance criteria, small and reversible Metered API (Gemini first, Claude Sonnet where quality needs it) plus the laptop runner or GitHub Actions, through mmdep jobs and the controller. ChatGPT scheduled runs can't write, so they can't do this Per task, inside the monthly cap
On-demand burst A larger item that needs a full sandbox for a few hours (multi-file refactor, dependency upgrade, long test-fix loop) Codespaces or a hosted container, started for the task and stopped after Per hour, measured per task

The controller picks the mode per issue: attended if the issue is labelled needs-atul or touches contracts or security; scheduled if small and clear; burst only when a scheduled attempt fails or the estimate is above one hour of compute. Every unattended or burst run records its cost in the ledger.

4. Gemini in real use — date

Planned for sprint 2 (D2). Proposal: pull a first real use into sprint 1, week 2 (about 25–31 Oct), as part of the Test harness: - Job: read the collector snapshot and failed pipelines, then draft one triage line per failure (likely cause, owner project, suggested label). - Model: Flash-Lite on the free tier (500 requests a day is ample; about 50 a day needed). Read-only. Output goes to a report file that Claude reviews. Only public or internal coordination data, never client data. - Measure: requests and tokens per day, and how often Claude agrees with its triage. These numbers decide the controller's scope in sprint 2.

Options

5. On-demand options vs subscriptions

Prices (USD per million tokens, standard tier; batch is half price on all three)

Provider Model Input Output Source
Anthropic Claude Opus 5.5 4.00 20.00 tracker, cites Anthropic pricing
Anthropic Claude Sonnet 5.5 2.00 10.00 tracker
Google Gemini 3.7 Flash 0.75 3.75 (doubles 1 Jan 2027) tracker
Google Gemini 3.1 Flash-Lite 0.25 1.50 tracker
Google Gemini 2.5 Flash-Lite 0.10 0.40 tracker
OpenAI current models — — tracker figures could not be matched to known model names; check OpenAI's pricing page before use

Compute for short bursts

Option Free each month Then Fits
Laptop runner (self-hosted, GitLab and GitHub) Unlimited Free; only when the laptop is on All CI today
GitHub Actions hosted (private repos) 2,000 minutes Per minute, cut up to 39% in Jan 2026 PE, MCRM, MMDEP CI while the laptop is off
GitLab hosted compute 400 minutes Per minute Small bursts only
GitHub Codespaces 120 core-hours (60 hours of 2-core) $0.18/h 2-core, $0.36/h 4-core An agent's coding or test sandbox for a few hours
AWS tools server (DR-004) — Small server, always on Collector, runner, controller without the laptop

Cost per task (estimates)

A typical agent coding or test-design session: about 2 million input tokens (mostly repeated context; caching cuts this further) and 0.2 million output.

Model Cost per session 20 sessions a month
Claude Opus 5.5 $12 $240
Claude Sonnet 5.5 $6 $120
Gemini 3.7 Flash $2.25 $45
Gemini 3.1 Flash-Lite $0.80 $16
Triage line (Flash-Lite, about 5k tokens) under $0.01 —

OpenRouter (checked 17 Oct)

One API key for models from many providers; provider prices passed through, plus a 5.5% fee on credit purchases (pay-as-you-go plan); bring-your-own-key option; automatic fallback to another provider when one fails; prompts not logged by default, and a data-policy filter limits which providers see them. Free plan: about 50 requests a day on free models (OpenRouter's page; third-party guides quote different limits), so evaluation only. Credits expire after a year.

Fit for MMDEP: useful for trying cheaper open models (preflight, triage, open work packages) and as a fallback route behind the controller without managing many keys. Not for client data (an extra intermediary) unless used with our own provider keys and an approved provider. Proposal: one small credit (about $10) to benchmark 3–4 cheap models against Gemini Flash-Lite on preflight and triage; keep Gemini direct as the default while its free tier lasts.

Recommendation

  1. Keep subscriptions for attended, interactive work (design, coding with a person present). They are flat-rate and already paid, and per-token pricing would cost more for long interactive sessions.
  2. Use metered APIs only for unattended work, which subscriptions can't do: test triage, scheduled test design, the controller. Start on the Gemini free tier, then paid Flash-Lite. Claude Sonnet API only for tasks where Gemini's quality is measured as insufficient.
  3. Hard monthly cap: ₹3,000 on API spend for sprints 1–2, enforced in the controller. Cost per task is recorded in the hub ledger (capacity register). Review at each sprint end against the BEP model.
  4. Compute: laptop runner first. GitHub Actions hosted minutes as the fallback while the laptop is offline (2,000 free minutes a month). Codespaces only for an agent sandbox burst. The AWS tools server (DR-004) when heartbeat history shows the laptop offline too often. Two weeks of heartbeat data will show this.
  5. Upgrades: no subscription upgrade now. Revisit when the ledger shows a subscription limit blocking planned sprint work in two consecutive weeks.

Decision

6. Decisions for Atul

  • HAR-1: Adopt the three-harness model (programme-wide Design and Test, per-product Code) with an MWMC pilot in sprint 1?
  • GEM-1: Pull Gemini's first real use (Flash-Lite test triage, read-only) into sprint 1, week 2?
  • ODM-1: Metered APIs for unattended work only, capped at ₹3,000 a month for sprints 1–2; subscriptions unchanged?

Consequences

3b. Phase 1: two harnesses, gated by capacity and dependency (Atul, 17 Oct 23:50)

Rule restated: nothing is allocated without checking capacity and dependencies first. Atul's capacity is constrained while he is on vacation, and any Design agent needs his time, so the Design harness is not started now.

Phase 1 (two harnesses, so roles don't change overnight):

Harness Who Does Starts when
Design + Preflight The existing ChatGPT project agents (API/MM-Cron, MWMC, MCRM, PE, WPDev), converted from "do everything" Design a whole epic (instead of up to 5 issues iteratively): LLD, acceptance tests, then preflight it into routed for:code items with routing blocks Code + Test harness is ready and Atul has time for the first epic reviews
Code + Test mmdep jobs + controller + laptop runner, models chosen per item (Gemini, Claude Sonnet API), ChatGPT in attended mode where needed Implement routed items, run tests, report test-pass or bug Built first, in sprint 1–2

Phase 2: split Code and Test into separate harnesses once phase 1 has run for two sprints with measured results.

Claude as super-designer: Claude (Opus) takes the super-designer role for 2–3 projects each, reviewing the epics the ChatGPT Design + Preflight agents produce, owning cross-project contracts and the HLD. A Claude project per group, for example: (1) API/MM-Cron + MCRM (data and catalogue), (2) MWMC + WPDev (websites), (3) PE + MMDEP (analytics and programme).

Order of set-up (dependency chain): routing block and label vocabulary → mmdep jobs (run queued items unattended) → Code + Test harness on one pilot project (MWMC) → convert the pilot's ChatGPT agent to Design + Preflight → Claude super-designer reviews its first epic with Atul → extend to the other projects.

Test layers and when they run:

Layer Example When
Unit PE's pytest suite, API route tests Every MR (CI)
Contract API routes against the catalogue/NAV contract that MWMC, PE and WPDev rely on Every API MR, nightly
Regression and route health MWMC route crawler, collector probes Nightly, and on demand after releases
UAT Scripted user journeys on MMCLONE/staging (fund page, category page, calculators) Every release candidate
Performance k6 or Locust against staging (/api/schemes takes 11 s today) Every release candidate, weekly baseline
Security Fast-lane checks (secrets, SQL injection regressions) Every MR in the security lane

Hand-off by labels (works on GitLab and GitHub, survives outages, readable by the collector): lld-ready → Code → ready-for-test → Test → bug (back to Code) or test-pass → release request to Atul.

Reuse and dependencies: - The shared-components register (docs/registers/components.md) lists what exists, who owns it and who uses it. Examples: PE's API client and normaliser, the route crawler, the collector probes, the test harness library. - Every LLD has a "reuse check" line: register searched, component reused or reason for building new. - The dependency map (from the existing milestone table) is kept by Design. The collector flags any lld-ready item whose upstream dependency is not done.

Rollout: pilot with MWMC in sprint 1, because its crawler and SEO checks are the first test assets. Add API/MM-Cron (contract tests) and PE in sprint 2. MCRM joins when it has code.

Sources