Skip to content

MMDEP Code + Test harness (phase 1)

Design: DR-007 §3b, routing DR-008. Code: mmdep/harness.py, config mmdep/harness_config.json.

What it does

Every hour at :16 (and minutes after the laptop wakes, unlocks or reconnects), the task MMDEP Harness runs mmdep harness run:

  1. Reads open GitHub issues in moneymarche-dev/MMDEP labelled for:code.
  2. Works only those whose routing block says mode: scheduled, state: ready, cx 1–2, ctx public or internal, no contract change, and a non-empty scope.paths. Everything else is skipped with the reason.
  3. Creates a worktree on auto/issue-<n> from origin/main.
  4. Sends the model only the scope files, the LLD (scope.docs), the issue text and the shared-components register (docs/registers/components.md); asks for whole-file JSON replacements.
  5. Writes only scope files; refuses anything else. Runs the issue's tests, feeds failures back, up to 4 attempts.
  6. Quality gates (rules §15): no new lint problems (ruff), no new dependencies (standard library, this repo's modules and declared packages only), tests changed whenever code changes. Failures are fed back to the model like test failures.
  7. Runs the full suite and accepts only if nothing fails that didn't already fail on main.
  8. Pushes auto/issue-<n> (never main); an issue whose auto/ branch is already pushed waits for review and isn't redone. Writes harness/LATEST.md and harness/ledger.csv (tokens, estimated cost, seconds) on the control branch.

From #55 (DR-012 §5, low risk): after a branch passes and is pushed, the harness opens the PR to main itself (Closes #n, the ledger line, Risk: low (harness, scope paths only)). With auto_merge (default true) it waits up to merge_wait_s (900 s) for CI and merges when every check is green, titled Merge auto/issue-n (harness, low risk, CI green; DR-012). Otherwise the PR stays open, the ledger reason says pr-open, and MMDEP-PM reviews it. With comment_on_issue (#54) the result is also posted on the issue.

Sources (#56, DR-013 lane S): harness_config.json sources lists where issues come from. With no sources key, only repo on GitHub is used, as before. - A GitLab source ({"forge": "gitlab", "repo": "moneymarche/api", "base": "api-phase1-development"}) takes open issues labelled for:code and lane:staging, and only those whose routing block says risk: low. - The harness works in a clone of that repository: path, or by default <work_root>/src/moneymarche__api, made with glab repo clone on first use. - It branches auto/issue-N from base, opens an MR to base with Closes #N, and with auto_merge merges it once the MR's pipeline succeeds. Results are posted as issue notes. - It never pushes or merges anything other than auto/* into base. - PE and MCRM (backlog B16, DR-014 §3.6): GitHub sources moneymarche-dev/portfolio-engine and moneymarche-dev/MCRM take open for:code issues whose routing block says risk: low, so their agents' low-risk items get worked between PROCEEDs. Every source other than MMDEP's own repository is low risk only (low_risk_only). Each is cloned with gh repo clone into <work_root>/src/<owner__name>, and PRs go to main, merged under the same CI-green rule (DR-012). - Requires: glab auth status shows logged in. python -m mmdep.cli harness check lists the GitLab issue count.

Free OpenRouter models (#65, backlog B1 #73): the backend openrouter-free uses OpenRouter's Free Models Router, model: "openrouter/free" ($0; OpenRouter picks a free model per request). It comes after the Gemini models and before local qwen-coder-7b in fallback_order, and needs OPENROUTER_API_KEY. - The ledger records the model that actually served the call, for example openrouter/free > minimax/minimax-m3.1-flash, so pass rates per underlying model can be measured. - On HTTP 429 it waits 30 s and tries once more; a second 429 hands over to the next backend. - harness check prints openrouter-free: openrouter/free (key set), or that the key is not set. - The older picker is still available as model: "auto-free". It reads /models once a day (cached in harness/state.json), ranks free models by free_prefer, tries up to 3 on 404/429, and records auto-free > <id>.

Exact strings (#55): text the issue quotes in single backticks (at least 3 characters) is exact. If the model removes such a string from a scope file that contained it, the attempt fails with changed a string the issue quotes exactly: …, and the prompt states the rule.

CI (from #42): every PR to main, and every push to main, runs GitHub Actions (.github/workflows/ci.yml): the full pytest -q on Linux, plus ruff check on the changed Python files only. The PR shows the result as a check, and the job runner's CI gate (mmdep jobs run) reads it.

Writing an issue for it

Use the template .github/ISSUE_TEMPLATE/harness-task.md. Keep scope to the few files the change needs; name the acceptance tests; one change per issue.

Models: free first, paid by rule (Atul, 18 Oct)

  • Each issue names a starting model; the harness then follows fallback_order in the config, free models first (Gemini free tier, local Ollama), moving on when a model hits its quota, is down, or fails tests twice (escalate_after).
  • Paid models ("paid": true: OpenRouter, the Claude API, the OpenAI API) are used only when the issue's routing block says allow_paid: true, and only while the monthly cap has room.
  • Paid API access uses separate API accounts (their own keys: ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY), not the chat subscriptions. Keys are user environment variables set by Atul through a hidden prompt; never in files or chat.
  • Aim: visible results each sprint at the lowest cost.

Limits and safety

  • Monthly cap monthly_cap_usd (35 ≈ ₹3,000, ODM-1) covers model calls plus one-off purchases (minimum top-ups or credits for Gemini API, OpenRouter, Hugging Face and similar). Record a purchase with python -m mmdep.cli harness spend --item "OpenRouter credit" --usd 10; it goes to harness/spend.csv on the control branch. At most 2 issues per run.
  • Protected paths are never writable (.github/, set-up scripts, pyproject.toml).
  • Stop switch: create a file harness/STOP on the control branch (or in C:\Projects\_mmdep-control\harness\); the next run does nothing.
  • Models: gemini-flash-lite (default, free tier), gemini-flash, and local qwen-coder-7b / qwen-coder-1.5b through Ollama once a local machine runs it (DR-010).

Commands (from the repo folder)

  • python -m mmdep.cli harness check — read-only: pytest, GitHub access, model names.
  • python -m mmdep.cli harness run --dry-run — show what it would do.
  • python -m mmdep.cli harness run --issue 12 — work one issue now.
  • python tools\harness\setup.py — create the scheduled task (once).

Atul's actions (#57, ACT-1, DR-014 §3.3): mmdep actions sync. After each collector snapshot (config actions_sync, default on; a failure is one line in collector.log, never a failed snapshot), or by hand with python -m mmdep.cli actions sync [--dry-run] [--no-model]: - Items: open question-atul and decision-log issues in every repository the collector reads, plus the open decisions in the newest reports/mmdep-assistant/panel-updates-*.md. IDs are the project prefix plus the issue number (PE330, WPDEV123; prefixes in mmdep/actions.py, override with action_prefixes in the collector config). - Output: actions/ACTIONS.md on branch reports, under 60 lines: questions with options and the recommendation; decision-log items as "FYI, veto within 48 h"; the latest decisions; and warnings for repositories not read or replies not understood. - Answers: Atul comments on the MMDEP issue Atul answers (created on the first sync), one per line: PE330 B, PE330 skip, MMDEP94 veto or ok, or all recommended. Only comments by the gh login (the owner account) count. Text that matches no pattern goes to Gemini Flash-Lite when GEMINI_API_KEY is set; only a known ID with a valid option is taken, and anything else is flagged in ACTIONS.md. - Applied once: a comment on the source issue (GitHub through gh, GitLab notes through glab), plus a row in docs/relay/answers.md on branch relay for mmdep relay merge. If that row disappears from relay before reaching main, it is written again. - State: actions/state.json on reports keeps the last answer comment handled and every decision. All forge access is behind ActionsStore; a database backend can replace GitHubStore later (DR-014 §6).

Knowledge extract (backlog B12, DR-014 §3.1): mmdep knowledge extract [--project NAME] [--limit N] [--dry-run] [--no-pr]. - Input and order: chats in the archive (MMDEP_CHAT_ARCHIVE, default C:\Projects\_chat-archive, one folder per project), in the order API/MM-Cron, PE, MCRM, WPDev, MWMC, then the rest. - Redaction: each chat is redacted by mmdep/redact.py before it is sent, and the model's answer is redacted again. - Model: knowledge_model in harness_config.json must be a Gemini model marked "tier": "paid", or the command refuses to run. Internal data never goes to a free tier. - Output: the model fills a fixed schema. The results go to docs/knowledge/<project>/ as DECISIONS, REQUIREMENTS, LEGACY, SOURCES, OPEN-QUESTIONS and LOST-ARTEFACTS, with every line citing its chat, plus GAPS.md (B37: decisions with no ADR, requirements with no design, lost artefacts; matched by words against ADR, decision and design titles, so a prompt to look, not proof). Per-chat extracts are kept in extracts/. - Incremental: docs/knowledge/state.json holds a hash per chat. Cost goes to harness/spend.csv, and a run stops at the monthly cap. - Publishing: work is committed on branch knowledge/<date> from origin/main, with a PR for review. Raw chats are never committed. --dry-run lists what would be sent and sends nothing.

Agent inbox (B39)

Agents write to Atul without a relay. mmdep agent-inbox sync runs hourly after the actions sync (switch inbox_sync in the collector config):

  • a file for-atul/<project>/<name>.md on main (front matter type: decide|do|question, choices, blocking, risk, project) becomes an MMDEP issue labelled for-atul, type:…, project:…, risk:…, with tick-boxes for the choices;
  • each open GitLab question-atul issue gets a mirror issue, closed when the source closes;
  • Atul answers in the GitHub app: tick a choice (or Done), or comment A / B / free text. The answer goes to docs/relay/answers.md and, for a file, to docs/relay/for-atul/<project>/<name>.md (branch relay, brought to main by mmdep relay merge); a mirrored GitLab question also gets a comment on the source. Then the issue is closed.

Prompt queue and post-step jobs (B40)

A design agent works through prompts/<project>/QUEUE.md (template: prompts/QUEUE-TEMPLATE.md) from one PROCEED. Each step names inputs, outputs, a check and a job. After a commit that adds runs/step-N.md, the standing job post-step-mmdep (jobs runner, trigger on runs/) runs mmdep queue post-step --latest: the step's job (design-check, design-issues, harness-run: #N, data-checks) and runs/step-N-feedback.md, which the next step reads first. mmdep queue next|check <project> shows the next step and validates the file. design-issues is a dry run unless the job is given --create.

The same sync renders the Actions page (B41): open for-atul issues by project and sprint (label sprint:<n>), blocking first, then oldest, with age, type, risk and a link, written to docs/status/actions.md on branch reports when it changed. main holds a placeholder; the site deploy (B38) publishes the reports version. mmdep agent-inbox page --out FILE makes it locally.

Model roles and per-model limits (B42)

harness_config.json has roles (code, review, knowledge, triage, impact): each an ordered list, first model first. The code chain is the issue's model, then roles.code (the old fallback_order is used only when there are no roles); the knowledge job takes the first paid Gemini of roles.knowledge that still has quota. Each model may carry rpm and rpd (defaults: Gemini 30 / 1,000, OpenRouter 20 / 1,000, local models unlimited; read the real values from AI Studio and set them). mmdep/ratelimit.py keeps a 60-second window per model, counts calls per day in <control>/harness/usage-<date>.json, and skips a model that reached rpd or answered 429 twice until its next day (midnight Pacific for Gemini, UTC for the rest). mmdep harness check prints today's use per model.