title: MM-Cron: phase 1 design (Oct–Dec 2026) component: MM-Cron level: phase period: sprints S2–S5 (1 Nov – 26 Dec 2026); S1 work carried in status: draft 1 (25 Oct) owner: MMDEP Design (Claude) decisions: TD-041, TD-046, TD-047, TD-048, TD-054, TD-056 overall: mm-cron/overall.md
MM-Cron: phase 1¶
1. Goal¶
By the end of phase 1, every mutual-fund fact Money Marche shows comes from a logged, tested MM-Cron job that writes through the API: AMFI complete, all 51 AMCs covered by one adapter framework, no silent failures.
This is what R1's programmatic fund pages (S3) and retention pages (S4) need, and what Portfolio Engine's upstream gate list asks for (capabilities note §1).
2. Scope¶
| In | Out (later phases) |
|---|---|
| AMFI: NAV daily and history, scheme master, AUM, TER, performance, NFO (6 jobs) and the factsheet index (S1) | ULIP fund NAVs (ICICI Prudential Life first): planned after phase 1 (relay 17 Oct) |
| 51 AMCs: monthly portfolio disclosures (holdings) and the factsheet fields AMFI does not publish (TD-054) | Insurance product data, SuiteCRM scripts, database backup scripts (stay as they are) |
| Adapter framework, job registry, raw archive, validator, run records | Moving production jobs off the production server (SRV-1 decides) |
| Write path through the API for every job above; removal of their direct SQL | The repair agent (augmentation layer 4): phase 2, once fingerprints exist |
| Freshness and failure alerts through the collector | Derived analytics (Portfolio Engine) |
What comes from where (TD-054)¶
AMFI already publishes several fund-detail fields for every scheme, so we take those from AMFI and parse factsheets only for the rest. This cuts the PDF work, the least reliable part, roughly in half.
| Field | Source | Why |
|---|---|---|
| TER (regular, direct) | AMFI TER (S7) | One file, all AMCs, daily |
| AUM | AMFI AUM (S6) | Quarterly and monthly, all AMCs |
| Scheme name, category, plan, option, ISINs, launch/closure | AMFI scheme master (S5) | Authoritative list |
| Returns (AMC-reported) and benchmark returns | AMFI performance (S8) | All AMCs |
| Fund managers, benchmark name, inception date where S5 lacks it, risk ratios (std dev, Sharpe, beta), riskometer, exit load | AMC factsheet | Not in AMFI files |
| Holdings | AMC monthly portfolio disclosure (Excel, SEBI-mandated) | Replaces AdvisorKhoj (S11) |
3. Milestones¶
| # | By | Milestone | Measured by |
|---|---|---|---|
| MC-1 | 7 Nov (S2 wk 1) | Framework ready: adapter schema v1 (sprint-2 LLD), fetcher, raw archive, validator, run record, registry with 51 AMC rows (URLs from #64 where known, UNKNOWN otherwise) and their format family | Framework tests green on the laptop runner; registry lists 51 rows |
| MC-2 | 14 Nov (S2 end) | Top 15 AMCs by AUM: portfolio adapters merged (harness, cx 1–2 each); AMFI NAV daily and TER through the framework on staging | 15/51 portfolio adapters; daily run records on staging |
| MC-3 | 28 Nov (S3 end) | All 51 portfolio adapters merged; AMFI 6/6 jobs on the framework, writing to stgapi daily; top 15 factsheet adapters | 51/51, 6/6, 15/51 factsheet; run success ≥ 95% over 7 days |
| MC-4 | 12 Dec (S4 end) | Production through the API: jobs enabled on production one family at a time (Atul approves each batch); direct SQL removed for these jobs; remaining 36 factsheet adapters | Writer inventory shows no direct writers for phase-1 data; 51/51 factsheet |
| MC-5 | 26 Dec (S5 end) | Stable: 14 days of production runs with alerts; AdvisorKhoj retired after one month of parallel runs shows agreement | Collector: no unexplained failure; agreement report |
Fallback (setup doc §2): if MC-3 slips, the portfolio adapters for AMCs 16–51 move one sprint to MC-4, and production enabling moves to S5. Factsheet adapters are always behind portfolio adapters.
4. How the 51 AMCs are delivered¶
- Group by format family. Many AMCs publish through the same registrar layouts (CAMS, KFintech). One family = one parser with per-AMC configuration (column map, sheet name, header row), so most AMCs are configuration plus fixtures rather than code. The API agent classifies the 51 in the registry (MC-1).
- One harness issue per AMC per document type, cx 1–2, with the public sample file, the expected output, and the fixture test (setup doc §2; allocation §2). Public data, so free models may build them (ADR-0009).
- Order: top 15 by AUM first (they cover most investor money), then the rest by family so each new family is built once.
- Variant sharing is measured, not assumed (TD-048, M8). Each portfolio adapter records the scheme(s) the disclosure names. The mapping table then records 1:1 or N:1 per AMC from that evidence.
- Escalation: two failed harness runs → stronger model; then the API agent fixes the spec; a family that keeps failing goes to Sonnet (allocation §3).
5. Interfaces that change¶
| Interface | Change | Owner | When |
|---|---|---|---|
| Common output schema v1 | New: records for nav, scheme, aum, ter, performance, nfo, fund_details, holding, index_value, run | MMDEP Design (sprint-2 LLD) | MC-1 |
| Adapter interface | New: fingerprint(), parse(); registry rows |
MMDEP Design (sprint-2 LLD) | MC-1 |
| API ingest routes | New write routes for each record type, keyed, idempotent (API phase 1 §4) | API agent (Lane P), Claude review | spec S2, staging S3, production S4 |
/save-nav |
Unchanged contract; key required; new duplicate rule after P2 | API agent | S3–S4 |
| Run record | New: JSON lines on the server, then POST /api/ingest/runs |
MMDEP Design / API agent | MC-1 / MC-3 |
| Crontab | Generated from the registry; retired scripts removed (v2 set only) | API agent | MC-2 onward |
6. Risks¶
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| AMC sites' terms forbid automated download | Medium | Blocks production fetching for some AMCs | for-atul question (option B): adapters are built and tested on hand-saved samples; scheduled fetching waits for the opinion. An AMC that forbids it is fetched by hand monthly into the same pipeline |
| AMFI factsheet links (S1, #64) still unknown for some AMCs | High | Registry URLs missing | Adapters don't need live URLs to be built; URL is a registry field filled as found |
| PDF factsheets too varied for cx 1–2 items | High | Factsheet adapters slip | TD-054 halves the PDF fields; factsheets are scheduled after portfolios; layout families again |
| Identity: disclosures give names, not codes | High | Wrong holdings on a fund | Name matches go to a review list (TD-048); nothing is published on a proposed match |
| Staging isolation not yet proven (X1) | Medium | A staging run writes production data | No staging write before the proof (TD-051); publisher refuses a target that doesn't match its key's environment |
| Harness capacity or quality | Medium | Fewer than 51 by MC-3 | Fallback above; adapters are independent, so partial delivery still has value |
| Write routes late (API dependency) | Medium | MC-4 slips | Adapters publish to the file target until the routes exist, so building never waits. A job keeps its legacy script until its route is live; the framework has no database mode (LLD §7) |
| The laptop is a single point of failure | Medium | Runs and harness stop | Production jobs stay on the server; SRV-1 in S2 |
7. What "done" means for phase 1¶
- [ ] 6/6 AMFI jobs and 51/51 portfolio adapters run through the framework, with fixtures and tests in CI.
- [ ] 51/51 factsheet adapters, or a list of the AMCs still pending with the reason.
- [ ] Every production write for phase-1 data goes through an API write route with the MM-Cron production key. The direct-access inventory (Lane S S5) lists no remaining direct writer for these tables.
- [ ] Every run leaves a run record. The collector alerts on a failed run or a late NAV (by 23:00 IST) within one check.
- [ ] Any published value can be replayed from its archived raw file (one replay test per record type).
- [ ] Holdings agree with the AMC disclosure totals (weights sum to 100% ± 0.5%, or the gap is recorded).
- [ ] The registry and the server crontab match; only the v2 set and the new jobs are live.