title: MM-Cron: overall design
component: MM-Cron (GitLab moneymarche/mm-crons)
level: overall
status: draft 2 (25 Oct): draft 1 corrected from the original agent's chats (KN, docs/knowledge/api-mmcron-original/, PR #150); code-level facts stay UNVERIFIED until the GitLab snapshot (MMDEP #185) reaches the repository
owner: MMDEP Design (Claude)
decisions: TD-041, TD-046, TD-047, TD-048, TD-053
adr: ADR-0009 (data classes), ADR-0013 (two lanes), ADR-0016 (design docs)
c4: mmcron, mmcronScheduler, mmcronFetch, mmcronAdapters, mmcronValidate, mmcronPublish, mmcronRegistry, mmcronRaw
MM-Cron: overall design
1. Purpose
MM-Cron collects public mutual-fund and market data (AMFI, 51 AMCs, NSE indices, later insurers' ULIP fund NAVs) and turns it into checked records in Money Marche's data stores. Everything the websites, Portfolio Engine and the future fund pages show about a fund starts here.
It must be true at all times that:
- Financial data is never fabricated or silently fixed. A value we can't get or can't trust is recorded as missing, with the reason (OR §3).
- Nothing is written that failed a check. A changed source format stops the job before any write (augmentation design §3, layer 1).
- Every record can be traced through the chain source attempt → fetch run → raw file → parse run → load run → API acknowledgement (TD-047; KN requirement). Operational times (
retrieved_at, parsed_at, published_at, persisted_at) are kept apart from business dates (nav_date, as_of).
- One write boundary. MM-Cron writes through the API's write routes, not to the databases (Atul's MM-Cron priority 1, relay 17 Oct). Direct SQL is the state today and is removed job by job in phase 1.
- Staging and production stay apart. A staging run writes only to stgapi; it never touches
mm_data (OR §5 X1, §16).
2. Context (C4 level 1)
| Actor or system |
Relation to MM-Cron |
Data class |
| AMFI (S2–S10) |
NAV daily and history, scheme master, AUM, TER, performance, NFO, factsheet index |
public |
| AMC websites (51; S1 links) |
Monthly factsheets (fund details) and monthly portfolio disclosures (holdings) |
public |
| NSE indices (S12) |
Index values for benchmarks |
public (source UNVERIFIED) |
| AdvisorKhoj (S11) |
Portfolio holdings today; format changed recently (OR §2) |
public, third-party site; to be replaced by AMC disclosures |
| Insurers (later) |
ULIP fund NAVs, ICICI Prudential Life first (relay 17 Oct) |
public |
| API (production, :5000) |
Write target in production through write routes, with a production key |
internal |
| stgapi (staging, :5001) |
Write target for staging runs and for every new adapter before production |
internal |
| MMDEP collector |
Reads freshness and run results; raises alerts |
internal |
| Ajay / Atul |
Run jobs by hand today; approve production writes |
— |
3. Containers and components (C4 levels 2–3)
MM-Cron is one container (mmcron in architecture/workspace.dsl) with seven components:
| Component |
Responsibility |
Today |
Target |
Scheduler (mmcronScheduler) |
Starts jobs on time; one entry per job in the registry |
bitnami crontab on the production server (OR §13; launchers run scripts from /home/bitnami/scripts/parsedata and /home/bitnami/scripts/portfolio, KN) plus two Lambdas: navdata (lambda/navdata/handler.py, daily 15:30) and icici-pru-sheet (trigger UNVERIFIED, KN) |
Same hosts in phase 1. Every schedule line comes from the job registry, so crontab and registry can be compared by a test |
Job registry (mmcronRegistry) |
One row per job: id, source IDs (S1–S12), schedule, adapter, destination route, owner, last good run |
Doesn't exist |
registry/jobs.yaml in mm-crons; the AMC list (51 rows, source URLs, format family) lives beside it as registry/amcs.yaml |
Fetcher (mmcronFetch) |
Downloads one source file; retries with back-off; records URL, HTTP status, size, sha256 |
Per script (nav-download.py, cliAum.py, NFO scraper, clihandler.py -a <amc>); lambda/navdata/lib/navdata.py runs fetch, parse and load in one step (KN) |
One shared fetcher, a separate stage from parse and load (KN requirement: independent Fetch, Parse, Load). The raw file goes to the raw archive before parsing |
Raw archive (mmcronRaw) |
Keeps every downloaded file with a manifest (source, URL, fetched-at, sha256) |
None |
Server disk /opt/bitnami/mm-crons-data/raw/<source>/<yyyy-mm>/ in phase 1 (TD-047); S3 later if size needs it |
Adapters (mmcronAdapters) |
Source-specific parsing into the common output schema. One adapter per AMC (TD-041), grouped by format family (CAMS/KFintech layouts, setup doc §2); AMFI adapters per AMFI source |
Parsers per script; per-AMC JSON column mappings in lambda/portfolio/lib/config/amc/ for 9 AMCs (augmentation §1, HO-18 #6); factsheet parsers factsheets-parse/icici-facts.py, parseFunds.py (KN) |
One interface: fingerprint(file) → ok/mismatch, parse(file) → records[]. Sample files and expected outputs as fixtures |
Validator (mmcronValidate) |
Checks before writing: schema, value ranges (NAV > 0, jump threshold), duplicate keys, row count against last run, freshness, identity known |
Mostly absent; the August NAV stop went unnoticed (augmentation §1). A fail-closed staging preflight exists: staging_routing/preflight.py (KN, 8 Oct) |
Shared rules in mm-crons/tools/data_validation/ (KN requirement), run on every record set. A failure stops the publish and writes the reason to the run log. The staging preflight is kept and becomes the publisher's environment check |
Publisher (mmcronPublish) |
Sends validated records to the target environment's write routes; idempotent (same records twice give the same state) |
/save-nav for NAV; direct SQL for scheme master, AUM, TER, performance, NFO, portfolios, index (OR §13, HO-18 #4) |
Write routes only, with a key per environment (API overall §6). Until a route exists, the job keeps its direct write and is listed as a migration item |
Every run, the same pipeline
registry row ─► fetch ─► raw archive (+ sha256) ─► fingerprint check ─┬─ mismatch → stop, log, alert (no write)
└─ ok → parse → validate ─┬─ fail → stop, log, alert
└─ pass → publish (stgapi or API) → run record
Run record (one per run; the collector reads these)
| Field |
Example |
job, source |
amc-factsheet-hdfc, S1 |
env |
staging / production |
started, finished |
ISO times, IST |
raw |
archive path and sha256 |
fingerprint |
ok / mismatch:<detail> |
records |
parsed, valid, rejected (with reasons), published |
status |
ok, stopped-format, stopped-validation, failed-fetch, failed-publish |
Phase 1 writes run records as JSON lines under logs/runs/ on the server; the publisher also posts a summary to a run-log write route once the API has one (API overall §5, POST /api/ingest/runs, proposed).
4. Data flows
| Flow |
Source |
Adapter output |
Destination (target) |
Destination today |
| NAV daily and history |
S2, S4 |
nav(scheme_code, nav_date, nav) |
/save-nav |
/save-nav, plus missing.py and tmp_funds_nav direct (OR §13) |
| Scheme master and discovery |
S5 |
scheme(scheme_code, isin_growth, isin_reinvest, name, amc, category_code, plan, option, status) |
scheme upsert route (to define) |
direct to mm_data.scheme_data, moneymarche_www.mm_product (cliAum.py, hand-run monthly) |
| AUM |
S6 |
aum(scheme_code or amc, period, value) |
AUM route (to define) |
mm_data.mm_funds_aum |
| TER |
S7 |
ter(scheme_code, date, regular, direct) |
fund-details route |
destination UNVERIFIED |
| Performance |
S8 |
performance(scheme_code, period, return, benchmark) |
performance route |
mm_data.mm_funds_performance |
| NFO |
S9 |
nfo(...) |
NFO route |
mm_data.mm_nfo_list |
| Index values |
S12 |
index_value(symbol, date, value) |
index route |
mm_data.nifty_index |
| Categories |
mm-new-cat.xlsx, files/master/category_codes.csv |
— |
catalogue (MCRM owns) |
scripts/categories/cat.py writes moneymarche_www.mm_category; which input is authoritative is open (KN) |
| Fund details (factsheets, 51 AMCs) |
S1 → AMC files |
fund_details(scheme_code, as_of, ter, aum, inception, benchmark, managers[], risk ratios) |
fund-details route |
not collected (Ajay: ~10 parsers, few working) |
| Holdings (portfolio disclosures, 51 AMCs) |
AMC files (replacing S11) |
holding(portfolio_key, as_of, isin, instrument, sector, weight, value) |
holdings route |
mm_data.mm_portfolio → mm_portfolio_items (the holdings architecture; portfolio_id is a snapshot id, not a product id), by hand monthly via portfolio.sh. moneymarche_www.mm_product_holdings is legacy, sparse and not to be implemented (KN, Atul 25 Sep) |
| ULIP fund NAVs (later) |
insurer sites |
nav(...) with a ULIP fund id |
ULIP NAV route (MCRM model) |
none |
5. Identity (TD-048)
- AMFI scheme code is the key for a scheme (one plan × option).
- ISIN is mapped to the scheme code in a mapping table maintained from the scheme master; a holding's instrument is identified by ISIN where the source gives one.
- Portfolio key: holdings attach to a portfolio, and a mapping table links schemes to it. The mapping's cardinality is data, not code: 1:1, or N:1 when variants share a portfolio. Whether Direct/Regular × Growth/IDCW always share one AMC portfolio is UNKNOWN, to verify per AMC (KN, MCRM 28 Sep), so the adapter records what the disclosure states and the mapping says how schemes share it. Holdings never key on a scheme name, and
portfolio_id is never equated with scheme_code (KN).
scheme_code, unique_code, product_code and category ids stay distinct fields (KN).- Name matching is never automatic (API !166 §17, OR §5 X2). When a source gives only a name, the adapter proposes a match, the record waits in a review list, and a person or the API agent confirms the mapping once; the confirmed mapping is then reused.
- Index identity is exact: NIFTY 50 is not NIFTYM150MOMNTM50, and no index is substituted for another (OR §3, HO-18 #11).
6. Non-functional requirements
| Area |
Requirement |
How it's checked |
| Correctness |
No record published without passing the validator; no duplicate (scheme_code, nav_date) produced by MM-Cron |
Validator tests; the collector's duplicate check on the API |
| Freshness |
NAV for the last business day by 23:00 IST; monthly factsheets and portfolios within 15 days of month end (AMCs publish by the 10th; window to confirm per AMC) |
Collector freshness rows; alert when late |
| Observability |
Every run writes a run record; failures alert |
Collector reads run records and API freshness (roadmap v4 §3, monitoring) |
| Resilience to format change |
Fingerprint mismatch stops the job before writing; the repair loop (augmentation §3, layer 4) proposes a mapping fix as an MR |
Fixture with a deliberately changed header must stop the job |
| Reproducibility |
Any published record can be re-derived from its raw file |
Replay test: archive file → same records |
| Security |
No credentials in code (secret scan; relay 14 Oct PF-1); keys from server env files; production key only on the production host |
gitleaks in CI; key per environment |
| Testability |
Every job runs on the laptop runner against fixtures with no network and no database |
CI job test with tags laptop, docker |
| Legal |
Automated download only where the source's terms allow it; polite rate (one request at a time, delay between requests, identifying user agent) |
Legal check before build (TD-041 L; for-atul/design/2026-10-25-amc-download-terms.md) |
- New or changed adapter: fixtures and tests on the branch → merge (risk by the usual rules) → runs against stgapi for at least one full cycle (one day for daily jobs; one month's file for monthly jobs) → production enable is a high-risk change: Lane P prepares it, Atul approves.
- Production jobs stay on the production server's crontab in phase 1. Moving them (to the new server, SRV-1) is a later decision.
- Retired scripts: everything outside Ajay's
mm-crons-v2 set is retired (relay 15 Oct); the registry lists only live jobs.
8. What is not in MM-Cron
- Catalogue editing (categories, product custom content): the WordPress admin panel, later MCRM (OR §12, §14).
- Client data of any kind. MM-Cron handles public data only, which is why free models may build adapters (ADR-0009).
- Derived analytics (returns, risk ratios computed by us): Portfolio Engine. MM-Cron stores what the source publishes.
9. Open points (to resolve in the phase design or from the snapshot)
| # |
Point |
Resolution path |
| M1 |
Which scripts actually run, and on which host (cron vs Lambda) |
Snapshot tree + Ajay's crontab (OR §13); registry built from both |
| M2 |
Is lambda/navdata the only NAV writer path in use? missing.py and tmp_funds_nav are recovery paths |
Lane S step S5 (direct-database-access inventory) |
| M3 |
AMFI factsheet link targets per AMC (S1, issue #64) |
API agent; blocks the AMC registry's URL column, not the framework |
| M4 |
TER destination table (S7 "to verify") |
Snapshot (TERParse.py) |
| M5 |
AdvisorKhoj replacement by AMC disclosures for the 5 current AMCs |
Phase design: run both for one month, compare, then switch |
| M6 |
icici-pru-sheet Lambda: what it does and its trigger |
Snapshot; Ajay |
| M7 |
Authoritative category input (category_codes.csv vs mm-new-cat.xlsx) |
MCRM with the API agent |
| M8 |
Do variants share one portfolio, per AMC? |
Adapter evidence (phase 1 §4); mapping table records it |