title: MM-Cron — augmenting the scripts with agents status: proposal for discussion (Atul, 11 Oct: jobs fail whenever AMFI or the AMCs change their formats) author: Claude (chat, MMDEP-PM role) date: 2026-10-11 sources: local clone of moneymarche/mm-crons (main, read only, contents not changed); MM-CRONS-PROJECT-PLAN-AND-SPECIFICATION-v1.0 (6 Sep); TECHNOLOGY-AUDIT-REFERENCE; TEA estate index and reconciliation (20 Sep) assumptions: A-23, A-25, A-29, A-30 (assumptions register)

MM-Cron: augmenting the scripts with agents¶

1. What MM-Cron is today¶

Python scripts and Lambda-style packages that pull public market data and write it to MoneyMarche's databases:
NAV from AMFI (lambda/navdata, nav-download.py, nav-history.py; posted to the API's /save-nav);
AMC monthly portfolios (lambda/portfolio, with per-AMC column mappings in JSON: Axis, Aditya Birla, HDFC, ICICI, Quant, SBI, Tata, UTI, 360 ONE);
AUM and TER (lambda/parsedata, quarterly and weekly);
NFO listings (scripts/nfo, daily);
NSE index data (nifty-data);
insurance product data (ICICI Prudential Life, Bajaj Allianz);
SuiteCRM data scripts, database backup scripts, SSL renewal.
They run from root cron on the production server (A-23). Examples: NFO daily at 21:15, TER weekly on Saturdays at 20:00, AUM quarterly, the NAV update daily at 04:00 (TEA audit).
A detailed project plan already exists (v1.0, 6 Sep), with a job contract, freshness, validation, idempotency and recovery. Little of it has been built.

Known failure: the TEA audit found that NAV persistence stopped advancing: the latest NAV date was 19 Aug 2026, while calls to /save-nav continued until 20 Aug. Nobody was alerted. Whether this has been fixed since is not known.

2. Why the jobs fail¶

Cause	Example	Today
Source format changes	An AMC renames a column or moves its header row in the monthly portfolio Excel; AMFI changes a field	The parser fails or, worse, loads wrong values
Source location changes	A new URL pattern for the monthly file	Download fails
Access changes	NSE cookies and headers (`nifty-data` handles cookies by hand)	Blocked
Silent breakage	Persistence stops while the API still answers (August NAV)	Not detected

3. Design: deterministic first, agent on exception¶

The scripts stay the primary path: cheap, repeatable and auditable. Agents are added around them, not in place of them.

Layer	What it does	Who	Cost
1. Contract checks before parsing	Each job checks the downloaded file against a stored format fingerprint: header row, column names and order, row-count range, a sample-row pattern. On a mismatch it stops before writing anything	Script (added to each job)	None
2. Output checks after parsing	Row counts against the last run; date freshness (NAV for the last business day); value ranges (NAV > 0; no jumps beyond a threshold); duplicates; referential integrity (scheme codes known)	Script	None
3. Freshness and source watch	Daily, independent of the jobs: the latest NAV date from the API, the latest portfolio month per AMC, plus a fingerprint of each source page or file, to spot format drift before a job fails	MCL collector (GitHub Actions)	None
4. Repair agent (on failure only)	Collects the evidence: the failed file sample, the error, the last good sample, the current mapping or config. A model proposes a fix, usually a change to the AMC's JSON mapping, sometimes a parser change. It runs the job against the file in a sandbox on the tools box, against staging and repeats layers 1–2. It then opens an MR in `mm-crons` with the evidence	Gemini (Flash-Lite, then Flash); Claude only if the change spans several jobs	Cents per repair
5. Human gate	Ajay or Atul reviews the MR, it is deployed to staging, then production by the normal release (spec §15.6–15.7)	People	—

Data class works in our favour: AMFI, AMC and NSE files are public, so free model tiers are acceptable for reading the files themselves. The mm-crons code and our database details are proprietary, so they go only to paid tiers.

Authority: - Always class D for now: writing financial data. No agent writes to the production databases or deploys. - Possibly class B later: after a run of correct repairs, a mapping-only change (JSON) that passes all checks on staging could be merged in a batch Atul approves.

4. Where it runs¶

Part	Where	Why
Production jobs	The production server for now (unchanged), behind the API's `/save-nav` boundary	The tools box must not hold production write credentials
Contract and output checks	Inside the jobs; deployed with them	—
Freshness and source watch	MCL (GitHub Actions)	Never sleeps, costs nothing
Repair agent and sandbox runs	The tools box, against staging (`stgapi`)	Isolated from production
Later	The jobs themselves move off the production server to a separate jobs runner, with a scoped write token	Part of separating production (estate proposal §2)

5. Security findings from reading the repository¶

An API write token is committed in plain text in lambda/navdata/readme.md. It's an Authorization header for /save-nav, the endpoint that writes NAV data. Action: rotate it now, move it to a secret store, and remove it from the file. It stays in git history, so rotation is the real fix.
The same example calls /save-nav over http://, not https://. Action: require HTTPS for write endpoints.
The folders flagged on 10 Oct (dbbackup, insurance, bajaj allianz) hold small scripts, not database dumps. The earlier concern is resolved. The backup scripts and other files may contain credentials; TEA should check them, without anyone copying them anywhere.

6. Plan¶

Step	What	Who	When
0	Rotate the `/save-nav` token; HTTPS only	Atul or Ajay	Now
1	Check NAV freshness today (has persistence resumed since 19 Aug?)	TEA, read only	This week
2	Job register (plan v1.0 §5) for the P0 jobs: NAV, index, master data	ChatGPT (mm-crons lead) with TEA	During the vacation, in the work repo
3	Contract and output checks for NAV first, then the portfolio jobs	mm-crons coder	After the register
4	Freshness and source watch in MCL	Claude Code (MCL story)	With MCL
5	Repair agent prototype on one AMC portfolio mapping, on the tools box against staging	Gemini controller (AC-1 pattern)	After the tools box is up