LLD: AMFI jobs on the adapter framework (NAV daily and history, scheme master, AUM, TER, performance, NFO)¶
status: draft (MMDEP Design, 27 Oct). Medium risk: the jobs move onto the framework and publish to file and staging only. Production enabling is a separate high-risk step (MC-4).
sprint: S2 (NAV daily, TER), S3 (the rest; MC-3: 6/6)
decisions: TD-054, TD-067
depends on: LLD-adapter-framework (MC-1); LLD-ingest-routes (staging publish from S3)
Context¶
AMFI publishes most of the data we need for every scheme (data-source register S2–S9). Today lambda/navdata and cliAum.py -m scheme|aum|ter|fund-performance fetch, parse and write directly in one step. Some of the destinations and live triggers are unknown (S7 "to verify", M1, M4). The phase-1 target is 6/6 AMFI jobs on the framework by 28 Nov (MC-3).
Decision¶
1. Six jobs¶
| Job id | Source | Family (new module) | Schedule (IST) | Record type | Freshness check |
|---|---|---|---|---|---|
amfi-nav-daily |
S4 NAVAll.txt (one text file, every scheme, today's NAV) |
amfi_text |
23:00 daily, retry 06:00 | nav (+ scheme check, §2) |
as_of = last business day for ≥ 95% of active schemes |
amfi-nav-history |
S2 portal, date range per request | amfi_history |
On demand (gap fill, back-fill) and weekly for the last 30 days | nav |
none (repair job) |
amfi-scheme-master |
S5 portal | amfi_scheme |
Weekly, Sunday 07:00 | scheme |
row count ±5% of last run |
amfi-aum |
S6 JSON API (fund-wise and scheme-wise) | amfi_json |
Monthly, 10th, and quarterly | aum |
the latest period present |
amfi-ter |
S7 Excel API | amfi_excel |
Daily 22:00 (AMFI updates TER when AMCs change it) | ter |
file date not older than 7 days |
amfi-performance |
S8 web/API | amfi_json or amfi_excel (decided by the sample) |
Weekly | performance |
as_of within 10 days |
NFO (S9) is a seventh, smaller job (amfi-nfo, daily). It is counted with the NAV and performance jobs under "6 AMFI jobs" in the setup doc only if AMFI turns out to be the source. Today the scraper reads AMC sites (S9 "partly known"); the API agent's S5 inventory decides.
Daily NAV comes from NAVAll.txt (S4), not the history portal (S2) (TD-067):
- it is one small file covering all schemes;
- it carries the scheme code, both ISINs and the scheme name on each line, so the daily run also catches new and renamed schemes;
- the history portal stays for back-fill and gap repair, which is what the NAV integrity LLD needs.
2. Scheme check inside the daily NAV run¶
amfi_text emits nav records. It also compares each line's scheme code, ISINs and name with the last amfi-scheme-master output:
- unknown code → review/new-schemes.csv (the NAV is published only after the scheme master knows the code, so a new fund appears within a week);
- changed name or ISIN → a warning in the run record, applied by the next scheme-master run.
This keeps one owner for scheme facts (the master job) while the daily file gives early notice.
3. Behaviour preserved¶
Each job's first framework version must produce the same rows as today's script for the same input file: a characterisation test. The API agent saves one real input and today's output (from staging or the script run locally on the file) as the fixture. Any intended difference (a de-duplicated date, a unit fix) is listed in the fixture's README.md.
4. Cut-over per job¶
file target for 7 days, then the staging target for one full cycle, then production with Atul (MC-4). The legacy script stays in the crontab until the job's production run has been clean for 7 days, then it is removed in the same MR that adds the framework schedule line (registry → crontab).
C4 element¶
mmcronAdaptersmmcronFetchmmcronSchedulermmcronRegistryamfi
Evaluation¶
| Assessment | |
|---|---|
| V value | High: the base data for every fund page and for PE; ends silent NAV stops |
| S simplicity | Five small family modules; one daily file for NAV |
| C cost | Harness cx 1–2 per job; the API agent's preflight to capture today's outputs |
| D data | Public (AMFI) |
| L legal | AMFI publishes these for public download; the request rate is a few files a day |
Test plan¶
- Fixture acceptance for each family (adapter LLD test plan), with one real AMFI file each.
- Characterisation: per job, the framework output equals today's script's rows on the same file, except for listed differences.
- Scheme check: a fixture with an unknown code and a renamed scheme → the review CSV and the warning; no NAV published for the unknown code.
- Freshness: a file whose dates are a week old →
stopped-validation. - Fingerprint: each family stops on a changed header (
NAVAll.txtsection headers; Excel column names). - Registry and crontab: the generated schedule lines equal the registry; a test fails if a legacy line and a framework line exist for the same job after cut-over.
Risk¶
risk: medium
Medium because these jobs only publish to file and staging in S2–S3. What would make it high: enabling production publishing (MC-4, Atul), or a characterisation difference that changes production numbers. Every listed difference is reviewed in the cut-over MR.
Work items¶
| # | Title | Risk | Model | Scope paths | Depends on | Cx | Tests |
|---|---|---|---|---|---|---|---|
| 1 | Preflight: confirm live triggers and destinations for navdata and cliAum.py modes (M1, M4); save one real input and today's output per job as fixtures | medium | chatgpt-api-mmcron | mm-crons: tests/fixtures/amfi/, docs/design/amfi-jobs-preflight.md | adapter LLD 12 | 2 | |
| 2 | amfi_text family and amfi-nav-daily job with the scheme check |
low | gemini-flash-lite | mm-crons: mmcron/adapters/families/amfi_text.py, registry/jobs.yaml, tests/ | 1 | 2 | tests/test_fixtures.py |
| 3 | amfi_excel family and amfi-ter job (characterisation against TERParse.py) |
low | gemini-flash-lite | mm-crons: mmcron/adapters/families/amfi_excel.py, registry/jobs.yaml, tests/ | 1 | 2 | tests/test_fixtures.py |
| 4 | amfi_scheme family and amfi-scheme-master job |
low | gemini-flash-lite | mm-crons: mmcron/adapters/families/amfi_scheme.py, registry/jobs.yaml, tests/ | 1 | 2 | tests/test_fixtures.py |
| 5 | amfi_json family, amfi-aum job |
low | gemini-flash-lite | mm-crons: mmcron/adapters/families/amfi_json.py, registry/jobs.yaml, tests/ | 1 | 2 | tests/test_fixtures.py |
| 6 | amfi-performance job (family from the sample) |
low | gemini-flash-lite | mm-crons: mmcron/adapters/families/, registry/jobs.yaml, tests/ | 5 | 2 | tests/test_fixtures.py |
| 7 | amfi_history family and amfi-nav-history job (date-range requests, polite rate) |
low | gemini-flash-lite | mm-crons: mmcron/adapters/families/amfi_history.py, registry/jobs.yaml, tests/ | 2 | 2 | tests/test_fixtures.py |
| 8 | Registry → crontab generator and its test | low | gemini-flash-lite | mm-crons: mmcron/schedule.py, tests/test_schedule.py | adapter LLD 12 | 2 | tests/test_schedule.py |