LLD: MM-Cron ingest routes (POST /api/ingest/<type>)

status: draft (MMDEP Design, 27 Oct). High-risk contract written by Claude; the API agent checks it against the schema and destinations (Lane P step P5) and builds it. sprint: S2 (spec in OpenAPI), S3 (staging), S4 (production, Atul approves) decisions: TD-057, TD-058, TD-060, TD-065 depends on: LLD-adapter-framework (schema v1), LLD-write-route-security (keys), LLD-nav-integrity (persist_nav)

Context

MM-Cron writes to the databases directly today, except for NAV through /save-nav (OR §13, HO-18 #4). Atul wants every writer to use the API (relay 17 Oct, MM-Cron priority 1). The adapter framework publishes envelope records in schema v1 (adapter LLD §5). This LLD gives each record type a route, a destination and a persistence rule.

Decision

1. One route pattern

POST /api/ingest/<type> for type ∈ nav, scheme, aum, ter, performance, nfo, index, holdings, fund-details, runs.

• Auth: class M, @require_key(scope="ingest:<type>") (security LLD). The MM-Cron keys get ingest:*.
• Body: {"run_id", "source_id", "raw_sha256", "schema": "mmcron/v1", "records": [<envelope>…]}, with at most 1,000 records.
• Validation: every record is checked against the same JSON Schema files MM-Cron uses (schema/v1/). They are vendored into the api repo at src/ingest/schema/v1/, and a test compares their sha256 against a pinned list, so a change on either side is a deliberate, reviewed change. Invalid records are rejected individually; valid ones go on.
• Reply 200: {"run_id", "accepted", "unchanged", "corrected", "rejected": [{"index", "reason"}]}. A wrong schema version or an over-size body gets 400; auth failures get 401/403.
• Idempotent: sending the same body again gives unchanged for every record.
• One transaction per request: a database error rolls back the whole request and returns 503, and MM-Cron retries.

2. Destinations and rules

Existing tables keep their meaning. New source facts go to new tables, so derived tables are not overwritten:

Type	Destination (UNVERIFIED names to confirm in P5 preflight)	Key	On an existing key
nav	mm_data.funds_nav via persist_nav	(scheme_code, nav_date)	NAV LLD stage B
index	mm_data.nifty_index via persist_nav	(symbol, date)	NAV LLD stage B
scheme	mm_data.scheme_data	scheme_code	update changed fields; log each change in ingest_changes
scheme (new code)	also a new MF product in moneymarche_www.mm_product, the way cliAum.py does it today (P5 preflight documents that logic)	—	—
aum	mm_data.mm_funds_aum	(scope, scheme_code or amc_code, period)	replace the value, log the change
ter	new mm_data.mm_fund_ter	(scheme_code, effective_from)	unchanged or log a correction
performance	mm_data.mm_funds_performance	(scheme_code, period, as_of)	replace, log
nfo	mm_data.mm_nfo_list	(amc_code, scheme_name, open_date)	update
holdings	mm_data.mm_portfolio (one snapshot per amc_code, portfolio_name, as_of) → mm_portfolio_items	the snapshot	a resend of the same snapshot = unchanged; a different one for the same month = a new snapshot version, the old one marked superseded (never edited in place)
fund-details	new mm_data.mm_fund_details_src (one row per portfolio and as_of; managers as JSON)	(amc_code, portfolio_name, as_of)	new version, old marked superseded
runs	new mm_data.ingest_runs	run_id	upsert counts

• Product creation stays as it is. Following #107 (MCRM: the Catalogue Manager sets ID rules; the API database is the store and writer), the API reproduces today's behaviour exactly and changes it only when MCRM publishes its ID rule. That is a contract change with MCRM UAT.
• New tables (mm_fund_ter, mm_fund_details_src, ingest_runs, ingest_changes, plus nav_corrections from the NAV LLD) are created by one migration file, applied on staging in S3 and on production with the S4 release.
• Read routes for the new data are not part of this LLD: holdings (TD-058), TER and fund details for the fund-factsheet contract (P4). They read these tables.

3. Every request leaves a trace

ingest_runs holds run_id, type, source_id, raw_sha256, the key id, received_at, and the counts. persisted_at is the database time of the commit. This closes MM-Cron's traceability chain (KN): source attempt → fetch → raw file → parse → load → API acknowledgement.

4. The OpenAPI entry

All ten routes go into docs/openapi.yaml in S2, with the request schema referencing the vendored JSON Schema files, the reply schema, the error codes, the auth scheme and the consumer (mm-cron). Contract tests (Sonnet T3) call them on stgapi from S3.

C4 element

• apiIngest
• apiSecurity
• apiDataAccess
• mmcronPublish
• mmData
• wwwDb

Evaluation

	Assessment
V value	High. It removes MM-Cron's direct database writes, gives one write path with checks, and gives an acknowledgement for every record
S simplicity	One route pattern, one shared schema, one transaction per request; existing tables keep their meaning
C cost	About 3–4 agent sessions; four new tables; no new service
D data	Public market data; key ids in logs, secrets nowhere
L legal	None beyond the sources' own terms (MM-Cron side)

Test plan

• Per type: a valid batch → accepted; resend → unchanged; an invalid record (bad decimal string, unknown enum) → rejected with index and reason while the rest are accepted; a wrong schema version → 400.
• Auth: no key → 401 in enforce; a key without ingest:<type> → 403; a staging key on production → 403.
• Transaction: a forced database error mid-batch → 503 and nothing written.
• Holdings versioning: the same snapshot twice → unchanged; a changed snapshot for the same month → a new version, the old one superseded, the items not edited in place.
• Scheme: a new scheme code creates exactly what cliAum.py creates today (a characterisation test built from the P5 preflight's reading of cliAum.py).
• Schema pin: the vendored schema files' sha256 match the pinned list; changing one without updating the pin fails CI.
• Contract: each route in docs/openapi.yaml; contract tests green on stgapi (S3).
• End to end (staging, S3): MM-Cron publishes one fixture per type to stgapi; the reply counts match; ingest_runs has the run.

Risk

risk: high

It's a new write contract into production data, and new tables. Mitigations: - the schema is shared and pinned; - per-record validation; - one transaction per request; - snapshots are versioned, never edited in place; - existing tables keep their meaning; - staging first for a full cycle; - the production release needs Atul's approval; - MM-Cron enables production family by family (MC-4).

Work items

#	Title	Risk	Model	Scope paths	Depends on	Cx	Tests
1	P5 preflight: confirm destination table names and keys; document cliAum.py's product-creation logic	medium	chatgpt-api-mmcron	api: docs/design/ingest-routes-preflight.md		2
2	OpenAPI entries for the 10 ingest routes	medium	chatgpt-api-mmcron	api: docs/openapi.yaml	1	2	contract lint
3	Vendor schema v1 with a sha256 pin test	low	gemini-flash-lite	api: src/ingest/schema/v1/, tests/test_ingest_schema_pin.py	adapter LLD item 1	1	tests/test_ingest_schema_pin.py
4	Migration: mm_fund_ter, mm_fund_details_src, ingest_runs, ingest_changes	high	chatgpt-api-mmcron	api: migrations/	1	2	migration test on a container
5	Ingest blueprint: auth, validation, reply, transaction; nav, index via persist_nav; runs	high	chatgpt-api-mmcron	api: src/ingest/, tests/test_ingest_core.py	3,4, NAV LLD 4, security LLD 2	3	tests/test_ingest_core.py
6	Types scheme (with product-creation characterisation), aum, ter, performance, nfo	high	chatgpt-api-mmcron	api: src/ingest/types/, tests/test_ingest_types.py	5	3	tests/test_ingest_types.py
7	Types holdings and fund-details with snapshot versioning	high	chatgpt-api-mmcron	api: src/ingest/types/, tests/test_ingest_snapshots.py	5	3	tests/test_ingest_snapshots.py
8	Staging end to end with MM-Cron fixtures; then the production release (Atul)	high	chatgpt-api-mmcron	api: release MR	6,7	2	staging e2e evidence