title: API (production): overall design
component: API, api.moneymarche.com (GitLab moneymarche/api, branch main)
level: overall
status: draft 2 (25 Oct): corrected from the original agent's chats (KN, docs/knowledge/api-mmcron-original/); route and table facts stay UNVERIFIED until the GitLab snapshot (MMDEP #185) and the OpenAPI draft (Lane S step S4) confirm them
owner: MMDEP Design (Claude)
decisions: TD-048, TD-049, TD-050, TD-051
adr: ADR-0013 (two lanes), ADR-0016 (design docs)
c4: api, apiCatalogue, apiMfData, apiIngest, apiApp, apiHealth, apiSecurity, apiDataAccess
API (production): overall design
1. Purpose
The API is the single door to Money Marche's product and market data: catalogue (categories, products, providers), mutual-fund data (schemes, NAV, index values, NFOs, holdings, fund details), and the application routes the websites use (quotes, enquiries, users). Legacy applications use it and every new application must (OR §16).
It must be true at all times that:
- Consumers read through the API, not the databases. Websites still read
moneymarche_www directly today; that ends in phase 1–2 (OR §14 #111, relay 17 Oct priority 2).
- Writers write through the API. MM-Cron and the admin panel move to write routes (MM-Cron overall §1; OR §12).
- The contract is written down and tested.
docs/openapi.yaml in the api repo is the contract (TD-049).
- Data is honest. A response says how fresh it is and marks missing data as missing; it never substitutes one series for another (TD-050; OR §3).
- Production changes only with Atul's approval (ADR-0013 Lane P).
2. Context (C4 level 1)
| Consumer or dependency |
Uses |
Contract owner |
| Production website (WordPress) |
Catalogue, NAV, categories; also direct DB reads today |
API agent (MWMC for website changes) |
Web app webapp-sakai (dashboard) |
Reads; KYC uploads to S3 mmkyc (client data) |
API agent |
| Portfolio Engine |
GET /api/schemes, /api/scheme/nav, /api/index/nav (capabilities note §1) |
API agent; PE's upstream gate list is the requirement |
| MCRM |
Catalogue contract; owns field meaning for catalogue (OR §14 #110) |
MCRM (meaning), API agent (running API) |
| MWMC fund pages |
Fund factsheet fields (Lane P step P4) |
API agent |
| MM-Cron |
Writes (today /save-nav; more routes in phase 1) |
API agent |
| MMDEP collector |
Read-only health and data checks |
MMDEP |
| SuiteCRM |
CRM entities, leads (lowest priority; secret to move to config) |
API agent |
moneymarche_www (MariaDB) |
Catalogue and application data |
— |
mm_data (MariaDB) |
NAV and market data (~37M NAV rows) |
— |
| MongoDB |
Not in use; code to be removed (Ajay, OR §13; legacy use for funds_nav and fund-statistics collections, KN) |
— |
update_products_nav.py (api repo, root cron 04:00 daily) |
A writer inside the API repo: refreshes derived metrics in mm_product_fund_details (KN). Counted in the writer inventory |
API agent |
3. Containers and components
Runtime: Python/Flask with gevent, /opt/bitnami/api/api_prod, 127.0.0.1:5000 behind Apache on the Lightsail host shared with WordPress and stgapi (RB §1). Code on main.
| Component |
Routes (as known; UNVERIFIED until OpenAPI) |
Notes |
Catalogue (apiCatalogue) |
/category, /product…, /product/owners, /product/owner/<code>, /api/home-page-categories |
Category flags defect: /category must return stored status and show_in_menu; add the effective menu tree (hidden categories collapsed, children promoted: Atul's roll-up rule, OR §11). Field meaning owned by MCRM |
MF data (apiMfData) |
/api/schemes, /api/scheme/nav, /api/index/nav, /api/nfos/active, portfolio, AUM, TER, performance routes; /api/indexes is deprecated and must not replace /api/index/nav (KN) |
Read-only. Every response carries freshness (TD-050). Holdings gap: the holdings route reads legacy mm_product_holdings / mm_product_sectors, while MM-Cron writes mm_portfolio / mm_portfolio_items (KN). Phase 1 adds a holdings route on the portfolio tables and keeps the old route's contract until consumers move |
Ingest (apiIngest) |
POST /save-nav, POST /jobs/upload; new: scheme, fund-details, holdings, AUM, TER, performance, NFO, index, run-log routes |
The only write path for MM-Cron (target). Idempotent upserts keyed as in §5. Keys required (§6) |
Application (apiApp) |
quote, enquiry, booking, user, customer routes (relay 19 Oct) |
Client data class. Out of scope of MM-Cron; included here for security only |
Health (apiHealth) |
/api/health (missing on production today: 404; P3) |
Returns release sha, DB reachability and NAV freshness; no secrets |
Security (apiSecurity) |
Middleware on every write route |
Key per consumer, constant-time compare, log-only then enforce (§6, P1 LLD) |
Data access (apiDataAccess) |
Two SQLAlchemy engines and sessions: moneymarche_www and mm_data (KN) |
pool_pre_ping and pool_recycle (stale-connection defect, RB §1); the NAV engine must use its own socket setting |
4. Interfaces and contracts (TD-049)
- One contract file:
docs/openapi.yaml in the api repo, generated from the Flask routes and then edited by hand where generation can't see (Lane S step S4; HO-18 #1). Every route has: method, path, parameters, response schema, error responses, auth (none / key), consumers.
- Contract tests (Sonnet T3) call each route on stgapi and validate against the file. A route not in the file is a defect.
- Change rules:
- Adding a route or an optional field: allowed; update the file in the same MR.
- Changing or removing a field, a type, an error code or a default: breaking. Needs a consumer check (an
ask-<project> issue to each consumer listed in the file) and Atul's approval (OR §14 #110). Prefer a new route or version (/api/v2/...) over changing an old one.
- Column dictionary:
docs/data-dictionary.md from the live schema (read-only), catalogue tables first (relay 17 Oct priority 3).
5. Data rules (TD-048, TD-050)
| Rule |
Detail |
| NAV key |
(scheme_code, nav_date) is unique in meaning. Duplicates exist today (collector: 3 dates for scheme 120245, 10 for NIFTY 50). De-dup and a unique constraint are designed in P2, reviewed by Claude, rehearsed on staging, approved by Atul; no constraint or upsert before that (HO-18 #5) |
| Upsert semantics |
A write of the same key with the same value is a no-op; with a different value it is recorded as a correction (old value, new value, source, time) and applied only if the writer is the authoritative source for that series. Detail in the P2 LLD |
| Missing data |
Returned as missing (null plus a reason), never filled, interpolated or substituted |
| Freshness |
Every MF data response carries as_of (latest data date) and, for series, last_updated |
| Identity |
Scheme code for schemes; ISIN mapping; fund family key for holdings; index symbol exact (MM-Cron overall §5) |
| Staging separation |
mm_data is written only by production. Staging has its own databases (stgapi overall §4) |
6. Security (non-functional)
| Control |
Design |
When |
| Write routes |
A key per consumer (MM-Cron production, admin panel, staging loader…) in a request header, compared in constant time, separate keys for staging and production, stored in server env files or CI secrets, never in browser code. Log-only for 7 days, then enforce on staging, then production |
P1 LLD (sprint 2); high risk; Claude reviews |
| Read routes |
Public catalogue and NAV reads stay open with caching and rate limits; non-public reads get keys |
Sprint 3 |
| SQL injection |
Fixed sort-key allow-lists (!132, !162) |
In release 1 |
| Database access |
A read-only user for audits and the dictionary; no shared admin credentials in apps |
Script by a person (relay 18 Oct C3) |
| Adminer |
Bound to localhost, used through an SSH tunnel (relay 18 Oct C5) |
Runbook |
| Secrets |
API_KEY, SuiteCRM, SMTP in env (MR !168); the /save-nav key is hard-coded in src/config.py and lambda/navdata/lib/navdata.py (KN), so it is treated as exposed and replaced by the per-consumer keys; rotation by a person |
Security fast lane |
| Client data |
Application routes and S3 mmkyc are client class: never in MMDEP, prompts or free models |
Always |
7. Observability
/api/health on production and staging (§3).- Structured logs (JSON lines: time, route, status, latency, consumer id, no bodies) under the app's
logs/.
- The MMDEP collector checks routes hourly and NAV freshness and duplicates daily; alerts go to the phone (roadmap v4 §3).
- Error tracking in sprint 3 (roadmap v4 §3).
8. Release and rollback
- Five persistent branches only:
main, feature/fund-list, api-production-baseline, api-phase1-development, api-phase1-staging-release; every other branch is a task branch (KN, 27 Sep).
main is production. A release candidate arrives as one MR from api-phase1-staging-release with evidence (tests, staging smoke, data checks) (ADR-0013 §3); Lane P plans the deploy; Atul approves.- Consumer gates (KN): catalogue changes pass MCRM UAT; any change that reaches the websites passes MWMC UAT on stgapi before promotion. A UAT is a recorded run of the consumer's contract tests plus its sign-off line in the RC MR.
- Each release has a note, the commit sha, and a rollback (the previous artifact stays on the host and can be reactivated with one command). The production runtime sha is unknown today (HO-18 #11): the first release must record it.
- Deploy and release jobs stay manual on the laptop runner (relay 13 Oct).
9. Open points
| # |
Point |
Resolution path |
| A1 |
Full route list, params and response shapes |
Snapshot + OpenAPI draft (S4) |
| A2 |
Does live mm_category hold real flags, or are they 1 in the database too? (HO-18 #7) |
Read-only comparison, then the fix |
| A3 |
Which websites and plugins read the databases directly, and which tables |
Direct-access inventory (S5); MWMC and WPDev findings |
| A4 |
Production release sha |
First release records it |
| A5 |
Writers of mm_product_type (form or direct entry) |
Ajay (OR §13 "to verify") |