title: stgapi (staging): overall design component: stgapi, stgapi.moneymarche.com (GitLab moneymarche/api, branches api-phase1-development and api-phase1-staging-release) level: overall status: draft 2 (25 Oct): corrected from the original agent's chats (KN); to confirm against the GitLab snapshot (MMDEP #185) and a read-only check of the staging configuration owner: MMDEP Design (Claude) decisions: TD-051, TD-052 adr: ADR-0013 (Lane S owns staging), ADR-0016 c4: stgapi, stgDbs
stgapi (staging): overall design¶
1. Purpose¶
stgapi is where every API and MM-Cron change proves itself before production. It runs the same code as the API, ahead of production, against its own databases, with enough real data that consumers (Portfolio Engine, WPDev, MWMC) can build and test against it.
Not to be confused with the staging website (staging.moneymarche.com), which is legacy code kept for its unreleased features and does not use stgapi (OR §15).
It must be true at all times that:
- stgapi never writes to production data. Its databases are
mm_api_stagingandmm_nav_staging; it has no credentials formm_data,moneymarche_wwworwp1(RB §1; TD-051). - stgapi cannot hurt production although it runs on the same host (resource caps, §5).
- Its data is real enough to test with: loaded from production through public read routes, then kept current by MM-Cron's staging runs (TD-052).
- What runs is known: the deployed sha is visible on
/api/health.
2. Context¶
| Actor or system | Relation | Notes |
|---|---|---|
| Lane S agent (ADR-0013, TD-033) | Owns staging: promotion, deploys, data load, smoke and contract tests | Merges itself on green CI; low risk |
| Lane P agent | Receives release candidates from staging | One MR per RC with evidence |
| MM-Cron (staging runs) | Writes through stgapi's write routes with the staging key | Every new adapter runs here first |
| Staging loader | One-time load and refresh from production public GET routes | No production database access (relay 19 Oct) |
| Portfolio Engine, WPDev, MWMC | Integration and contract testing against stgapi | PE's client already switches base URL (capabilities note §1) |
| MMDEP collector | Smoke checks hourly; route parity daily | Read-only |
| Contract tests (Sonnet T3) | Validate stgapi responses against docs/openapi.yaml |
Run in CI |
| MCRM and MWMC UAT | Catalogue changes (MCRM) and anything reaching the websites (MWMC) are accepted here before promotion (KN; API overall §8) | Sign-off line in the RC MR |
3. Runtime (as restored 19 Oct)¶
| Item | Value | Source |
|---|---|---|
| Host | Same Lightsail server as production (ip-172-26-9-240) |
RB §1 |
| Path | https://stgapi.moneymarche.com → Apache → https://127.0.0.1:5001 |
RB §1 |
| Service | systemd mm-stgapi.service, restart on failure, enabled at boot |
RB, restored 19 Oct |
| App directory | /opt/bitnami/api/api_stg_release/app |
RB §1 |
| Env file | /opt/bitnami/api/staging-secrets/db.env (root, 600; never printed) |
RB §1 |
| Release | Release-C 825127e0 from api-phase1-development |
RB |
| Must never run on 5001 | /opt/bitnami/api/api (2024 copy) |
RB, relay 19 Oct |
4. Data (TD-051, TD-052)¶
Databases: mm_api_staging (catalogue: products, owners, categories, fund details) and mm_nav_staging (NAV, index values), each with its own user limited to that database (RB §1).
Conflict to close first (OR §5 X1): three accounts disagree. (a) Ajay: mm_data is common to staging and production (12 Oct). (b) The runbook: stgapi uses its own mm_api_staging and mm_nav_staging (TEA, 14 Sep). (c) The original agent's decision of 7 Oct: staging phase 1 manages a separate copy of mm_data, while the staging API reads the production WordPress database read-only (KN). Each may describe a different build or a different consumer (the legacy staging website, an older stgapi). Rule until proven: no staging job writes anything. Proof: a read-only check on the server that lists the database names and users in stgapi's effective configuration (database names only, no secrets), plus SHOW GRANTS for the two staging users. Lane S step S2 starts only after this proof is recorded in its run file.
How data gets in:
| Data | Loader | Cadence |
|---|---|---|
| Catalogue (products, owners, categories, fund details) | Staging loader reads production public GET routes, writes mm_api_staging with the staging credentials |
One-time, then weekly |
| Scheme NAV | Loader reads production GET /api/scheme/nav, de-duplicates dates, flags gaps, posts to stgapi /save-nav (one scheme first) |
One-time, then daily by MM-Cron staging runs |
| Index values | Loader from production GET /api/index/nav into mm_nav_staging |
One-time, then daily by MM-Cron |
| New data (fund details, holdings from 51 AMC adapters) | MM-Cron staging runs through stgapi write routes | As each adapter lands |
Data differences are expected and recorded: staging NAV is de-duplicated and may differ from production until P2 lands. The parity check (§6) reports differences by type (duplicate removed, gap, value differs) instead of failing on them.
5. Isolation on a shared host (TD-051)¶
| Risk | Control |
|---|---|
| stgapi starves production (issue #30: host starved by WordPress/Apache) | systemd limits on mm-stgapi.service: MemoryMax, CPUQuota, TasksMax (values set from the host's size in the stgapi phase design) |
| A staging writer reaches production data | Staging DB users have grants only on staging databases; staging keys are rejected by production write routes and vice versa; MM-Cron's fail-closed staging preflight (staging_routing/preflight.py, KN) runs before every staging job |
| Loader load on production | Loader reads with a delay between requests and at most one request at a time, outside 09:00–18:00 IST |
| Wrong code started on 5001 | Only systemd starts the app; deploy-staging restarts it with sudo systemctl restart mm-stgapi through a narrow sudoers rule (RB §3) |
| Stale DB connections (500 "MySQL server has gone away") | pool_pre_ping, pool_recycle (API overall §3) |
6. Branches, deploy and promotion¶
task branch ─► api-phase1-development ─► api-phase1-staging-release ─► deploy-staging (automatic on green merge) ─► smoke
│
Lane P review, Atul approves ◄── RC MR to main with evidence ◄──┘
- Smoke tests after every deploy:
/api/health,/api/schemes,/api/scheme/nav?scheme_code=120245,/api/index/nav?symbol=NIFTYM150MOMNTM50,/category(ADR-0013 §5). - Contract tests against
docs/openapi.yamlon every deploy. - Route parity (daily): for a fixed sample (10 schemes across AMCs and categories, 2 indices, the category tree), compare stgapi with production. Differences are classified (new field, duplicate removed, gap, value differs, missing route) and written to the collector's report. Unexplained "value differs" rows become issues.
7. Non-functional¶
| Area | Target |
|---|---|
| Availability | Best effort, 07:00–23:00 IST (agents' working hours); restart on failure; the collector alerts if down for two checks |
| Data freshness | NAV no more than one business day behind production once MM-Cron staging runs start |
| Security | Same write-route keys as production, separate values; TLS through Apache; staging DB users rotated after the TEA exposure (RB §3) |
| Cost | ₹0 extra: same host |
8. Open points¶
| # | Point | Resolution path |
|---|---|---|
| G1 | X1: is stgapi provably isolated from mm_data? |
Read-only configuration check (§4) before any staging write |
| G2 | Host headroom for the resource caps | Memory and CPU figures from the server; set in the phase design |
| G3 | Does deploy-staging still start python itself? |
Snapshot of .gitlab-ci.yml; change to systemd restart |
| G4 | Retire moneymarche_staging and moneymarche_staging_backup schemas (unused) |
After G1; backup first; a person runs it |