LLD: stgapi isolation proof, resource caps, systemd deploy and the staging data load¶
status: draft (MMDEP Design, 27 Oct). The isolation proof is high risk (it decides whether staging may write at all); the rest is medium. Lane S builds it (queue S2, S3). sprint: S2 (SG-1 by 4 Nov, SG-2 by 14 Nov) decisions: TD-051, TD-052, TD-055, TD-066 overall: stgapi/overall.md §4–6; phase: stgapi/phase-1.md §3–4
Context¶
stgapi serves 1 scheme and 1 category. Its NAV stops on 19 Aug. /api/index/nav?symbol=NIFTY 50 returns 404 and /api/nfos/active returns 500 (collector, 25 Oct). Three accounts disagree on which databases it uses (stgapi overall §4, X1). Until that is settled, no staging job may write (TD-051). The service runs under systemd, but deploy-staging may still start python itself (G3), and it shares the host with production and WordPress without limits.
Decision¶
1. Isolation proof (SG-1, before any staging write)¶
The api repo gets a read-only script, tools/staging/isolation_check.sh, run by a person over SSH. It prints names and grants only, never a secret value:
- Configured targets:
Passwords are not matched by the pattern.
sudo grep -E '^(MYSQL_DATABASE|MYSQL_NAV_DATABASE|MYSQL_HOST|MYSQL_UNIX_SOCKET|MYSQL_NAV_UNIX_SOCKET|MYSQL_USERNAME|MYSQL_NAV_USERNAME|ENV)=' /opt/bitnami/api/staging-secrets/db.env - Live connections of the running service: for the users from step 1,
SELECT USER, DB, COUNT(*) FROM information_schema.PROCESSLIST GROUP BY USER, DB, taken after a smoke run so the pool is open. - Grants:
SHOW GRANTS FOReach of the two staging users. - Code defaults: the database names the app falls back to when an env var is missing, as
greplines from the deployedconfig.py(no values that look like secrets). - Verdict line:
ISOLATEDonly if every configured and live database is one ofmm_api_stagingormm_nav_staging, and the grants cover nothing else. OtherwiseNOT ISOLATED: <db list>.
The output goes into runs/S2.md (MMDEP relay, api-lane-s).
| Verdict | Next |
|---|---|
ISOLATED |
Data load (§4) may start |
NOT ISOLATED |
Stop. Lane P plans the configuration fix (staging users, env, code defaults) as a high-risk change with Atul. The load waits; the read-only promotion (§3) may still go ahead |
2. Resource caps (SG-1)¶
- A person records the host's memory, CPUs and the steady memory use of the production API and stgapi (
free -m,nproc,systemctl status mm-stgapi,ps -o rss). - The caps are set with a systemd drop-in,
/etc/systemd/system/mm-stgapi.service.d/limits.conf. Defaults, to adjust from the measurement: MemoryMax: the lower of 512M and 15% of RAM;CPUQuota=50%;TasksMax=64.- If the host can't spare that, stgapi moves to the new server (SRV-1) instead of being starved (phase-1 risk).
3. Deploy through systemd and promote (SG-1, SG-2)¶
deploy-staging(api.gitlab-ci.yml):- uploads and activates the artifact as today;
- then runs
sudo systemctl restart mm-stgapi, with a sudoers rulebitnami ALL=(root) NOPASSWD: /bin/systemctl restart mm-stgapi(person installs it); - then runs the smoke routes.
- Any
nohup,pkillorpython3 app.pylines are removed. - The job is automatic on merges to
api-phase1-staging-releaseand keeps itslaptop,dockertags. - Promotion MR:
api-phase1-development→api-phase1-staging-release, bringing !147, !132, !162 and the S3 route fixes. Risk medium; merged by Lane S on green CI and a green staging smoke.
4. Staging data load (SG-2, only after ISOLATED)¶
The api repo gets a loader, tools/staging_loader/, run from the laptop runner (or by a person), that reads production's public GET routes and writes only to staging:
| Data | Read from production | Write to staging | Notes |
|---|---|---|---|
| Catalogue: categories, owners, products, fund details | /category, /product/owners, product and fund-detail routes |
mm_api_staging with the staging DB credentials (no save routes exist; relay 19 Oct) |
Full; weekly refresh |
| Scheme master | /api/schemes |
mm_nav_staging scheme table |
Full |
| NAV history | /api/scheme/nav?scheme_code=… for each scheme in docs/staging-set.yaml |
stgapi POST /save-nav with the staging key |
De-duplicated by the D1/D3 rules of the NAV LLD; D2 dates are left out and listed; gaps listed |
| Index history | /api/index/nav?symbol=… for the set's indices (NIFTY 50, NIFTYM150MOMNTM50, benchmarks) |
mm_nav_staging index table |
Fixes the NIFTY 50 404 for the data part |
Rules:
- One request at a time, 1 s apart, outside 09:00–18:00 IST.
- Resumable: a state file records the last scheme done.
- Before writing, the loader re-runs a quick check: the configured staging DB names must be the two staging databases, or it stops (the same check as §1, steps 1 and 5).
- No production credential is ever read or held.
- Order: one scheme first (120245) through /save-nav as proof (queue S2); then the catalogue; then the rest of the set.
docs/staging-set.yaml (api repo) lists:
- pe_universe (PE's 25 funds);
- parity_sample (10 schemes across AMCs and categories, 2 indices);
- mwmc_pilot (the fund-page pilot schemes, from P4);
- category_reps (one per category, chosen by the loader from /api/schemes, then frozen).
Each entry gives the scheme code and the reason. Target size: about 200 schemes.
5. Daily parity (SG-3; the build is a harness item in MMDEP's collector)¶
For parity_sample, the collector calls the same routes on both hosts and classifies each difference:
- extra_field;
- missing_route;
- duplicate_removed (production has a duplicate date that staging doesn't);
- gap (a date missing on one side);
- value_differs.
It writes a table to latest.md. Only value_differs and missing_route count as defects.
C4 element¶
stgapistgDbsapimmdep
Evaluation¶
| Assessment | |
|---|---|
| V value | High. Staging becomes usable for PE, MWMC and MM-Cron, and the isolation question that blocks everything is settled with evidence |
| S simplicity | Shell checks, a systemd drop-in, one loader on public routes |
| C cost | About 2 agent sessions; three short person-runs (check, caps and sudoers, loader start); about one night of gentle load on production for ~200 schemes' history |
| D data | Public NAV and catalogue data. Staging credentials only; db.env values never printed; outputs carry names and grants |
| L legal | None |
Test plan¶
- Isolation check: run against a fixture env file and a stub
mariadbcommand (CI). Mixed databases →NOT ISOLATED: <list>; staging-only →ISOLATED; a line matchingPASSWORDis never printed (asserted on output). - Loader: with a stub production server and a stub stgapi, a scheme with D1 and D3 duplicates is written once per date; D2 dates are skipped and listed; resume after a stop continues from the state file; a production-looking target database makes it stop before any write.
- Deploy: a CI lint checks
.gitlab-ci.ymlhas nonohup,pkillorpython3 app.pyindeploy-staging, and does callsystemctl restart mm-stgapi. - Staging after load (collector):
/api/schemes≈ production count;/category= 180; NIFTY 50 returns 200; 120245 latest date within one business day of production once MM-Cron staging runs start. - Parity: the collector's classifier test with fixture pairs, one per difference type.
Risk¶
risk: high
The isolation verdict decides whether staging writes. A wrong ISOLATED would let staging jobs write production data. Mitigations:
- the verdict needs config, live connections and grants to agree;
- the loader repeats the check before writing;
- staging keys are refused on production (security LLD).
The caps, the deploy change and the loader are medium risk on their own.
Work items¶
| # | Title | Risk | Model | Scope paths | Depends on | Cx | Tests |
|---|---|---|---|---|---|---|---|
| 1 | isolation_check.sh with fixture tests; runbook for the person |
high | chatgpt-api-mmcron | api: tools/staging/isolation_check.sh, tests/test_isolation_check.py, docs/runbooks/stgapi-isolation.md | 2 | tests/test_isolation_check.py | |
| 2 | Person runs the check; verdict into runs/S2.md | high | chatgpt-api-mmcron | MMDEP: docs/relay/api-lane-s/runs/S2.md | 1 | 1 | |
| 3 | Host measurement and the systemd limits drop-in (runbook; person applies) | medium | chatgpt-api-mmcron | api: docs/runbooks/stgapi-limits.md | 1 | ||
| 4 | deploy-staging restarts via systemctl; sudoers rule runbook; CI lint |
medium | gemini-flash-lite | api: .gitlab-ci.yml, tests/test_ci_deploy_staging.py, docs/runbooks/stgapi-sudoers.md | 2 | tests/test_ci_deploy_staging.py | |
| 5 | Promotion MR development → staging-release | medium | chatgpt-api-mmcron | api: branches | 4 | 1 | staging smoke |
| 6 | docs/staging-set.yaml |
low | chatgpt-api-mmcron | api: docs/staging-set.yaml | 1 | yaml lint | |
| 7 | Staging loader (catalogue, schemes, NAV via /save-nav, indices), resumable, guarded | medium | chatgpt-api-mmcron | api: tools/staging_loader/, tests/test_staging_loader.py | 2,6 | 3 | tests/test_staging_loader.py |
| 8 | Load run: 120245 first, then catalogue, then the set | medium | chatgpt-api-mmcron | runs/S2.md evidence | 7 | 1 | collector rows |
| 9 | Collector parity classifier and report table | low | gemini-flash-lite | MMDEP: tools/collector/parity.py, tests/test_parity.py | 6 | 2 | tests/test_parity.py |