Skip to content

API and MM-Cron data architecture — owner record

The old API agent kept no record of Atul's explanation (its knowledge-transfer reply, 13 Oct). This file rebuilds it from the written sources Atul supplied. Labels: AJAY (Ajay's own words), ATUL (Atul in the meetings, from Gemini notes, which can mis-transcribe), CODE (agent findings in API !153, !157, !166).

1. Which database holds what

Data Database Source
NAV, Nifty index, portfolio data mm_data AJAY, 24 Sep
Product category, product, owners/AMC moneymarche_www (WordPress) AJAY, 24 Sep
Sector breakup Not maintained AJAY, 24 Sep
moneymarche_www.mm_product_holdings Not in use; no relation to mm_portfolio_items AJAY, MM Questions doc
WordPress databases Different for staging and production AJAY, 12 Oct
mm_data Common to staging and production AJAY, 12 Oct

2. Portfolio pipeline (MM-Cron)

All AJAY, MM Questions doc: - Runs on the production server from /home/bitnami/scripts/portfolio: python3 clihandler.py -a <amc>, one .sh per fund house. - AMC codes: icici, hdfcmf, quant (not downloading), sbimf, axis, adityabirla. - Cron is scheduled but not working, so it is triggered by hand every month. - Source: AdvisorKhoj portfolio pages. For the last few months the URL or file format changed; to be fixed. A file can also be downloaded and uploaded through the Admin Dashboard (React). - scheme_name is taken from the downloaded sheet. portfolio_id is auto-increment, internal only. - Each instrument row in the sheet becomes one mm_portfolio_items row. The source has ISIN for some rows, not all. - Portfolio is linked to the product by scheme name, not by code. - Known defect: ICICI Prudential Banking & Financial Services Fund has 1 item for May, June and July 2026 (52, 51, 51 items before). To be reviewed and fixed.

3. Environment principles (ATUL, meetings 26–27 Sep)

  • Production is the protected legacy system. Its actual runtime behaviour is preserved and characterised, not changed during investigation.
  • stgapi (the staging API, new development) is the controlled, reconstructed and modernised version of the capability. It was meant to be isolated from production with separate staging database connections, to verify API contracts and catalogue identity. It is not a production clone.
  • Financial data problems are documented, never silently fixed. Financial data may be shown as unavailable, never fabricated.
  • API index NAV must not be substituted with another index just because one exists.
  • Scheme catalogue needs: stable scheme identity, complete NAV history, freshness, duplicate/replace rules, error rules.
  • Trusted sources: an official industry source first (Gemini notes say "M"; probably AMFI, to confirm), then insurer and AMC records.
  • WPDev's React layer stays independent of WordPress IDs and metadata. Portfolio Engine aims at one canonical dataset.

4. stgapi ownership (ATUL, 13 Oct; clarified 14 Oct)

stgapi (the staging API) was built by the API agent and installed by hand during the TEA audit. Ajay's team does not run it. Staging is ours to restore and operate.

5. Conflicts to resolve

# Conflict Next step
X1 Design intent: staging isolated with its own database connections (26 Sep). Ajay, 12 Oct: mm_data is common to both. Check the staging API's effective configuration on the server (read-only). Until proven isolated, no staging job writes to mm_data.
X2 Portfolio-to-product link is by scheme name (AJAY). Standing rule: never map by name (API !166 §17). Plan !167 Phase B: a deterministic mapping table (scheme code / ISIN), name match only as a reviewed fallback.
X3 Writers for catalogue tables (mm_product, mm_category, owners, fund details) are not stated anywhere. Ajay says they live in WordPress. Ajay's questions doc Part 2, item H: WordPress admin, import script or job.

6. Target

The target database follows the API architecture. For non-MF products and the extended category tree it uses MCRM's catalogue design (MCRM research/catalogue/2026-10/); mutual funds are standardised onto it later (PM decisions log, 13 Oct).

7. Catalogue ownership and MF data (ATUL, 13 Oct 09:14)

  • Category and product data live inside the WordPress database (moneymarche_www).
  • Categories: the API has read and write operations on categories, but no MM-Cron job updates them. Categories form a tree; the knowledge of this tree must be documented so agents can use it. The API may need to maintain the mutual-fund categories because AMFI changes them from time to time. Category and product APIs will be taken over by MCRM catalogue management.
  • Products (called schemes for mutual funds) are updated by the API from MM-Cron data.
  • MF data sources: two main sources: (1) AMFI endpoints and URLs, some leading to AMC sites; (2) monthly factsheets on AMFI and AMC sites (Ajay: parsers for 10 AMCs, few working, about 40 more to plan).
  • Mapping Atul asked the old agent for (status to check): (1) MM-Cron scripts against each source and back, for coverage and gaps; (2) source to database, since each source has several sections of data points; (3) API to database. Database redesign or renaming is acceptable where needed.
  • Known design clash: MF composition portfolio (holdings) clashes with Portfolio Engine's portfolio (client holdings of MF and fund-like investments). MCRM introduces product family and variant: in MF, four products share a name across Direct/Regular and Growth/IDCW; they are one family sharing one fund structure, so the composition portfolio should key on the family. Today its primary key is loosely defined (linked by scheme name).

8. Website ownership (ATUL, 13 Oct 09:14)

  • MWMC maintains production (moneymarche.com, moneymarche/webapp-sakai, branch dashboard) and MMCLONE. Every production change is made on MMCLONE first. MMCLONE holds only the incremental code that differs from production, plus the history of the current agent's work.
  • MMCLONE uses a static menu: the production menu reads the category and product tables (and other API tables) directly from the database although APIs exist for them; the static menu avoids giving MMCLONE extra database credentials.
  • MWMC's role: build knowledge and tests; find and fix immediate bugs; build SEO; then coordinate with MCRM and WPDev to introduce their architecture, design and features. Its repo has 5 project tags for its objectives and work streams.
  • MCRM holds the future of the catalogue, Mautic, calculators, the customer dashboard and related features. Much code is expected to become standard CRM features, leaving only website interfaces in WordPress in the long run.
  • WPDev holds the future direction of the website itself.

9. MWMC working repository and labels (from the MWMC chat, 10–13 Oct)

  • MWMC's working repository is money-marche-testing/money-marche-sandbox (MMCLONE): 21 open issues, route and SEO evidence work (#20, #21), MRs !24 (route-family reconciliation contract) and !25 (README label taxonomy), both documentation only.
  • Labels (created by Atul, 12 Oct): exactly one project:: label per issue: project::production, project::mmclone, project::wpdev, project::mcrm, project::mmdep, project::tea, project::cross-project; functional labels (seo, inventory, testing, qa, database, coverage, maintenance, mutual-funds) stay as a second dimension. The "5 project tags" Atul mentioned are these labels.
  • Its October plan (MR !7) and work packages #52 to #56 were filed in moneymarche/webapp-sakai (the Next.js app); they belong in the sandbox repository with the labels above.

10. Production site code (ATUL, 13 Oct 10:12)

  • moneymarche/webapp-sakai holds the main site code served at moneymarche.com; the dashboard code is for logged-in users. MCRM has researched it from the repository.
  • The production website was built after work on the staging website stopped, losing some unimplemented features; that is why staging is preserved. Work probably started on the dashboard branch with code copied from staging.
  • WordPress plugins deployed in production live in a separate repository, moneymarche/wordpress_plugins.

11. Category tree flags (ATUL, 13 Oct 20:33)

  • The category table and API already carry two flags: status (activates or deactivates a category) and show in menu. The production menu code reads them correctly from the database and builds collapsible menus. Example: "Market Cap Oriented Funds" is an administrative branch and is not shown in the menu.
  • The /category API currently returns both flags as 1 for every category: a defect to fix, so consumers can rely on the API instead of reading the database.
  • The staging website (staging.moneymarche.com, repo money-marche-group/money-marche-website-staging) shows a larger category tree than production; compare the two before proposing the target tree.
  • Roll-up rule (Atul, 17 Oct): the production website "wraps up" categories: when a category is hidden from the menu, its children appear one level up in its place. The API must reproduce this so websites can stop reading the database: return the stored flags, and offer the effective menu tree (hidden categories collapsed, children promoted) as computed by the production menu code. Ajay, MWMC and WPDev were asked to locate where production uses status and show in menu; their findings define the exact rule.
  • MCRM proposes the category tree first (structure, status, show-in-menu, administrative branches), in the same way as the product design.

12. Admin panel, product discovery and sources (ATUL, 13 Oct 20:57)

  • Earlier design: the production website's admin panel edits both the category tree and products. For each category it auto-creates pages whose data comes from the API, and an editor can add custom content that is saved with the page.
  • Today the admin panel reads and writes the database directly. Next phase: move those reads and writes to the API.
  • Mutual fund products are discovered from AMFI NAV data: when a new scheme appears, MM-Cron jobs look up its details on AMFI sites and update the database.
  • Insurance products: primary source is the IRDAI site, then the insurance companies' own sites.
  • This answers part of the writer inventory (table H in Ajay's doc): categories and product custom content are written by the website admin panel; MF products by MM-Cron from AMFI.

13. Ajay's answers, 13 Oct evening (FROM AJAY, "MMDEP — Questions for Ajay", Part 2)

Writers and schedules

Table Writer Schedule
moneymarche_www.mm_product WordPress form; and MM-Cron lambda/parsedata/cliAum.py -m scheme Script run by hand monthly; to be scheduled
moneymarche_www.mm_category WordPress form —
moneymarche_www.mm_product_owner WordPress form —
moneymarche_www.mm_product_type WordPress form or direct database entry (to verify) —
moneymarche_www.mm_product_fund_details API api/update_products_nav.py Daily, through the API engine
mm_data.funds_nav NAV Lambda; missing.py for gaps Lambda daily 15:30; recovery by hand
mm_data.tmp_funds_nav Recovery script only (staging table for faster inserts) —
mm_data.nifty_index nifty-data/nifty_download.py Daily
mm_data.mm_nfo_list MM-Cron scripts/nfo/nfo-new.py Daily
mm_data.mm_portfolio, mm_portfolio_items scripts/portfolio/clihandler.py, one .sh per AMC (icici, hdfcmf, sbimf, axis, adityabirla) Monthly
mm_data.scheme_data Temporary table for faster upserts —
Mongo Not in use; the code can be removed —

A database view vw_portfolio_coverage now shows portfolio coverage.

Production crontab (bitnami user)

When Job
Daily 22:26 Let's Encrypt renewal (bncert/lego) for www.moneymarche.com only, then Apache graceful restart
Daily 21:15 scripts/nfo/nfo.sh
Saturdays 20:00 scripts/parsedata/ter.sh
7 Jan, Apr, Jul, Oct 00:00 scripts/parsedata/aum.sh
7th of each month 03:00 scripts/portfolio/portfolio.sh
Daily 15:30 scripts/nifty-data/v2/getdata.sh

Other answers

  • No read-only database user exists; one can be created and used.
  • Adminer is reachable from the internet; Ajay uses it to review data and agrees extra security should be added.
  • NAV stopped on 19 Aug because the AMFI endpoint changed its response structure; handled in the 7 Oct fix.
  • NIFTY 50 stopped on 25 Sep because getdata.sh still called the old Python script; updated on 13 Oct, so data should resume from the next run.
  • The SuiteCRM secret can move to configuration, but the SuiteCRM integration itself needs validating and fixing.
  • Staging: no answer needed (built by our agents).

14. Catalogue ownership answers (ATUL, 14 Oct 00:15, MCRM questions #107 to #112)

  • #107 Product IDs: MCRM's Catalogue Manager sets the canonical ID rules; the API database stays the store and writer.
  • #108 UIN/version conflicts: the regulator governs the UIN, the provider governs product wording, the API owner publishes.
  • #109 Category tree editing: the WordPress admin stays the edit screen for now. In future, categories and products will live in the SuiteCRM catalogue.
  • #110 API contract: MCRM owns what the fields mean; the API agent owns the running API; breaking changes need a consumer check and Atul's approval. Everything other than mutual funds is MCRM's domain.
  • #111 Migration: in the current phase everything stays where it is. First, direct database calls move to the API, and the API exposes information to the same level as the database reads.
  • #112 Further domains: decide after WP1 from real traffic and business need.

15. Staging website versus stgapi (ATUL, 14 Oct)

Two different things share the word "staging":

  • Staging website (staging.moneymarche.com, repo money-marche-group/money-marche-website-staging) is legacy code. It holds a lot of earlier work that never reached the production website (calculators, a larger category tree, more). It does not run against stgapi. It is an important product to preserve. Downstream projects studied only the production website, so their baseline starts there; only MCRM studied the staging website, from the repository (calculators and categories), and has open questions it cannot answer without the database or a running deployment. Lower priority: it does not block production work.
  • stgapi (stgapi.moneymarche.com) is the new API development. It was down and no project reported it: nobody uses it yet.

Sections 3 and 4 refer to stgapi, not to the staging website.

16. API versus stgapi, and the staging database (ATUL, 14 Oct)

  • API (api.moneymarche.com) is the production route handed over by the legacy team. Legacy applications use it and all new applications use it. Its code is on main; anything deployed to the production route must be merged to main.
  • stgapi is a little more advanced: built on requirements from Portfolio Engine and WPDev, on separate branch(es) of the same repository. Neither could use it because its database had no data.
  • MM-Cron work this implies: (1) a script to copy data from the production API database into the staging database (one-time load, then refresh); (2) scripts that keep staging records current; (3) housekeeping scripts for the database: cleanse data and fix missing records. Staging scripts must write only to the staging database (rule in §5, X1).
  • Portfolio Engine's Gateway extends the APIs for two purposes; see docs/notes/capabilities-pe-mcrm-2026-10-14.md.