Skip to content

Runbook: rotate the /save-nav API token

1. How the token works (from the API code)

  • src/utils/decorators.py defines token_required. It compares the request's Authorization header with one value, API_KEY, from the app's configuration (config.py, or the runtime environment as the sample file advises).
  • Only /save-nav uses it (src/controllers/nav_controller.py; no other route has token_required).
  • There is one key and no grace period: the moment the API's key changes, the old key stops working, and so does every producer still using it.
  • Known producer: MM-Cron's NAV publisher (production server: /home/bitnami/scripts/navdata/, root cron at 04:00 daily). Possibly also an AWS Lambda function: the TEA audit couldn't confirm or rule it out.

2. Risks and how to contain them

Risk Effect Containment
A producer is missed (for example a Lambda) NAV saving stops silently, as in August Search for every user of the old key first (step 2); check the next morning's NAV date (step 8)
The production API doesn't come back after the restart Website pages that use the API fail Snapshot first; restart at low traffic; know the restart command beforehand (Ajay); keep the old value to revert
Config location wrong (file vs environment) The new key doesn't take effect Confirm where API_KEY is set on the server before changing
Staging shares the same key Staging can write with the production key Give staging its own, different key

Severity: medium, not an emergency. The repository is private and the key only allows NAV writes. But people and agents with repository access, including AI agents, have seen it. Rotating this week is right.

3. Steps (≈ 30–45 minutes; daytime, after the 04:00 NAV run)

  1. Snapshot the production Lightsail server.
  2. Find every user of the old key on the server, without printing the value to shared screens. Search the MM-Cron script folders and crontabs for the Authorization header or the variable that holds it. In the AWS console, check Lambda (region ap-south-1) for any function with a NAV or save-nav environment variable.
  3. Confirm where the API reads API_KEY in production (/opt/bitnami/api/api_prod: config.py or an environment file), and how the API is restarted (service manager, process manager or script). Ajay knows both.
  4. Generate a new key on the server: python3 -c "import secrets; print(secrets.token_urlsafe(32))". Keep it only on the server and in a password manager. Never in chat, issues, repositories or tickets.
  5. Update the producers first (script config, and the Lambda environment variable if one exists). Then update the API's API_KEY and restart the production API.
  6. Staging (/opt/bitnami/api/api_stg_release, port 5001): set a different key, so staging can never write with production's key.
  7. Quick auth check: a request with the old key must return 403. Whether an empty {"records": []} request is a harmless no-op isn't known, so don't send test data to production. Rely on step 8.
  8. Next morning after 04:00: confirm the latest NAV date advanced (/api/health, or the NAV read route Ajay uses). If not, check the producer logs.
  9. Clean up the repository: an MR removing the token from mm-crons/lambda/navdata/readme.md, and changing the example to https:// with a placeholder. The old value stays in git history, but it no longer works.
  10. Tell Ajay what changed and where the new key lives. Record the rotation (date, who) in the TEA estate notes, without the value.

4. Follow-ups (for the API owners)

  • Accept two keys briefly during a rotation (a list instead of one value), so future rotations need no exact cutover.
  • Use a constant-time comparison (hmac.compare_digest) for the key check.
  • HTTPS only for /save-nav (redirect or refuse plain HTTP).
  • Producer reliability (API !156): check the HTTP status, set a timeout, retry safely.