CRP system architecture

flow2 · CRIO write path (mutating)

Every change funnels through the book-and-verify chokepoint (calendar) or the interaction-write chokepoint (notes). Reads are CDC-replicated into a CRIO replica dataset; a single read engine handles all reads. Any direct CRIO calendar write outside the chokepoint is blocked by a static lint rule in CI.

flow3 · Inbound triage

Fax: Telnyx → a scheduled classify step (Claude) → a fax-classification record → a scheduled route step → auto-dispatch email for the categories on the auto-send allowlist.

Voicemail: Google Voice → Gmail forward → a scheduled triage job → Claude classifies (11 categories × 4 urgency × 5 transfer targets) → merges into the voicemail record → SMS confirmation to caller via Telnyx → urgent items routed through the alert router to the task board.

flow4 · Comms-tab contract

Any auto-comm goes through the comms approval queue. Two near-in-time comms → the later one is canonical; earlier comms must NOT contradict it. An auto-send allowlist defines which message types fire automatically vs. land in a human-review tier for approval.

flow5 · Bulk mutations

Every N-row external mutation goes through the bulk-action guard (canary → apply → fail-fast: abort after 3 of 5 same-error). Direct N-row apply loops without the guard are lint-blocked in CI. The doctrine originated from an early incident where a bulk site-correction ran 25 cancel-without-create deletions for lack of such a guard.

• CRIO is read-mostly via CDC replication, write-rarely via API. Don't try to push every state change to CRIO — its API surface is narrow (calendar, interactions, traits). Audit lives in our own CRM data store.
• Firestore is for keyed state + audit, NEVER for analytical queries. BigQuery is for analytical + CRM data. Firestore docs must stay <1 MiB (a lesson from a 2026-05-13 fax outage).
• Email goes through the standard email chokepoint for everything except finance digests, which use a separate direct sender (off the Comms tab — financial-action emails, not coordinator workflow).
• Change-controlled files (the dashboard server entrypoint, the core cloud-function source, all CI workflows, git hooks, and the agent instructions file) are code-owner-gated + pre-commit-enforced + re-checked in CI.
• Cloud SQL is NOT used by the dashboard. The one Postgres instance is an experimental side project. BigQuery + Firestore cover the dashboard's analytical + state needs.

Project: a single Google Cloud project (id and billing redacted).

us-east1 (production):

• A Cloud Run operations dashboard — serves the internal IAP-gated dashboard
• A Cloud Run read API
• A single chokepoint Cloud Function Gen 2 — ~126 read feeds, ~125 read/action capabilities
• A Cloud Run finance dashboard — the internal finance dashboard (owner-only)
• ~91 Cloud Scheduler jobs (85 enabled)

A second region (experimental):

• A small experimental agent stack (Cloud Run + a micro Postgres) — side project
• A handful of ad-hoc BigQuery tooling services

BigQuery region: US multi-region (job location).

A single role→tabs map is the source of truth, mirrored into a change-controlled region of the app shell. Admin is a strict superset of leadership — owners see every leadership surface automatically. Section-level role gating inside a shared tab is a lint-blocked anti-pattern; RBAC is tab-level only.

Finance cluster decommissioned 2026-04-24: a separate internal finance dashboard. The finance role above maps to the main dashboard's read-only subset, not the legacy 6 finance tabs (which are change-controlled and no longer served).

The dashboard front end is built from 26 source modules (concatenated at build time into a single hashed bundle) plus a shared design system, loaded ahead of a small late-binding inline block in the app shell. Rather than one monolith, responsibility is split into focused modules grouped by capability:

A shared design system exports the phone-formatting, link, chip, and button helpers; a fallback shim mirrors every render-path helper so a missing import never breaks a render.

All schedules run on Eastern time. Each job's apply/send behavior follows the same safety defaults as the rest of the platform: read-only by default, with mutation and outbound-send gated behind explicit flags. The live count is ~91 jobs (85 enabled · 6 paused). Rather than enumerate them, here is the breakdown by capability domain — what each category of automation does and roughly how many jobs sit in it.

Kill-switch pattern: any scheduled job can be paused independently without a code deploy; pausing is documented per-job in the incident-response runbook.

Leads / patient-acquisition pipeline

Public intake + Meta Lead Ads → a prescreening / intake table → eligibility match → recruiter. Hardened across waves 1–3 (2026-06).

Comms engine

One queue, three channels (email via Gmail, SMS + fax via Telnyx). Every auto-comm flows through a single enqueue chokepoint; allowlisted types auto-fire, everything else lands in a human-review tier for approval. The later comm wins — earlier comms must not carry contradicting calls-to-action.

• Rule engine + role-inbox registry — a single source of truth for reply-to inboxes (recruitment / billing / operations / owner); raw inbox literals were ratcheted down from 163 to 84 across migration waves. Outbound always sends from a no-reply address; finance uses a separate direct sender (off the Comms tab).
• Data model — a comms queue (state) · a comms audit log (chokepoint) · a send log (the only place carrying both the sent record and the rule that produced it; SMS is audited here too).
• Send-time guards — toll-free phone guard · duplicate-send detector (a standing alarm) · email-volume detector · per-kind frequency guards · stuck-message escalation · structurally-impossible suppression (rolling window) · suppress-empty owner digests + a weekly heartbeat.
• Phone-vs-in-person copy — phone-screen studies render "phone call" copy: no address, no maps link, no "bring photo ID".
• Claims-coverage registry — every appointment-bearing comm declares its five claims (when / when / where / which / whom); a new kind can't ship without a guard for each (a CI tripwire).

CRIO has no payments/finance write API, so this drives the CRIO web UI like a bookkeeper to close the status-flip gap (sponsor pays → invoice stays "unpaid" in CRIO). Two paths in one tree:

Recipes: payment + reconciliation, reconcile-only, and invoice-create — all LIVE behind the approval queue. Mark-invoice-paid ("Full Payment") is built but test-only — it requires an explicit execute flag, is hard-gated to a single test study, and its enqueuer is deliberately unwired (double-pay risk).

Write-then-verify (mirrors the calendar writer)

• In-recipe vision-confirmation + URL navigation guards on every commit step.
• An out-of-band warehouse read-back flips submitted → verified only on a deterministic remittance-id + amount match — the same phantom-confirmation doctrine as the calendar writer.

Local background agents (on the Mac · invisible to cloud logs)

• A warm-browser daemon that consumes the approved-write queue one row at a time.
• A no-browser read-back poller (every few minutes) that flips verification status.
• A daily session / navigation smoke watchdog that performs no writes.

Kill switches

A soft kill-file · an auto-tripping circuit breaker · a short canary hold + STOP-via-SMS · schema validation refuses any row not explicitly approved · dry-by-default (mutation requires an explicit apply flag) · and a hard "unload the agents" stop.

Remote ops

The Mac sits on a private identity-based mesh network: SSH over the mesh (never exposed to the public internet) lets us restart the writer, the read-back poller, or the inbound tunnel when they die — no physical trip to the box. Deliberately on a separate path from the tunnel the agents use: the way in must not depend on the thing that breaks.

The CRIO calendar write chokepoint. Any direct CRIO calendar write outside this helper is lint-blocked in CI. Doctrine origin: a 2026-05-01 incident (149 patient appointments stored four hours late since the prior autumn).

The 8-stack hardening that surrounds it

1. Real-CRIO smoke test on every deploy — exercises a designated test patient and fails the deploy on any non-success or read-back mismatch. No skip flag by design.
2. Preflight subject-on-study check — against live CRIO (~200 ms overhead). Catches replication lag.
3. Atomic create-first saga — earlier the order was cancel-first; a create-fail then left the patient with no appointment.
4. Eastern-naive datetime contract — a single formatting chokepoint; tests block UTC drift.
5. Phantom-confirmation gating — every consumer reads the verified-success flag, never "we tried."
6. Bulk-action canary → apply gate + fail-fast — 3 of 5 same-error aborts the loop.
7. Lead-status reconciliation alerts on dangerous demotions — phantom bookings can't hide.
8. CRIO write monitoring — a failure-rate sweep over the audit trail (deferred · tracked as a coverage gap).

Three morning defense-in-depth audits

Run by the daily data-health scan (early AM): CRIO time drift (booking notes vs. stored start), physical impossibility (overlap / cross-site <90 min), and zero-duration upcoming visits. Independent of the chokepoint, so they catch direct-in-CRIO-UI mistakes too.

Newer validation layers (2026-06)

Generalization: the mutate-and-verify chokepoint extracts the validate → write → read-back → assert → audit contract so any phantom-confirming external mutation (task board, stipend provider, e-sign provider, second CRIO surface) reports success only when read-back confirms intent. New write sites use it now; the ~36 legacy task-board / stipend-provider sites migrate in phases.

eSource QC engine

A 15-module subsystem (engine, chokepoint, LLM narrator, rules, registry, wizard, digest, a tracker ingest/classify loop, plus five proposers: consent, eligibility, IP-canon, and required-docs). The engine runs on a business-day morning cadence, an all-day ingest + classify loop feeds it, and a digest emails ops@ each morning. The LLM is a narrator: schema-validated output cites subject + visit + evidence row, with severity capped by a ground-truth tier (1–4). Auto-pause on a 3σ findings spike (an append-only run log, 21 CFR Part 11). A deterministic pre-classifier runs before the LLM.

Screen-fail intelligence

Classifier precedence: a deterministic pre-classifier (mirrors 18 months of real coordinator cancel-reason vocabulary) → Claude over the visit notes → a locked 10-value reason enum (drift blocked by a CI lint rule). The inbound webhook is primary; a scheduled sweep is the dropped-webhook backstop. About ten capabilities: manual-tag · reclassify · classifier-preview · protocol-level recurrence patterns · context · outcome detection (a rematch learning loop) · cross-study indication patterns · weekly digest. Surfaces sponsor-level signals (inclusion criteria too tight) that per-event triage misses.

• IT-security monitor — ~14 monitoring sections · an account-suspension check runs every 15 min (auto-suspends risky Workspace accounts) · an owner-only suppress-empty digest each morning + a weekly heartbeat.
• PHI perimeter — a sensitive/PHI feed allowlist contract (403 to anonymous callers) · enforced by a CI lint rule + contract tests · every dashboard fetch is wrapped by the IAP session guard.
• Tracing fail-closed — the agent-tracing integration refuses to enable if pointed at a public, out-of-perimeter endpoint (traces can carry PHI; they must stay in-perimeter). Currently dark after a 2026-05-22 leak.
• Change-controlled sections — 4-layer enforcement: code-owner review + pre-commit + CI re-check + branch protection. Author-domain gate (@phillyresearch.com only, SSH-signed commits).
• Supply-chain / IAM — secret scanning · static security scan · IaC static analysis · SBOM + a nightly security sweep · a weekly IAM-drift audit (project + service accounts + critical secrets + dataset ACL vs. an allowlist).

The fourth production surface: a token-gated provider portal at joinresearchstudies.com/refer for direct-from-clinic ordering. Referring physicians place orders, get a fax-back referral, and see a study list with eligibility criteria. It also renders live PI rheumatology availability — it reads a dedicated "PI schedule" calendar, subtracts booked visits, and honors out-of-office blackouts. PHI-free by construction (only dates + open windows; never patient / coordinator / subject identifiers). A reviewer view renders a no-code page (an operator eyeballs it, edits the calendar, refreshes) — to change what providers see, edit the calendar event times; no code change.

Source surfaces: the 26 front-end modules + the design-system sources + a small inline late-bind block (and inline styles) in the app shell, delimited by build markers.

Build: a build script concatenates the modules + design system into a single hashed bundle, transforms the app shell, and extracts a stylesheet.

11 CI/CD workflows (auto-deploy on push to main, ~11 min):

Pre-commit pipeline

1. Step 0 — author-domain gate (must be @phillyresearch.com)
2. Step 0b — rule-drift advisory (prose rules ↔ coverage rows)
3. Step 0c — orphan-stash warning (other-agent work-in-progress stashes)
4. Step 1 — change-controlled-sections check
5. Step 2 — invariant-coverage check (coverage rows pointing at non-existent backstops fail)
6. Step 3 — the custom lint suite (56 rules)
7. Step 3b — an AST-based no-undef check across the front end + inline block
8. Step 4 — the unit-test suite (1,559 tests)
9. Step 5b — render-smoke (executes a key render path against fixtures in a headless DOM)

Change-controlled files (code-owner review + pre-commit + CI re-check): the dashboard server entrypoint · the core cloud-function source · git hooks · the code-owners file · all CI workflows · the change-control checker · the custom lint suite · the agent instructions file · the architecture + ground-truth docs. Inside the app shell, a handful of region tags mark change-controlled admin, finance, and RBAC blocks.

56 active rules in the custom lint suite · 1,559 passing unit tests · 39 invariant rows (34 enforced) in the coverage checker · 5 cloud-function contract tests · a headless-DOM render-smoke. Selected high-leverage rules, described by what they catch:

The table above is a curated subset — 56 rules total. Many enforce the doctrine chokepoints (each NEVER/ALWAYS prose rule is paired with a backstop and a coverage row — 405 rows).

Post-deploy smoke canaries — 14 (run in the cloud-function deploy and able to fail it): book-and-verify · email · AI caller · real AI dial · SMS · fax · task board · stipends · ad-platform · ad-platform conversions · rides · intake forms · voicemail triage · screen-fail (+ fax routing).

A small set of primary keys carries most analytical queries — roughly: study, patient, subject (one patient-on-one-study), subject-visit, calendar-appointment, and user. The accounting/revenue tables are the exception: they use string UUIDs and join to the clinical side through the study's external id rather than its integer key (a common gotcha). A handful of additional keys connect the document, web-form, and invoice-line chains.

Universal filter gotchas

• CDC soft-deletes: always exclude soft-deleted rows on the CRIO replica.
• Cancelled appointments: always exclude the deleted-status sentinel (it means DELETED, not cancelled — otherwise it adds dozens of phantom rows).
• Publisher / template studies: dropped at source by a study-fetch helper.
• Sponsor coordinators: a normalization helper silently drops a known set of sponsor-side coordinator names.
• Timezone: always use Eastern-explicit current-date SQL, never the bare (UTC) form; enforced by a lint rule.

A headless Cloud Run scraper that observes a sponsor portal for protocol updates, query responses, and document expirations. It writes additively to a scraper-runs audit table via an open/close merge chokepoint.

Cost-telemetry chokepoint

Every warehouse read in the scraper's query layer routes through one accumulator that also reads each query's bytes-processed metric. The per-run sum is reset at run start and flushed at run end into the audit row's cost column. This mirrors the cloud-function-side read engine's cost sink so cost telemetry lands end-to-end across both services. Without it, the column is always null and any future cost-regression alert has nothing to work with.

Audit-table contract

• Scraper-runs rows carry an open/closed state — run start merges an open row; run end closes it.
• A stale-run alert fires through the alert router when a run stays open past its expected window.
• A watcher endpoint + a Resources-tab health card are open follow-ups (tracked in the scraper runbook).

Auth: the portal vendor's bookmark-redirect URLs were unmasked in a one-shot step (2026-04-13). A cookie-based session is held in a secret. It is re-run ad hoc when bookmarks change — intentionally NOT on a schedule.

• RBAC tab matrix — 8 roles × 10 tabs
• Frontend modules — 26 source modules + design system
• Scheduled automation — ~91 jobs (85 enabled), grouped by capability
• Agent system — 6 active agents + phase roadmap
• AI voice telephony — Telnyx-native · enrollment/Lp(a) pre-screen · caps + consent + audits
• CRIO writer — browser-automation write-back · payments/invoices/mark-paid
• Book & verify saga — 4-step sequence + 8-stack hardening + capacity/cross-site guards
• eSource QC & screen-fail intelligence — 15-module QC engine + classifier + rematch loop
• Security, compliance & PHI perimeter — auto-suspend + PHI contract + tracing fail-closed
• Provider portal — direct ordering + live rheum availability
• Build & deploy pipeline — 11 workflows + pre-commit + change-controlled files
• Lint rules & test backstops — 56 rules + 1,559 tests + 14 smoke canaries
• BigQuery join model — primary keys + universal filter gotchas
• Revenue & receivables platform — 4 layers · margin engine · dollar-journey FMEA
• Portal scraper — sponsor-portal observer · audit table · cost-telemetry chokepoint