Architecture

System Context

The Insight Service is deployed as a set of AWS Lambda functions. It exposes a REST API behind API Gateway and participates in the broader FloatMe platform as both an event-driven consumer (Kinesis, SQS, EventBridge) and producer (EventBridge).

system context insight service

Inbound Traffic

Source Description

prod-insight-service-api (API Gateway, IAM + JWT)

IAM-authenticated requests from internal FloatMe services and JWT-authenticated requests from the mobile app. Routes to the prod-insight-api Lambda.

prod-insight-webhook-api (API Gateway, HMAC-verified)

Pave webhook callbacks. The prod-insight-webhook Lambda validates the HMAC signature on every request before processing.

prod-txn-plaid-transactions (Kinesis, LATEST)

Plaid transaction events from the Transactions Service. Consumed by the prod-insight-feeder Lambda.

prod-txn-floatme-transactions (Kinesis, LATEST)

FloatMe transaction events from the Transactions Service. Consumed by the prod-insight-income-signaller Lambda for real-time income detection.

prod-insight-miner (SQS)

Mining job messages enqueued by the webhook Lambda. Consumed by prod-insight-miner.

prod-insight-replay-feeder (SQS)

Manual replay job messages. Consumed by prod-insight-replay-feeder.

prod-insight-balance-alert-event-tap (SQS)

new_account events from txn-service, routed via an EventBridge rule. Consumed by prod-insight-balance-alert.

prod-income-event-tap (SQS)

income_txn events emitted by this service’s own income-signaller, routed via an EventBridge rule. Consumed by prod-insight-funds-notifier.

prod-insight-institution-change-handler-sqs-tap (SQS)

new_bank_account_added events from txn-service, routed via an EventBridge rule. Consumed by prod-insight-institution-change-handler.

Outbound Traffic

Destination Description

EventBridge (default bus, source insight-service.income)

income_txn events emitted by the income-signaller Lambda when a negative-amount transaction dated today (America/Chicago) is detected on the FloatMe Kinesis stream.

EventBridge (default bus, source insight-service.miner)

user_new_insights_available events emitted by the miner Lambda after all Pave insight pages for a user have been processed and saved.

Pave API

Transaction and balance uploads (feeder, replay-feeder); unified insight and cash-advance score fetches (miner, API).

SageMaker (income-detection-endpoint)

ML inference requests for per-transaction income classification, invoked by the API Lambda on POST /{user_id}/insights/employment/detect.

Segment

Push notifications and analytics events from the funds-notifier and balance-alert Lambdas.

Credit Card Service

Credit card funding notifications sent by the funds-notifier Lambda on income detection events.

User Service

User profile lookups by the API Lambda.

Transactions Service

Account and balance queries by the miner and API Lambdas.

Underwriting Service

Eligibility checks by the API Lambda (income verification) and the balance-alert Lambda.

Lambda Functions

Function Trigger Responsibility

prod-insight-api

API Gateway (IAM + JWT)

Primary REST API. Handles all insight CRUD, employment management, payday prediction, income detection via SageMaker, cash-advance score retrieval, and expense forecasting. Composes Pave data, RDS employment records, and DynamoDB insight cache.

prod-insight-feeder

Kinesis (prod-txn-plaid-transactions, LATEST)

Receives Plaid transaction events, uploads the user’s transactions and balances to Pave in paginated batches, and tracks pagination progress per user in DynamoDB (pave_label, last_page).

prod-insight-webhook

API Gateway (Pave webhook, HMAC-verified)

Receives Pave webhook callbacks. Validates HMAC signatures. Acknowledges USER_DATA_UPLOAD_SUCCESS and USER_DATA_DELETE_SUCCESS without side-effects. Enqueues a mining job to prod-insight-miner SQS on USER_DATA_INSIGHTS_READY.

prod-insight-miner

SQS (prod-insight-miner)

Dequeues mining jobs. Fetches unified insights (expenses, income, scores) from Pave. Writes all entity types to DynamoDB. Checks last_page state and emits user_new_insights_available to EventBridge only after all pages for a user have been processed.

prod-insight-replay-feeder

SQS (prod-insight-replay-feeder)

Manual replay path. Refetches Pave data with custom date or parameter overrides and updates pagination state in DynamoDB. Does not trigger mining or emit events.

prod-insight-income-signaller

Kinesis (prod-txn-floatme-transactions, LATEST)

Filters the FloatMe transaction stream for negative-amount transactions dated today (America/Chicago timezone). Emits income_txn events to EventBridge for each match.

prod-insight-funds-notifier

SQS (prod-income-event-tap, via EventBridge rule on income_txn where amount < CC notification limit)

On income detection, calls the Credit Card Service and emits a Segment event to notify the user. Stores a cc_income_notification entity in DynamoDB for deduplication.

prod-insight-balance-alert

SQS (prod-insight-balance-alert-event-tap, via EventBridge rule on new_account from txn-service)

When a new bank account is linked, validates eligibility via the Underwriting Service and sends a balance alert push notification via Segment. Uses a DynamoDB distributed lock to prevent concurrent alerts for the same user.

prod-insight-institution-change-handler

SQS (prod-insight-institution-change-handler-sqs-tap, via EventBridge rule on new_bank_account_added from txn-service)

When a user’s bank institution changes, clears stale employment records from the RDS employment table.

Pave Mining Architecture

The Pave mining sub-system handles the full lifecycle of ingesting transaction data into Pave and retrieving the resulting financial insights. The feeder and miner are decoupled via Pave’s own processing pipeline and a webhook callback.

system context pave mining

See Pave Mining for detailed flow diagrams and per-component decision logic.

Data Storage

DynamoDB: prod-pave

The primary data store for all insight entities. Uses a single-table design. Entities stored:

Entity Type Content

insights

Aggregate metadata record for a user’s latest insight run (timestamp, version)

recurring

Recurring expense records sourced from Pave (merchant, due date, amount, bill type)

ritual

Ritual expense records sourced from Pave (normalized merchant, frequency, average amount)

income

Income source records sourced from Pave (recurring sources, total recurring income)

balance

Bank account balance snapshots computed from Transactions Service data during mining (checking + savings accounts, average monthly income/expenses)

scores

Cash-advance repayment probability scores sourced from Pave (15d, 30d, 45d intervals)

payday

Predicted next payday date and supporting data

pave_label

Per-user label tracking whether the user’s Plaid data has been labeled in Pave

last_page

Per-user, per-request pagination state for tracking whether all Pave insight pages have been processed

income_verification

Manual income verification data entered by the user; augmented by MX or Ollie verifications

alert

Balance alert state for deduplication

cc_income_notification

Credit card income notification deduplication state

See DynamoDB Tables for full key construction, access patterns, and TTL configuration.

RDS PostgreSQL: employment

The employment table stores user-submitted employment records and is the source of truth for payday prediction inputs. Both a main (read-write) and replica (read-only) connection pool are maintained.

See PostgreSQL Schema for column reference and access patterns.

Additional DynamoDB Tables

Table Purpose

prod-incomedetection

Stores income detection run results written by the API Lambda during employment detection. Each record captures the output of all detection methods (A, B, C, D, and ML) for a user’s transaction batch, along with ML metadata. TTL is 7 days to allow downstream Snowflake ingestion. Separate from prod-pave to allow independent TTL and scaling configuration.

fmdatacapture-*

Shared data-capture table used to record PAYDAY_PREDICTION and CASHFLOW_ANALYTICS events with a 2-day TTL for analytics pipeline consumption.

user-balance-settings (legacy, us-east-1)

Legacy user settings table accessed by the balance-alert Lambda for pre-existing user state.

locks (legacy, us-east-1)

Legacy distributed lock table used by the balance-alert Lambda to prevent concurrent alerts per user.

External Service Integrations

Service Integration

Pave

Core analytics provider. The feeder uploads paginated Plaid transactions and balances. The miner fetches unified insights (expenses, income, ritual, scores) after Pave processes them. The API Lambda fetches scores and expense data directly for synchronous API responses.

Transactions Service

Queried by the miner to fetch account balances for the balance snapshot entity. Also queried by the API Lambda for account and balance data used in payday prediction and forecasting.

User Service

Queried by the API Lambda for user profile data needed in insight and employment responses.

Underwriting Service

Queried by the API Lambda during income verification flows and by the balance-alert Lambda to validate eligibility before sending a balance alert notification.

Credit Card Service

Called by the funds-notifier Lambda to trigger credit card funding notifications when an income transaction is detected.

SageMaker

The income-detection-endpoint ML model is invoked by the API Lambda on POST /{user_id}/insights/employment/detect. Classifies individual transactions as income or non-income based on a trained model. Results are cached in the income-detection DynamoDB table.

GrowthBook

Feature flags fetched at Lambda startup. Controls income detection method selection, SageMaker model thresholds, payday prediction behaviour, and transaction batch sizes in the feeder.

Segment

User analytics and push notification delivery. Called by funds-notifier and balance-alert with structured event payloads.

EventBridge (default bus)

Both a producer and a consumer. Produces income_txn (income-signaller) and user_new_insights_available (miner). Consumes new_account and new_bank_account_added events from the Transactions Service, routed via EventBridge rules to the balance-alert and institution-change-handler SQS queues.

Related Pages