Pave Mining

Overview

The Pave pipeline uploads a user’s Plaid transactions into the Pave third-party analytics platform. Pave’s computed insights are no longer retrieved or cached; the only remaining read of Pave is the live recurring_income call on the payday-prediction path. The pipeline is split across four Lambdas:

Lambda Trigger Role

prod-insight-feeder

Kinesis (prod-txn-plaid-transactions, LATEST)

Uploads transactions and balances to Pave for users on an eligible pay frequency; tracks per-user pagination state

prod-insight-webhook

API Gateway (Pave webhook, HMAC-verified)

Receives Pave callbacks; enqueues mining jobs on USER_DATA_INSIGHTS_READY

prod-insight-miner

SQS (prod-insight-miner)

Emits user_new_insights_available on the last page of an insight run

prod-insight-replay-feeder

SQS (prod-insight-replay-feeder)

Forwards deferred transaction deletions to Pave; no downstream events

The feeder and miner are intentionally decoupled. The feeder pushes data to Pave asynchronously; Pave does its own processing and signals completion via a webhook callback. The webhook Lambda bridges these two phases by enqueuing a mining job.

pave mining sequence

Feeder

Lambda: prod-insight-feeder
Trigger: Kinesis stream prod-txn-plaid-transactions (LATEST position, batched)

The feeder runs on every Plaid transaction event from the Transactions Service. Its job is to upload the user’s current transactions and balances to Pave and record that the upload happened.

Pay-Frequency Gate

Only users whose pay frequency Pave predicts better than the FloatMe algorithms are sent to Pave. Every other user is dropped before any Pave call is made.

The allow list is the insight.pave.upload.pay_frequencies.config GrowthBook flag, a comma-separated list of pay frequencies. An empty value — the default in code — means no user is eligible and acts as the kill switch for the whole integration.

The feeder reads the user’s pay_frequency from the monolith’s Postgres (FloatMeAPI_employmentinfomodel), which is why it runs in the private subnets with RDS credentials. Matching folds both sides to lowercase, since the column holds the same frequency in several casings (MONTHLY, Monthly). A user is skipped when:

  • their pay frequency is not in the allow list;

  • they have no employment record yet — common for a freshly linked bank, since transactions arrive before the user enters employment details;

  • the Postgres lookup fails, which is logged rather than returned so a struggling database does not cause the whole Kinesis batch to be re-driven.

The gate covers uploads, deletions, and the pave_label write alike, and applies identically in the replay feeder.

insight.pave.upload.gated counts each decision, tagged with the normalized pay_frequency and eligible, so the eligible share of the population is visible before and after the allow list changes.

Becoming Eligible

Because ineligible users are dropped, a user who later moves onto an eligible pay frequency has no history in Pave at all, and their payday prediction would silently stay on the FloatMe algorithm.

CreateEmployment and UpdateEmployment therefore compare the pay frequency being saved against the one already on file, and on a transition from ineligible to eligible ask the Transactions Service to re-mine each active item (POST /{user_id}/transactions/items/{item_id}/remine). That refetches from Plaid and republishes onto the Kinesis stream the feeder consumes, which now passes the gate.

The transition check matters: an edit that stays within eligible frequencies — correcting an employer name, say — must not trigger a Plaid refetch, since the user’s transactions are already in Pave. Re-mine failures are logged and never surfaced, as they only degrade a prediction, whereas failing the request would lose the user’s input. insight.pave.remine.triggered counts each re-mine.

Pagination State

Pave returns insights in pages. The feeder tracks two DynamoDB entities per user:

Entity Purpose

pave_label

Marks whether the user’s Plaid data has ever been labeled in Pave. Written once on first upload; used to avoid re-labeling.

last_page

Records the latest pagination request ID for a user. The miner checks this entity to decide whether all pages for a given insight run have been processed before emitting the user_new_insights_available event.

Upload Logic

For each Kinesis event, the feeder:

  1. Collapses the records into one batch per user, so a user’s transactions reach Pave in a single upload rather than one per record.

  2. Drops every user who fails the pay-frequency gate above.

  3. Uploads the transaction batch and account balances to the Pave API, and forwards any removed transactions as deletions.

  4. Saves the last_page entity with the request ID returned by Pave, when the batch contained the last page.

  5. Checks and sets the pave_label entity if this is the user’s first upload.

Webhook

Lambda: prod-insight-webhook
Trigger: API Gateway (HMAC signature-verified endpoint)

The webhook Lambda receives all callbacks Pave sends to the FloatMe webhook URL. Each request is validated via HMAC signature before any processing occurs; requests with invalid signatures are rejected with a 401.

Supported Event Types

Pave Event Type Behaviour

USER_DATA_INSIGHTS_READY

Insights are ready for this user. Enqueues a mining job (containing user_id and request_id from the webhook metadata) to the prod-insight-miner SQS queue.

USER_DATA_UPLOAD_SUCCESS

Acknowledged and logged. No downstream action taken.

USER_DATA_DELETE_SUCCESS

Acknowledged and logged. No downstream action taken.

Any other type

Logged as a warning. No downstream action taken.

Non-200 status fields in the Pave webhook payload (indicating Pave-side processing errors) are also logged as warnings without enqueuing a mining job.

Miner SQS Message Shape

The SQS message body enqueued for each USER_DATA_INSIGHTS_READY event:

{
  "user_id": "<floatme_user_id>",
  "request_id": "<pave_request_id_from_webhook_metadata>"
}

The request_id is carried through from the Pave webhook metadata and used by the miner to correlate with the last_page entity in DynamoDB when deciding whether to emit the user_new_insights_available event.

Miner

Lambda: prod-insight-miner
Trigger: SQS (prod-insight-miner, batch processing)

The miner turns a Pave "insights ready" webhook into the user_new_insights_available EventBridge event. That is all it does.

What It No Longer Does

It fetches nothing from Pave and writes nothing to DynamoDB. It previously cached recurring, ritual, income, and scores from Pave on every webhook; nothing needs those refreshed, since GET /{user_id}/forecasts is now served entirely from FloatMe recurring detection.

The only readers those entities have left are the deprecated GET /{user_id}/transactions/expenses and GET /{user_id}/scores/cash_advance endpoints. Their data is no longer refreshed and ages out via its TTLs, after which both endpoints return empty results. Both are documented as deprecated with no replacement — see API spec.

The monthly bank-account balance snapshot is also no longer the miner’s job; it is written by prod-insight-savings-snapshot from transaction activity instead. See Architecture.

For each SQS message the miner reads the last_page entity to decide whether this webhook completed the user’s insight run, and emits the event if so.

user_new_insights_available Emission

The miner only emits the user_new_insights_available EventBridge event when it determines it has processed the last page of a user’s insight run. This avoids notifying downstream services (e.g., the Float Service) multiple times per insight cycle when Pave returns results across multiple pages.

The check is done by reading the last_page DynamoDB entity for the {user_id, request_id} pair. If is_last_page = true, the event is emitted.

EventBridge Field Value

Bus

default

Source

insight-service.miner

Detail-Type

user_new_insights_available

Payload

 
{
  "user_id": "<floatme_user_id>",
  "last_page": true
}

Failures to emit the event are logged as errors but do not cause the SQS record to be retried — this prevents insight data from being rewritten unnecessarily when only the event emission failed.

Error Handling

The miner processes SQS messages in batch mode and returns per-message failure responses (SQSBatchItemFailure). A message is added to the failure list only when its body cannot be unmarshalled. Everything else is best-effort: with no data writes left, re-driving a record cannot recover anything, so last_page lookup and event-emission failures are logged without adding to the batch failure list.

Failed messages are retried up to the queue’s configured maximum receive count before being routed to the dead-letter queue (prod-insight-miner-dlq).

Replay Feeder

Lambda: prod-insight-replay-feeder
Trigger: SQS (prod-insight-replay-feeder)

The replay feeder handles deletions that arrived before the transaction they refer to had been mined. The Transactions Service retries those for a while and, once the transaction exists, places it on this queue so Pave can be told to drop it.

Each SQS message carries a user ID and a list of removed transaction IDs. The replay feeder issues DeleteTransactions against Pave and nothing else — it writes to no DynamoDB table, does not enqueue to the miner SQS, and emits no EventBridge events. The same pay-frequency gate as the main feeder applies, so deletions are only forwarded for users whose data is in Pave to begin with.

Replay SQS message
  └─▶ prod-insight-replay-feeder
        └─▶ Pave API (delete transactions)

Related Pages

  • Architecture — System context and full Lambda inventory

  • Event Flows — user_new_insights_available event routing and downstream consumers

  • DynamoDB Tables — recurring, ritual, income, scores, pave_label, last_page entity schemas