Pave Mining
The Pave pipeline uploads a user’s Plaid transactions and balances into Pave and turns Pave’s completion webhook into the user_new_insights_available event. Pave’s recurring income sources are read live during payday prediction; nothing else in the service reads from Pave.
Pipeline
| Lambda | Trigger | Role |
|---|---|---|
|
Kinesis |
Uploads transactions and balances for gate-eligible users; records |
|
Shared webhooks gateway, |
Verifies the HMAC signature; enqueues a miner job on |
|
SQS |
Publishes |
|
SQS |
Forwards deferred transaction deletions to Pave |
Pay-Frequency Gate
The gate decides, per user, whether any data goes to Pave and whether Pave is consulted for payday. The feeder, the replay feeder, the payday predictor, and the employment re-mine check all evaluate the same conditions, in order:
| Condition | Result | pay_frequency tag |
|---|---|---|
|
Eligible (no Postgres lookup) |
|
|
Ineligible (kill switch) |
|
No employment record, or the Postgres read fails |
Ineligible |
|
Normalized |
Eligible |
the value |
Otherwise |
Ineligible |
the value |
Source: pkg/feeder/feeder.go (sendToPave), pkg/employment/pave_eligibility.go
Normalization lowercases and trims both sides; matching is on the literal value, not a parsed cadence. Each decision emits insight.pave.upload.gated tagged pay_frequency and eligible. An ineligible user gets no upload, no deletion, and no pave_label. A user who becomes eligible through an employment edit is backfilled by the re-mine check.
Feeder
Each Kinesis record is one page of a user’s Plaid sync: user_id, transactions (a Plaid TransactionsGetResponse), removed transaction IDs, and last_page. Per invocation the feeder:
-
Collapses records into one batch per user (transactions, accounts, removals;
last_pageis true if any record set it). -
Drops users the Pay-Frequency Gate rejects and returns early if none remain.
-
Per remaining user, concurrently: uploads transactions plus the most recently updated balance per account (accounts whose available and current balances are both null are dropped); forwards
removedIDs as a Pave delete; whenlast_pageis set and the upload returned a request ID, writes thelast_pageentity (USERS#<user_id>/LAST_PAGE#REQUEST_ID#<request_id>, 5-day TTL). -
Per user, if no
pave_labelentity exists, uploads theREFRESH_INSIGHTSlabel to Pave and writes the entity.
Pave 4xx, 500, and 502 responses are logged and swallowed; any other error fails the batch, which Kinesis re-drives.
Webhook
The HMAC middleware runs before the handler. Pave-Signature is t=<unix>,sig=<hex>; a missing or malformed header returns 403, a signature mismatch or a timestamp more than WEBHOOK_TTL seconds from now returns 401. The payload schema is in spec/webhook.yaml.
| Webhook | Handling |
|---|---|
|
Logged; |
|
Logged; |
|
Enqueues |
Source: pkg/webhook/webhook.go, pkg/middlewares/webhook.go
Miner
For each SQS message the miner decodes {user_id, request_id}, reads the last_page entity for that pair, and, when it exists, publishes user_new_insights_available to the default bus — see Events. It reads nothing from Pave and writes nothing to DynamoDB.
Only a decode failure is reported as a batch item failure; last_page lookup and publish errors are logged and the message is consumed. Nothing deduplicates: site-insight-miner is a standard SQS queue, and a redelivered message republishes the event for the same request_id. Metrics: insight.pave.insights.mined, insight.pave.insights.mined.failures.
Replay Feeder
The Transactions Service places {user_id, removed[]} messages on site-insight-replay-feeder for deletions it could not apply when they first arrived. The replay feeder batches by user, applies the Pay-Frequency Gate, and calls Pave DeleteTransactions for each eligible user. It writes no DynamoDB rows, enqueues nothing, and publishes no events. Any Pave error fails the batch.