Plaid Mining

Mining fetches transactions, accounts, balances, liabilities, and KYC results from Plaid and writes them to DynamoDB. It is triggered by Plaid webhooks and by synthetic webhooks from the API and the listener.

system context plaid mining

Pipeline

Stage Component Input Output

Intake

webhook

POST /v1/plaid from Plaid via site-floatme-webhook-api

Message on site-txn-plaid-webhooks

Synthetic intake

api, listener

Item add, Layer creation, remine; Auth0 login

Message on site-txn-plaid-webhooks with a source

Route and fetch

miner

Webhook message

Plaid calls; raw records on site-txn-plaid-transactions; MineRequest on site-txn-plaid-throttle; WEBHOOK_INFO row

Paginate

throttle-miner

MineRequest

Next page on Kinesis; further MineRequest

Refine

refiner

Kinesis records

ACCOUNT, BALANCE, TRANSACTION, REMOVE rows; user_new_txns_batch_completed

Feed

feeder

DynamoDB Streams

new_account; item and Layer-stub cleanup at Plaid

The miner and throttle-miner report failed messages as batch item failures; a message that fails five times lands in its queue’s DLQ. The refiner processes records in order with fmsdk kinesis.ProcessRecords; the first record that fails is reported by its sequence number as the batch item failure, and Lambda bisects the batch and retries from that record. No retry limit or failure destination is configured, so a failed record is retried until it expires from the stream. A record whose failure a retry cannot fix is logged and skipped instead, and its data is not written: a record that does not decode, Plaid data that cannot be converted to rows (nil transaction, bad date, unknown currency), or a DynamoDB ValidationException. Records that fail to refine count toward txn.refiner.record.failed, tagged result:retried (with throttle:true|false) or result:skipped. The listener logs processing errors and does not retry.

Webhook Intake

The webhook Lambda serves POST /v1/plaid (spec: pkg/webhook/spec/plaid.yaml).

  1. Plaid-Verification header missing: 401.

  2. Verify the header as an ES256 JWT against Plaid’s JWK for its kid, and the request body against the token’s body hash. A rejected token or hash mismatch: 401. An error reading the body or running the validation: 500.

  3. Body is not JSON: 422.

  4. Send the payload to site-txn-plaid-webhooks; failure: 500. Success: 200.

Source: pkg/webhook/middleware.go, pkg/webhook/server.go

Miner Routing

For each queue message:

  1. Parse. webhook_type must be TRANSACTIONS, ITEM, AUTH, LIABILITIES, IDENTITY_VERIFICATION, or SCREENING; an ITEM code must be WEBHOOK_UPDATE_ACKNOWLEDGED, LOGIN_REPAIRED, USER_PERMISSION_REVOKED, USER_ACCOUNT_REVOKED, or ERROR. Anything else is dropped without retry. Source: pkg/plaid/webhook.go.

  2. WEBHOOK_UPDATE_ACKNOWLEDGED: done.

  3. IDENTITY_VERIFICATION and SCREENING carry no item id and go straight to KYC webhooks.

  4. Look up the item by id; not found: dropped.

  5. Item is REMOVED and source=FROM_LISTENER: dropped.

  6. Refresh the item from Plaid: item_error_code, status (ERROR when a code is present, else ACTIVE), available and billed products, last successful and failed update times, item_last_webhook_sent_at.

  7. Route by type per the tables below. Every handled webhook writes a WEBHOOK_INFO row.

A user.ErrNotFound from the User Service drops the message; other errors return it for retry.

TRANSACTIONS

Code Window Action

DEFAULT_UPDATE

14 days

Mine; a Plaid ITEM_LOGIN_REQUIRED or NO_ACCOUNTS error ends the mine without retry

HISTORICAL_UPDATE

180 days

Mine

INITIAL_UPDATE

30 days

Mine

TRANSACTIONS_REMOVED

—

Put removed_transactions on Kinesis with last_page=true

Source: pkg/miner/transactions.go

Pagination

A mine calls Plaid for up to 500 transactions at the request’s offset, puts the response on Kinesis with last_page = total - offset ⇐ 500, and, when more remain, sends a MineRequest (item_id, user_id, start_date, end_date, offset + page length) to site-txn-plaid-throttle. The queue’s 30-second delivery delay spaces the pages. The throttle-miner repeats the same step until last_page. The WEBHOOK_INFO row for the webhook records the first page’s totals and accounts.

ITEM

Code Action

LOGIN_REPAIRED

Segment event plaid_item_repaired

ERROR

Segment event plaid_item_errored; when item_error_code is ITEM_LOGIN_REQUIRED, ITEM_LOCKED, PASSWORD_RESET_REQUIRED, or USER_SETUP_REQUIRED, publish plaid_item_connection_lost

USER_PERMISSION_REVOKED

Revoke the token at Plaid; set REMOVED; Admin API note; publish plaid_item_connection_revoked (reason=USER_PERMISSION_REVOKED)

USER_ACCOUNT_REVOKED

Publish plaid_item_connection_revoked (reason=USER_ACCOUNT_REVOKED); the item is unchanged

WEBHOOK_UPDATE_ACKNOWLEDGED

No action

Source: pkg/miner/item.go, pkg/miner/events.go

AUTH

DEFAULT_UPDATE: when account_ids_with_updated_auth includes the item’s main account, call the Payments Service to delete the stored bank account so the next fetch re-reads the numbers from Plaid. Other accounts: no action.

LIABILITIES

DEFAULT_UPDATE: when ENABLE_LIABILITY_MINING is off (prod), return without writing. Otherwise fetch liabilities for the new and updated account ids and store each as a current row plus a history row — Liability.

Listener

Auth0 login events reach site-txn-listener-eventbridge through the rule in EventBridge.

  1. Parse; detail.data.type must be gd_auth_succeed, sepft, or sertft (case-insensitive), else skip. The user id is detail.data.user_id with any auth0| prefix stripped.

  2. Read the user’s items from site-txn-plaid; none: skip. Items whose status is not ACTIVE are logged and skipped; the remaining steps run once per item.

  3. Read the item’s TRANSACTIONS WEBHOOK_INFO rows (GSI2) and decide:

History Action

A DEFAULT_UPDATE, INITIAL_UPDATE, or HISTORICAL_UPDATE within 24 hours

Skip

No HISTORICAL_UPDATE and the item was created less than 30 days ago (the WEBHOOK_INFO retention)

Synthetic HISTORICAL_UPDATE

Anything else, including no rows

Synthetic DEFAULT_UPDATE

Synthetic messages carry source=FROM_LISTENER. Processing errors are logged and the message is not retried.

Source: pkg/listener/listener.go

Refiner

For each Kinesis record:

  1. Parse the wrapper (user_id, item_id, transactions, removed, last_page).

  2. When a transactions response is present:

    • Each account becomes an ACCOUNT row and a BALANCE row (accounts with an unknown currency are skipped).

    • Each transaction becomes a TRANSACTION row, batch-written. Transactions that already have a REMOVE tombstone are re-tombstoned instead, and their ids are sent to site-insight-replay-feeder as {user_id, removed}.

    • last_page with at least one transaction: publish user_new_txns_batch_completed.

  3. When removed is present: write a REMOVE tombstone per id and mark existing TRANSACTION rows as removed — Transaction.

Source: pkg/refiner/refiner.go

Feeder

Stream record Action

TTL expiry of a LAYER row (site-txn-transactions)

For each stub entry with no saved ITEM, revoke the token at Plaid

TTL expiry of an ITEM row (site-txn-plaid)

A prequal item that still has a main_account_id and whose user is ACTIVE in the User Service is written back without ttl and stays connected (txn.item.prequal.expiry, outcome:persisted); a failed user lookup fails the record so it is retried. Otherwise set REMOVED with item_removed_reason for prequal (outcome:removed) or reactivation and revoke the token at Plaid (ITEM_NOT_FOUND tolerated)

Insert or modify of an ACCOUNT row

Build the account, set is_main when it matches an active item’s main account, publish new_account

Anything else

Skipped (and filtered before invocation — Event Source Mappings)

Source: pkg/feeder/feeder.go

Institution Sync

Every Saturday at 00:00 UTC the institution Lambda pages through Plaid’s institution list and overwrites each row in site-txn-plaid-institutions with a full PutItem.

Source: pkg/institution/syncer.go