Collections Engine

The collections engine recovers subscription fees through a scheduled path (three weekday runs: scheduled, retry, pause) and a webhook path that reacts to real-time balance events. GrowthBook flag keys are referenced inline; the flag catalog is GrowthBook Flags.

Overview

collections subsystem
Path Trigger Scope

Scheduled Collection

EventBridge rule, weekdays 08:00 UTC

SCHEDULED subscriptions with billing_date on or before today

Retry Collection

EventBridge rule, weekdays 07:00 UTC

ERROR subscriptions with billing_date within the last month

Pause Collection

EventBridge rule, weekdays 22:00 UTC

PAUSED subscriptions; advances the pause counter, moves no money

Balance-Update Collection

new_account balance events via SQS tap

ERROR subscriptions from the last 2 months for the event’s user

The three scheduled runs share one pipeline: an EventBridge rule invokes the collections-job Lambda with a process field (scheduled / retry / pause); the job pages qualifying subscriptions out of DynamoDB, enqueues each page to the process’s worker queue for the collections-worker Lambda, and re-invokes itself through a paging queue when more pages remain. Queue configuration and schedules are in Infrastructure — SQS Queues and Schedules.

Collection outcomes set the subscription’s billing_status — the status reference is Subscription Lifecycle — Status Machine. ACH results arrive later as settlement callbacks on the site-payments Kinesis stream (ACH Processing).

Every money-moving path first acquires the per-user billing lock (key subscription-billing:user_id:{userID}); a held lock skips the attempt with no status change. Lock schema and semantics: DynamoDB — Locks.

Every attempt that submits a payment also records its decision context (rail, fallback, observed balances, gate states) as data_* attributes on its history row — reference and per-path coverage: DynamoDB — Decision Context.

Scheduled Collection

When the collections-worker dequeues a SCHEDULED subscription, ProcessCollection runs four gates — the first failure ends the attempt:

  1. User is a FloatMe employee (isEmployee) → subscription is WAIVED, next cycle is scheduled, done.

  2. User is not active (user-service) → status CANCELLED, done.

  3. User has a pending cancellation (isPendingCancellation) and the subscription date is after the cancel date → updated_event: PENDING_CANCELLATION.

  4. Billing lock acquired; already locked → skip silently.

checkScheduled then stamps the initial run date, applies any pending downgrade (handlePendingDowngrade — a failure here is returned for SQS retry), and continues:

  1. User on the payments blocklist → status ERROR.

  2. Plaid balance fetched via txn-service; no Plaid account or fetch error → status ERROR.

  3. Balance below subscription_amount → status ERROR, unless the ML decider overrides (ML Decider).

  4. Institution in the pinless pilot (subscriptions.pinless.institutions) → pinless only; ACH is never submitted for these institutions. With a valid debit card → pinless debit: success sets COMPLETED and emits a receipt + AppsFlyer event; decline sets ERROR; a refused submit also sets ERROR (Retry-Limit Blocks). No valid card (or a failed card lookup) → status ERROR with usio_error No valid debit card, no payment attempt and no member comms; the check metric carries reason:no_valid_debit_card.

  5. Otherwise → ACH submit: accepted sets ACHSENT (awaits settlement callback); rejected sets ERROR.

Source: pkg/collections/worker.go, scheduled.go.

Scheduled and retry collection decision flow

Whatever the outcome, a deferred step writes the next cycle’s SCHEDULED record, priced from the membership tier version in global.tiers.config (falling back to the previous amount — Renewal) and repriced or held by subscriptions.collection.increase.temp when a price override targets the renewal (Price Override).

Retry Collection

Candidates are ERROR subscriptions whose billing_date falls within the last month — the job queries with CollectionDate = now − 1 month and the page query filters billing_date > CollectionDate (GetRetrySubscriptionPage, pkg/dynamo/subscriptions/subscriptions.go).

checkRetry gates, in order — the first failure skips the subscription with no status change (each skip emits the check metric with a reason: tag):

  1. ACH attempts in the subscription’s history at or above ACHRetryLimit (2, pkg/collections/worker.go) → skip. A history-read error also skips.

  2. User on the payments blocklist → skip.

  3. Plaid balance below subscription_amount (or balance fetch failed) → skip.

  4. Institution in subscriptions.pinless.institutions (pinless-only) → skip; these institutions are collected by the webhook path instead.

  5. Otherwise → ACH submit: accepted sets ACHSENT; rejected sets ERROR.

Source: pkg/collections/retry.go.

Pause Collection

When the collections-worker dequeues a PAUSED subscription, handlePaused:

  1. Finalizes any pending tier downgrade (user-service FinalizeDowngradeMembershipTier).

  2. Marks the current record PAUSED_SKIPPED (UpdatedEvent = PauseSkipped) as the audit record for the skipped cycle.

  3. Writes the next record with pause_duration_months decremented by 1. An indefinite pause (duration 0) stays indefinite; when the countdown reaches 0 the next record is written as SCHEDULED with UpdatedEvent = PausePendingResume.

When a record carrying PausePendingResume is later processed, the worker transitions it to active collection via UpdateEventTypePauseResume, which has user-service remove the pause tag from the account.

Source: pkg/collections/worker.go.

Balance-Update Collection

The webhook-balance-worker Lambda consumes new_account events (source txn-service.feeder, primary account only) whose available, current, or calc_available balance is ≥ 0, via the balance-event SQS tap (infrastructure.adoc#queues; Events for the contract).

Per event, the worker:

  1. Acquires the billing lock; a held lock returns an error so SQS redelivers.

  2. Populates collection state from the event and lookups: balances, institution ID, user status, ERROR subscriptions from the last 2 months, per-subscription ACH history, debit-card validity, blocklist status.

  3. Returns with no action when the user is inactive or has no ERROR subscriptions in the window.

  4. For each collectable subscription, determines the method, then evaluates the rule chain in order; the first failing rule skips that subscription with a reason:-tagged check metric.

When subscriptions.webhook.balance.prefer_ach is on for the user, the method is determined by evaluating ACH first — ruleUserIsActive, ruleNotBlocklisted, ruleAchBalanceCheck, ruleACHLimitNotExceeded — and is ACH when all pass. When any fail, the attempt is marked as a fallback (data_is_fallback on the record) and the method is left to the regular chain below, exactly as when the flag is off:

  • ruleUserIsActive

  • ruleDefaultPinlessIfNotSet

  • ruleAchIfInvalidDebitCard

  • rulePinlessIfBlocklisted

  • ruleHasValidDebitCard

  • rulePinlessBalanceCheck — threshold from subscriptions.webhook.balance.threshold (standard tier) or .threshold_premium (premium tier); compares calc_available instead of available when subscriptions.webhook.balance.use_calculated_balance is on

  • ruleAchBalanceCheck — threshold from subscriptions.webhook.balance.ach_threshold

  • ruleACHLimitNotExceeded

Source: pkg/collections/webhook_balance_worker.go.

Balance-update collection decision flow

A subscription that passes is attempted per the rules' routing: pinless success sets COMPLETED and sends the debit receipt; an accepted ACH sets ACHSENT; a failed attempt — including a refused pinless submit (Retry-Limit Blocks) — stops processing of the remaining subscriptions. ACH and prenotes route to USIO or JPM per subscriptions.payments.provider. When multiple subscriptions are attempted, the worker sleeps 11 seconds between submissions (usioTimeToWait).

Retry-Limit Blocks

The Payments Service blocks some pinless submits before they reach the processor, returning HTTP 200 with status: FAILED, an empty confirmation_id, and a block_reason — RETRY_LIMIT_EXCEEDED, CARD_INVALID, or CARD_SUSPENDED. Any block_reason value marks a block, including one outside that set.

The scheduled worker (pkg/collections/worker.go), the balance-update collector (pkg/collections/collector.go), and the batch worker (pkg/collections/batch-worker.go) each handle a block as a skip rather than a decline:

  • No member comms — neither the Iterable user_subscription_payment_failed nor the AppsFlyer ad_subscription_payment_failed event a decline sends.

  • billing_status: ERROR with the block reason in usio_error and transaction_id left empty.

  • subscriptions.collections.attempt tagged status:blocked instead of status:failed — the scheduled path adds block_reason:, the balance-update path carries the reason in its error: tag. The batch path emits subscriptions.collections.batch.debit.blocked with block_reason:.

On the balance-update path a block stops the remaining subscriptions for that event, as a decline does.

Member-initiated payments set bypass_retry_limit and are never blocked — Manual Pay and Reactivation.

ML Decider

pkg/mldecider calls a SageMaker binary classifier that predicts whether a scheduled collection attempt will succeed; it runs only in the scheduled path. Predictions are always logged, and the prediction may override a failing balance check only when subscriptions.initial.override_balance_with_ml is on. The model result must meet the subscriptions.initial.ml_threshold confidence to count as true. When no 7-day balance history exists (ErrNoDataForML), the decider returns true and collection proceeds on the normal balance gate alone.

Feature assembly and the SageMaker request are in Source: pkg/mldecider/MLDecider.go.