Collections Engine

ACH settlement callbacks for all paths arrive on the prod-payments Kinesis stream and are processed by the ACH Handler — see ACH Processing.

Attempts update float status (status reference); declined runs write an outcome-only collection-history row and leave status unchanged (Skip outcome codes). A pinless submit the payments service blocks before it reaches the processor is a skip as well, not a decline (Blocked pinless submits).

All collections-jobs subcommands run through one shared loop (pkg/collections/jobs/run.go). Each job implements the Processor interface (JobTag, RunTitle, ListCandidates, ProcessFloat); RunJob owns the rest: job started/completed/duration metrics, the worker pool and shared rate limiter, run counters, Slack reporting (one generic reporter for every job, Slack run report), and the floats.collections.error per-float hard-error metric. A run whose candidate query returns zero floats completes as a clean no-op (metrics only, no Slack message). Run dates come from the shared SettlementDate helper (midnight-truncated, weekends rolled to Monday).

Candidates load up front as refs — one query returning (loan_id, user_id) pairs (rds.FloatRef, ~120 bytes each; a 500k-float run holds ~60 MB against the task’s 1 GiB) — and each worker fetches the full float only after acquiring the per-user billing lock. That post-lock fetch is the single authoritative read — stale candidate state cannot cause a double-collection; a failed fetch is a fetch_current_float_failed skip picked up on the next run. There is no retry pass: a billing-lock failure (or any other per-float error) counts as a hard error and the float waits for the next scheduled run. After any accepted ACH the job triggers an underwriting recalculation for the user (failures logged, never propagated).

Every collection calendar decision — which day a run belongs to, which weekday it is, and how many days past due a float is — is anchored on America/New_York, and every collections-jobs schedule is pinned to the same zone. A collection attempt at 9pm ET therefore belongs to that ET day, not to the UTC day that has already begun.

Two paths deliberately use a different zone:

Collection-history rows are unaffected: run_date and runtime are written in UTC as a stable data-interchange contract. Anything that needs the business day derives it from runtime, which is an absolute stamp.

Source: pkg/collections/shared.go (Eastern, BusinessDay, SameBusinessDay).

collections.ACHPolicy decides whether an overdue float still gets an ACH and in which settlement window. The Balance update webhook path applies it to the ACH that follows an unsuccessful pinless attempt; the batch jobs do not use it. Given a float’s due date and the clock, it answers whether an ACH should go out and in which settlement window. Windows are local time in America/Chicago (DST-aware; tzdata is embedded in the binaries) on banking days — ACH does not settle at the weekend, so Saturday and Sunday never submit one:

Ahead of the windows the policy checks the float’s age: more than floats.collections.ach.days_past_due days (default 28) past its due date and no ACH is submitted whatever the hour — collection rates fall off sharply after that while a late ACH still risks a return fee. Age is counted in whole calendar days on the business day; the limit day itself still submits.

A withheld ACH records outcome:ach_skipped_past_due or outcome:ach_skipped_outside_window on float.collections.payment; the pinless result that preceded it stands. Per-float gates that are not about timing — the payments blocklist, the NACHA cap, pinless-only institutions — stay with the caller and are applied after the policy.

Source: pkg/collections/ach_policy.go.

The day before a float’s due date, the day-before-ach Fargate job (pkg/collections/jobs) runs at 19:00 ET — approximately 2.5 hours before the Usio next-day ACH cutoff (20:30 America/Chicago = 21:30 ET, see ACH submission policy). The job loads SCHEDULING floats due the next business day (Friday targets Monday) from the RDS replica (Shared batch loop (RunJob)); workers process each float under the per-user billing lock and run six rules in order, first route wins:

Ranks 2 through 6 are owned by neither rule: the float falls through to the due-date pinless run.

Source: pkg/collections/jobs/day_before_ach.go, rules.go.

If no rule routes the float, it stays in SCHEDULING for the due-date pinless run. An accepted ACH marks the float ACHSENT; a failed submission leaves it in SCHEDULING with the attempt logged (process TOMORROW, or the first/returning-float tag), and the due-date run still fires.

Each weekday at 06:00 ET — one hour after the daily retry — the initial Fargate job (pkg/collections/jobs/initial.go, Shared batch loop (RunJob)) loads SCHEDULING floats whose due date is today or earlier and gives each its first attempt. Status writes and collection-history rows carry process TODAY6AM.

Pinless debit leads when the user has a valid primary debit card; ACH is submitted directly when there is no valid card, and as a fallback when the pinless attempt fails with an NSF code (62 or 05) or is blocked before submission. Outcomes: pinless success → COMPLETED; accepted ACH → ACHSENT; anything else → RETRY, moving the float into the Daily Retry (RETRY) pool. Just before each submit the run reads the primary account’s balances from txn-service for the row’s balance snapshot; nothing is gated on them.

The payments blocklist is the only ACH gate on this run — pinless is never gated. The NACHA cap is not applied: a SCHEDULING float has not been collected yet, so it holds no automated ACHSENT row to count against the cap. This job never writes DEFAULTED.

A blocklisted user with a valid debit card still gets the pinless attempt; only the ACH route is withheld, and a withheld pinless→ACH fallback records the pinless decline (or the block’s skip outcome) as the attempt’s outcome.

A blocklisted user with no valid debit card has no route left, so the float is moved to RETRY and handed to the daily retry, which owns the 90-day DEFAULTED rule. No collection-history row is written — nothing was attempted. Leaving it in SCHEDULING would strand it: only this job selects SCHEDULING, and SCHEDULING is outside the overdue set the retry pool draws from.

A float skipped for a transient reason — a debit-card fetch error, or a blocklist lookup failure, neither of which establishes the user’s status — stays in SCHEDULING and is a candidate again the next morning.

The task definition passes --workers=18 (deploy/fargate_collections_jobs_initial.tf), sized the same way as the daily retry.

Overdue floats (RETRY, FAILED, UNCOLLECTABLE, ACHFAILED) due today or earlier get one retry pass each weekday morning from the daily-retry Fargate job (pkg/collections/jobs/daily_retry.go, 05:00 ET, Shared batch loop (RunJob)). Status writes and collection-history rows carry process RETRY (RETRY_EVENING for the evening run).

The run evaluates exit conditions in strict precedence — first match wins — then attempts collection with the same pinless-then-ACH routing as the due-date run:

ACH routes are additionally gated — pinless never is. Two ACH-only gates apply: the payments blocklist and the NACHA cap (floats.collections.max_ach_attempts, default 3, automated ACHSENT rows only). A gated ACH is silently withheld — no row, no status change; when the pinless→ACH fallback is withheld, the pinless decline (or the block’s skip outcome) is recorded as the attempt’s outcome.

The task definition passes --workers=18 (deploy/fargate_collections_jobs_daily_retry.tf). The submit rate is the Usio processor cap, not a tuning knob — workers are sized to keep that cap saturated across the submit round trip, and cannot breach it. At the default of 5 workers this run takes roughly 10 hours on Thursday and Friday, the blind-debit days when the most attempts fire.

The webhook-worker Lambda consumes multiple EventBridge-tapped triggers from the Transactions Service — one SQS tap/DLQ/rule per trigger — and dispatches per record by the envelope’s detail-type: new_account → the balance path; plaid_item_connection_lost / plaid_item_connection_revoked → the Plaid-disconnected path.

Both paths take the user’s billing lock once, then walk every overdue float the user has (RETRY/FAILED/ACHFAILED/UNCOLLECTABLE), oldest due date first. The event moves on to the next float after a pinless COMPLETED or an accepted ACH, and ends at the first float it does not collect from — a failed gate, a submit error, or a pinless miss (no usable card, a decline, a blocked submit) with no ACH to fall back on — leaving the remaining floats untouched. Every row a float after the first writes carries reason SUBSEQUENTFLOAT. User, debit card, blocklist status and pinless-only institutions are read once per event; collection history, fee and the ACH decision are per float.

The payments service can refuse a pinless debit before it reaches the processor: its rolling-window retry limiter (block_reason RETRY_LIMIT_EXCEEDED, transient — clears as the windows roll or on any approval) or its card gate (CARD_INVALID, CARD_SUSPENDED, until the member provides another card). The response is FAILED with a block_reason and no confirmation ID: no network call was made and no payment record exists. Only pinless submits can be blocked; ACH is unaffected, and member-initiated paybacks bypass the limiter (retry-limit bypass).

Every automated flow branches on block_reason (collections.BlockReason), never on error_message, and treats a block as a skip:

CARD_INVALID and CARD_SUSPENDED need no float-side state: the payments service has already invalidated the card, so the existing card-validity checks route the next run to ACH where allowed.

Source: pkg/collections/shared.go (BlockReason, BlockedOutcome, BlockedError, ShouldRetryAsACH), pkg/collections/jobs/base.go.

Prenotes are zero-dollar ACH verification transactions submitted 5 calendar days before a float’s due date, giving the processor time to surface returns (e.g. closed accounts) before real debits go out. The prenote path is decoupled from collection: it does not change float status, acquire the billing lock, or write collection history. Eligibility (an ACH-routable bank account) is enforced by the Payments Service at submission.

Every collection path acquires a per-user lock (loan-processing:user_id:{userID}, 60-second lease) in the locks DynamoDB table before processing; when the lock is held, the attempt is skipped without a status update. The webhook paths hold it across every float they attempt for the event. The prenote job acquires no lock — prenotes are zero-dollar and touch no float state. Schema: dynamo.adoc#locks.