Payback Predictor V2

This page documents the payback predictor v2 model integration in pkg/underwriting_model — a separate mechanism from the ML rules documented in Rules Reference (RuleMLPaybackPrediction, RuleMLPaybackPredictionVariableThreshold), which run inside the rule engine as ordinary rulebook rules. Payback predictor v2 instead runs after the rule engine has already produced a decision, as a final step in the API Lambda, and can override that decision in either direction.

Purpose

The model predicts a risk score for a float request and, once fully rolled out for a user, can approve floats the rule engine denied or deny floats the rule engine approved — a second opinion layered on top of the rulebook-driven decision described in Rule Engine.

Where It Runs

Unlike the Rule Runner / Result Runner pipeline, payback predictor v2 is invoked synchronously inside the API Lambda, from a single choke point (Predictor.getEvaluationResult in pkg/api/float_check.go) shared by all five eligibility/loan endpoints:

FloatRequest (POST float draw)
GetEligibilityCheck
LoanEvaluationCheck
GetLoanEligibility
InstantCheck

Each of these calls Predictor.MaybeRunFloatCheck and then underwritingmodel.ApplyOverride (both in pkg/underwriting_model/predictor.go) against the same cached EvaluationResult.ResultID for as long as the Rule Runner/Result Runner haven’t produced a fresh evaluation. Any one of the five endpoints can be the first call that generates and saves a prediction (an MLResult row); any subsequent call for that same ResultID reuses the saved prediction rather than re-invoking SageMaker.

Cross-Service Flow

The mobile client (floatme-flutter) calls underwriting-api’s eligibility endpoints directly to show the user a decision, then calls float-service’s create-float endpoint, which independently calls back into underwriting-api’s `FloatRequest to re-check eligibility server-side before actually creating the float — float-service correctly respects whatever Approved/Result underwriting-api returns (including an authoritative override in either direction) and refuses to create the float if not approved, and note float-service uses the client-requested amount for the float record, not underwriting-api’s Amount field (irrelevant during shadow mode, but a change here would matter to the amount side of an override).

No evaluation/result ID is threaded from the client’s check into float-service’s re-check — each independently resolves "whatever evaluation is currently latest" for the user (getEvaluationResult, pkg/api/float_check.go). In the common case (no recalculation lands in between) both calls hit the same EvaluationResult/MLResult, so float-service’s check is the "reuse" path described below. If a recalculation does land in that window (a Plaid transaction webhook, another event), float-service’s check gets a genuinely fresh evaluation and authoritative is re-checked live for it — independently of what the client’s earlier check saw. This means the client-displayed decision and float-service’s authoritative decision can legitimately disagree across a live rollout-percentage or targeting change in that window; this is an inherent characteristic of there being no idempotency key between the two calls, not a bug.

Two-Stage Rollout: `enabled` vs `authoritative`

Two independent GrowthBook flags gate the model, both evaluated per-user (GetFeatureEnabledForUser), so rollout can be staged and targeted by percentage or user segment:

Flag Effect

Flag	Effect
`underwriting.payback_predictor_v2`	Master kill switch. When off, `MaybeRunFloatCheck` returns immediately — inference never runs at all.
`underwriting.payback_predictor_v2_authoritative`	Shadow-mode gate. When off, the model still runs, saves its prediction, and reports metrics, but `runAndSave` returns `nil` so `ApplyOverride` no-ops and the rule engine’s decision stands untouched. When on, the saved prediction is returned and `ApplyOverride` can flip the decision.

underwriting.payback_predictor_v2

Master kill switch. When off, MaybeRunFloatCheck returns immediately — inference never runs at all.

underwriting.payback_predictor_v2_authoritative

Shadow-mode gate. When off, the model still runs, saves its prediction, and reports metrics, but runAndSave returns nil so ApplyOverride no-ops and the rule engine’s decision stands untouched. When on, the saved prediction is returned and ApplyOverride can flip the decision.

Important: the authoritative flag’s value is snapshotted into MLResult.Authoritative at the moment the prediction is first generated and saved. Every subsequent call that reuses the saved prediction (via GetByEvaluationID) trusts that persisted snapshot rather than re-checking the live flag — this is deliberate: since all five endpoints share the same ResultID, re-checking live would let a later call (e.g. the actual float request) disagree with a decision already shown to the user on an earlier call (e.g. an eligibility check) if the flag’s effective value for that user changed in between, and it keeps the reuse decision consistent with floatcreated.settleMLResult (below), which also keys off the persisted field.

Decision Logic

ApplyOverride (pkg/underwriting_model/predictor.go) compares the model’s risk score against the rank/term threshold resolved when the prediction was generated (see Rank / Term Threshold Table):

Model approves (risk_score >= threshold) and rules denied → override to approve (uses the legacy -1 "approved" signal; the actual dollar amount comes from the float profile).
Model denies (risk_score < threshold) and rules approved → override to deny.
Otherwise (model and rules agree, or v2Result is nil — shadow mode, kill switch off, or no saved prediction) → no-op, rule engine’s decision stands.

Rank / Term Threshold Table

The risk score is compared against a threshold looked up by the user’s rank (1–8, derived from prior_borrow_count) and term (contract_lag in days, bucketed into rungs) — see pkg/underwriting_model/threshold_table.go. The table is GrowthBook-configurable (underwriting.payback_predictor_v2_threshold_table); if it can’t be loaded, parsed, or is missing a row for the user’s rank, a hardcoded defaultThresholdTable is used instead. Changing the number of term rungs or their day cutoffs is a GrowthBook edit, not a code change.

Features

Cold features — extracted from rule outcomes already stored by the model-feature rules registered in pkg/rulerunner/model_feature_rules.go (e.g. RuleFloatRank, RuleTransactionCategoryFeatures, RuleSubscriptionRank). Every cold feature must be present; a missing or unparseable one aborts the evaluation (extractPaybackPredictorV2Features), so a partially-featured request is never sent to SageMaker.
Hot features — computed live at call time from external services (login counts from the SageMaker Feature Store, account age from the user service, contract lag from the insight service, days since last float paid back from RuleRecentFloat’s outcome). Unlike cold features, a hot-feature fetch failure does not abort the evaluation — the field is left `nil (→ NaN for XGBoost) and the model still runs, which means an authoritative override can happen on a partially-featured request. This is intentional (XGBoost handles NaN natively) but isn’t currently visible in the metrics below.
- last_float_paid_back is nil only when prior_borrow_count is nil, i.e. the user has never taken out a float at all. A user who has borrowed before but never completed a repayment instead gets a huge day count computed off Go’s zero time.Time — that’s a distinct signal from never having borrowed, and reproduces a training-time quirk where RuleRecentFloat used to write that zero-time value into recent_float_date for "no completed float" (see `extractLastFloatPaidBackFeatures’s doc comment).

Good Standing Gate

The model is never run for a user who isn’t in good standing — regardless of the prediction, they wouldn’t be approved, so there’s no reason to spend a SageMaker call or produce a score. checkGoodStanding (predictor.go) runs first in runAndSave, right after rule outcomes are fetched and before cold-feature extraction even starts:

Looks for the dedicated model-feature outcome (RuleGoodStandingV2, registered in pkg/rulerunner/model_feature_rules.go alongside the other model-feature rules — same RuleGoodStanding Go function, RuleTypeTxns, matching the RuleType early-advance data (ruleData.EarlyAdvance) is populated for, so this registration computes early-advance features correctly too, same as a real rulebook’s own registration).
If that hasn’t run yet for this user (a brand-new registration — every current rulebook already runs RuleGoodStanding, just under the unsuffixed "good_standing" name), falls back to that existing rulebook-configured outcome instead of blocking the model outright. This is what lets the gate roll out silently: most users already have a "good_standing" outcome from their normal rulebook run, so the fallback succeeds immediately for them, while the dedicated V2 outcome fills in gradually as users get a fresh rule run.
Checks outcome.Loan directly (-1 = good standing, 0 = not, Error = can’t be determined — treated the same as not in good standing, fail closed).

Any of these — not found, errored, or not in good standing — logs the same "payback predictor v2: not running model" warning the other cold-feature/rank gates use, with a reason distinguishing which case it was.

Failure Handling

Every failure path in runAndSave fails closed — returns nil so ApplyOverride no-ops and the rule engine’s decision is what’s used: rule-outcome fetch errors, not being in good standing, missing/unparseable cold features, an errored RuleFloatRank outcome, unresolvable contract_lag, SageMaker inference errors, and MLResult save failures. GrowthBook itself being unreachable defaults every flag lookup to false (fail-safe).

Persistence & Audit (`MLResult`)

Each generated prediction is saved as an MLResult row (pkg/dynamo/ml_result.go) keyed to the EvaluationResult, with a 168-hour TTL. When a float is actually created, floatcreated.Handler.settleMLResult (pkg/floatcreated/handler.go) looks up the MLResult for that evaluation and — only if its persisted Authoritative field is true — records the float_id and strips the TTL so the record is retained permanently for audit. This runs as a separate step after the handler’s normal historical-evaluation save (CreateHistoricalFloatEval); it doesn’t gate or replace that save. Predictions that were never authoritative simply expire after 7 days.

CreateHistoricalFloatEval’s historical-evaluation save is unchanged: it still resolves "whatever evaluation is latest right now" via `GetLatest. settleMLResult’s evaluation ID is resolved separately, though, preferring the float event’s `EvaluationId (set by float-service to the exact ResultID that decided the float) over that latest evaluation’s ResultID — falling back to the latter only when EvaluationId is absent (older events). This matters because settlement runs asynchronously, after the float already exists; using "whatever is latest right now" instead of the evaluation that actually decided the float would settle (or fail to settle) the wrong MLResult whenever a recalculation (e.g. a Plaid webhook, or another concurrently-processing float for the same user) lands in that window. The historical-eval record itself doesn’t carry this same risk in scope here — only MLResult settlement was changed.

Observability

underwriting.payback_predictor_v2.inference — emitted for every saved prediction (emitPaybackPredictorV2Metric), tagged with model_version, risk_score, authoritative, rank, term, threshold, exceeds_threshold.
underwriting.payback_predictor_v2.override — emitted on every authoritative ApplyOverride call (i.e. whenever v2Result is non-nil), tagged with model_pass and rulebook_pass (both true/false) reflecting the model’s and rule engine’s decisions independently, rather than pre-computing an "overridden" tag — group by both in the dashboard to see agreement vs. override, independently of the per-inference metric above.

There is currently no Datadog monitor configured against either metric (see deploy/datadog.tf) — alerting on override rate is left to whoever sets one up during/after rollout.