Payback Predictor V2

An ML risk model (pkg/underwriting_model) that runs after the rule engine has produced a decision and, when authoritative for the user, can override it in either direction. It is separate from the in-rulebook ML rules (Rules Reference). The data/training side lives in ML Feature Pipeline. The risk score also drives the Float Tiers assignment, computed alongside every prediction.

Where it runs

Synchronously in the API Lambda, from a single choke point (pkg/floatdecision) shared by all five eligibility/loan endpoints: FloatRequest, GetEligibilityCheck, LoanEvaluationCheck, GetLoanEligibility, InstantCheck. It calls Predictor.RunAndSave against the current EvaluationResult.ResultID and applies the result (pkg/floatdecision/model.go). Whichever endpoint runs first generates and saves the prediction (MLResult); later calls for the same ResultID reuse it on the same UTC day (same-day reuse).

The model is wired only when PAYBACK_PREDICTOR_ENABLED is true (Infrastructure). Otherwise pkg/floatdecision runs with no predictor: every decision is the rule engine’s, no MLResult is saved, and Float Tiers enforcement is treated as off regardless of its flag.

Float creation is a separate re-check: float-service calls back into FloatRequest before creating a float and honors the returned decision. No evaluation ID is threaded between the client’s check and float-service’s re-check — each resolves the latest evaluation independently, so the two can disagree when a recalculation or a rollout change lands between them (float-service also uses the client-requested amount for the float record, not this service’s Amount field).

Rollout flags

All per-user (Feature Flags):

Flag Effect

underwriting.payback_predictor_v2_authoritative

Shadow-mode gate for ranks 1–3. Off: the model runs, saves its prediction, and reports metrics, but the decision is not overridden. On: the prediction can flip the decision.

underwriting.payback_predictor_v2_authoritative_high_rank

Same gate for rank 4 and above.

Exactly one authoritative flag is consulted per prediction, chosen by rank (authoritativeFlagForRank) — the same rank the threshold lookup uses. The flag consulted is logged as authoritative_flag; the Datadog authoritative tag records only the resolved boolean, so split by the rank tag to attribute a rate to a cohort.

The applied flag value is snapshotted into MLResult.Authoritative when the prediction is saved; every reuse of the saved prediction trusts the snapshot rather than re-checking the live flag, keeping all five endpoints and floatcreated.settleMLResult consistent with the decision already shown to the user.

Decision logic

pkg/floatdecision first applies an enforced Float Tiers assignment when the MLResult carries one — it becomes the float decision outright (approval, amount, status, deciding rulebook) and the model’s own override is not applied. Otherwise the model branch (model.go) compares the saved risk score with the saved threshold:

  • Model approves (risk_score >= threshold), rules denied → override to approve (result -1; the dollar amount comes from the float profile).

  • Model denies (risk_score < threshold), rules approved → override to deny.

  • Otherwise (agreement, shadow mode, or no saved prediction) → no-op.

Threshold table

The threshold is looked up by rank (1–8, from prior_borrow_count) and term (contract_lag days, bucketed into rungs) — pkg/underwriting_model/threshold_table.go. The table is GrowthBook-configurable (underwriting.payback_predictor_v2_threshold_table); on load/parse failure or a missing rank row, the hardcoded defaultThresholdTable applies. Rung counts and day cutoffs are GrowthBook edits, not code changes.

Features

  • Cold features — read from the rule outcomes stored by the model-feature rules. Every cold feature must be present; a missing or unparseable one aborts the evaluation (extractPaybackPredictorV2Features) — a partially cold-featured request is never sent.

  • Hot features — computed at call time (login counts from the SageMaker feature store, account age from user-service, contract_lag from insight-service, days since last float paid back from RuleRecentFloat’s outcome). A hot-feature failure leaves the field `nil (NaN to XGBoost) and the model still runs — an authoritative override can happen on a partially hot-featured request.

  • last_float_paid_back is nil only for users who have never borrowed. A user with floats but no completed repayment gets a large day count computed from Go’s zero time.Time — a distinct signal matching the training data (extractLastFloatPaidBackFeatures).

Good-standing gate

checkGoodStanding runs before feature extraction: it reads the RuleGoodStandingV2 model-feature outcome, falling back to the rulebook-configured good_standing outcome when the dedicated one hasn’t run yet for the user. Not found, errored, or not in good standing all skip the model (fail closed) with a "payback predictor v2: not running model" log and a reason.

Failure handling

Every failure path in RunAndSave returns an error and no MLResult: outcome-fetch errors, bad standing, missing cold features, an errored RuleFloatRank outcome, unresolvable contract_lag, SageMaker errors, and save failures. Bad standing and an errored standing outcome wrap ErrNotInGoodStanding. For a user off float tiers no override is applied and the rule engine’s decision stands; a float-tiers user is handled by Float Tiers. GrowthBook being unreachable resolves every flag to false.

Persistence and settlement

Each prediction is saved as an MLResult keyed to the evaluation, TTL 168h, carrying the Float Tiers snapshot. When a float is created, floatcreated.Handler.settleMLResult records the float_id and strips the TTL — only when the persisted Authoritative is true or the float-tier assignment’s authoritative_v2 is set; other predictions expire. The handler also copies the float-tier snapshot onto the HistoricalEvaluation. Settlement resolves the evaluation from the float event’s EvaluationId (the exact ResultID that decided the float), falling back to the latest evaluation only for events without it — settlement runs asynchronously, after recalculations may have landed.

Reason codes

The endpoint returns reasons with each score: up to three codes naming the feature families whose summed TreeSHAP contributions pulled the score down the most, most negative first, relative to the model’s average user. Every model feature belongs to exactly one code (REASON_CODE_FEATURES in machine-learning’s `underwriting_v2/constants/constants.py), and NO_FLOATME_HISTORY replaces REPAYMENT_HISTORY for a user with no prior borrows. This service doesn’t interpret them and they don’t affect the decision: RunAndSave saves them on the MLResult as returned, and they are exposed read-only as model_result.reasons.

Observability

  • underwriting.payback_predictor_v2.inference — per saved prediction; tags model_version, risk_score, authoritative, rank, term, threshold, exceeds_threshold.

  • underwriting.payback_predictor_v2.override — per authoritative model decision (pkg/floatdecision/model.go); tags model_pass, rulebook_pass — group by both to separate agreement from override.

  • underwriting.payback_predictor_v2.float_created — on settlement.

No Datadog monitor is configured against these metrics (deploy/datadog.tf). Source: pkg/datadog/metrics.go