ML Feature Pipeline

Every model-feature field in PaybackPredictorV2Request (pkg/underwriting_model/predictor.go) comes from one of three places:

Cold features must all be present or the evaluation aborts; hot features degrade to nil → NaN — failure modes in Payback Predictor V2.

Several features are day-relative ("days since X", "count in the last N days"), computed at inference time; the training SQL computes them relative to the float’s creation time, so a prediction reused on a later day stops matching what the model learned. MLResult records InvokedAt, and Predictor.RunAndSave reuses a cached prediction only when InvokedAt’s UTC calendar date matches today’s (`MLResult.IsFresh) — otherwise it re-invokes SageMaker with fresh features. The check is same-day, not same-evaluation: calls hours apart on one UTC day still reuse the prediction. Profile does not apply this rule: it never runs the model, so it shapes from the latest evaluation’s saved snapshot until its TTL expires.

Go computes every feature value and POSTs flat JSON to the SageMaker endpoint. machine-learning’s `underwriting_v2_service.py builds the model row by looking up each name in constants.py’s `MODEL_FEATURES list (106 entries, order-sensitive for the XGBoost Booster); names not in the list fall out silently. Go’s request carries 119 named fields — the 13 extras (max_services_accounting, transactions_food_groceries, and 11 std_* category fields) are declared in the Pydantic model, so they are accepted and ignored. When a new RuleTransactionCategoryFeatures field doesn’t move the risk score, check MODEL_FEATURES first. user_id travels alongside the payload for auditing and is not a model input.

underwriting_v2/model_trainer.py trains an XGBoost Booster via the native API (binary:logistic, auc, tree_method: hist, early stopping up to 10,000 rounds). The label repayment_target is 1 when due_repay_lag ⇐ 7 days. Missing values stay NaN — the same convention Go’s nullable pointer fields produce at inference time.

Terraform in machine-learning/deploy/ provisions the endpoint underwriting-v2-inference-endpoint-site (matched by this repo’s PAYBACK_PREDICTOR_ENDPOINT_NAME) with a content-addressed endpoint config that rotates when the serving image or model artifact changes. Model artifacts resolve from a literal S3 URI; the use_mlflow_for_underwriting_v2 gate is off, so MLflow promotion does not apply.

machine-learning/deploy/feature_store.tf provisions the feature group site-machine-learning-underwriting-v2-user-features (user_id identifier), holding exactly app_opens_count_30d and app_opens_count_90d — read at call time by extractLoginCountFeatures (pkg/underwriting_model/hot_features.go).

The group is populated by a daily Glue job in the data-engineering repo (feature_store_uw2_user_features). Despite the name, it counts distinct calendar days with any Segment track event per user over trailing 30/90-day windows anchored to yesterday (UTC), window [anchor - N, anchor) — matching the training SQL’s < created_date filter; the job’s window logic must change in lockstep with underwriting_v2/query.sql or it introduces train/serve skew. These features are as fresh as yesterday’s run regardless of when the float check happens — a coarser staleness bound than the same-day MLResult reuse above.