Runbook: Float Creation Has Dropped to Critical Lows

Datadog Resources

Symptoms

There is a drop in the number of floats created events in Datadog.
There are a lot of payment errors in the logs.

Likely Causes

USIO payments are down.
A recent deployment to production has caused an issue.

Diagnosis

Is USIO Down?

Look at the payments dashboard and check if there are any errors.
Check the logs for any error messages.
Check the USIO status page: https://paymentdatasystems.statuspage.io/

Is there a production issue?

Check production float-service logs for any increase in errors.
Check production logs for any overall increase in errors.

Mitigation / Resolution

If USIO is down:

Alert USIO via Slack channel if their system appears to be down: https://floatme-crew.slack.com/archives/C0AT6TFKCCR
If USIO payments are in fact down, we may need to turn off floats and collections during the outage.

If there is a production issue:

Alert the team via Slack.
Which service deployment caused the issue, and who released it? Escalate to that person for help.
Rollback that deployment if necessary, or deploy a hotfix if that is more appropriate.
Consider turning off floats and collections.

Turning off floats and collections

Escalation

Escalate to USIO via Slack.
Escalate to the team or team member who did the production release for assistance.