Around 11:30 am EST we identified an issue that impacted the timeliness of disclosures being sent to Blend. Around 11:50 am EST the issue was resolved and disclosures were being sent to Blend in near real-time as usual. We waited until around 12:25 pm EST to update our status center due to the fact we were monitoring the integration closely and wanted to ensure it was fully operational.

The root cause of this incident was a log drive that filled up / exhausted the free disk space available. We use a hosted monitoring solution, New Relic to monitor disk space and a variety of other metrics to ensure our systems are running within the metrics we deem acceptable. In this case, we didn’t receive notification the disk space was below 20%. After further inspection it was due to the disk filling up faster than our polling / monitoring interval (10 minutes). Once the disk filled up, the agent couldn’t report back to New Relic due to no disk space available.

We changed a few things to prevent this from happening again. First we expanded the disk drive giving us a larger amount of available space and will continue to monitor percent available as we progress through the increased volume of disclosures and loans in the coming months. Second, we updated the infrastructure monitoring software to poll the server every 1 minute rather than 10 minutes. This will enable the agent to report back before the disk free space is consumed. Third, our support team is going to closely manually monitor the systems and processes over the next few months in case of any further issue with the increased volume.

Have a great day!

Shawn