Blend Integration Outage
Incident Report for Mace Innovations
Postmortem

We wanted to take the time to explain today’s issue and be transparent. Today around 12:27 pm EST we noticed one of our service providers for our cloud technology released a new feature. This feature and/release was not part of their release notes, notifications or documentation the make publicly available. Our automated alerting system notified us that events we received from Blend were being captured but not processed in a timely manner. We identified the issue with our cloud service provider and escalated the issue. Due to the design of our systems we were able to still maintain up time and receive events from Blend real time, but there was a delay in processing those events. After further analysis we determined in majority of cases most of our clients utilizing this integration were unaware of the latency. However, we did have an issue and wanted to make everyone aware.

We are including the response from the service provider as a reference.

...our team has rolled back the deployment that introduced the new functionality. As a result theDeployview will temporarily be unavailable...”

We are working with them closely to ensure this does not happen again. We are also taking internal steps to further reduce any latency should this happen again.

Thanks,

Shawn & Derik

Posted Jul 12, 2019 - 12:58 MDT

Resolved
All systems supporting the Blend integration are operational and functioning normally. The only impact was in the timeliness of processing events. No events were missed and all clients are caught up at this time. Email [email protected] should you have any further questions or issues.
Posted Jul 12, 2019 - 12:01 MDT
Update
As we continue to work this outage systems may experience slower than normal response.
Posted Jul 12, 2019 - 11:33 MDT
Identified
We identified the root cause and are now working to ensure we are caught up. We will update again when all systems have resumed normal operation.
Posted Jul 12, 2019 - 11:32 MDT
Investigating
We are currently investigating this issue.
Posted Jul 12, 2019 - 10:30 MDT
This incident affected: Blend (Integration).