We wanted to take the time to explain today’s issue and be transparent. Today around 12:27 pm EST we noticed one of our service providers for our cloud technology released a new feature. This feature and/release was not part of their release notes, notifications or documentation the make publicly available. Our automated alerting system notified us that events we received from Blend were being captured but not processed in a timely manner. We identified the issue with our cloud service provider and escalated the issue. Due to the design of our systems we were able to still maintain up time and receive events from Blend real time, but there was a delay in processing those events. After further analysis we determined in majority of cases most of our clients utilizing this integration were unaware of the latency. However, we did have an issue and wanted to make everyone aware.
We are including the response from the service provider as a reference.
...our team has rolled back the deployment that introduced the new functionality. As a result theDeploy
view will temporarily be unavailable...”
We are working with them closely to ensure this does not happen again. We are also taking internal steps to further reduce any latency should this happen again.
Shawn & Derik