SF eventually came back with the following details on the incident:
Errors when accessing custom HTTPS domains for Force.com, Salesforce Communities or Site.com –
March 9th and 10th, 2015
At Salesforce, trusted customer success is our #1 value, and delivering the highest standard in system availability, performance, and security is our top priority. We know that trust begins with transparency, and with that in mind, we are providing the full root cause analysis for the recent performance and service disruption below:
At approximately 12:00 UTC on March 9th, 2015, the Salesforce technical team observed a partial failure of the load balancer that facilitates TLS communications to the SR2 instance. During this time, some customers who use custom HTTPS domains with Force.com sites, Salesforce Communities, or Site.com received errors when attemping to access their domain from some regions around the world, such as the eastern United States or Europe.
During the investigation, it was determined that the replication between two load balancers in SR2 was not operating properly. A failover to the SR1 instance was performed to bring the domains back into worldwide operation. An all-clear was called at 17:10 UTC on March 9th, 2015.
At 02:00 UTC on March 10th, 2015, while executing remediation steps to correct the original incident, the failover was inadvertently overwritten when the backup configuration was applied, which resulted in a recurrence of errors when attempting to access the custom HTTPS domains of Force.com sites, Salesforce Communities, or Site.com sites from the same impacted regions of the world as the previous day's incident. This error was corrected at 09:36 UTC on March 10th, 2015. In order to prevent a third recurrence, Network operations blocked the SR2 ip range at the edge of our network to prevent automated monitoring tools reenabling the SR2 VIPs until the team was satisfied that configuration on load balancers were consistent and restored successfully.
On March 11th, 2015, the configurations of the load balancers in SR2 were rebuilt, including their replication connections. SR2 was put back into service for custom HTTPS domains at 00:55 UTC on March 12th, 2015.
The root cause of the initial incident was the load balancer failure. Despite continued investigation alongside our third party load balancer vendor, a definitive root cause for why the load balancer failed is not yet known. This remains under investigation in collaboration with the vendor of the load balancer.
The root cause for the recurrence of the issue on March 10th, 2015, was the failover details being erroneously overwritten by the application of the backup load balancer configuration.
Next Steps and Preventive Actions:
We are working with the vendor of the load balancer to determine the root cause of the failure.
This incident also highlighted a monitoring deficiency in the DNS-based global server load balancer (GSLB) configuration. The HTTP port is monitored and used for testing end-to-end reachability, but the HTTPS port was not being monitored by the GSLB configuration. The HTTP port was operational on SR2 during this incident, but the HTTPS was not. The need to monitor the HTTPS port in the GSLB configuration has been escalated to the technology team.
Load balancing remediation procedures for Network operations updated to include blocking the SR2 ip range at the edge of our network to prevent automated monitoring tools from re-enabling the impacted VIPs before the team intends these to be re-enabled.
Our commitment to you:
We sincerely apologize for the impact this incident may have caused you and your business. It is our goal to provide world-class service to our customers, and we are continuously assessing and improving our tools, processes, and architecture in order to provide customers with the best service possible.