So the response from R&D via Support was:
As discussed on call, I went back to RnD to confirm your request. They
have confirmed that It is the nature of multi tenet architecture of
Salesforce platform when asyc request is queued, we cannot determine a
fair time for the request to go through.
So basically, the delay is working as designed and hour long waits for an IDE to save/deploy a single class is not abnormal despite what we know to be the case.
After pushing the issue on the partner community the answer I received was a little more encouraging yet still vague. At least it has been acknowledged that there IS an issue:
I discussed this with the product team responsible for MD API. They
acknowledge that we have been having several issues with MD API based
deployments. There are several large deployments being processed (like
packages) and since the deployments are transactional, breaking them
into batches isn't possible, which causes some deployments to be
queued for a longer period. The current situation is not the norm and
hopefully will be resolved soon as teams are working on it. The
product team also mentioned a strategy which they are working on to
alleviate the queuing latency in the next couple of releases.
I have done development using the IDE for a long time so I very well
understand everyone's frustration here. Please be assured that the
product team is aware of and working on a better solution.
So if you are having issues PLEASE open a case with support even if it gets closed (Or wherever you post your issues) so that if it continues it is known that it is an issue vs just me being crazy :)
As a point of reference - When I posted my case I provided the Deployment ID so at least the logs could be looked into. As other comments suggest there is no pattern. Sometimes it works fin, then so slow for hours, then fine, then 5 minutes later so slow. At least with a deployment ID they can focus the efforts
UPDATE 8-19-2016
Been seeing this on other instances although not to the same degree. Other instances are 5 minutes or so. CS19 seems to be the worst. Update from SF today
For everyone interested. What I'm finding out is that the Deployment
jobs are getting demoted within instances. This happens due to a fair
usage algorithm that evaluates all messages (jobs) that are executed
on the Pods shared MQ system. When demoted, our deployment jobs go
from being dequeued from many app servers to only a few, which
obviously affects our ability to scale. Once we confirm the behavior,
we'll be able to prioritize specific improvements.
