Mdapi deployment stucks in queue, even when there is no other deployments in progress

Question

Mdapi deployment stucks in the queue, even when there are no other deployment in progress

Steps To Reproduce: Prepare package for deployment run force:legacy:mdapi:deploy Expected result Deploymnt in sandbox starts immediately when there is no any other deployment in progress, or gets into a queue when there is another deployment/validation in progress

Actual result Deployment job gets into queue, and stays there for hours, even when there is no any deployment in progress Each next submitted deployment lands in queue as well

I tested this on multiple orgs, and for 2 out of 5 of them, it happens currently. Also, nature of this wrong behavior is a bit random, because it starts to happen this way one day, and the next day it's all fine again, then after some days strikes again.

System Information sfdx-cli/7.82.0-3d0c527ac6 linux-x64 node-v12.18.3

Answer 1

We also noticed a lot of deployment issues using salesforce source:deploy since 11 may 2022 to our Sandbox on instance CS88. Both deploying locally or via CI/CD using bitbucket.

On status.salesforce.com several performance degradations where reported for CS88 instance since then:

1. https://status.salesforce.com/incidents/9038
2. https://status.salesforce.com/incidents/9057
3. https://status.salesforce.com/incidents/9101

When deploying we also noticed that deployments where not starting and in a pending stage. When running the source:deploy command, most of the time we saw prompts like this:

Error: socket hang up
at connResetException (node:internal/errors:692:14)
at TLSSocket.socketOnEnd (node:_http_client:478:23)
at TLSSocket.emit (node:events:539:35)
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'ECONNRESET'
}
(node:121) Warning: Network error occurred.  Continuing to poll.
(Use `node --trace-warnings ...` to show where the warning was created)

Most of the time the deployments failed due to the max waiting time being reached (-w parameter). Also we didn't notice any difference using SOAP or REST deploy.

Yesterday (17 may 2022) during office hours (CEST) we noticed some of the same issues although no incident report was made. But as of yesterday night (17 may 2022 after 22:00 CEST) deployments are running as expected again. Will continue to monitor during the day again.

23-5-2022 Update: There is now also a known issue logged by salesforce: https://trailblazer.salesforce.com/issues_view?title=performance-degradation-message-queue-processing-on-multiple-sandbox-instances&Id=a1p4V00000212Ja

Answer 2

I believe this is a Summer 22 issue if your source environment is on that release. Nothing has officially been released from SF about this yet. We are seeing some orgs completely blocked and others on this Summer 22 release completely blocked from any Metadata API calls. Old skool change sets still work if you are really stuck!

Answer 3

Thank you so much for the update! We had the similar issue on CS109 instance but now the deployments are picking up as expected.

Hope the issue is permanently resolved now. thanks !