Metadata.DeployCallback is not called when deployment status is Fatal Error

Question

BACKGROUND

I am working on a feature that deploys custom metadata record. I am using a scratch org with a linked namespace.

This is a my method which initiates deployment:

@AuraEnabled
public static String deployDemo()
{
    Metadata.CustomMetadata customMetadataRecord;
    List<String> componentNameList = new List<String>{'mynamespace__ObjectSetting__mdt.mynamespace__Config'};
    List<Metadata.Metadata> components = Metadata.Operations.retrieve(Metadata.MetadataType.CustomMetadata, componentNameList);
    customMetadataRecord = (Metadata.CustomMetadata)components.get(0);

    Metadata.CustomMetadataValue customField = new Metadata.CustomMetadataValue();
    customField.field = 'mynamespace__Objects__c';
    customField.value = 'dummy text';
    customMetadataRecord.values.add(customField);

    Metadata.DeployContainer mdContainer = new Metadata.DeployContainer();
    mdContainer.addMetadata(customMetadataRecord);
    ID jobID = Metadata.Operations.enqueueDeployment(mdContainer, new MetadataDeployCallback());

    return String.valueOf(jobID);
}

And here the implementation of Metadata.DeployCallback:

public with sharing class MetadataDeployCallback implements Metadata.DeployCallback
{
    public void handleResult(Metadata.DeployResult result, Metadata.DeployCallbackContext context)
    {
        System.debug('MetadataDeployCallback with DeployResult ' + JSON.serialize(result));
        System.debug('MetadataDeployCallback with DeployCallbackContext ' + JSON.serialize(context));

        MetadataDeployComplete__e mdEvent = new MetadataDeployComplete__e();
        mdEvent.Success__c = result.success;
        mdEvent.JobID__c = result.id;
        mdEvent.Status__c = String.valueOf(result.status);

        if(result.status != Metadata.DeployStatus.Succeeded)
        {
            // Verbose error handling is skipped for brevity
            mdEvent.ErrorMessage__c = 'Error';
        }

        Database.SaveResult eventResult = EventBus.publish(mdEvent);
        if( !eventResult.success)
        {
            system.debug('Failed to publish event');
        }
    }
}

PROBLEM

Every time I call the deployDemo() method I get a valid jobID value. However, the handleResult from the MetadataDeployCallback class is invoked only when deployment is Successful. If a deployment has been completed with Fatal Error, handleResult is never executed (this is my observation based on absence of debug logs ). What is more, this issue occurs randomly, sometimes deployment is successful, sometimes it ends up with fatal error.

QUESTION

Why is Metadata.DeployCallback not called when deployment status is Fatal Error? And why does deployments fail randomly? If there was an error in the code, I would expect it to fail all the time.

RELATED QUESTIONS

Cannot update packaged custom metadata records from packaged Apex

Edit Custom Metadata Records from a Managed Package

Cannot update Custom Metadata records from Subscriber org in managed app