ANT Migration Tool struggles

Question

I performed a dress rehearsal into a Developer Pro sandbox using the sfdx force:source:deploy command successfully. Now that I'm trying to use the same source metadata with ANT migration tool & Jenkins my customField(s), recordTypes are not deploying at all.

The biggest difference i notice is that when I use sfdx to deploy, 52 components are deployed (no errors), while using ANT I only see 17 components attempting to be deployed (10 of them are errors) so clearly stuff is being skipped over causing the errors.

Is there anything I am missing? In the Jenkins job there is an ANT step to copy all of the source into a root directory. The root directory looks find and has the following structure for example:

sourceRoot/objects/Lead/fields
sourceRoot/objects/Lead/recordTypes
sourceRoot/objects/Lead/listViews

So I figured that all the metadata is getting picked up alright for the deploy step, but I guess not.

Package.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">
    <types>
        <members>*</members>
        <name>ApexClass</name>
    </types>
    <types>
         <members>*</members>
        <name>ApexComponent</name>
    </types>
    <types>
        <members>*</members>
        <name>StaticResource</name>
    </types>
     <types>
        <members>*</members>
        <name>ApexTrigger</name>
    </types>
     <types>
        <members>*</members>
        <name>ApexPage</name>
    </types>
    <types>
        <members>*</members>
        <name>CustomObject</name>
    </types>
    <types>
        <members>*</members>
        <name>FlexiPage</name>
    </types>
    <types>
        <members>*</members>
        <name>Layout</name>
    </types>
    <types>
         <members>*</members>
        <name>AuraDefinitionBundle</name>
    </types>
    <types>
         <members>*</members>
        <name>EmailTemplate</name>
    </types>
    <types>
         <members>*</members>
        <name>CustomMetadata</name>
    </types>
    <types>
         <members>*</members>
        <name>Flow</name>
    </types>
    <types>
         <members>*</members>
        <name>FlowDefinition</name>
    </types>
    <types>
         <members>*</members>
        <name>PathAssistant</name>
    </types>
    <types>
         <members>*</members>
        <name>Profile</name>
    </types>
    <types>
         <members>*</members>
        <name>Report</name>
    </types>
    <types>
         <members>*</members>
        <name>ReportType</name>
    </types>
    <types>
         <members>*</members>
        <name>Group</name>
    </types>
    <types>
        <members>Lead.Account__c</members>
        <members>Lead.Rating</members>
        <members>Lead.LeadSource</members>
        <members>Campaign.Type</members>
        <name>CustomField</name>
    </types>
    <types>
        <members>Lead.GTM</members>
        <name>RecordType</name>
    </types>
    <types>
        <members>Lead.Lead_Classification_Required</members>
        <name>ValidationRule</name>
    </types>
    <types>
        <members>*</members>
        <name>BusinessProcess</name>
    </types>
    <version>37.0</version>
</Package>

Answer 1

Your source is in Salesforce DX format, as evidenced by the fact that you have subdirectories within your object metadata:

sourceRoot/objects/Lead/fields

Metadata API source format is more monolithic; all metadata related to a Custom Object, for example, is stored in a single .object file.

Metadata API clients like Ant cannot directly deploy Salesforce DX-format source code. You'll need to ask Salesforce DX to convert the code to Metadata API format to get a package you can deploy:

sfdx force:source:convert -r sourceRoot -d <where you want MDAPI source>

Then you can deploy the MDAPI output with Ant.

You don't need Ant, though. Salesforce DX itself can deploy Metadata API source just fine with sfdx force:mdapi:deploy.