Does Test.stopTest() cause ALL waiting asynchronous functions to fire?

Question

For practice, I'm composing on a unit test for a Schedulable class:

/**
 * Example of scheduling Apex Batch Processing
 * @see Apex Workbook, page 68
 **/
global class WARE_CleanMerchandiseScheduler implements Schedulable 
{
    global void execute (SchedulableContext context)
    {
        // The query used by the batch job.
        String query = 'SELECT Id, CreatedDate FROM Merchandise__c WHERE Id NOT IN (SELECT Merchandise__c FROM Line_Item__c)';

        BAT_CleanUpRecords cleanRecords = new BAT_CleanUpRecords (query);
        Database.executeBatch(cleanRecords);
    }
}

This Schedulable class is used to execute a Batchable class:

/**
 * Example of Apex Batch Processing
 * @see Apex Workbook, page 64
 **/
global class BAT_CleanUpRecords implements Database.Batchable<sObject>
{
    global final String QUERY;

    global BAT_CleanUpRecords (String query) { this.query = query; }

    global Database.QueryLocator start (Database.BatchableContext batchableContext) { return Database.getQueryLocator (QUERY); }

    global void execute (Database.BatchableContext batchableContext, List<sObject> sObjectScopeList)
    {
        delete sObjectScopeList;
        Database.emptyRecycleBin (sObjectScopeList);
    }

    global void finish (Database.BatchableContext batchableContext)
    {
        AsyncApexJob asyncJob = [
                                    SELECT id, status, numberOfErrors, jobItemsProcessed, totalJobItems, createdBy.Email
                                    FROM AsyncApexJob
                                    WHERE id = :batchableContext.getJobId()
                                ];

        String[] toAddresses = new String[] { asyncJob.CreatedBy.Email };

        // Send an email to the Apex job's submitter notifying of job completion
        Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
                                     mail.setToAddresses(toAddresses);
                                     mail.setSubject('Record Clean Up Status: ' + asyncJob.Status);
                                     mail.setPlainTextBody(
                                            'The batch Apex job processed ' + asyncJob.TotalJobItems
                                            + ' batches with ' + asyncJob.NumberOfErrors + ' failures.'
                                        );

        Messaging.sendEmail (new Messaging.SingleEmailMessage[]{mail});
    }
}

I've already created a unit test for the Batchable class, and it appears to function correctly.

While it is therefore arguably unnecessary overkill to test that the Batchable class has functioned correctly when I am unit testing the Schedulable class, I decided to do so anyway:

/**
 * Unit Tests for example of scheduling Apex Batch Processing
 * @see Apex Workbook, page 68
 **/
 @isTest
private class WARE_CleanMerchandiseSchedulerTest 
{
    public static final User TestRunner = TEST_RunAsUserFactory.create();

    // CRON expression: midnight on March 15. Because this is a test, job executes immediately after Test.stopTest().
    public static final String CRON_EXP  = '0 0 0 15 3 ? 2022';
    public static final String NEXT_FIRE = '2022-03-15 00:00:00';


    private static testMethod void scheduleTest()
    {
        WARE_MerchandiseCleanTestLists merchandiseTestLists = new WARE_MerchandiseCleanTestLists (5, 7); 

        List<Merchandise__c> dummyMerchandiseNotInLineItemsList = merchandiseTestLists.dummyMerchandiseNotInLineItemsList;
        List<Merchandise__c> dummyMerchandiseInLineItemsList    = merchandiseTestLists.dummyMerchandiseInLineItemsList;

        System.runAs(TestRunner)
        {
            Test.startTest();
            {

                // Schedule the test job
                Id jobId = System.schedule('ScheduledApexClassTest', CRON_EXP, new WARE_CleanMerchandiseScheduler());
                TEST_SchedulableHelper.assertCronTrigger(jobId, CRON_EXP, NEXT_FIRE);

                TEST_DummyMerchandiseFactory.assertMerchandise (dummyMerchandiseNotInLineItemsList, true);  // Verify these are there to start with.
                TEST_DummyMerchandiseFactory.assertMerchandise (dummyMerchandiseInLineItemsList, true);         // Verify these are there to start with.
            }
            Test.stopTest();
        }

        TEST_DummyMerchandiseFactory.assertMerchandise (dummyMerchandiseNotInLineItemsList, false); // Verify merchandise items without line items got deleted.
        TEST_DummyMerchandiseFactory.assertMerchandise (dummyMerchandiseInLineItemsList, true);     // Verify merchandise items with line items did not get deleted.
    }   

}

However, the penultimate assertion is failing, informing me that there are still the original five merchandise items in the "dummyMerchandiseNotInLineItemsList", which I expected to be deleted.

This should only be possible if either (a) they have line items [they don't: not only did I never create them, but in some debug code {since removed} I checked to make sure they weren't there]; (b) my query is wrong [but this is the same exact query I used to test my Batchable class, and it worked fine there]; or (c) the Batchable class never executed.

So, I'm wondering whether either (A) my understanding of Test.stopTest() is incorrect or (B) I'm overlooking something else?

In case it helps to expose the problems, here is the WARE_MerchandiseCleanTestLists class:

 /**
 * Unit Tests for example of Apex Batch Processing
 * @see Apex Workbook, page 65
 **/
@isTest
public class WARE_MerchandiseCleanTestLists
{
        public List<Merchandise__c> dummyMerchandiseNotInLineItemsList = null;
        public List<Merchandise__c> dummyMerchandiseInLineItemsList    = null;

        public WARE_MerchandiseCleanTestLists() {}

        public WARE_MerchandiseCleanTestLists(Integer numberOfMerchandiseNotInLineItems, Integer numberOfMerchandiseInLineItems)
        {
            // Create some test merchandise items to be deleted by the batch job.
            this.dummyMerchandiseNotInLineItemsList = TEST_DummyMerchandiseFactory.createDummyList(numberOfMerchandiseNotInLineItems, true);

            // Create some test merchandise items to be kept after the batch job (to test against false positives).
            this.dummyMerchandiseInLineItemsList    = TEST_DummyMerchandiseFactory.createDummyList(numberOfMerchandiseInLineItems, true);
            addLineItemsForMerchandise(this.dummyMerchandiseInLineItemsList);
        }

        private static void addLineItemsForMerchandise(List<Merchandise__c> merchandiseList)
        {
            List<Invoice__c> dummyInvoiceInLineItemsList            = TEST_DummyInvoiceFactory.createDummyList(merchandiseList.size(), true);

            List<Line_Item__c> dummyLineItemList = new List<Line_Item__c>();
            Integer i = 0;
            for (Merchandise__c dummyMerchandise : merchandiseList)
            {
                dummyLineItemList.add(TEST_DummyLineItemFactory.createDummy(dummyInvoiceInLineItemsList[i], dummyMerchandise , false));
                i++;
            }

            insert dummyLineItemList;
        }

}

Answer 1

Understanding. You understanding is correct, as per the documentation here.

The System.schedule method starts an asynchronous process. This means that when you test scheduled Apex, you must ensure that the scheduled job is finished before testing against the results. Use the Test methods startTest and stopTest around the System.schedule method to ensure it finishes before continuing your test. All asynchronous calls made after the startTest method are collected by the system. When stopTest is executed, all asynchronous processes are run synchronously. If you don’t include the System.schedule method within the startTest and stopTest methods, the scheduled job executes at the end of your test method for Apex saved using Salesforce.com API version 25.0 and later, but not in earlier versions.

A Hunch. Its just a hunch, but what I suspect might be happening is your using a Mock approach to test the result of your Schedule job? Using 'static' variables? If so these are not shared between the context running your test and the one Salesforce spins up in the background to run your Schedule job (or for that matter the job it runs). Meaning you will likely have to assert using a none Mock approach for this test.