Are @future methods ignored in @TestSetup blocks?

Question

Ok, this is going to become a quite long question. The main goal is to document strange platform behavior and maybe learn more why this happens.

It all started with that test which failed because setup code called an @future method which failed. (BTW: I was really confused to learn that @future are called at all without a Test.stopTest() which led to this question.)

Fails

@isTest
private class TestSetupTest1 {

    @isTest
    private static void willFail() {

        // Setup
        callFailingFutureMethod();

        // Verify 
        ...
    }
}

More or less by accident I made it work moving the setup code into a @TestSetup method:

Succeeds

@isTest
private class TestSetupTest2 {

    @TestSetup
    private static void setup() {
        callFailingFutureMethod();
    }


    @isTest
    private static void wontFail() {
        // Verify 
        ...
    }
}

It seems like the @future is not called at all or in a separate execution context here. That is my assumption because by adding Test.stopTest() here I can make it fail again.

Again fails

@isTest
private class TestSetupTest3 {

    @TestSetup
    private static void setup() {
        Test.startTest();
           callFailingFutureMethod();
        Test.stopTest();
    }


    @isTest
    private static void willAlsoFail() {
        // Verify 
        ...
    }
}

Answer 1

Your setup definitely can run future methods, so they are not ignored entirely. This test passes for me on cs14 using any API Version between 35 and 40:

@IsTest
public class DemoFuture
{
    @TestSetup
    static void setupData()
    {
        createSomeData();
    }
    @future
    public static void createSomeData()
    {
        SObjectFactory.create(Account.sObjectType);
    }
    static testmethod void testFutureCall()
    {
        system.assertEquals(1, [SELECT count() FROM Account], 'Future should be run');
    }
}

If I change the body of the createSomeData() method to system.assert(false), the test fails. So future calls in @TestSetup are definitely respected at least some of the time.