Does Database.QueryLocator and Iterable both have same limit?

Question

I wrote a batch class to insert dummy data into big object and have used Iterable to insert data. I was under impression that Iterable should only be able to insert in normal limit of 50K records.

But when I tried giving it 1 Millions, it processed even that. As this is the max I can insert in my developer edition.

Here is the sample :

global class BigObjectInsertHelper implements Database.Batchable<Integer> {

    global Iterable<Integer> start(Database.BatchableContext param1) {
        return new IntegerIterator(1000000);
    }

    global void execute(Database.BatchableContext param1, List<Object> param2) {
        List<BigAccountRecord__b> bigAccountRecordBS = new List<BigAccountRecord__b>();
        for (Integer i = 0; i < param2.size(); i++) {
            BigAccountRecord__b accountRecordB = new BigAccountRecord__b();
            accountRecordB.FirstName__c = 'Nagendra' + generateRandomString(5);
            accountRecordB.LastName__c = 'Nagendra' + generateRandomString(5);
            accountRecordB.Name__c = accountRecordB.FirstName__c + ' ' + accountRecordB.LastName__c;
            accountRecordB.PersonalEmail__c = generateRandomString(6) + '@gmail.' + generateRandomString(3);
            accountRecordB.Description__c = generateRandomString(40);
            bigAccountRecordBS.add(accountRecordB);
        }
        Database.insertImmediate(bigAccountRecordBS);
    }

    global void finish(Database.BatchableContext param1) {
        Messaging.SingleEmailMessage message = new Messaging.SingleEmailMessage();
        // Replace IDs with valid record IDs in your org.
        message.toAddresses = new String[]{
                UserInfo.getUserEmail()
        };
        message.subject = 'Batch Status';
        message.plainTextBody = 'Finished with BigObject Insertion';
        Messaging.SingleEmailMessage[] messages =
                new List<Messaging.SingleEmailMessage>{
                        message
                };
        Messaging.SendEmailResult[] results = Messaging.sendEmail(messages);
        if (results[0].success) {
            System.debug('The email was sent successfully.');
        } else {
            System.debug('The email failed to send: ' + results[0].errors[0].message);
        }
    }

    public static String generateRandomString(Integer len) {
        final String chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz';
        String randStr = '';
        while (randStr.length() < len) {
            Integer idx = Math.mod(Math.abs(Crypto.getRandomInteger()), chars.length());
            randStr += chars.substring(idx, idx + 1);
        }
        return randStr;
    }
}

Also, when I tried to pass 50 Million / 60 Million in my Iterable, it starts preparing the batch for it. But it never gives an error saying Maximum record limit reached.

Though it fails while preparing the batch, I think this is because there is a time limit in which start should return the result, 2 minutes probably.

MY question is what is the limit of records which can be passed in Iterable in a batch class? Like we have for QueryLocator as 50 million max, do we have something for Iterable too?

Answer 1

The limit for a QueryLocator is 50,000,000 records, the maximum size for any query (including API-based SOQL calls). The there is no hard limit for Iterable, though you're still limited by both CPU time (limit 60,000 ms/1 minute) and total start time (10 minutes); in practice, it might be hard to get up to even 50 million rows in that time, but it is likely theoretically possible.

In addition, Iterable return types have no hard maximum scope size, unlike QueryLocator, which will not allow more than 2,000 records per execute call. Other limits may limit how many items you can actually process (e.g. heap or CPU time), but there's no inherent hard limit for scope size.

Hypothetically speaking, using Iterable, you could process 100,000,000 items, or more, assuming you can somehow generate that many items in the time allotted.

Answer 2

The only limit is what you can actually pass to it. At a certain point, you will hit a CPU timeout. If you try to query records and return the List<SObject> as your iterable, you will be confined to Limits.getLimitQueryRows().