Why did some batches execute successfully despite the Report REST API Limit?

Question

BACKGROUND

We have a legacy batch process that selects all contacts and runs a report for each account. Below is a simplified example of that class (all the necessary details are preserved):

MyBatchExample.cls

public class MyBatchExample implements Database.Batchable<SObject>, 
                                       Database.AllowsCallouts
{
    public Database.QueryLocator start(Database.BatchableContext BC)
    {
        String query = 'SELECT Id, Name FROM Account';
        return Database.getQueryLocator(query);
    }

    public void execute(Database.BatchableContext BC, List<Account> scope)
    {
        Id reportId = 'any_report_id';
        // in the real implementation the reportMetadata
        // is initialized with some report filters,
        // but it's omitted here for brevity 
        Reports.ReportDescribeResult describe = Reports.ReportManager.describeReport(reportId);
        Reports.ReportMetadata reportMetadata = describe.getReportMetadata();

        for (Account acc : scope)
        {
            Reports.ReportResults results = Reports.ReportManager.runReport(reportId, reportMetadata, true);
            // do something with the results, e.g. convert to csv and send an email - skipped for brevity
        }
    }

    public void finish(Database.BatchableContext BC)
    {
        // execute final actions
    }
}

This batch gets called from another process:

MyBatchExample batch = new MyBatchExample();
Database.executeBatch(batch, 200);

PROBLEM

With about 700 accounts this logic is doomed to hit the Reports API Limits as described at Dashboard Limits, Limitations, and Allocations and System.LimitException: You can't run more than 500 reports synchronously every 60 minutes

It's obvious why we see the following error in the Apex Jobs:

First error: You can't run more than 500 reports synchronously every 60 minutes. Try again later.

What's not clear is better to see in the screenshot:

How is it even possible that 3 jobs started at the same time, failed 2 and succeeded 2 batches each? My understanding is that once 2 batches completes successfully (the report has been run 400 times by this time), the 3rd batch fails. This seems to be true if we consider 1 job. However, there were 3 jobs ran simultaneously totaling in 6 successful batches, i.e. the report has been run ~1200 times.

I can vaguely explain the above by assuming that 3 distinct jobs were considered as asynchronous requests:

Your organization can request up to 1,200 asynchronous requests per hour.

BUT, what I completely cannot understand why the next 4th job had only one batch and it was successful even though Reports API Limits had been reached?

Answer 1

Batch jobs can run in interleaved order (up to five batches can run at once). Further, most governor limits of this nature are not necessarily accurate to the second, so most likely you did exceed the limit, and did so sufficiently quick enough to get about 1200 reports in. Asynchronous is a bit of a word that's thrown around a bit, but in regards to the 1200/hour limit, that would be using the runAsyncReport method instead of the runReport method.

You shouldn't rely on this behavior in the future, because you're only allowed 500. So, while technically, the limit should be hit by the third batch, it simply wasn't detected in time. Consider using the runAsyncReport method to use the higher limit, and perhaps consider just using a query to get the results that you want, since queries per hour are essentially unlimited.