Bulk API V1 Query - ResultId missing when running concurrent Bulk Jobs

Question

When querying for the result ids of a batch on a closed Bulk (v1) query job, sometimes the result id is not returned; instead the full returned xml is:

<result-list xmlns="http://www.force.com/2009/06/asyncapi/dataload"/>

The expected response is:

<result-list xmlns="http://www.force.com/2009/06/asyncapi/dataload"><result>7520S000002q9sJ</result></result-list>

Where 7520S000002q9sJ is the result id for the requested batch.

Here is my situation:

• My custom code (C#.NET) creates a query job, closes it, check for completion [EDIT: this was the faulty part], then retrieves the batch results via a result id
• This issue has only occurred when running concurrent runs of my custom code. A re-run of my custom code with the same query and same results yields the expected results returned. A single run of the custom code has no issue.
• Results of these queries is between 2,000 to 88,000 records and completes in a matter of seconds. From this, and subsequent reruns, I surmise that Salesforce limitations on size returned is not the issue.
• Each of the query jobs created run against different objects. There is not any link between queried object and which query has this issue.
• Each of the runs of this custom code uses the same login account. There does not appear to be a direct issue of concurrency because (1) each of the jobs is able to create, run, and close a bulk job without issue and (2) there are only 3 jobs running at once and I believe the concurrency limit is 5.
• Which of the 3 runs of the custom code that fails to retrieve a result id is inconsistent, usually only 1 or 2 fail but occasionally all 3 fail in this manner.
• When viewed from the Salesforce web interface, all of the jobs have a valid query, and valid results and a valid batch id.
• No error message from Salesforce has ever been given (such as too many concurrent users, thresholds exceeded, etc)
• log messages verify that the correct request is being sent to Salesforce (correct jobid and correct batchid) for any given run

What is causing Salesforce to return an empty result-list, when by all appearances, there are valid and available results to be retrieved?

My hunch is that there is an issue with threading or concurrent Salesforce sessions.

Answer 1

Result was a bug in the custom code that prematurely determined results were ready to retrieve from the bulk query job. My criteria was checking job completion instead of batch completion.

What is causing Salesforce to return an empty result-list, when by all appearances, there are valid and available results to be retrieved?

This happens when the batch is queried before it has completed processing.

Resolution was finally tracked down by viewing the timestamp on debug statements of "Checking when Bulk job is completed..." and "... completed". The "... completed" message had a timestamp before the End Time listed on the batch job via Salesforce interface.

A few questions answered:

• why was the issue only present with concurrent runs of my custom job? - concurrent jobs creates a higher demand on Salesforce which often manifests itself as batches taking longer to process but the latency for job closure did not change.
• why wasn't the issue detectable in Sandbox? - our test/Sandbox has significantly less data than production. This meant that the difference between the time a job closed and its batch completion was negligible due to smaller query results to process.