How to ensure all Leads are processed in batch job before firing off next step?

Question

We're using Omni-Channel to route leads to specialists, and have a requirement that certain leads come before others. In this case, we're relying on a custom field that is, in essence, a priority score.

Based on conversations with Salesforce Support, the only way to change the order of a queue is to set the ownership of the queue members. Thus, to re-order a queue, we must first reassign all queue members to a holding queue, then pick out the ones we want and assign them in groups. For example, after reassigning to the holding queue, we pull all the priority 1 leads, then all the priority 2, and so on.

The problem I'm running into is that we have to use queueables as part of any job that updates Leads, because we use callouts to an external service, and (as I recall) that's a no-no in batch jobs.

Thus, my pattern is something along these lines:

1. Get a list of all leads of a particular type (campaign, in this case)
2. Batch those leads up
3. For each batch, fire off a queueable to reassign to holding
4. In the finish method of the batch job, fire off the next step

My problem is this: I can't guarantee that all of the queueables finish by the time the finish method of the parent batch job fires. That means I might actually have some leads that have not been assigned to holding yet, and would missed getting processed.

I played with adding a check method in between the two jobs, to see whether the main queue was really empty, but it has to run in what is in essence an infinite loop; I added a counter to get around this, but had to add an artificial delay to make that work, and now I'm running into Apex CPU limits...so it's back to square one.

Oh, and here's the rub: I can't just load all the leads into the queue as a one-off. I have to load them by time zone, based on when they're callable, and I have to re-order them each time. (For example, when 9am CT rolls around, a lead in Dallas with a priority of 1 should be called before a New York City lead with a priority of 2, even though the NYC lead was there first.)

Thus, the question: What sort of pattern would work for this setup? Assume that I have a sufficient number of leads that I have to process them in batches of some sort (I typically have thousands, if not tens of thousands), and that I have to run the job at least three times per day.

Answer 1

I'd highly recommend you read Dan Appleman's Advanced Apex Programming and look at some of his asynchronous patterns, especially the ones that use custom objects to help with queuing up the queueables, batchables, etc. He also explains patterns that allow you to dynamically choose to call a schedulable instead of another batch or queueable (as appropriate). You might find that helpful in your situation.

More than anything, it sounds to me as though you'd benefit from doing some kind of pre-sorting before running your batches. You've not shared any code, so perhaps there's a way to include it as part of your batchable's query to help you get them closer to the correct processing order before running them through the remainder of your code. Without having more detail so the problem can be broken into smaller pieces, it's difficult to give you further guidance.

Answer 2

You're allowed callouts in Batchable classes, but you have to specifically request it via Database.AllowsCallouts. This might make your design a lot more feasible. Keep in mind that you only get 100 callouts per transaction, so you might want to reduce your batch size when calling Database.executeBatch (the second parameter is number of records per execute method, from 1 to 2,000).