Both Apex Flex Queue and Scheduled Jobs used up

Question

The system has batch call out APEX classes to sync Salesforce information with another Email sending provider, Active Campaign. Every time we insert or update a custom lead or custom opportunity, the system will call out the batch classes to communicate with Active Campaign so we can send proper emails to our customers.

Everyday we have a plenty of records to change on Lead and Opportunity. Occasionally, the system reaches the 100 limit for Apex Flex Queue. Therefore, I modified the code when the 100 limit for Apex Flex Queue has been reached, new tasks will be scheduled in Scheduled Jobs. However, today both the Apex Flex Queue and the Scheduled jobs limit have been reached, which means at the backend Salesforce queued 200 tasks in the system.

I would like to know whether there are people who have experienced the same situation and how they solved this problem. I believe the code has been optimised to its maximum capacity, but if you want to see my code, please tell me. One of the reasons I do not put my code here is that there are four batch classes and three triggers to execute the whole process. There would be too much to put on the forum.

Thanks,

Answer 1

In situations like this I've added an intermediate queue custom object to represent the records that need to be processed by the batch.

Then, rather than immediately starting a batch job every time a record changes, it gets added as a reference in the custom object. The batch jobs now run over the records in the queue object.

This has a number of advantages:

1. You only need a single batch job running on a schedule
2. The individual batch job can process all the outstanding records in one go. This might reduce the number of outbound API calls you need to make. Bulkification!
3. You can flag individual records in the queue that have problems while continuing to process other records.
4. If a record is already in the queue to be processed you don't need to add it again. This improves the situation where someone makes several changes to a record in rapid succession.

A more elaborate example of this approach was presented at Dreamforce in the Message Queuing - Building a Messaging Framework on the Salesforce1 Platform talk by @anup.

Answer 2

As a beginner of Stack Exchange, I am not sure whether I need to open a new question or make it a sub topic of this question. Anyway, if you think I should make a new question, later I will make one.

For this callout, there is a class to delete existing "tags" in the external system and then later add the new "tags". Therefore, an update will use two classes to callout. The first class to delete "tags" and the second class as the finish method of the first class to add "tags".

The delete class make an array to contain the full options of "tags" and use a loop to callout the delete API one by one. This is really not efficient. However, after checking the API, I found out that there isn't an API code to "check whether a tag is existing and delete it". That is why the system is deleting the whole array one by one now.

The API document is here http://www.activecampaign.com/api/overview.php

Our classes has only used the "Contact" part of the API. contact_sync and tag_remove are the major commands to use.

Answer 3

Perhaps we are going to use @future method as a temporary measure to see how it performs. Later if still have problems, I think this object reference method is a good long term solution.