0

We have to connect/integrate Service Cloud and ServiceNow cloud applications. Apart from the usual features of Case - Incident sync the requirement is to handle Retries and Sequencing in the event of "transient faults".

The design envisaged (for retries) is to use Scheduler and Batch Apex to scan through the custom log Object and reconstruct the message (based on the logged source row id) before calling out. The retry threshold could also be configured to limit the maximum retries.

The design envisaged (for sequencing) is to log all outbound requests into a custom log object and use scheduler and batch apex to scan and process the messages in order. This seems to be highly inefficient and error-prone.

The limitations we see for the above designs are the Scheduled Apex limits, Callout Limits and Batch Apex Job limits.

We are looking at suggestions if these have been successfully implemented in Salesforce. If not then we would think of some other 3rd party alternatives.

  • If the desired features retries and sequencing have not been implemented in Salesforce then could anyone suggest an integration platform (iPaaS) solution that have been successfully implemented catering to present and future needs. We are considering Service Cloud and ServiceNow as just two constituents in our overall Integration Landscape. – sHiBuKaLiDhAsAn Oct 6 '16 at 1:02
0
+50

I have implemented pretty much exactly this design for callouts in a variety of contexts where we had to ensure correct ordering of the messages.

It has worked successfully and reliably for us (a consultancy) in a number of clients' systems.

Given your question, you may already appreciate subtleties such as using an auto-number field to ensure the sequencing. If you try to use CreatedDate or ModifiedDate, then you will realise that the granularity of those is only down to seconds, not milliseconds, so they are not reliable in high volume systems. Auto-numbers also mean that if you insert a list of log entries at once, they are numbered in the order of that list, so you have predictable ordering within the list.

The easiest solution to implement is to have a scheduled job running frequently and processing all of the pending logs. If you can live with having a delay of minutes between the log being inserted and being processed, then this is the simplest solution.

An alternative that we have used is to have batch apex launched on-demand when a log is created (Queueable would be a better fit, but it can only be chained once if you are using callouts). This would mean that the log object is processed almost immediately. But, you need to implement some sort of semaphore system to make sure that you only run one batch at a time. And there are a whole bunch of other details around that. But it is achievable.

In any solution, you will need to check your limits as you go along: The two we had to be careful of are the maximum callout time, and number of callouts. In our system, we stop the batch as those limits approach and then just pick up the unprocessed items later.

And you also need to be really careful about how you update your log records by using the Database.update(logsToUpdate, false) version and then looking into any errors.

TBH, so much work has gone into our code for doing this we've been pretty tempted to put it on the appexchange as a managed package!

  • Nice explanation .. pretty much what we thought ... could you explain how you are managing the faults/errors when a callout is made and where are you logging them ... do you have a child/detail Object for the Main Log Object? This way the Main Log object can be used to maintain the sequence while the child object logs the details of every attempts. – sHiBuKaLiDhAsAn Oct 26 '16 at 6:01
  • Thanks! For logging, I use a completely separate object with a text field to store the id of the object which caused the error. I do this for 2 reasons: 1. sometimes, we use the framework to manage callouts to different systems, so there might be different log objects for each and a text field can work like a polymorphic lookup 2. we never want an error log to fail to insert. If it were M-D to the main log object, an error on the main log might cause the error log to fail as well. We then use workflow emails and scheduled reports to monitor the system performance. – Aidan Oct 26 '16 at 7:02
  • A couple of questions - you mentioned 2 solutions - one using Scheduled Job and the other with Batch Apex. When you use Scheduled job to clean up the logs .. should we be aware of the limits pertaining to callouts? I believe only 100 callouts/transaction could be made. And for the Batch Apex although the callout limts and the payload size are enhanced (200, 12MB), does it guarantee the order of the callouts made and are the callouts synchronous? – sHiBuKaLiDhAsAn Nov 9 '16 at 2:25
  • As I understand it, SF doesn't guarantee the order of execution of chunks of batches. So, what I did was write my start() method to query with LIMIT 1. Then SF only ever runs one chunk. Then, in the execute() method, I query the number of records I actually want to process. And in the finish() method, I check to see if another batch is needed, and start it if I need to. It's abusing the batch a little bit, but it ensures safety with the concurrency. – Aidan Nov 9 '16 at 9:10
  • If I understand it well the scope is 1 record (via LIMIT clause) and you execute database.executeBatch() command repeatedly from finish() method if more records exist. Is it correct? When you mention Concurrency, did you mean the limit pertaining to "Number of Batch Apex Jobs running in parallel", doesn't running the batch repeatedly cause to violate the limit? especially dealing with high volume transactions involving multiple interfaces. Also this limit is global just not for the interface concerned, is it not? Could you give some insights? – sHiBuKaLiDhAsAn Dec 18 '16 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.