Service Cloud - ServiceNow Integration (with retries and sequencing)

Question

We have to connect/integrate Service Cloud and ServiceNow cloud applications. Apart from the usual features of Case - Incident sync the requirement is to handle Retries and Sequencing in the event of "transient faults".

The design envisaged (for retries) is to use Scheduler and Batch Apex to scan through the custom log Object and reconstruct the message (based on the logged source row id) before calling out. The retry threshold could also be configured to limit the maximum retries.

The design envisaged (for sequencing) is to log all outbound requests into a custom log object and use scheduler and batch apex to scan and process the messages in order. This seems to be highly inefficient and error-prone.

The limitations we see for the above designs are the Scheduled Apex limits, Callout Limits and Batch Apex Job limits.

We are looking at suggestions if these have been successfully implemented in Salesforce. If not then we would think of some other 3rd party alternatives.

Answer 1

I have implemented pretty much exactly this design for callouts in a variety of contexts where we had to ensure correct ordering of the messages.

It has worked successfully and reliably for us (a consultancy) in a number of clients' systems.

Given your question, you may already appreciate subtleties such as using an auto-number field to ensure the sequencing. If you try to use CreatedDate or ModifiedDate, then you will realise that the granularity of those is only down to seconds, not milliseconds, so they are not reliable in high volume systems. Auto-numbers also mean that if you insert a list of log entries at once, they are numbered in the order of that list, so you have predictable ordering within the list.

The easiest solution to implement is to have a scheduled job running frequently and processing all of the pending logs. If you can live with having a delay of minutes between the log being inserted and being processed, then this is the simplest solution.

An alternative that we have used is to have batch apex launched on-demand when a log is created (Queueable would be a better fit, but it can only be chained once if you are using callouts). This would mean that the log object is processed almost immediately. But, you need to implement some sort of semaphore system to make sure that you only run one batch at a time. And there are a whole bunch of other details around that. But it is achievable.

In any solution, you will need to check your limits as you go along: The two we had to be careful of are the maximum callout time, and number of callouts. In our system, we stop the batch as those limits approach and then just pick up the unprocessed items later.

And you also need to be really careful about how you update your log records by using the Database.update(logsToUpdate, false) version and then looking into any errors.

TBH, so much work has gone into our code for doing this we've been pretty tempted to put it on the appexchange as a managed package!