1

I need some help in designing the solution.

My requirement is I have around 50,000 contacts in my org. I need to send contacts(external id) to a Vendor-product, get the latest information(eg: email id, phone, address) and update the contact in our Salesforce.com CRM.

Detailed requirement.

  1. From Salesforce, take chunk of 10,000 contacts, pass the contact keys(external id) as HTTP REQ Parameter and make a HTTP call to Vendor-Product.

  2. Vendor-product accepts this request and creates a batch Job.

  3. Make another HTTP call to Vendor-Product to check the status of the Job.

  4. Wait for the Vendor-Product job to complete.

  5. When Vendor-Product job is complete, make another HTTP call to read the

  6. Response (result of the job is in either Jason or CSV format based on my wish).

  7. After reading the response (CSV or Jason format), update all 10,000 contacts (email, phonee, address) with latest information from the response.

  8. Above process must run for 5 times to completer all 50,000 records.

Please provide some insight on how to implement it. Any code please.

Additional comments Please note that this process won't run daily, may be I would execute this process twice in a year to refresh all my contacts with fresh data(email, phone, address, etc)

DESIGN

  • Mainjob –with multiple batch size of 1000 records.
  • Mainjob – use Batchable

    • Start
    • Execute
    • Finish
    • Inside Execute() method – Do following
      • Make a HTTP request out to remote service
    • Inside Finish method() – Do following
      • Schedule new Job – JobChild for checking the status and request the result (may be after 10 minutes).
    • Questions:

      • I put a time delay between each batch, so that I can allow remote service to complete the previous request successfully?
      • Can I pass a parameter( that is Remote service JOBID) to a new job –ChildJob ?
    • ChildJob use batchable

      • Make HTTP request to remote service to check the status of the job – Pass JOBID
      • If not complete, reschedule Childjob itself after a time gap.
      • If complete,

        • Make HTTP request to remote service, get the result.
        • Read the response and update the contact records with response.
        • Process the errors.
      • Questions:
        • ChildJob - I don't have any scope. Is that okay to have batachable job without scope ?
4
  • What's going to start this whole process off? Is it going to be a one off process or something that you do frequently? Nov 30 '15 at 20:14
  • Trying to do bulk async processing from Salesforce will be problematic. If you could reduce the chuck/batch size down to the point there the external system can respond with the result in a single callout then you could create a batch job in Salesforce to easily work through small chunks, updating them as it proceeds. Nov 30 '15 at 20:16
  • @DanielBallinger, Pls look at my comments.
    – Razu
    Nov 30 '15 at 20:29
  • A batch job already breaks it into chunks of 200, so you just run a single batch to do the work and you are good.
    – dphil
    Dec 1 '15 at 15:40
1

Are you required to do 10,000 at a time? If you schedule a batch, you can iterate over all 50k w/o having to run the job multiple times.

In addition, the processing would be sped up - As it is, you may run into several issues, including running times, heap size, and handling any errors for fields you may update.

Generally, as a design, I recommend this:

  1. Create a batch job with a size of between 500 and 1000 contacts.
  2. Within your controller, schedule a regular job to check back. This job will either reschedule itself, or it will execute a batch job to retrieve the results.

A few things to note:

  • Limits are for a single batch in a batch job, so you could make up to 100 callouts per batch
  • Depending on how the remote process wants you to get your results, you may want to alter the scheduling so that it waits until all of the batches are complete.

UPDATE: Since you can only process two jobs at a time, and there is no way to force the process to sleep, I recommend allowing the first job to kick off either the wait job or the next job.

This link has two good examples for dealing with parameters to scheduled jobs: Passing Parameter Into Schedulable Class

What you'd want to do for the schedulable job:

global class ContactUpdate_SendBatch implements Schedulable{

    global void execute(SchedulableContext sc) {
        //get next batch of contacts
        List<Contact> contacts = ContactUpdate_BatchingHelper.getNextBatch();
        String jobId = RemoteService.sendBatch(contacts);

        System.schedule('Wait on Batch',nextRunTime, new ContactUpdate_WaitOnBatch(jobId));
    }

    global void finish(Database.BatchableContext BC)
    {
        // Get the ID of the AsyncApexJob representing this batch job from Database.BatchableContext.
        // Query the AsyncApexJob object to retrieve the current job's information.
        AsyncApexJob a = [SELECT Id, Status, NumberOfErrors, JobItemsProcessed, TotalJobItems, CreatedBy.Email 
                          FROM AsyncApexJob WHERE Id = :BC.getJobId()];

        //then use the active job id and abort it
        system.abortJob(a.id);
    }
}


global class ContactUpdate_WaitOnBatch implements Schedulable{
    global String JobId {get;set;}

    global ContactUpdate_WaitOnBatch(String jobIdVal)
    {
        JobId = jobIdVal;
    }

    global void execute(SchedulableContext sc) {
        if(RemoteService.isCompleted(JobId) == false)
        {
            //Reschedule job, since the the finish method removes it
            System.schedule('Wait on Batch',nextRunTime, new ContactUpdate_WaitOnBatch(jobId));
        }
        else
        {
            RemoteService.ProcessBatch(JobId);
            System.schedule('Process Next Batch',nextRunTime, new ContactUpdate_SendBatch(jobId));
        }
    }

    global void finish(Database.BatchableContext BC)
    {
        // Get the ID of the AsyncApexJob representing this batch job from Database.BatchableContext.
        // Query the AsyncApexJob object to retrieve the current job's information.
        AsyncApexJob a = [SELECT Id, Status, NumberOfErrors, JobItemsProcessed, TotalJobItems, CreatedBy.Email 
                          FROM AsyncApexJob WHERE Id = :BC.getJobId()];

        //then use the active job id and abort it
        system.abortJob(a.id);
    }
}

I've glossed over some of the finer details, such as the nextRunTime variable, but you can find this information online quite easily. In addition, I pulled some info from here: https://help.salesforce.com/apex/HTViewSolution?id=000002809&language=en_US

2
  • I can provide some samples, but how does the remote service handle these requests? Does it expect you to request the results for the last run, or can you submit multiple batches, and the remote service gives you a "receipt" or ID to request the results for each batch when finished? Dec 1 '15 at 14:51
  • Thanks for the response. I really appreciate. Remote service can take 2 HTTP requests at a a time, for each request, they return a JOBID. Using JOBID, I can make request to check the status and get the result(Job output). Please look at my design above and additional questions.
    – Razu
    Dec 1 '15 at 16:03
0

Solutions below.

1-Make a batch class (Batchsize-10,000).

2-Make a HTTP request from web service class.

3-Make a HTTP request for status in while loop with delay 5 minutes.

4-Get the response from status HTTP request, if status done then we make another HTTP request for result.

5-We can save all information in CSV/Jason format after result.

If you want to discuss more about this please email me at sandy@mirketa.com

Thanks, Sandy

1
  • any code example please. I sent a note via email as well.
    – Razu
    Dec 1 '15 at 20:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.