Best Practices for Batch Processing

Question

We have a business process to execute batch inserts/updates and Searches. Here is the scenario, We have a community Portal where external users will select List of records (custom records) based on a certain logic. This list may have up to 100-500 records.

And based on their selection, we have to process them. Like create Accounts, Contacts and create set of Custom records, send email alerts etc.. + Searches. This process is significantly heavy. And after inserting some of above custom records, there are triggers + Apex classes to run too. This will simply exceeds most of the governor limits.

Apparently this community was in a Client side application previously, where we used SOAP API to handle the Process. In that way we didn't have to bother about governor limits.

So I seek the help from yours to understand the best option to follow to implement this. Few options we are considering right now is,

1. Using Batch Apex
2. Using BulkTK - BulkTK
3. Using @future
4. Use Apex REST?

Appreciate your help here to understand the best possible approach. Or else please share your thoughts

Many Thanks, Lahiru

Answer 1

It seems to me that your questions go far beyond "best practices" for asynchronous or batch processing or data. What you're really asking about is architecture for a complete solution to data processing to overcome limits issues from within your community and I suspect potentially what may be license limitations depending on what community license you're using; something you've not shared with us. I say the latter because unless you're using a Partner Community License, your users won't have access to any Accounts other than their own. This would prevent you from processing these requests from within your community.

Without knowing how the selections are made, your question is very difficult to answer or give you much guidance. There's also the question of how quickly the results need to be returned. Is it 24 hour turnaround, 1 hour, 15 minutes, 5 minutes or as close to real time as possible? How do you want to notify your community users that the request is complete? These are all considerations that become factors in creating a solution of this kind.

As to Batch and ForceTK, if you read the comments below the link you provided, near the end you'll see where @metadaddy said:

Unfortunately, external (community/portal) users do not have access to the Bulk API. See https://success.salesforce.com/ideaView?id=08730000000Dr21AAC.

What that means is this work needs to be sent to inside the boundaries of your org if you want to be able to process it asynchronously using Batch processes or queueables classes. Depending on your schema, you may be able to do that without using REST; most likely by creating a custom "wrapper object" to "mirror" the data into your org using a trigger. Otherwise, you'd probably need to do this using REST which will of course consume API requests.