In Visualforce I need to ASYNChronously do 999 (n) multi-table queries and upsert all results (asynchronously, and presumably in chunks), to 1 custom_object__c, all from a visualforce page. The page would have a few elements reserved for reporting job/chunk progress and error(s) of the async chain. Would queueable or batch apex be ideal? Note there's potential for many users in an Org to use the page, and thus enqueue large volumes of similar async queries/upserts. What design would support having the most concurrent/parallel async jobs (many users can view the page init these async processes)

Scenario and Design Constraints:

  • Suppose that a custom Visualforce Page allows a user to define many SOQL statements in a List<String> soqlStrings. Suppose soqlStrings.size() is 999, or any number larger than the Governor Limit (200) for "Total number of SOQL queries issued" in "Asynchronous" contexts.

  • On page load, the visualforce application must begin the work of querying all strings in soqlStrings (in this case 999 soql statements), and after each chunk is processed it must Upsert() the query(s) results into a single custom_object__c

  • While the asynchronous process is doing the aforementioned work of querying and upserting, the visualforce page must show a "% Processed" progress element, SPECIFIC to the chunk being processed

  • finally, after the asynchronous process finishes, the visualforce page must hide the same progress element

  • soqlStrings.size() can be larger than 200 and the strings can be different for different users.

  • The init code for firing the necessary async job(s) is intended for a managed package

Additional design thoughts: I think that *"Future Methods"** are not suitable here because it seems they are for "fire and forget" use-cases, whereas I need visualforce "aware" of the async progress

If I learn more, I will update the question.

Thank you!

  • 2
    What is your requirement from business point of view? It seems to be weird from user point of view to see spinner where you are firing 999 soql queries synchronously – Santanu Boral Mar 6 at 19:27
  • 2
    Daniel Peter put out a neat solution that he called HyperBatch that you may be able to adapt to your needs here. (presentation link included in the repo as well) github.com/danieljpeter/HyperBatch – Mark Pond Mar 6 at 19:30
  • 1
    As a user, I'd rather see a table with a "status" column than for the page to be unusable. I can't tell if the page has frozen or lagging, or it a specific job has failed. – battery.cord Mar 6 at 19:31
  • Thanks Mark, I will check out HyperBatch. Thanks Battery , your comment about seeing feedback for specific jobs has helped me narrow my search, which led me to a post about monitoring batch jobs and displaying progress in visualforce using action poller salesforce.stackexchange.com/a/17575/10073 . – Peter Noges Mar 6 at 20:43

I did something similar to this by using a custom setting - at the beginning of your process insert a custom setting record, then update that custom setting with each async batch that processes successfully. I used a number field on the cs definition to basically keep count of how many records had been processed at a certain point in the process. In your vf page use an action poller to query that cs record and get new # of records processed.

Not the answer you're looking for? Browse other questions tagged or ask your own question.