In Visualforce I need to ASYNChronously do 999 (n) multi-table queries and upsert all results (asynchronously, and presumably in chunks), to 1 custom_object__c, all from a visualforce page. The page would have a few elements reserved for reporting job/chunk progress and error(s) of the async chain. Would queueable or batch apex be ideal? Note there's potential for many users in an Org to use the page, and thus enqueue large volumes of similar async queries/upserts. What design would support having the most concurrent/parallel async jobs (many users can view the page init these async processes)
Scenario and Design Constraints:
Suppose that a custom Visualforce Page allows a user to define many SOQL statements in a
List<String> soqlStrings. Suppose
soqlStrings.size()is 999, or any number larger than the Governor Limit (200) for "Total number of SOQL queries issued" in "Asynchronous" contexts.
On page load, the visualforce application must begin the work of querying all strings in
soqlStrings(in this case 999 soql statements), and after each chunk is processed it must Upsert() the query(s) results into a single custom_object__c
While the asynchronous process is doing the aforementioned work of querying and upserting, the visualforce page must show a "% Processed" progress element, SPECIFIC to the chunk being processed
finally, after the asynchronous process finishes, the visualforce page must hide the same progress element
soqlStrings.size() can be larger than 200 and the strings can be different for different users.
The init code for firing the necessary async job(s) is intended for a managed package
Additional design thoughts: I think that *"Future Methods"** are not suitable here because it seems they are for "fire and forget" use-cases, whereas I need visualforce "aware" of the async progress
If I learn more, I will update the question.