I have a requirement to build a lightning component to allow users upload csv files. These files have 100k rows and multiple users can upload large files at the same time. It's possible to have 15 files uploaded at the same time each with 100k rows. I need to have an apex class that processes these files and creates either leads or contacts from the rows based on some business logic. My first thought was to run an apex batch to process the files, but only up to 5 queued or active batch jobs are allowed. So if I run an apex batch for each file, up to 5 file will be processed but if the 6th one fires while the first 5 batches are active or in queue, the 6th one will fail. Any thoughts or recommendations? Thanks in advance!
This is not something that should be done in Apex, and it most likely can't be.
If your file contains 100,000 rows, and each row contains just 60 characters, you've already hit the string length limit of 6 million characters. Even in asynchronous context, there's an extremely high probability you would also hit the heap limit.
Plus, Apex is really bad at parsing CSV data.
I recommend you do one of two things:
Note that your user would have to remain on the page throughout the import process, something likely to be quite painful at 1.5 million imported records (100,000 * 15 files).
- Utilize an enterprise ETL middleware solution that can talk to the Bulk API, and ideally offers you deduplication or in which you can build deduplication and match/upsert logic. If you're really proposing to insert 1.5 million records into your database, per user, on a regular basis, you need to be thinking in a much more holistic way about your data intake and management strategy, and utilizing tools that are designed to support that type of data flow.
5 queued or active batch jobs are allowed
This is not accurate; you can have 100 jobs in the Apex Flex Queue. That does not change the fact that this is architecturally a non-starter.