0

As it stands currently, I have created a scheduled apex job that is supposed to run nightly and compare records being imported into a 'staging' object against records already in custom objects, and either creating or modifying these custom object records depending on what it finds in the system already. The process works as intended, and runs for any amount of records X that is passed into the method.

The problem that I am finding, is that it does not handle the sheer number of record necessary, as we would like for this class to be able to process around 1000 records nightly, but currently it can only handle around 300. I have looked through my code and optimized/bulkified it as much as possible, but when I look in the logs I am finding that huge amounts of time are being wasted running validation rules over and over on objects relative to the time spent in the code I have written, as we have several validation rules and a couple trigger calls on basically every object being touched by this process.

We have a pattern that I see repeating in the logs several time when using the Database.insert function that essentially - inserts 100 records, run a custom object validation rule over all records, fire a workflow over all 100 records, fire account validation rules over 100 records, fire account trigger over 100 records.

This then repeats until all of the custom object records are created/updated.

This takes the bulk of my Apex CPU time limit to the point that it times out for any number of records significantly higher than 300, which is quite obviously a far cry from 1,000.

Someone suggested that I look into the Bulk API, but having never used it before or had anyone knowledgeable to introduce me to it, I am hesitant to get too far into redesigning it for the bulk api until I know that it will both solve the problem I am currently having and not run into any others.

In the Salesforce documentation, it says that the bulk api 'provides a programmatic option to quickly load your org’s data into Salesforce.'. However, since the process I have designed is far more complicated than simply inserting data and needs to compare a record against the entire system's records at a time, it is not clear to me if this would be a good use case for the bulk api. I am under the impression that the bulk api is more meant for inserting or updating static records of absurdly high volumes, and it would be difficult or even impossible to leverage it to run through complicated logic of comparing and taking specific action based on differences between records.

Given that my problem is CPU Apex time caused by the insertion and not the size or number of insertions themselves, does this seem like a good use case for the bulk api? If not, does anyone have any suggestions as to how we may get around this CPU time issue in order to be able to process that number of records a day??

I am happy to provide some or all of the code if requested, but it is quite a large and complex code base, so I felt it would be easier to describe the problem and see if I could get an answer just from that. If code would help to answer, please let me know and I will provide. Otherwise if there is anything that needs clarified please let me know, I realize this is a bit long winded and hard to follow, so happy to help clear things up if I can.

  • On Top of head, Bulk API is used to load data in bulk, I dont think it can solve CPU timeout issues. – Pranay Jaiswal Jan 22 at 21:14
  • CPU timeout limit and Bulk API are unrelated. You utilize Bulk API to perform operations in bulk. The CPU timeout issues are mostly code problems which deviate from best practices thus leading to that scenario. If you are looking to solve CPU limit issues, this is a good starting point -- Code more efficiently to avoid 'Apex CPU time limit exceeded' – Jayant Das Jan 22 at 21:16
  • Bulk API won't help. You need to convert your code to batchable/queueable so you can do your processing in smaller pieces. – sfdcfox Jan 22 at 21:17
  • Appreciate the feedback and confirming what I thought everyone. @Jayant Das I am fairly confident that the code is not the problem, as I have gone through with one of our more experienced apex developers and confirmed that it is following all the best practices and about as optimismed as we can make it just using the base library. it seems to be the time spend on validation rules and firing other triggers and work flows after insertion that are causing problems, as those apparently still count towards the limit and our org has abused them a bit. – Frank Evers Jan 22 at 21:19
  • 1
    If you’re think your data is clean you can disable validation rules to stop them for running. You can also customize this for certain user using hierarchy custom setting – Pranay Jaiswal Jan 22 at 22:08
3

I don't think Bulk API is your answer.

Your scheduled job should invoke batch apex. Try batch size of 1.

If you are hitting CPU limit (or other governor limit(s) ) with batch size of 1, you have bigger issues... you must rework and optimize your "downstream" work (e.g. object validation rule over all records, fire a workflow over all 100 records, fire account validation rules over 100 records, fire account trigger over 100 records, etc).

  • Thanks for the feedback @Krigi! i will say, batch sounds a bit more like what I will need to use to make this work. Just a couple bits of follow up, Is a batch size of 1 literally a single record size for the batch? and how does the batch processing differ from the Database.DML class methods which I was under the impression were already batching by themselves in sets of 200? – Frank Evers Jan 22 at 21:24
  • @Frank Evers batch apex works like this: Step 1) define the SCOPE of records to process (this can be millions of records, up to 50 million assuming your start method returns a QueryLocator), Step 2) iterate over SCOPE in batches of 1-200 records AT A TIME. You specify the size of each batch, e.g. 1 to 200... each iteration gets a new set of governor limits, 3) a single finish method is fired... you can do stuff, post-process, etc. read up on docs here: developer.salesforce.com/docs/atlas.en-us.apexcode.meta/… – krigi Jan 22 at 21:28
  • also batch size of 1 is just your starting point, to prove out that you can process a single record without blowing up governor limits. You get that... if you cannot process a single record without blowing up governor limits... your issue is not your processing the records, your issue is your downstream mess. Definitely FINE TUNE batch size, but start at 1 just to tell yourself "ahh, nice, ok, I can do this once... how about I bump to 10? to 100? If throughput is ever an issue (i.e. you NEED a batch size of 200) than you WILL need to refactor your downstream work to be more efficient. – krigi Jan 22 at 21:32
  • Excellent thanks for the references. I am going to read through the documentation and maybe run through the trailhead to see if I can get this figured out. – Frank Evers Jan 22 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.