0

Currently, I utilize an AWS endpoint that returns a small JSON response based on my request.

I am considering updating all of my records (300,000+) daily.

My AWS counterparts can generate a JSON file of all records in S3 which I could then GET through a rest call at which point I am thinking I could parse the response with batch apex. However, I am concerned I will run into heap size, CPU processing, transaction or other governor limits.

The other approach I have considered is having a client access Salesforce through the Bulk API and upsert the records in this way.

What is the best practice for this many records?

I have read through the update mass records documentation: https://help.salesforce.com/articleView?id=000267082&language=en_US&type=1

Improve this question

1 Answer 1

Reset to default
2

You'd want to use the Bulk API. This would give you 30 API calls to upload 300,000 records. Of course, this needs to be in CSV rather than JSON, but it would be the most efficient method of all the choices you probably have. Otherwise, you'd need some sort of API that can use pagination if you wanted to process these records in Apex. It would be less efficient, but might be easier to code overall. It really depends on what kind of developer resources you have.

Improve this answer
4
  • That's what I was thinking based on the documentation, but our Signature Success rep was leaning towards reaching out to the endpoint and trying to minimize the response. Let's say we still want to update all 300,000 records, is it possible to do schedule a Bulk API job from within SF. For example, S3 stores a .CSV with all 300,000 records, could I reach out with a REST call get the file and schedule a Bulk API job within the Apex Class? Or some other roundabout way to accomplish this? Ideally I would like to avoid using creating client to do the daily scheduled upload.
    – S.B.
    Oct 2, 2018 at 1:21
  • @S.B. No, the volume is entirely too large to do in one pass in Apex. If you want to do it, you need pagination of some sort, and by that point, you could use JSON to parse the records if you wanted to. However, It will be much easier to do it from the AWS side, e.g. an Elastic Beanstalk or something, because you will have fewer limits and much better performance.
    – sfdcfox
    Oct 2, 2018 at 1:32
  • I took look of the following Java client: developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/…, It seems I can use an external Id field which is good. If so, how would duplicates be handled / does the external id field have to be unique? Ex: Unique_Identifier__c is ABC on two contacts and I want to update both contacts with the same information in the CSV where Unique_Identifier__c is ABC. If I'm getting off-topic, let me know and I can open a new thread.
    – S.B.
    Oct 2, 2018 at 14:22
  • 1
    @S.B. The external ID must be unique, or you will get errors during uploads. If you need to deal with this scenario, you'll be forced with running a query to download all of the Salesforce records, then match them up, and then deploy a bulk API call. At some point, you're going to have to compromise somewhere. Either get pagination for your Apex, or write a client.
    – sfdcfox
    Oct 2, 2018 at 14:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .