0

We have a "dirty" leads custom object.

It has around 1000 records. We will be getting similar files which we will load into dirty leads object every once in a while.

Our lead object has around 800k records.

I would like to create a dedup process that checks whether the incoming records in the dirty lead custom objects already exist in Lead.

My problem here is how to design code for this process.

How am I supposed to store the 800k records in memory as I am sure I will hit the heap limit...then what's the best way to achieve this ?

Would appreciate any help on this

Improve this question

1 Answer 1

Reset to default
1

Don't read them all in memory, but do specific queries to see if one or more of the 800k record matches the 1000 dirty lead records.

The searching should be in the database, and less in code. What I would do is to make the Database search as specific as possible (not necessarily 100% specific, but the results should be duplicate candidates), and then for the results that you get back, loop over them in code and do the 100% check.

For the database, the LIKE operator might come in useful. See https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_select.htm and https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_select_conditionexpression.htm

Improve this answer
2
  • So I can design a batch process with batch size of say 80 records... And I will need to issue one SOQL per record so that comes to 80 Soql per batch... Right?..,I am assuming that the governor limit is 100 soql per batch...am I right?
    – Varun
    Sep 25, 2015 at 12:31
  • Yes, if you can't query per multiple records (which of course is more efficient, working with OR in the SOQL query), then batch them up in chunks small enough that you won't hit governor limits. Oh, and yes, 100 SOQL queries is the limit per execution.
    – Willem Mulder
    Sep 25, 2015 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.