We have a "dirty" leads custom object.
It has around 1000 records. We will be getting similar files which we will load into dirty leads object every once in a while.
Our lead object has around 800k records.
I would like to create a dedup process that checks whether the incoming records in the dirty lead custom objects already exist in Lead.
My problem here is how to design code for this process.
How am I supposed to store the 800k records in memory as I am sure I will hit the heap limit...then what's the best way to achieve this ?
Would appreciate any help on this