6

I have the requirement to store and retrieve the information that a set of arbitrary sobject records are "dirty" and need to be touch by asynch processes.

I am talking about 10.000 to 100.000 records that I want to mark by calling a method. I also need a method that tells me if a record XYZ is dirty.

Setting an isDirty field on thousands of record would consume multiple Limits, so that's not an option.

I want to persist the information about a set of records that need to be processed later.

Another idea is to store comma-delimited Id list in multiple long text fields, but the only way to search an Id in such fields seems to be Sosl which takes a while to index.

I am out of Ideas?

EDIT: I need a solution which works instantly (synchronous) without any batch. I know that this cannot be done by really modifying all records. That's why I thought about Large Text fields storing Ids and SOSL.

6 Answers 6

8

As the "mark dirty" process, can you consider submitting a Batch Job using a Query Locator? It permits up to 50,000,000 records.

public class DirtyJob implements Database.Batchable<sObject> {

    public Database.QueryLocator start(Database.BatchableContext context) {
        return Database.getQueryLocator('SELECT Id FROM Filthy__c');
    }

    public void execute(Database.BatchableContext context, List<Filthy__c> scopes) {
        for (Filthy__c filthy : scope) {
            filthy.IsDirty__c = true;
        }
        update scopes;
    }

    public void finish(Database.BatchableContext context) {}
}

If one sets the maximum scope size of 2,000 it can power through them alright:

Database.executeBatch(new DirtyJob(), 2000);
3
  • 1
    Sorry that's not an answer. I know Batch Apex and know that it's good to process hundreds of thousands of records, but I want to persist the information that a set of records need to be processed later. Feb 20, 2014 at 22:38
  • Ahhhh! That could be a very long and skinny table indeed @RobertSösemann Feb 20, 2014 at 22:40
  • Please see the clarification of my question. Feb 24, 2014 at 8:40
6

This sounds like a job for batch apex. Batch apex runs asynchronolously based on available resources and will chunk up your data per batch that i'll process. You should be good upto 50.000.000 records. Else you may want to export and do it externally, or use multiple batch iterations/processes depending on your specific scenario.

Use the Database.QueryLocator object when you are using a simple query (SELECT) to generate the scope of objects used in the batch job. If you use a querylocator object, the governor limit for the total number of records retrieved by SOQL queries is bypassed. For example, a batch Apex job for the Account object can return a QueryLocator for all account records (up to 50 million records) in an organization. Another example is a sharing recalculation for the Contact object that returns a QueryLocator for all account records in an organization.

5
  • Sorry that's not an answer. I know Batch Apex and know that it's good to process hundreds of thousands of records, but I want to persist the information that a set of records need to be processed later. Feb 20, 2014 at 22:37
  • @RobertSösemann Why not store the flag that a record needs to be processed on the record in question? Feb 20, 2014 at 22:38
  • 1
    Could you then please clarify you question more clearly? Setting a flag using batch is precisely a quite common strategy. I thought you question was HOW to set that flag on a high quantity of data. (100k is not a lot tbh) Feb 20, 2014 at 22:40
  • Please see the clarification of my question. Feb 24, 2014 at 8:41
  • I'm sorry, I still find your question not very clear, can you explain why you want such an alternative solution or what the true constrains are you are facing ? Feb 25, 2014 at 21:54
6

Batch Apex is absolutely the way to go for marking records as dirty and for the subsequent processing. When it comes to flagging them you could either set a specific field on each record, such as a checkbox, or potentially use a second custom object. In that object you'd create an entry for every record that needs processing (with a lookup to the first object), indicating it's current status etc..

One advantage with using a second object for this is you could even create a 3rd object, called "Process Job" which is a master to the first custom object. That way you could maintain batches of work to do, using rollup fields to get some basic analytics on processing progress, failures etc.

2
  • Please see the clarification of my question. Feb 24, 2014 at 8:41
  • @RobertSösemann, I'm afraid there's no way you can process that many records without using batch apex... you're limited by DML to 10,000 updates/inserts which means you'll never be able to mark all the dirty records you want.
    – Matt Lacey
    Feb 24, 2014 at 22:48
5

Sounds like you want to persist 10,000 to 100,000 IDs in the database without actually doing a DML operation on 10,000 - 100,000 records.

Well, since an ID is 18 characters, and long text fields are only 32768 - that is 32768 / (18 + 1 [for a delimiter]) = 1724 IDs per long text field - so you'd need 59 instances of a custom object called DirtyBoxCar__c for 100,000 IDs.

Your async process would read all instances of DirtyBoxCar__c and explode into a List using String class split() method. Then you can process the actual DML using traditional batch job techniques.

If all the SObjects to be dirtied were of the same type, you could strip out the first three and last three characters; adding the first three back in your async process - this would mean you could store 2521 IDs per record and need only 40 boxcar records.

You would also have to do a giant concat in order to save the data (or use JSON serialize which will occupy more space). This might run out of CPU time - but you could test this using anonymous APEX.

I'm surmising that the list of IDs to stage for async dirtying are fetched using a SOQL call so rather than loop through the results, using JSON serialize into a JSON array might be faster. Only experimentation will let you know. Maybe you will need to distribute the staging work to multiple @future methods.

I suppose you could also use a Blob datatype and save in an Attachment or ContentVersion SObject - this would generally avoid the need for multiple Boxcars records

3
  • Not a bad idea but the String.split() likes to die with "REGEX too complicated" error. We might need more car(t)s with less payload ;)
    – eyescream
    Feb 25, 2014 at 21:48
  • ah yes, I've seen that 'regex too complicated' message. Tx for reminding me. The Json serialize should work better (although I dont know about hitting cpu limits) - heap space becomes an issue too -- ["<someid>","<someid2>"...] as JSON will take up 21 chars * 100,000 + 2 = 2,100,002 bytes which is a heap issue. Hence farming the list of ids out to 10 future jobs that do the json serialize into a ContentVersion or Attachment blob. I'm sure some other limit will get hit and the whole solution is hack-y in a big way but it was thought provoking
    – cropredy
    Feb 26, 2014 at 0:09
  • long text fields now 128K
    – cropredy
    Jun 3, 2015 at 19:10
2

How about @future? It still won't be truly synchronous but could happen somewhat faster & more reliable than a batch execution. Batch can take a while to run the initial query, fail if there are 5 batches already running, get aborted if the query took over 2 minutes to run...

You could have up to 10 calls of @future method in 1 transaction and each will get its own 10K DML limit...

http://www.salesforce.com/us/developer/docs/apexcode/Content/apex_gov_limits.htm

4
  • That was my first thought too. I didn't post that answer simply because he said he doesn't want it to be asynchronous.
    – dphil
    Feb 24, 2014 at 21:49
  • there is no guarantee @future will be processed faster than kicking of a batch process is there ? Feb 25, 2014 at 21:56
  • @SamuelDeRycke I've used "faster" reluctantly and I've upvoted the batch-related answers as they're definitely logically cleaner way to do it that can scale as much as we need... Then again all the stuff I've written out should be considered too. In the end - both are async. And the fact the sync way was requested tells me there will be trouble with scheduling multiple batches from different transactions that might clash into each other...
    – eyescream
    Feb 25, 2014 at 22:15
  • @SamuelDeRycke they really do execute faster in my experience - Batch has shitty days where you wait 4... 8... minutes for it to progress from 'Queued' to 'Processing' Feb 25, 2014 at 22:32
0

Personally the use of a Batch Job is your best approach, and this is one I would take. But I do not know the extend of your project / issue. Another approach, is to create a 'queue' table to hold which records are marked as dirty, and you can reprocess them. Maybe have a flag if the record which is considered dirty because it has been created has been processed already. You'll be able to run reports by using this type of logic as well. To process all of your new generated records, I would still suggest running a batch job.

This senerio is just an alternative suggestion to what everyone else has suggested. I like @user320's suggestion better.

tl;dr; Account record is considered dirty -> create queue record on custom object. Batch job run -> process all records which are in queue table and either deletes them or marks them processed.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .