2

To migrate "trees of related data" from an external system to Salesforce I am using dynamic chains of Apex Queueables. Imagine I am "synchronizing" Accounts, Opportunities and Contacts from an external CRM into Salesforce.

There is a separate Queueable class for each object type and to stay within limits each Queueable is just migrating a certain amount of objects. This is how a typical flow could look like:

  1. AccountQueueable: Get 10 Accounts
  2. ContactQueueable: Get 1000 Contacts of that 10 Accounts
  3. OpportunityQueueable: Get 200 Opptys from Contacts and Accounts in 1./2.

  4. AccountQueueable: Rerun for next 10 accounts

  5. ContactQueueable: Get related Contacts ...you get the scheme

This works with 3 related object types but if I get more I am unable to run even the smallest scenario in my dev org because I hit the limit document here https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_queueing_jobs.htm

For Developer Edition and Trial organizations, the maximum stack depth for chained jobs is 5, which means that you can chain jobs four times and the maximum number of jobs in the chain is 5, including the initial parent queueable job.

I can't use Batch as I am not working on a single object. I also can't increase the limit as Salesforce told me that's a hard limit of dev orgs.

Maybe my overall approach is wrong?! What would you do here?

  • 1
    If I understand this right, you have custom queueable classes for data synchronization purposes? If that's right, have you explored the options of ETL tools? I understand that it comes with a price, but with an upfront investment, at least you will not end up with some complex logic hitting limits at times which becomes difficult to troubleshoot in future. You will end up a lesser capex vs. opex if you opt ETL. – Jayant Das Mar 8 at 15:07
  • I fully agree. This is legacy code... Any recommendations on how to start small. I know there is Mulesoft and I guess it costs a fortune. What route do you recommend for me to start small ETL wise without being locked in by a vendor to early. – Robert Sösemann Mar 8 at 15:10
  • 2
    I use Talend, it's free and has great sf connector. It also provides you java /python code as well for the transformation you did so you can just plug that code anywhere if needed. Bang on product – Pranay Jaiswal Mar 8 at 15:13
  • I have worked with Informatica and that's quite useful too. There are others viz., MuleSoft available in the market. And in your situation, I think the best will be to invest a bit on researching the best suited for your use case and then take the final call. I think there's always a trial version available for most of the products. – Jayant Das Mar 8 at 15:17
  • 1
    I agree with Pranay about Talend. It is free and we have been using it from last 4 years to pump data in our dev sandboxes from prod. I can’t think of anything we couldn’t achieve with free version of Talend. – javanoob Mar 9 at 18:39
3

I'd say use a batchable class. What you need is a dynamic approach. Even though you're working with multiple objects, a batch class can still be used here. Here's a design pattern for you:

public class DynamicBatch implements Database.Batchable<batchAction>, Database.Stateful {
  class StateInfo {
    public Account[] accounts = new Account[0];
    public Contact[] contacts = new Contact[0];
    public Opportunity[] opps = new Opportunity[0];
    // ...
  }
  StateInfo state = new StateInfo();
  interface batchAction {
    void execute(StateInfo state) {
  }
  class AccountAction implements BatchAction {
    void execute(StateInfo state) {
      // ...
    }
  }
  class ContactAction implements BatchAction {
    void execute(StateInfo state) {
      // ...
    }
  }
  class OpportunityAction implements BatchAction {
    void execute(StateInfo state) {
      // ...
    }
  }
  public batchAction[] start(Database.BatchableContext context) {
    return new batchAction[] { new AccountAction(), new ContactAction(), new OpportunityAction() };
  }
  public void execute(Database.BatchableContext context, batchAction[] scope) {
    scope[0].execute(state);
  }
  public void finish(Database.BatchableContext context) {
    if(!finished()) {
      Database.executeBatch(new DynamicBatch());
    }
  }
  // ...
}

You can adjust this as you like, but hopefully you get the general idea. This batch class is called with a scope size of 1. This behaves like an unkillable Queueable and can be chained indefinitely, unlike Queueable calls. This also avoids "hacks" like swapping back and forth between future/queueable or some other design.

  • Awesome! As always! – Robert Sösemann Mar 8 at 15:56
  • Let me try that out. I leave the question open as I am sure I will have some follow up questions. – Robert Sösemann Mar 8 at 16:01
  • Sorry, maybe it's already a bit late but I still don't get it. Imagine I am going to tranfer Accounts, Contact and Opptys from an external CRM (same structure and dependencies as in SFDC). Accounts are my dependency-less objects and I am moving over 10.000 accounts with related Contacts and Opptys. As I understand your code the batch runs on generic jobs where I triple consist of Account, Contact and Opptys. Could you assume its moving from one org to the other and add some code to showcase what happens where?! Or do you have a Github repo for this ? – Robert Sösemann Mar 8 at 22:06
  • 1
    @RobertSösemann Each interface-implementing class would perform the appropriate callouts (don't forget Database.AllowsCallouts), then insert/update/upsert whatever based on criteria, then go on the net step. I suppose I could write in a more comprehensive edit if you'd like. – sfdcfox Mar 8 at 22:14
  • I’d spend you a beer or two next Dreamforce. Maybe in a github gist?! – Robert Sösemann Mar 8 at 23:55
1

If you run a Batchable from Account, the execute method could query all the necessary child records and act accordingly. If the transaction limits would get busted there, spawn a Queueable from the batch. You won't reach maximum stack depth unless any one Queueable launched by the batch passes 5 deep.

1

Well, I have a hack

We cant call future from future, but we can call Future from a Queueable and Queueable from future.

So from the 5th Queuable call the future, and then that Future can call another Queueable to have an infinite chain in Developer orgs.

Edit: I did a demo of recursion, calling Future from Queuable and from Queuable call future, I was able to chain over 400+ levels deep before my Daily Async Apex Limit ended.

  • Aren't there similar limitations regarding futures in Dev Orgs? – Robert Sösemann Mar 8 at 14:42
  • 1
    We cant call future from future, so thats there, You can call 1 Queuable from Future. Thats what you need to restart your chain. No more than 0 in batch and future contexts; 1 in queueable context method calls per Apex invocation – Pranay Jaiswal Mar 8 at 14:44
  • Sounds great. What's the drawback. Or where does this start to be hacky? – Robert Sösemann Mar 8 at 15:01
  • 1
    The future method does not return enqueued job id, thus you lose track of what's going on. I cant think of anything else at the moment. I made an engine which would go 8 level deep for some data import work like you, didnt disappoint me. – Pranay Jaiswal Mar 8 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.