5

We use a batched approach for read and update CampaignMember records within an external application (Webservice Connector, or WSC SOAP API).

The approach can be described like this:

  1. Query campaign members (result size is unkown - query bulk size is 2000).
  2. Process the bulk and update the members of this bulk immediately afterwards (update bulk size is 200 - results in up to 10 update calls).
  3. If the initial result contains more than 2000 objects, queryMore is executed for the initial query to fetch the next bulk. Then proceed at step 2 unless the result is completely processed.

The described process is used in a generic integration and worked for a few years without any problems for our customers. Since a few days ago we have one customer/organization that throws an InvalidQueryLocatorFault after the first bulk was updated and the query result contains more than 2000 objects (queryMore at step 3).

It seems that the update calls trigger some actions in salesforce that open query locators (limit is 10) that will make the QueryLocator of the initial query invalid during the update process.

We have tried to identify the action that will open the query cursors in this specific organization during the update, but we were not able identify the action that will open the query locators.

Components that we looked at are:

  • Triggers
  • Workflows
  • Flows

There is only one related Apex Trigger for the CampaignMember object. We also only see the execution of this trigger in the debug log of an update call.

Debug log of one update call:

26.0 APEX_CODE,FINEST;APEX_PROFILING,FINEST;CALLOUT,FINEST;DB,FINEST;SYSTEM,FINEST;VALIDATION,FINEST;VISUALFORCE,FINEST;WORKFLOW,FINEST
11:36:42.2 (2125816)|ENTERING_MANAGED_PKG|XXX
11:36:42.2 (62134253)|SOQL_EXECUTE_BEGIN|[15]|Aggregations:0|SELECT Id, DeveloperName FROM RecordType WHERE (DeveloperName = 'XXX' AND SObjectType = 'Campaign' AND IsActive = TRUE) LIMIT 1
11:36:42.2 (68739522)|SOQL_EXECUTE_END|[15]|Rows:1
11:36:42.2 (70073261)|SOQL_EXECUTE_BEGIN|[67]|Aggregations:0|SELECT Id, ParentId, Parent.ParentId FROM Campaign WHERE (Id = :tmpVar1 AND RecordTypeId = :tmpVar2 AND ParentId != NULL)
11:36:42.2 (76135866)|SOQL_EXECUTE_END|[67]|Rows:0
11:36:42.76 (76446324)|CUMULATIVE_LIMIT_USAGE
11:36:42.76 (76446324)|LIMIT_USAGE_FOR_NS|(default)|
  Number of SOQL queries: 0 out of 100
  Number of query rows: 0 out of 50000
  Number of SOSL queries: 0 out of 20
  Number of DML statements: 0 out of 150
  Number of DML rows: 0 out of 10000
  Maximum CPU time: 0 out of 219000
  Maximum heap size: 0 out of 6000000
  Number of callouts: 0 out of 100
  Number of Email Invocations: 0 out of 10
  Number of future calls: 0 out of 50
  Number of queueable jobs added to the queue: 0 out of 50
  Number of Mobile Apex push calls: 0 out of 10

11:36:42.76 (76446324)|LIMIT_USAGE_FOR_NS|XXX|
  Number of SOQL queries: 2 out of 100
  Number of query rows: 1 out of 50000
  Number of SOSL queries: 0 out of 20
  Number of DML statements: 0 out of 150
  Number of DML rows: 0 out of 10000
  Maximum CPU time: 0 out of 219000
  Maximum heap size: 0 out of 6000000
  Number of callouts: 0 out of 100
  Number of Email Invocations: 0 out of 10
  Number of future calls: 0 out of 50
  Number of queueable jobs added to the queue: 0 out of 50
  Number of Mobile Apex push calls: 0 out of 10

11:36:42.76 (76446324)|CUMULATIVE_LIMIT_USAGE_END

When I look at the documentation these queries should be fine in terms of query locators and should not open any query locators (no sub-queries used and result size should be under threshold). I also tried to reproduce the problem with an API sample tool, where I executed these queries additionally to the update call and they do not opened a query locator (at least when they are executed with the SOAP API).

Questions

Are there any other components/actions in salesforce that can open query locators during an update call (and are not shown in the debug logs of an update call)?

Are there any additional ways in salesforce to gather more runtime information's (about queries, query locator count, etc.) for the update call/transaction in question except the user debug log?

The problem occurs for our customer the first time shortly after the summer'16 release, are there any changes that can be related to this problem?

PS: We are aware that the approach we have taken is somewhat fragile in terms of the query locator limit for a user. But the past has shown that this limit is barely reached (this is the first time). For now we would still prefer this solution compared to loading the result with an unknown size in memory, that is in other terms also fragile (OOM).

3
  • What about callouts? Maybe you hit some external system and it tries to query via the REST API and that opens a query locator?
    – Adrian Larson
    Jul 13, 2016 at 13:05
  • @AdrianLarson That's a good point, thank you. What's the best way to find these callouts? They can be hidden in any APEX class, right? On the other hand if a callout is involved shouldn't it be listed in the debug log from the update transaction? Can we trust the debug logs or can there be cases that not all executed code from an transaction is included?
    – lgraf
    Jul 13, 2016 at 13:35
  • 2
    Never trust the debug logs to be complete.
    – Adrian Larson
    Jul 13, 2016 at 13:43

2 Answers 2

3

After a long support journey, it has been found out that there is internal salesforce code that get executed under specific conditions (depending on organization configuration), that can open a QueryLocator during the CampaignMember update. A bug was logged by the R&D team an they are working on them right now.

Nevertheless we have implemented a workaround for now to overcome the problem. We now completely avoid to open Querylocators for the initial query. Instead of using QueryMore we build multiple queries that are limited to return a maximum of 2000 objects.

To achieve this we use a slightly adjusted PK Chunking approach, as described here.

Update: The issue was fixed by the R&D Team.

1

Just one thing to check, with no guarantees it's anything but a dead-end. Take a look around for callouts. If you call out to an external system, it may try to query back data using the REST API. To search for such callouts, you would look for code that instantiates a new HttpRequest(). Outbound Messaging can also make such calls, so it's worth re-examining your Workflow Rules specifically to see if you send any Outbound Messages.

Even if you make a callout, it does not mean a query locator will be generated. But it is a possible source.

2
  • Because it is troublesome to go trough all the code and also there is a lot of managed code where I have no access to, I temporarily deactivated all remote sites (there are no named credentials). This should block all callouts, right?
    – lgraf
    Jul 14, 2016 at 9:49
  • No, try unit testing without a mock.
    – Adrian Larson
    Jul 14, 2016 at 12:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .