The cause of UNABLE_TO_LOCK_ROW error?

Question

My scenario is as followed, I have a single Parent account with about 600 child accounts. Every child account has a single Opportunity with 2 or more lines. I have a monthly batch job to create records in a Custom Object (Opp_Revenue__c) for each Opportunity Line Item. This batch runs for 100 oppportunities at a time. Opp_Revenue__c has a master-detail relationship to Account, the lines in that get created in the batch are all detail lines to the Parent account. However, also the Opportunities get updated (those are related to the child records).

Starting from last month, I got the UNABLE_TO_LOCK_ROW error. I only get the problem on a single iteration of the batch. The system is unable lock rows only of type account, but both the parent and about 60 child accounts are in the list.

The org in which this problem occurs has had Private sharing for accounts enabled from the start. There used to be a single Sharing rule, based on a custom text field on Account to share certain records with a community user. Since last month, some sharing rules were added to restrict access to certain public groups based on the accounts standard type field.

I'm 100% sure that no people were even logged in to Salesforce during the execution of the batch. There were no other Apex jobs or scheduled reports running at the same time. My batch does not call any future methods or queueable classes and does not schedule any other Apex job or batch.

My questions are; could the locking issue in a batch be caused by the addition of the sharing rules? Am I correct in the assumption that this problem can not be caused by interference between two iterations of the same batch? I have read a lot about the error and there could be a lot of causes for this, but almost none of the articles are about batches.

Answer 1

The issue is some combination of data skew and (potentially) ownership skew. Furthermore, sharing rules might lengthen the time of parent record locks, leading to increased chances for row locks.

Data Skew

Because all records of both objects are related to the (single) Parent Account, the update of each Opportunity and the insertion of each Opp_Revenue__c will briefly lock the Parent Account in order to maintain DB integrity and to recalculate sharing. There is a risk that an update to a child record will fail because a previous child record is still holding the lock on the Parent Account.

Remedy: Divide the single Parent Account into several and evenly distribute the child Accounts between them. Group the Opportunity batches by limiting the scope to one (grand)Parent account at a time.

SF Developer Blog: Data Skew Row Locks

Ownership Skew

If there is a default/"generic" owner for many/all of these records, and they have a nested (not top-level) role in the Role Hierarchy, sharing recalculations for Opportunities and Opp_Revenue__c will take longer to complete, increasing the chances of row locks.

Remedy: Either re-assign the Accounts to different individuals, remove the skewed owner's role, or place the skewed owner in their own role at the top of the hierarchy without child nodes. Also make sure any sharing rules used to grant visibility to skewed records are as restrictive as possible by granting access specifically to users who need it.

SF Developer Blog: Ownership Data Skew Row Locks

Sharing Rules

As stated off-handedly in the first section entitled "Problem Diagnosis" of this SF Developer Blog, "Lydia also doesn’t realize that the sharing configuration that she and Mike set up prior to the data load is further contributing to the poor loading performance which is then magnifying her lock contention problems." Although it doesn't mention sharing rules explicitly, it seems to make sense to me that including certain sharing rules that call for many sharing records to be created might lengthen the time of row locks.

Remedy: Change the OWD sharing defaults for the Account object, or grant "view all" or "modify all" permissions via permission sets or at the profile level for those users who need access to these records, if feasible.

Additional Resource: SF Record Locking Cheatsheet

NOTE: If you are using the Bulk API rather than batch apex, disable parallel loading.

It still strikes me as odd that row locks are an issue in your case, since common wisdom says this typically only becomes a skew issue when there are >10k child records or >10k records of the same object owned by a single owner...

Answer 2

You absolutely can get this error by having 2 successive iterations in a batch try to update the same group of parents in a hierarchy. This is one of the worst parts about having data skew and why there is 1 parent account in your error message.

The Opportunity and Opp_Revenue__c are causing a lock row in the Account hierarchy, since there is no guarantee you batch is segmented across 2 distinct Account Trees. This could even happen within 1 batch since you are updating 2 different objects that result in the same account hierarchy locking (in practice I find this much less likely).

A hackey way that might fix this is to order by account.parent in your soql query locator. Could also reduce your batch size. These will reduce the likelihood of locks but aren't true fixes. True fix would be to segment by distinct sets of accounts.

I have never seen sharing rows cause this issue.