5

Fairly new to Salesforce and come from a non-SFDC enterprise development background. Using FinancialForce framework and enterprise patterns in our development.

Question as it relates to best practice for development given the following:

Requirement Example - A Purchase Order must have at least 1 item and the item must have at least 1 delivery schedule and the SUM(DeliverySchedule.Quantity) must equal the Item.Quantity.

Details - In order to be able to support the above, all data must be "insert"ed at the same time within a single transaction so all UI development is custom VF. The challenge is that if there are more than 200 Delivery Schedule records for an Item, OnBefore/ValidationRules/OnAfter fires in blocks of 200. In this case, we can't validate yet because there is more data coming and we would fail our "SUM(DeliverySchedule.Quantity)=Item.Quantity rule.

In SFDC, if you "insert List" for 1000 records, you're validation rule will fire 5 times (if I'm understanding how it works properly). Also, as soon as you insert the header, it's validation rule would fire and it would fail because it doesn't have an item yet.

In thinking through the possible solutions, a couple of ideas come to mind but none of which I really like and they all have one shortcoming or another.

Ideas are:

1) Extra field on each SObject that gets "toggled" by API layer, then upsert, then toggle back. This will fire Validation rules multiple times but the last time the validation will be enforced. This has unnecessary extra DML statements, multiple "execution process" iterations, requires an extra field and lots of coordination of the flipping of that field by the API layer.

2) Static variable (request context specific as I understand it) which gets flipped, ValidationRules in APEX onBefore and check static variable. This has downside of transaction coordination across objects that have rollups so it gets messy and likely wouldn't work.

The only way I can think to do this is to have a true Service API layer that locks records, processes in-memory (including keeping track of rollup values, etc.), calling "IsValid()" on each SObject in unitOfwork and then rollback/commit. The downside to this approach is that we lose a lot of the stock features of SFDC and/or third-party packages because everything MUST go through our API to save. There's also a data integrity risk if someone didn't go through our API (e.g. rogue user using REST API, Unsuspecting Admin using Standard Layout page, etc.). We can protect against the data integrity with some of the concepts in #1 or #2 above but it's not ideal.

Question - What are the best practices for development/architecture in these scenarios on SFDC platform and using FinancialForce?

Thank you for any assistance, greatly appreciated!

-jd

1

You can't generally enforce "rows of child data on insert" in salesforce.com. What I usually recommend is that a particular record can't progress to another stage until the conditions are satisfied. For example, you might have a Status field on the purchase order. This might be one of "Preparing", "Ready to Fulfill", etc. At that point, it becomes trivial to key off of the status field. This is essentially (#1), as you've described, except that it's intentional, and the user can use the UI to enter as much data as they want until the stage is advanced, in which case the record must be in a sane state. The most usual cases for this type of behavior includes opportunities, orders, and quotes, in order to add product line items before attempting to move them to a more defined phase. And, no, your static variable idea generally won't work for several reasons, including the fact that validation rules can't see Apex variables, except custom settings, which would bleed across transactions.

  • Thanks sfdcfox, appreciate your insight. Unfortunately, using a "workflow" type of approach won't meet the apps requirements. It's needs to be an all or nothing scenario. An example would be when "editing" an existing Delivery Schedule. The user might want to increase one record and decrease another. The validation must be enforced across both changes. For less than 200, onBefore/ValidationRules/etc. would be OK but we have data cases where there could be more than 200 edits at a time. Thoughts? – Jon Davis Jun 25 '15 at 22:57
  • I agree with sfdcfox here, the trouble with triggers is they act in isolation from each others logic, by design. There are some validations you can do (for example preventing the last child from being deleted), but not all (insisting that a master has a child on insertion). As you say an API 'transaction' can handle this validation, but that breaks your data integrity through other routes, in this use case. So i agree with this answer, its a life cycle design solution. Sorry if thats not what you where hoping for though. – Andrew Fawcett Jun 26 '15 at 9:40
  • Thanks Andrew. Coming from a non-SFDC background, this seems like a shortcoming of the platform. Essentially you need to allow invalid data in to your system and then build a business process around handling that invalid data. In the case of FF, where you have an existing invoice with 300 distributions, if the user or batch process needs to change 201 distributions and the business rule states that the SUM(distributions.total) = Invoice total, how do you handle statuses and what does the business process look like since it's already an "existing" invoice? Appreciate your time and thoughts! – Jon Davis Jun 26 '15 at 14:01
  • Sorry I'm not at liberty to get into details on internal FF application implementation here, i hope you understand. As i said, i'm in support of this answer from sfdxfox as being the basis for a general approach to such things on this platform. – Andrew Fawcett Jun 26 '15 at 15:23
  • Totally understand regarding not being able to share internal FF details. Generally speaking, can you or sfdcfox recommend how I could apply the "stage" concept to an existing record? It works easily for records that do not yet have any activity. However, let's say you have a Purchase Order Item that has been partially received and now need to change the quantities. What's the best practice for handling invalid data since I can't take the entire PO or even the item itself back to a previous stage? Thanks! – Jon Davis Jun 26 '15 at 15:40
1

I was able to develop a solution for this scenario which "delays" final evaluation of validation until all "DML" operations are complete.

Understanding that this solution is not the traditional approach to development on the SFDC platform, unfortunately we can not change our business requirements and therefore must find a way to meet them (or leave the platform which isn't an ideal choice either).

Given that we must find a way, would appreciate anyone's thoughts on trying to find gaps, improvements, etc. in the below approach.

Requirement Summary - Validations are dependent on child/related data and DML operations will occur on more than 200 objects per SObject and therefore "final" validation must occur only after all DML operations are complete. For example, an order must have at least 1 item and an item quantity must be exactly equal to the SUM of it's delivery schedule quantities (there could be several hundred of these).

Approach

  1. Implement a TransactionManager that contains a static variable to track the state of a transaction (NOTSTARTED, INPROGRESS, VALIDATING, COMPLETE).
  2. Service API method starts a new transaction
  3. DML operations occur as normal. During DML processing, trigger code (it's actually in fflib common) immediately validates data that is not dependent on child data (this helps fail as soon as possible). For data that is dependent, if a Transaction is not detected, it immediately validates that as well (this provides support for standard layouts, etc.) If one is in progress, it registers with TransactionManager with a list of any errors that it contains on that data (TrxMgr.markValid / markInvalid) instead of calling addError.
  4. Once all DML is complete, Service API calls commitWork() on TransactionManager
  5. TransactionManager fires off any post-processing checks (using fflib IDoWork concept). Note that no DML is being performed here, just "validation" checks. For now, this is simple as checking an in-memory map of Id to List of errors.
  6. If commitWork fails, it throws exception which is caught by Service API and handled "gracefully" to user. If no errors occur in commitWork, transaction completes and everything is committed. When errors do occur, we can't call addError because we are out of Trigger context so for now, this is just a list of messages (not ideal). Thoughts on how to improve this welcomed! One idea I had was to force another DML on the faulty SOBject which would enable addError to be used but then there are extra unnecessary DML statements and with our record amounts, governors are something we need to be mindful of (yes- we are failing anyway, but if we hit DML limit we fail with DML error instead of the real error)

There are some variants of this approach that I've also POC'd but the above is the simplest concept to explain the approach. One of the variants involved an optimization to delay the execution of "rollups" to the post-process stage (to reduce DML statement and record limits) prior to validation using the Declarative Lookup Summary Tool (thanks Andrew!) in "Developer" mode.

Appreciate any thoughts, holes identified, feedback, etc. Thank you!

1

You mentioned that you have to have all custom code and custom Visualforce. My suggestion would be to abandon Validation Rules. Instead do all of the transactional validation within the page. The concept would be 1. Database.Savepoint -> take a snapshot of the database 2. Do your code, save the header, save the child records in batches of 200, do whatever logic you need etc. 3. Call a validation routine - validate that you have enough child records or that the sum isn't greater than the total, etc. 4. If validation fails, Database.Rollback

Alternatively if you can NOT save in step #2 and still accomplish your logic, then do your validation BEFORE you call Save. But I'm thinking that with your data volumes you may need to have pagination and you may be saving as you go? So maybe you can't run the validation before you save everything.

This is fine because Database.Rollback will allow you to maintain transactional integrity.

Personally I would advise my customer to try to avoid insert integrity and instead insert in status "Draft" and then update to status "Final" and do validations then. If needed you could delete the draft, but I find customers like the ability to save partial records. Let's them come back later and make adjustments before "committing". Even if integration is involved, you just don't integrate until "Final".

Hope this helps, Caleb

  • Thanks for your time and thoughts Caleb. The approach I've come up (see my post below in this thread for details) with is essentially exactly what you describe (save point, process, validate). The challenge is that the validation logic is extremely complex (much more than just the simple scenario I provided in the op). Given the complexity, each service API (page) would need to be intimately aware of every possible validation that would need to be done even if it didn't directly "touch" the record. Keeping the validation in the domain layer allows us to abstract the validation. – Jon Davis Jul 1 '15 at 3:54
  • Contd... for "inserts" we can save as draft and if users doesn't mark "permanent" just abandon record and clean-up later (not ideal but would work). "Existing" records are the issue since we can't make a "valid" item invalid and leave it that way for a user to fix later. Definitely agree that "draft/invalid" concept is recommended approach in this case, users "do not want to manage drafts" and leaving invalid data in the DB would have disastrous consequences to the business. Regarding pre-validate, the risk here is data integrity. We can "check before" and everything might look good but – Jon Davis Jul 1 '15 at 4:02
  • Contd... a concurrent transaction could slip in after validation and before commit which could result in the data being invalid but it would get committed. Validation in onAfter (ideal because of domain specific validations) or service layer (after DML before commit - not ideal because of validation complexity) seems to be the only way to ensure data integrity. Any additional thoughts are greatly appreciated! - happy to provide more details if it helps. – Jon Davis Jul 1 '15 at 4:06
  • While a series of validation rules is certainly easier to maintain than those same rules in some Apex code. It should be possible to create separate Apex class whose sole job is validations. You could abstract it more using interfaces/abstract classes to try to keep future updates/enhancements more flexible, but I'm simple minded on this - single class, easy for developer to read and modify later. While the validations may certainly be complex and "additive" if you really want a transactional system I don't see much of an alternative. You could – Caleb Sidel Jul 2 '15 at 4:23
  • 1
    Thanks Caleb. The approach I put together essentially follows the principles of that pattern by using a static variable to track what "stage" of processing we're in and the triggers conditionally do things based on that. – Jon Davis Jul 7 '15 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.