3

I've seen a few threads on this subject, but they're pretty old. Thought I'd reach out and see if there's there were any developments on this subject.

We have a highly complex data structure (500+ objects, multiple dependencies), and trying to move away from developing in a full/partial sandbox, to developing in personal developer sandboxes. However, we need a small subset of data to work with. That also includes many custom settings which of course are being created empty in dev sandbox after refresh, and our code highly relies on their data (too many to try to move them to MDT, would be a long project).

We don't want to have to move every piece of half baked code to full sandbox just for testing during development, that misses the point of developing in personal sandboxes in my opinion.

Has anyone ever tried to tackle this challenge? What did you do? Any thoughts and recommendations would be highly appreciated!

Thanks

1
5

Disclosure: I am the author of this free and open source project.

I had exactly the same problem, so I wrote Amaxa. It is an open source, command line data loader built in Python, whose raison d'etre is extracting self-consistent data subsets from a production Salesforce org and loading them into sandboxes.

Features that specifically target this use case:

  • Select a set of data from Production by specifying top-level objects, such as one or several Accounts, and let Amaxa follow relationships to extract a complete, self-consistent set of related records.
  • Load data set in a single operation, rebuilding all relationships including reference cycles and hierarchies.
  • Store data in version control as readable, diffable CSV files.
  • Emphasis on correctness and validation (never load bad data).

The one thing that I'm not sure about is your Custom Settings. Amaxa doesn't have functionality that specifically targets Custom Settings, and I'm not entirely sure off the cuff how they'll behave if you include them in your operation definition. However, sObject data is well supported, regardless of the size or complexity of your data model.

Also worth a look is CumulusCI (I am on the CumulusCI team at Salesforce.org). CumulusCI is primarily targeted at scratch orgs, not sandboxes, but also has sophisticated tools for data seeding, and explicitly supports Custom Settings with a special task. It doesn't have the same data subsetting features as Amaxa, however, due to its focus on scratch orgs.

1
  • 1
    Dude your tool is amazing! It even worked with custom settings! I only tested it in a very small scale with sObjects so more complex testing is required, if it does it's just awesome. Great job, and thanks so much for letting me know! – EranV Aug 12 '20 at 23:27
1

Salesforce DX is tailor-suited for this method of development, except you would work in Scratch Orgs instead of Developer Edition orgs. This lets you set up temporary orgs, deploy whatever subsets of metadata you want to, and even deploy a limited set of related data (force:data:tree:import) drawn from data you've built (force:data:tree:export).

Once you've set up the scripts you'd need, you can spin up a new org in just a few minutes with minimal fuss. No waiting in long server queues for a new Sandbox to appear, no need to move full sets of metadata just to add a feature to one specific functionality you've built, etc. Other solutions are indeed out there, but if you haven't looked at DX yet, you may want to take a look. It also comes with a Trailhead to get started quickly.

4
  • Yep, I've had my fair share of experimenting with sfdx. But how would you go about exporting and importing a very complex structure of data? force:data:tree:export requires a specified SOQL query, it would be a pretty huge project to perform all these exports, even for a one time effort. – EranV Aug 12 '20 at 19:33
  • @EranV Honestly, "one step at a time." Any moderately big org isn't going to be converted to any new development model, period. There's no such thing as a fully automated conversion where you just click a button and you're done. Tools can get you part way, but I've never seen a fully automated system that "just works" with all possible scenarios. – sfdcfox Aug 12 '20 at 19:54
  • Gotcha. And then of course, it can never be just code, right? Scratch orgs come completely empty, and even the most basic metadata like objects and fields would need to be pushed to make them usable, correct? – EranV Aug 12 '20 at 20:09
  • 1
    @EranV Correct. I recommend a three step process. Deploy all fields/objects without any "extras" (no validation rules, etc), then deploy data, then finally code, validation rules,etc. DX makes this pretty easy by allowing multiple folders to be used as deployments. Although this does increase the complexity a bit, I've found it incredibly useful so far in our DX migration. – sfdcfox Aug 12 '20 at 20:14
0

I have created Forceea, an open-source data factory, which can easily generate data of very complex data structures for both real data (e.g. in a sandbox or scratch org) or testing data. It works synchronously and asynchronously, if you want to create thousands (or even millions) of records.

Forceea uses Dadela data generation language, which makes it very flexible and powerful. Have a look at the User Guide (you can download it from https://nmitrakis.com/Forceea-UserGuide).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.