CumulusCI - Data Automation in pipeline?

Question

I recently starting evaluating CumulusCI to see if it meets our data automation needs. One use-case is populating new sandboxes with a bunch of foundational data (e.g. schools, academic majors, police departments, etc) - and it looks prefect for that. Another use-case is automating occasional changes to that foundational data. For example if we need to add a school to our data, a developer would add it (directly to a sandbox or a data file), and the pipeline would push that new record to other orgs.
What would be the best way to handle that scenario? Thinking the dev would create the record in an sfdc org, do a cci extract_dataset (configured so that each object has its own dataset), then the pipeline would run a load_dataset on the modified dataset's .sql file? Thanks for your time.

Answer 1

I am on the CumulusCI team.

One use-case is populating new sandboxes with a bunch of foundational data (e.g. schools, academic majors, police departments, etc) - and it looks prefect for that.

Yes, CumulusCI can do this for you easily. See Automate Data Operations in the CumulusCI documentation for details.

Another use-case is automating occasional changes to that foundational data. For example if we need to add a school to our data, a developer would add it (directly to a sandbox or a data file), and the pipeline would push that new record to other orgs.

CumulusCI does not, in the general case, support progressive changes to an existing data set in a persistent org.

You can absolutely load a data set into an org, modify it, and then capture your changes. However, to move these changes to another persistent org (as opposed to a new scratch org), you will typically want to delete the existing data from that org and then reload the entire data set fresh. CumulusCI does provide a task to automate this deletion, but note that it isn't selective - if your users have added to the data set, their additions are going to be wiped out too.

In general, CumulusCI assumes that it gets to manage the entire set of data for a given sObject in an org, and it assumes that every load operation is creating a new org from scratch via automation.

You can use our allied Snowfakery data generation tool to perform some progressive updates of your data set. However, the smoothest pathway is always going to be "remove and reload", not attempting to update an existing data set in a target org.

---

As a general principle, CumulusCI automation encourages externalizing the entire state of truth for your orgs into source control. Then, org creation becomes cheap, and you're encouraged to use fresh orgs frequently rather than attempting to maintain a persistent org. This ensures that you're always working with a known-good state. The principle of frequent org refreshes applies whether you are using scratch orgs, developer sandboxes, or developer edition orgs.