0

The project we're working on has a robust CI/CD pipeline (Circle CI) using scratch orgs for development Git branches. If approved, Pull Requests from dev branches are merged into an integration branch, which kicks off an automated deploy to a scratch org and then all unit tests are run. Also, all dev branches pushed to remote dev branches trigger a similar automated deployment to a scratch org (without merge).

The project will soon add Salesforce Data Pipelines (sub-set of Tableau CRM / CRM Analytics), which will connect to External Data Sources and push Salesforce data to an external system on a daily schedule.

My question: Do you know of best practices / lessons / documentation for SFDX deployment of Salesforce Data Pipelines and / or its associated Salesforce Metadata types of waveapplication, wavedataset, waverecipe, etc? In particular, I'm interested to see if there are limitations / issues for these "wave" metadata types when it comes to converting to MDAPI format and other things which may impact the CI/CD process.

I'm unable to find much documentation on this area, so --> Thanks in advance.

1 Answer 1

1

UPDATE: We started out with trying to do this using a scratch org flow, and so far so good. There are some small tweaks needed in the project-scratch-def.json file and it has been helpful to install the SFDX CLI Analytics plugin.

In the scratch-def.json file, in "features" entries, added 2 items:

  • DevelopmentWave
  • AnalyticsAdminPerms

Another step which must be accomplished before doing 'sfdx force source push' is to 'enable Analytics' in the scratch org. To do this, the CI/CD script needs to do 2 additional steps:

  1. Install the SFDX Analytics plugin: sfdx plugins:install @salesforce/analytics Install the Analytics CLI Plugin
  2. Enable Analytics using the Analytics plugin: sfdx analytics:enable

After getting Tableau CRM / Analytics set up in the org, we have been able to successfully sfdx force source push which included a simple Analytics WaveDataFlow to the scratch org!

Here are some best practices and guidelines I was able to dig up from some additional searches - they are mostly applicable to the org development model (instead of the SFDX Source Development Model with Scratch orgs which we used above).

  1. Note that Data Pipelines are not directly supported in SFDX. To deploy Data Pipelines, you must deploy their associated metadata types like waveapplication, wavedataset, waverecipe, etc.

    To deploy Data Pipelines using SFDX, you include the metadata types in the package.xml file and deploy them using the force:source:deploy command. Alternatively, use the sfdx force:mdapi:convert command to convert the metadata to MDAPI format and then deploy them using the force:mdapi:deploy command.

  2. Use version control for Data Pipelines, as it allows you to track changes to the pipeline over time and roll back to previous versions if needed. Use Git to version control your Data Pipelines just like other metadata types.

  3. Converting the "wave" metadata types to MDAPI format, there are some considerations to keep in mind. For example, some "wave" metadata types like wavedataset and waverecipe have complex JSON structures that can be difficult to work with when using MDAPI. Additionally, some features that are available in the UI for creating and editing Data Pipelines may not be available through the metadata API.

  4. A possible approach is to create a separate CI/CD pipeline specifically for deploying and testing Data Pipelines. This can help to isolate any issues that arise with Data Pipelines from the main CI/CD pipeline.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .