2

It feels like groundhog day. As a Salesforce Partner building AppExchange apps one application aspect keeps repeating and the custom solutions I build just don't satisfy my need for reuse.

It's about synchronizing data between Salesforce and a custom REST API. So on user interaction or a schedule data is eighter updated /deleted/ created:

  1. in the external System using REST PUT and POST based on the state in Salesforce OR
  2. in Salesforce using REST GET

Every solution contains those ingredients:

  • Apex classes modeling REST resources
  • Apex that calls endpoint URLs
  • Code or Metadata-stored Mapping between API objects and SObject

AND always a ton of clunky code to wire that together. Isn't there a smarter way to do that? less clunky, more flexible?

On my search, I found some overly complicated standards as OData (that most of my customers don't support) or expensive Salesforce solutions like Apex Connector Framework over Salesforce Connect (that nobody wants to pay for).

Come on, there must be a smart and lean and flexible library out that there that does 80% of that plumbing.

5
  • I think, unfortunately, Salesforce would point at Connect as the solution here.
    – nbrown
    Sep 17 '20 at 19:54
  • 3
    You are on the right track, re: standard and that standard is OpenAPI. If the target supports it, you can auto-gen the client. On SF side External Services is where the product team is investing resources first, e.g. Winter 21 pilot. Clicks, not code. The DIY equivalent is openapi-generator. It supports Apex but it does need some love.
    – identigral
    Sep 17 '20 at 20:01
  • @nbrown look at how clunky and ugly the Apex code looks you need to write for Apex Connector Framework. I even wouldn't want that when it was for free: developer.salesforce.com/docs/atlas.en-us.apexcode.meta/… Sep 17 '20 at 20:03
  • @identigral External Services looked good but they dont move. And why Flow? I need Apex. Why know APIs? I need the APIs of my customers. Do you have sample code for Salesforce using OpenAPI generator? Would like to see how much love it needs. Sep 17 '20 at 20:06
  • 3
    Flow or Ext Svcs + Flow is not targeting developers. You already know why - most of SF users aren't devs with skills to do API-based integration. openapi-gen ships with samples
    – identigral
    Sep 17 '20 at 20:24
8

Sorry in advance for a long answer, but this topic is near and dear to my heart and I think I can provide some interesting context/perspective.

I am the founder of Valence (https://valence.app), a native AppExchange app written entirely in Apex that is designed to be exactly the sort of library/engine/framework you are wishing existed.

It originated from the same experience you describe above. My team and I were building integrations over and over and over that were 80% the same every time (scheduled movement, mapping layer, transformations, error recovery, delta syncs, etc, etc). We were rather confused about why a starter library didn't exist in Apex to facilitate projects like the ones we were doing.

So, we decided to stop being consultants and started a new ISV company that would develop and support such a framework. I'll share a bit of our journey and some of the design decisions; I hope that you find it informative.

I can tell you from several years of developing such a framework that the devil is in the details. Though many integrations seem quite similar, in fact there are so many small details that cause them to be slightly different from each other. We have put thousands of developer hours into Valence, and there is still so much more we could do.

Many REST APIs are not implemented as "true" REST, but something inspired by REST and a little unique.

To even begin to tackle this complexity I think you have to create some kind of plugin system so that you can encapsulate strange endpoints and their unique needs with as much universality as you can.

https://docs.valence.app/en/latest/concepts/extensions.html

We decided to support plugins for both external systems that can be talked to, and transformations that can be applied to "records in flight". Each plugin is just an Apex class that implements one or more Apex interfaces, where the interfaces dictate expected behavior and the engine exercises the plugins through those interfaces.

Now you have to tackle the issue of systems that are at odds with Salesforce in how they treat and move data. What happens if you have a REST API you call that returns more than 10,000 records, the limit of a single Apex execution context? Now you have to do some kind of storage / caching / allocation system, or pagination, etc. Like a duck that is moving through the water gracefully but paddling its feet quickly underwater, a huge amount of the work that has gone into Valence has been making Salesforce execution limits irrelevant to the plugin implementers. You can give Valence more than 10k records in a single call and it will handle it. You can explode the running execution context with an exception (even an uncatchable one) and Valence will recover. This sort of resiliency is probably one of the most important aspects of an enterprise-grade integration.

So you've solved for the basic mechanics of record movement, now you get into schema, mapping, and transformation work. Someone mentioned OpenAPI in the comments; I think Swagger/OpenAPI is a great way to define expectations. So are JSON Schema, and WSDLs, and all sorts of other specifications designed to lay out a contract for message shape. They are important components of a much larger whole. Beyond the shape itself you want to have your bindings into records in Salesforce, and of course data transformations that apply in both directions.

And don't forget about endpoints that don't have machine-readable (or even human-readable) documentation about their schemas.

Or, a personal favorite of mine, endpoints that HAVE a machine-readable spec like JSON Schema and it's just...incorrect. (Think that doesn't happen? It does.)

We tackled this lack of conformity by having a very fluid definition of schema. The addressable space for mappings/transformations is the superset of what an endpoint told us was available, and what was actually seen when records started to flow. We also have an idea of "stale" schema so that over time as things change (and they always do, don't they?) we can try to help users be proactive.

Ok, I'll wrap up by getting out of the weeds a little and talking about vision.

I started Valence to help people skip boilerplate stuff when building integrations in Salesforce. We have three types of customers: people buy it to use in their own orgs, they buy it as consultants to use with their clients, and they buy it as app vendors to use as part of the plumbing/infrastructure of their app so they don't have to deal with it.

It is my ambition to build an ecosystem of plugins, both open-source and otherwise, to begin to tame the madness of this huge problem space with integrations and Salesforce, and make it more and more common for there to be something quick and easy to use as a jumping off point (or perhaps the entire project).

Because of Salesforce's natural ISV packaging extension system, paired with the fact that we've used Custom Metadata Types for all of our extension/configuration records, people can develop "kits" for Valence that essentially pre-package:

  1. An Apex class that knows how to talk to a certain system or endpoint
  2. Apex classes that know how to apply transformations to records as they go by
  3. Configuration for data flows, including mappings, transformations, etc
  4. Anything else they like, perhaps a new SObject that is the target of certain data flows, etc

Integrations with Salesforce are hard. There's no easy button, and one team of developers chasing all the variance of how they exist in the wild is an impossible task.

Best to get the core, must-have stuff 100% solved with grace and resiliency, and then get the heck out of the way with a good plugin architecture to let people work their magic.

5
  • 2
    I forgot to mention that I wrote a JSON parser for Apex that we use inside Valence, that I open-sourced: github.com/open-force/jsonparse It really helped reduce how much code we were writing to break down REST responses.
    – Grekker
    Sep 18 '20 at 3:33
  • 1
    This looks super useful. Sep 18 '20 at 4:31
  • 1
    Thanks @grekker that's a very useful answer. Yes even as being someone on the business / architecture side of integrations one API is not like the other. Salesforce's implementation of odata is not complete and it can't handle many aspects of odata that other devs build, so just chucking money at Salesforce is not the answer. External Services is not good enough either, for similar reasons, it just doesn't handle enough of the swagger schema to be useful.
    – JodieM
    Sep 18 '20 at 5:07
  • So if you do not have free reign to modify the external API to work the way Salesforce works then you are always going to be manipulating your code to tweak it to exactly the way the external API works. It's a nightmare to even get it to the point of understanding it to spec it so it can be developed, let alone the nuances of actually developing it. From the little I've seen of Valence it really does address all these issues and the next time I'm involved in an integration project I will start there.
    – JodieM
    Sep 18 '20 at 5:07
  • 1
    @Grekker Great answer Chuck and I marked it as "the" answer. It doesn't solve my problem at hand but I kills my hope to find a quick and cheap solution. Thanks for your insights. When I stumble over a customer how is willing to make a Valenced backed app I will reach out. Sep 18 '20 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.