I am having few issues designing an architecture to use with Platform events and Microservices.
According to microservices principle(Kubernetes), there should be at least 3(odd number) services running so that it can handle the load properly. It works for callouts but for platform events it just gets bit tricky.
Here is our architecture :
Whenever some conditions, to send uploaded file to 3rd party for processing and get that data back in Salesforce. So flow goes like this
- Person X uploads a file(File can be big so can't do HTTP callouts to a 3rd party with file data)
- We fire a platform event with fileId(content version id)
- The 3rd party microservice listens to it and fetches that file from SF rest endpoint and then processes and updates its data back in SF
Now the problem is in microservice. There can be multiple instances of it listening and follows the independent processing architecture so that they can work without worrying what's going on in other instances.
So, in order to stop single point of failure there will be at least 3 services running,(Even more if peak load) and they listen to the same event and act independently. This is problematic as all 3(or evn more) will process and respond same data to SF.
We can't let a single microservice to listen to all events as it will be again a single point of failure. Also, we cannot pop an event out of the event bus as its an event bus and not an event queue.
Has anyone faced this and what possible solution can be built around it?