Though I’m working on Salesforce platform for some time but I’m still new to webservices and reading a lot about it recently.
I’m working independently and recently my customer( little tech Savvy) asked me to use web services to fetch approx. 4-5 K(thousands) from one of their other system ,daily into his SFDC org. He said , he can provide me WSDL that I should use in some Batch process running once per day.
This left me confused. As far my understanding from POCs I’ve done, Call outs doesn't seem like ideal choice.. when you've to pick thousands of records in response.
If they want to insert/update data in SFDC, then better choice would be , they should consume SFDC enterprise WSDL in their system to make changes in SFDC. Isn’t it ?
SO in nutshell , I’ve these questions :
Would it be fine choice to to use WSDL to call that external system and in response get that many records and make changes in SFDC ?
If above is fine, I’m thinking, I won’t seem logical to get all data in one call. What kind of issues I could face if volume in response is large?
As far what I got, I should be recommending using bulk API(in future, volume could be more) to get such large amount of data. But what I know, I could use data loader or some Middleware (informatica etc) with Bulk API. But customer doesn’t have budget for middleware.
Is data loader ideal choice in that scenario? Customer doesn’t want that any manual step. I know Data loader CLI is also used to automate loading of data. But what I information do I need to ask from customer? would that be direct database connection ?(JDBC /ODBC , etc.)
What If customer doesn’t agree to Data loader, what is the next best choice?
Lastly Customer is particular about security. What kind of security/authentication mechanism should I recommend to securely pick data from that system?
I know , it is getting long, Please provide any suggestions or some directions atleast, as I’m not getting much help just by reading articles/forums.
Thanks for all the help. It would be greatly appreciated.