Is there any best practice or configs available to set API call quotas per Connected App so that Salesforce API call limit isn't exceeded?
No. You can run reports and determine which clients are using the most resources, but there is no way to enforce quotas to prevent this from happening. Note that API usage limits roll off hourly, so fixing/blocking an offending client will ensure that your org isn't down for very long.
You can implement a "quota" on the client (caller) side by looking at
Sforce-Limit-Info header that is included by SF in REST and SOAP API responses. From docs:
api-usage — Specifies the API usage for the organization against which the call was made in the format nn/nnnn. The first number is the number of API calls used, and the second number is the API limit for the organization