I am a developer for a Salesforce org that serves as a community portal for thousands of users. We have been consulting with a large team of developers for years and everything we have is custom coded. Our pages are built using AngularJS to remotely call Apex code. We are on an Unlimited Edition org and we are nearing 90% of our total allowed code. We have a lot of callouts to external services and many times these callouts can run long.
We are now running into issues where users are seeing the "Unable to Process Request - Concurrent requests limit exceeded" error. The issue is there is so much code written by so many different developers it is hard to identify what/where the problematic code is. Sometimes users see this error when they are trying to perform a callout. Sometimes it's when they are inserting a record and triggers are firing. The long-running code issue can be anywhere in our org. Another issue is error handling has been done very poorly, so if any process does run long, we most likely are not looking for it or logging it.
Other than enhancing the logging on our callouts, I do not know where to even begin to find other problematic code. Every developer before me has written code with no regards to limits, so inefficient code can be anywhere. Is there any way or best practice to being able to analyze such a complex org and find where the problems may be? All I can think of is enhanced logging EVERYWHERE, but that may not be the best approach and would be very time consuming, especially considering how many developers we have constantly writing and pushing new code.
Any help would be greatly appreciated. I am not familiar with the majority of the code/functionality of our org, so I do not know how to really solve this issue considering I don't even know what could be problematic aside from long running callouts. Thanks!