We allow our Salesforce customers to login to our own Salesforce Service Community by using OAuth ("Login with Salesforce"). To do so, we created in our customer's org a Connected App, set the Callback URL correctly and entered the Client ID/Secret on our side in an Auth. Provider. Everything works smoothly when logging in.
But now I realized that even users from other orgs (without having the Connected App set up!) can "Login with Salesforce".
This means for us, that we need to implement a user validation mechanism in our Registration Handler, to "filter" those who are allowed to login and those who are not.
Until now, I expected that this would be done during communication between Auth. Provider and Connected App.
Is there any way to prevent users to login without having the Connected App (+the correct Client ID/Secret)?