We have setup Google login via OAuth for our Salesforce (Not SAML) and authentication via our custom domain page works fine. But as soon as we try to restrict our users to login only from our domain page (In Setup->Domain Management->My Domain->Login Policy: Prevent login from https://login.salesforce.com), this method for authentication fails when initiated from our custom domain, with the following error:
Problem Logging In
We can’t log you in because of an authentication error. For help, contact your Salesforce administrator.
When I go and visit the login history I notice the following entry associated with this failed login attempt:
Login Type: Third Party SSO
Status: Restricted Domain
As soon as I loosen that restriction, login via Google starts working again.
I've followed this guide to setup SSO: https://help.salesforce.com/articleView?id=sso_provider_google.htm&type=5
I've used this code for my registration handler: https://github.com/trineo/sfdc-rego-handler/blob/master/src/classes/GoogleAppsRegistrationHandler.cls