For creating a user account - you can call an out of the box REST API, it sounds like you're walking down this path. While you can (for example) create a single User object (record) in one REST call via an out of the box API, more than one record of one or more object type may need to be created for a user to be fully provisioned. You can use a Composite REST resource to combine multiple REST calls to out of the box APIs into one REST call. Alternatively you can create your own REST resource by developing the REST service/endpoint in Apex.
Once you have the user account created, you can implement single sign-on. There are two major, interoperable and industry-standard options: SAML or OpenID Connect. Since you aren't going to be rolling your own client for either option, this comes down to your comfort level with one of these and your application / development stack supporting your chosen approach.
Last but not least, you can combine user account provisioning with SSO via SAML by using Just-in-Time (JIT) provisioning. In this option you can use the payload of a SAML assertion to create your users on the fly the first time they try to log in from an identity provider. This eliminates the need to create user accounts in advance. The con of this approach is that you'll be limited to a very simple user account data model and minimal error handling. It might be good enough to start.
