"Invalid redirect_uri" on OauthKickoffUrl of Custom AuthProvider

Question

The following Apex code uses Metadata and SOAP API to create a NamedCredential and Custom Auth Provider to connect to a public REST API that is authenticated via OAuth.

public override PageReference run() {
    createAuthProvider();
    createNamedCredential();

    return oAuthKickoffUrl();
}

private void createAuthProvider() {
    createSObject('<urn1:type>AuthProvider</urn1:type>' +
        '<urn1:DeveloperName>' + DEV_NAME + '</urn1:DeveloperName>' +
        '<urn1:FriendlyName>' + DEV_NAME + '</urn1:FriendlyName>' +
        '<urn1:ProviderType>OpenIdConnect</urn1:ProviderType>' +
        '<urn1:OptionsSendAccessTokenInHeader>true</urn1:OptionsSendAccessTokenInHeader>' +
        '<urn1:OptionsSendClientCredentialsInHeader>false</urn1:OptionsSendClientCredentialsInHeader>' +
        '<urn1:ConsumerKey>' + CLIENT_KEY + '</urn1:ConsumerKey>' +
        '<urn1:ConsumerSecret>' + CLIENT_SECRET + '</urn1:ConsumerSecret>' +
        '<urn1:AuthorizeUrl>' + API_ENDPOINT + '/oauth/authorize</urn1:AuthorizeUrl>' +
        '<urn1:TokenUrl>' + API_ENDPOINT + '/api/v1/oauth/token/</urn1:TokenUrl>'
    );
}

private void createNamedCredential() {
    MetadataService.NamedCredential cred = new MetadataService.NamedCredential();
    cred.fullName = DEV_NAME;
    cred.label = DEV_NAME;
    cred.allowMergeFieldsInBody = false;
    cred.allowMergeFieldsInHeader = false;
    cred.authProvider = DEV_NAME;
    cred.generateAuthorizationHeader = true;
    cred.principalType = 'NamedUser';
    cred.protocol = 'Oauth';
    cred.authProvider = DEV_NAME;
    cred.endpoint = API_ENDPOINT + '/v3';

    create(cred);
}

private PageReference oAuthKickoffUrl() {
    PageReference result = new PageReference(authProvider.OauthKickoffUrl);

    String startUrl = '/' + String.valueOf(namedCredential.Id).left(15);
    result.getParameters().put('startURL', startUrl);
    
    return result;
}

Everything is created properly and looks ok but the Kickoff Call fails with a page of the API provider saying

Error: invalid_request: Invalid "redirect_uri" in request.

    https://provider.com/oauth/authorize
    ?response_type=code
    &client_id=bJE3456345njoH77n5hdGYMV3a
    &redirect_uri=https%3A%2F%2Ftrusted-saltedcaramel-234-dev-ed.cs163.my.salesforce.com
    %2Fservices%2Fauthcallback%2FProvider_Integration&state=CAAAAX94cbTCMDAwM

(I have used this approach successfully with many other APIs without such issues.)

Answer 1

Note: Even if it is a bit embarrassing that this question just showed my own misunderstanding of OAuth, let me add as an answer how we resolved the issue.

Everything on the Salesforce side was correct but our assumption that the External Service Provider could just accept arbitrary Callback URL was plain wrong.

Even if some tools allow wildcards on Redirect URIs this is super insecure and should not be done.

We now register every new client / Salesforce subscriber at the External Provider first so they can know the exact Redirect UI in advance.