You may have performed callout not to a canonical URL but to Visualforce domain or some other domain. You can actually perform successfully self-callouts without any RemoteSiteSettings, but you need to get the correct Domain URL by calling getBaseURL()
method
/**
* Determines the true API hostname for a Salesforce org using the Identity API.
* eg 'https://pod.salesforce.com' (most orgs)
* eg 'https://custom.my.salesforce.com' (my domain)
* eg 'https://custom--dev.sandbox.my.salesforce.com' (sandbox orgs)
*/
public static String getBaseURL() {
String orgId = UserInfo.getOrganizationId();
String userId = UserInfo.getUserId();
String sessionId = getLexSid();
//use getSalesforceBaseUrl within batches and schedules (not Visualforce), and fix inconsistent protocol
if (sessionId == null) {
return Url.getSalesforceBaseUrl().toExternalForm().replace('http:', 'https:');
}
PageReference pr = new PageReference('/id/' + orgId + '/' + userId);
pr.getParameters().put('oauth_token', sessionId);
pr.getParameters().put('format', 'json');
//within test context use url class, else derive from identity api
String data = Test.isRunningTest() ? '{"urls": {"rest": "' + Url.getSalesforceBaseUrl().toExternalForm() + '"}}' : pr.getContent().toString();
Url rest = new Url(new Parser(data).readString('urls.rest').replace('{version}', '55'));
return rest.getProtocol() + '://' + rest.getHost();
}
Once you know the correct domain (https://custom.my.salesforce.com or https://custom--dev.sandbox.my.salesforce.com), you may perform Metadata API calls to create Connected Apps or Remote Site Settings to different sites or Visualforce domain or Lightning domain.