Yes you can put IP addresses in there as well. We've been doing this for years.
Do note that salesforce does a 1:1 comparison between the domain name used in the callout and the Remote Site Setting. So you will have to use the IP address in the callout as well (and not the literal domain name).
External objects in Salesforce require an OData interface to the external data source. MySQL does not natively support OData, you need OData-speaking middleware that sits between SF and MySQL. There are commercial and OSS oData providers/libraries that help with this.
Pick an OData provider that will let you define an OData service endpoint that includes the ...