You cannot detect these failures in a safe manner purely in Apex Code. Attempting to give a StandardSetController an unsupported type causes an immediate halt of the program. Even using a try-catch block will result in the same immediate failure, even if the list is determined dynamically using a query string (e.g. Database.query). I suppose you could get a global describe of all SObjects in your organization, then query them 100 at a time using Visualforce remoting, catching those exceptions client-side and generating a list. However, I daresay that such a program would probably take a while to run.
As Itai's answer points out, however, you do know the list of standard supported types; if it's not in the documentation, you should presume it is not supported, even if your code seems to work with that type. Custom objects that use "__c" as the suffix should all be safe, as well as the standard objects listed. Other types of related metadata, like "__Share" are also not supported.
All that said, you can choose to use your own custom pagination. It's not as handy as using a StandardSetController, because each time you query the database, you might get a different subset as records are added/removed from the database.