Casting collections is a bit different than casting objects of non-collection types.
Based on the Collection Casting documentation, what you're trying to do with your map should likely work. However, most examples of casting that you'll find are examples of up-casting.
The following works just fine
Map<Id, Account> acctMap = new Map<Id, Account>();
Map<Id, SObject> sobjMap = (Map<Id, SObject>)acctMap;
What you're trying to do here, however, is down-casting. Down-casting is a bit of an oddity. It can work, but the requirements are more strict than with up-casting.
The following example will work
Map<Id, Account> acctMap1 = new Map<Id, Account>();
acctMap1.put('001000000000000AAA', new Account(Id = '001000000000000AAA'));
Map<Id, SObject> sobjMap = (Map<Id, Account>)acctMap1;
Map<Id, Account> acctMap2 = (Map<Id, Account>)sobjMap;
If you start with a map of the correct type, you can up-cast it, and then later down-cast it. Why this works and your example doesn't is something I'm not well-equipped to answer. Like Phil W mentioned, that's something that you'd need to ask the people at Salesforce that maintain the Apex language about.
I did some investigation on why we can't up-cast Sets, and managed to come up with a plausible explanation with type-erasure in Java, but I'm not sure if that applies to this situation as well.
instanceOf isn't really reliable when working with collections, see sfdcfox's answer on this question for more details.
Keeping track of collection types in your head is one option. Another option is to use the
getSObjectType() method of the List and Map collection types. It doesn't seem to work with
Map<Id, SObject> though...