This is certainly possible. You just need to build the map yourself instead of trying to create it form a a single query.
As an example lets say we have a custom lookup on the contact record to another contact, say 'Emergency Contact'. Its possible that a person is listed as the emergency contact for more 1 person, so we could create a map where the key is the contact id, and the value is actually a list of contacts in which they are the emergency contact for.
Obviously this is a bit different from your case, but the same idea and you should easily be able to adjust the code for your scenario.
Map<Id, List<Contact>> conMap = new Map<Id, List<Contact>>();
for(Contact parent : [Select Id, Name, (Select Id, Name From EmergencyContacts__r) From Contact]) {
conMap.put(parent.Id, new list<Contact>());
for(Contact child : parent.EmergencyContacts__r){
conMap.get(parent.Id).add(child);
}
}
system.debug('My Map: ' + conMap);
You now have a map where a contact Id is the key, and a list of contacts that person is the emergency contact for is the value.
Hope that helps
EDIT
Thinking about it more, you could likely make trim this down a bit more and just do this as well.
Map<Id, List<Contact>> conMap = new Map<Id, List<Contact>>();
for(Contact parent : [Select Id, Name, (Select Id, Name From EmergencyContacts) From Contact]) {
conMap.put(parent.Id, parent.EmergencyContacts__r);
}
system.debug('My Map: ' + conMap);