RemoveAll on set of records before insert doesn't remove matching records in set of existing records

Question

I've run into an issue comparing sets of records to insert vs sets of records that already exist as the set of existing records contains a record ID and therefore the removeAll method doesnt see a match

I've got these two sets in my code, both of which only have one element in my test. The elements have the same values, except the existing record has an Id.

cgmembers: a set of all existing collaborationgroupmember records

{CollaborationGroupMember:{CollaborationGroupId=0F9L00000005hpDKAQ, Id=0FBL0000000gDtWOAU, MemberId=00540000001CkxPAAS}}

cgUsersToInsertSet: set of records I want to insert. This set also has one element (without an Id, as they are not inserted yet)

{CollaborationGroupMember:{MemberId=00540000001CkxPAAS, CollaborationGroupId=0F9L00000005hpDKAQ}}

I need to remove the existing collaborationGroupMembers from the set of records to insert before inserting them, else I get a duplicate record error. Using two sets and RemoveAll seemed like a good approach...until it didnt work.

So how can I do this efficiently?

Is there a way to populate the set of existing records without the ID field?
Or if not whats the most efficient way to do the comparison?

Below is a snippet of the code that shows how I'm populating the cgmembers set (the existing collaborationGroupMembers, which has the ID field)

     Map<String, CollaborationGroup> chatterGroups = new Map<String, CollaborationGroup>();
           FOR (CollaborationGroup chatterGroup : 
           [SELECT Name, (SELECT CollaborationGroupId, MemberId FROM GroupMembers) FROM CollaborationGroup WHERE Name IN :chatterGroupNames]) 
           chatterGroups.put(chatterGroup.Name, chatterGroup);


            FOR (PermissionSet_Map__c psmap : psmaps){

                string cg = psmap.collaborationgroup__c;

                //cgx = current chatter group in loop
                CollaborationGroup cgx = ChatterGroups.get(cg);

                cgmembers.addAll(cgx.GroupMembers)

Thank you!

EDIT - using a FOR loop to create sObject without ID field

so far the best I've been able to do is to use a FOR loop to create a collaborationGroupMember sObject without the ID field. Seems redundant, hopefully there is a better way.

set <collaborationGroupMember> cgmembersNoId = new set <collaborationGroupMember>();
                FOR (collaborationGroupMember cgm : cgmembers){
                   collaborationGroupMember cgmx = new collaborationGroupMember();
                    cgmx.MemberId = cgm.MemberId;
                    cgmx.CollaborationGroupId = cgm.CollaborationGroupId;
                    cgMembersNoId.add(cgmx);
                }

Answer 1

The best solution I've found is to run the list of collaborationgroupmembers through a FOR loop to create a new list that does not contain the Id field, and use that in the removeAll method.

set <collaborationGroupMember> cgmembersNoId = new set <collaborationGroupMember>();
                FOR (collaborationGroupMember cgm : cgmembers){
                   collaborationGroupMember cgmx = new collaborationGroupMember();
                    cgmx.MemberId = cgm.MemberId;
                    cgmx.CollaborationGroupId = cgm.CollaborationGroupId;
                    cgMembersNoId.add(cgmx);
                }