S2S: INSUFFICIENT_ACCESS_ON_CROSS_REFERENCE_ENTITY error when attempting to sync Notes and Attachments

Question

I have S2S set up between 2 sandboxes. I'm successfully sharing opportunities between them.

However, I then added notes and attachments to the sync:

newConnection = new PartnerNetworkRecordConnection (ConnectionId = conn.Id,
                                                   LocalRecordId = opp.Id,
                                                   RelatedRecords = 'NoteAndAttachment',
                                                   SendClosedTasks = false,
                                                   SendOpenTasks = false,
                                                   SendEmails = false);

Now in the source sandbox when I modify an opportunity that has a note I get this error: ****System.DmlException: Insert failed. First exception on row 0; first error: INSUFFICIENT_ACCESS_ON_CROSS_REFERENCE_ENTITY, insufficient access rights on cross-reference id: []****

I presume this has something to do with ownership of the note in the receiving org. However, the opportunity and the note have the same owner in the source org, and S2S has no problem syncing the opp when it has no note attached. So I'm puzzled about why I'm getting this error.