I needed to do the same thing and unfortunately there's not an easy (and dynamic) way to do this as far as I know, so I ended up creating a set of blacklisted system fields that I know I didn't want and if the field API name showed up in the list when I was gathering fields via the Schema class I just ignored it. My final list ended up being:
private Set<String> blacklistedFields = new Set<String>{'CompletedDateTime','RecurrenceRegeneratedType','RecurrenceMonthOfYear','RecurrenceInstance','RecurrenceDayOfMonth','RecurrenceDayOfWeekMask','RecurrenceType','RecurrenceTimeZoneSidKey','RecurrenceEndDateOnly','RecurrenceStartDateOnly ','RecurrenceActivityId','ReminderDateTime','ActivityOriginType','ArchivedDate','IsArchived','IsClosed','PrioritySortOrder','IsHighPriority','AccountId','Jigsaw Contact Id','ReportsToName','EmailBouncedReason','EmailBouncedDate','IsEmailBounced','JigsawContactId','Id','IsDeleted','CreatedDate','LastModifiedDate','SystemModstamp','LastActivityDate','LastViewedDate','LastReferencedDate','UserRecordAccessId','MasterRecordId','AccountSource','IsCssEnabled','CssLastLoginDate','CompareName','PhotoUrl','CompareSite','OwnerAlias','JigSawCompanyId','ConnectionReceivedDate','ConnectionSentDate','AccountRollupId','ProductIsArchived'};
Very long, I know! It would really be nice if Salesforce had an easy way to get only fields that show up in the Object Manager.