I am using a custom object to track students (outside of Contact and Leads) and also has an Email field we use as one of the contact methods.
How do I flag undeliverable messages so I can mark them as invalid like in the Contacts bounce management? I don't see where to get at the returned email within Salesforce - only get it forwarded to may Admin email.
You can't. Bounce Management only works for emails sent from Salesforce, and you can only email Users, Contacts, and Leads. It is not recommended that you use Custom Objects for managing people of any sort; they almost certainly should have been Contact records, perhaps with a unique Record Type to distinguish them from other types of Contacts you have in Salesforce. Here's an Idea you can vote on in the meantime.
