We had a situation similar to this where users were not selecting the right boxes and selecting the wrong ones.
The solution I ended up doing was to remove the Account Owner field from the page layout and replaced with with a custom account owner field:
HYPERLINK('/' & OwnerId, Owner.FullName__c) & ' ' &
HYPERLINK('/apex/AccountOwnerChange?Id=' & Id, '[Change]')
(Note: we opted not to care about the profile image, there are some tricks that can be done to facilitate this if you care).
Then I made a VF page that used the standard controller with a custom extension.
This allowed me to remove the unwanted checkboxes and customize the behavior completely. We did think about taking the approach that you are trying to take with using a trigger to correct the mistakes but we opted against this for usability; very confusing to a person when the click a box or don't click a box and see the opposite of what they thought would happen. We also wanted effectively non of the built in functionality to work and this was a much easier way to achieve that as opposed to writing a bunch of logic to undo each and every one of the checkboxes. There is also the risk that when the changes are being made manually (not necessarily by changing the owner) that your trigger jumps in inappropriately.