You could create a formula field on the object that holds the Owner.UserRole.Name then use that in your validation rule like:
AND(
ISCHANGED(OwnerId),
PRIORVALUE(Owner_Role__c) <> Owner.UserRole.Name
)
Which would only allow users to change the owner if the existing owner has the same Role as the new owner.