You could create a formula field on the object that holds the Owner.UserRole.Name then use that in your validation rule like:

AND(
   ISCHANGED(OwnerId),
   PRIORVALUE(Owner_Role__c) <> $UserRole.Name
)  

Which would only allow users to change the owner if the existing owner has the same Role as the running user.

You could create a formula field on the object that holds the Owner.UserRole.Name then use that in your validation rule like:

AND(
   ISCHANGED(OwnerId),
   PRIORVALUE(Owner_Role__c) <> Owner.UserRole.Name
)  

Which would only allow users to change the owner if the existing owner has the same Role as the new owner.