You do not have direct access to the ownership change page so there's no direct way of doing this. I can think of two option and both these options will work only if the sidebar is enabled in your org:
1) Create a custom VF page which will mimic the standard change owner page. As this is a custom page you will have full control to check / uncheck the checkbox based on your condition. Create a home page sidebar component which will contain a script (jQuery / javascript) which will override the [change Owner] link to you VF page. So whenever a user click on the [change owner] link, he will be redirected to your VF page.
2) Now, If you do not want to create a VF page then create a home page component which will contain a script, this script will run only when you click on the change owner link, the script will check if the case record record type and the owner that is being changed to and based on this it will check / uncheck the "send email notification" checkbox.