You should use Visualforce behind your button rather than javascript.
In the action attribute of the page you can do all sorts of complex logic rather than using formula field.
lets see some sample example of how you should proceed with the code
<apex:page standardcontroller="Object__c" extensions="MycustomCtrl" action="navigate">
and then use the logic as below in your extension controller
public class MycustomCtrl{
private final object__c obj;
public MycustomCtrl(ApexPages.StandardController stdController) {
this.obj= (Account)stdController.getRecord();
}
public pagereference navigate(){
pagereference p=new pagereference();//Use pagereference variables
//Query for the recordtype and name and other fields
//Use if logic and construct your navigation logic with pagereference variables
}
}