I'd say it would be easier to do away with the apex:detail and use apex:repeat to render your own related list rather than try to override the Edit behaviour.
I reckon the only way to do this would be to override the Edit action on the Contact with a custom Vf page, where you check where the contact edit is being invoked from (using perhaps a url query param) and accordingly redirect to your custom vf page to either the standard or the custom contact edit page.
Note however that overriding the Edit behaviour will mean you can't inline edit Contacts on standard page layouts anymore.
You'd still have to build your custom vf page to edit the contact. Or else do RecordType gymnastics to dance to the desired Page Layout assignments.
In summary, I'd think it would be a lot less trouble to render your own related list using apex:repeat