I believe you can actually do this without resorting to the use of JavaScript. I used the following blog post by our very own Andrew Fawcett to use as a reference and create a List Button based on your example.
I have not had a chance to completely test it, but I am fairly confident this will work.
Using this information I can create the following controller:
public with sharing class CreateNewRelationshipListController
{
private ApexPages.StandardSetController AccountSetController;
public CreateNewRelationshipListController(ApexPages.StandardSetController inheritedSetController)
{
AccountSetController = inheritedSetController;
}
public PageReference SetContactRelationshipsToNew()
{
List<Account> selectedAccounts = (List<Account>)AccountSetController.getSelected();
if(selectedAccounts.size() > 0)
{
Set<Id> accountIdSet = (new Map<Id, Account>(selectedAccounts)).keyset();
List<Contact> contactsToUpdateRelationships = [SELECT Id, Relationship__c FROM Contact WHERE AccountId IN :accountIdSet];
if(contactsToUpdateRelationships.size() > 0)
{
for(Contact contactToUpdateRelationship : contactsToUpdateRelationships)
contactToUpdateRelationship.Relationship__c = 'New';
UPDATE contactsToUpdateRelationships;
}
}
return null;
}
}
Next you would have to create a custom VisualForce page to specify the action once your user clicks the custom button:
<apex:page standardController="Account" extensions="CreateNewRelationshipListController"
action="{!SetContactRelationshipsToNew}" recordSetVar="accountsToProcess">
Lastly, you would create a custom list button, choose display checkboxes (so you can select multiple Accounts), and choose the VisualForce created above as the Content Source.
There are reservations to this. Mainly this is open CSRF attacks. So you should be weary of that.
For more information on how to accomplish this, please take a look at this blog post I used as a reference for more information.
There is actually another link that the blog mentions, where you can use the same controller for either a List Button and Detail Page Button. You can find more information here, but it's not the most elegant solution (which the author admits himself).