So if I understand correctly, when a new custom object is inserted, you want to ensure that its name is unique within those associated with a particular contact?
Unfortunately, I think you will have to resort to a trigger for this.
It is tempting to assume that you can do this via the VLOOKUP function, but this won't allow you to constrain the records that you consider to those associated with the contact.
I've also tried using a field on the custom object set via workflow to a concatenation of the contact name and custom object name, and then using a VLOOKUP function in a validation rule to ensure that this is globally unique.
However, as per the Triggers and Order of Execution in the Apex Developer's Guide, my custom validation rule didn't fire after the update.
11 . If the record was updated with workflow field updates, fires before and after triggers one more time (and only one
more time), in addition to standard validations. Custom validation rules are not run again.