I am a little unclear whether you want to customize the fields displayed in the search result, or if you want to customize the lookup filter itself to get different search results.
If it is a matter of filtering the result, you need to do this as a Lookup Filter. You will modify this in the field parameters for the lookup field (in this case the Contact field). Here is an example of doing this for the standard Case
object.
- Case fields:
- Account
- Contact (Only where
Contact.AccountId
is same as Case.AccountId
)
You can alternatively make this optional if you want the user to only be strongly urged in the direction of using the filter.
If the desire is to change which fields in Contact are displayed when you do the lookup search, you can do this in the Search Layout for Contact (Setup > Customize > Contacts > Search Layouts).
Here is the contact search layouts list. The Lookup Dialogs search layout is what you want to modify.
Here's how you pick which fields will appear.
And here is what it looks like in the end:
These can be modified for any object that is allowed to be the related lookup or Master-Detail object in such fields. Custom object modifications are all found on their custom object page (Setup > Create > Objects > My_Custom_Object__c)