I have a lookup field in a Visualforce and I need to add one filter in that. I have to do with apex because with the standard way I can't create this filter. Could you help me? How can I create the custom filter in a lookup?
Without any code or knowing what you've done so far, Jeff Douglas wrote a great tutorial that will let you Roll your Own Salesforce Lookup Pop Up Window here.
The tutorial is too long to quote here, but essentially you need to have:
two Visualforce pages (the record you are editing and the popup window) and two Apex controllers (a simple one for the record you are editing and the controller for the search and new record popup).
One VF Page uses jQuery to intercept the popup showing your custom one instead of the standard lookup.
Within the VF Page and Controller for the custom popup, you can then define all the filters you want.
Lookup Filters are used to filter down the related records and they are set on filed definition.
If you can't able to specify and filter condition in Lookup Filter then you can write a code(controller/extension/trigger) which will restrict user at the time of saving a record. You can display a error message to the user based on conditions.
As @Dan Jones has suggested, you can build a custom lookup window to display filtered records on pop up window.