Firstly, create a List Button under App Setup > Customize > Opportunities > Opportunity Products > Buttons and Links. Uncheck "Display Checkboxes (for Multi-Record Selection)", as last time I checked you can't have list controllers for OpportunityLineItem.
Then, when editing the Opportunity page layout click the small spanner above the products related list. The Buttons section can be a bit hard to find and will need to be expanded up from the button of the dialog. From there you can add your new button.
UPDATE Linking to a Visualforce Controller
You can't use a custom visualforce page against a non top-level standard object. See StandardController recordSetVar OpportunityLineItem issue (my emphasis)
Yes we restrict list controllers ( standardCOntroller + recordSetVar attribute) to only top-level standard and custom objects so effectively you can not create a list button to a Visualforce page for OpportunityLineItem today. - Andrew, Product Manager - salesforce.com
Instead, you can set the Content Source to URL and point the URL at your Visualforce page.
