I think this extract from the excellent Visualforce Development Cookbook by Keir Bowden will really help point you in the right direction.
Opening a pop-up window
In this recipe, we will create a page that renders a list of accounts,
displaying a very small subset of fields per row. A link will be
provided on each row to allow the user to view full details of the
account in a pop-up window.
Note - there is no way to ensure that a browser will display a pop-up
window. Pop-up blockers generally allow windows to be opened in
response to an action by the user, such as clicking on a link, but it
is possible for users to configure their browser to block all pop-ups
regardless of how they were triggered.
This recipe requires 2 Visualforce pages to be created: the main page
containing the list of accounts and the pop-up window page. The pop-up
page is referenced by the main page, so this will be created first.
Create a VF page called 'Popup'. Paste the contents of the Popup.page
file into the VF page markup area. Save the page.
<apex:page standardController="Account" sidebar="false" showHeader="false">
<apex:sectionHeader title="Account" subtitle="{!Account.Name}"/>
<apex:detail relatedList="false"/>
</apex:page>
Then do the same but for a VF page called 'PopupMain'.
<apex:page standardController="Account" recordSetVar="accs">
<apex:pageBlock >
<apex:pageBlockTable value="{!accs}" var="acc">
<apex:column headerValue="Action">
<apex:outputLink title="View detail in a popup window" onclick="return openPopup('{!acc.Id}');">Details</apex:outputLink>
</apex:column>
<apex:column value="{!acc.Name}" />
<apex:column value="{!acc.Industry}" />
<apex:column value="{!acc.Type}" />
</apex:pageBlockTable>
</apex:pageBlock>
<script>
function openPopup(id)
{
var newWin=window.open('{!$Page.Popup}?id=' + id, 'Popup',
'height=600,width=650,left=100,top=100,resizable=no,scrollbars=yes,toolbar=no,status=no');
if (window.focus)
{
newWin.focus();
}
return false;
}
</script>
</apex:page>
How it works
Opening the following URL into your browser displays a list of
accounts:
https://<instance>/apex/PopupMain
Note - as this page uses a standard list controller, the list of
accounts displayed will be that of the last list view that the user
accessed.
The detail link markup is a follows:
Details
The onclick attribute defines the JavaScript function to be invoked
when the link is clicked; note the {!acc.Id} merge field, which passes
the ID of the chosen account to the function.
The JavaScript function uses the window.open function to open the new
window.
var newWin=window.open('{!$Page.Popup}?id=' + id, 'Popup',
'height=600,width=650,left=100,top=100,resizable=no,scrollbars=yes,toolbar=no,status=no');
The final parameter details the features required for the new window
as a comma separated list of name=value pairs.
In this book, there is another chapter about how to capture
information from a pop-up window and pass the information back to the
main window to populate input fields.